I got AssertionError: If you want to use `ReformerModelWithLMHead` make sure that `is_decoder=True`. with model = ReformerModelWithLMHead.from_pretrained("hf-internal-testing/tiny-random-reformer").

For ReformerModelWithLMHead, we could use the checkpoint google/reformer-crime-and-punishment, and we won't have the issue regarding is_decoder.

print loss instead of loss.backward() because I got AssertionError: If you want to train `ReformerModel` and its variations, make sure to use `model.train()` to put the model into training mode..

Maybe it is a good idea to set model.train() before loss = model(**inputs, labels=labels).loss?

So the users will know it is required :-)

cc @patrickvonplaten for comment.

Thank you for the suggestion!
I tried setting model.train(), but got ValueError: If training, sequence length 11 has to be a multiple of least common multiple chunk_length 4. Please consider padding the input to a length of 12. at loss = model(**inputs, labels=labels).loss.

Yes indeed that's very Reformer specific, but nice that you caught it :-) Could we use a sequence length of 12 here?

Thank you for the comment.
I tried a sequence length of 12 like below:

import torch
from transformers import ReformerTokenizer, ReformerForSequenceClassification

tokenizer = ReformerTokenizer.from_pretrained("hf-internal-testing/tiny-random-reformer")
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model = ReformerForSequenceClassification.from_pretrained("hf-internal-testing/tiny-random-reformer", problem_type="multi_label_classification")

inputs = tokenizer("Hello, my dog is cute", max_length=12, padding="max_length", return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

predicted_class_id = logits.argmax().item()
model.config.id2label[predicted_class_id]

# To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
num_labels = len(model.config.id2label)
model = ReformerForSequenceClassification.from_pretrained("hf-internal-testing/tiny-random-reformer", num_labels=num_labels)
model.train()

num_labels = len(model.config.id2label)
labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to(
    torch.float
)
loss = model(**inputs, labels=labels).loss
loss.backward()

However, I got ValueError: If training, make sure that config.axial_pos_shape factors: (4, 25) multiply to sequence length. Got prod((4, 25)) != sequence_length: 12. You might want to consider padding your sequence length to 100 or changing config.axial_pos_shape. at loss = model(**inputs, labels=labels).loss in this time.

Then, I tried a sequence length of 100.
It seems to work fine!

@ydshieh @patrickvonplaten
I pushed this change. Could you please check that?

I will take a look. However, we will wait Patrick's review anyway as he has more knowledge on this model than me (he is currently not available).

The reason for using replace_return_docstrings here is below:

• We need to load the model with is_decoder=True. (line:2230)

The reason for using replace_return_docstrings here is below:

• ReformerTokenizer's mask_token is None, so we need to set it. (line: 2360)
• len(tokenizer.tokenize("The capital of France is [MASK].")) is 16, but len(tokenizer.tokenize("The capital of France is Paris.")) is 17. So, we need to match the lengths. (line: 2378)

OK, Reformer is in fact an auto-regressive generative model, not really intended to be used as Masked LM model.
I saw that you already use hf-internal-testing/tiny-random-reformer for ReformerForMaskedLM, so we don't have issue regarding is_decoder.

We still have issue regarding mask token, so good to use replace_return_docstrings! Thank you.

The reason for using replace_return_docstrings here is below:

• I got an error when calling loss.backward(). So, I replaced it with simply print loss. (line: 2530)

Update:

• adding pad_token to pad inputs. (line: 2509)
• padding max_length=100 to avoid the error. (line: 2510)
• model.train() to call loss.backward(). (line: 2526)

This part is required for the next block example, not this current block.
So a good reason to overwrite the code sample here.

@patrickvonplaten I am not sure if we should add this in doc.py. (Currently this is not added in doc.py) Without this, we are not in the train model, and might be a bit misleading.

Confirmed the special treatment here is required to make the example run.
(Better to have a good way to handle this in doc.py in the future)