The documentation is not available anymore as the PR was closed or merged.

Unfortunately, we can't change the EvalPredictions namedtuple structure like this as it would be a massive breaking change. Users would suddenly have to unpack it with three objects instead of two, so every compute_metrics function out there in the wild would suddenly fail.

Ok, spent a bit of time on this and to enable an EvalPredictions that work with/without inputs, the class should be replaced by the following one:

class EvalPrediction:
    """
    Evaluation output (always contains labels), to be used to compute metrics.

    Parameters:
        predictions (`np.ndarray`): Predictions of the model.
        label_ids (`np.ndarray`): Targets to be matched.
        inputs (`np.ndarray`, *optional*): Inputs of the model.
    """
    def __init__(
        self,
        predictions: Union[np.ndarray, Tuple[np.ndarray]],
        label_ids: Union[np.ndarray, Tuple[np.ndarray]],
        inputs: Optional[Union[np.ndarray, Tuple[np.ndarray]]] = None,
    ):
        self.predictions = predictions
        self.label_ids = label_ids
        self.inputs = inputs
    
    def __iter__(self):
        if self.inputs is not None:
            return iter((self.predictions, self.label_ids, self.inputs))
        else:
            return iter((self.predictions, self.label_ids))
    
    def __getitem__(self, idx):
        if idx < 0 or idx > 2:
            raise IndexError("tuple index out of range")
        if idx == 2 and self.inputs is None:
            raise IndexError("tuple index out of range")
        if idx == 0:
            return self.predictions
        elif idx ==1:
            return self.label_ids
        elif idx == 2:
            return self.inputs

Then we should add a flag so the user can choose whether or not they want the inputs included for metrics or not (default False so that there is no change). With those two things, we can enable your use case while maintaining backward compatibility.

Can you include them in your PR?

Hi @sgugger ,

Thanks for adding this change. I've tested this code with the summarization examples from pytorch and I'm afraid this change will break other people's code. The problem is that since the inputs are passed in the trainer by default (based on my PR):

trainer.py

self.compute_metrics(EvalPrediction(inputs=all_inputs, predictions=all_preds, label_ids=all_labels))

The inputs in EvalPrediction will never be None, and then, it will fail when returning the predictions in compute_metrics:

run_summarization.py

def compute_metrics(eval_preds):
    preds, labels = eval_preds

  File "./transformers/examples/pytorch/summarization/run_summarization.py", line 575, in compute_metrics
    preds, labels = eval_preds
ValueError: too many values to unpack (expected 2)

You can see in the debugger that all 3 (inputs, preds and labels) are coming:

In my case, I have my own implementation for compute_metrics, however, for other users it will fail by default. When you suggest the flag, where is it located? In the class Trainer and then an if before each call of EvalPrediction?

This is my testing case:

./transformers/examples/pytorch/summarization/run_summarization.py
--model_name_or_path
t5-small
--do_train
--do_eval
--train_file test.json
--validation_file test.json
--source_prefix
"summarize: "
--output_dir
/tmp/tst-summarization
--overwrite_output_dir
--per_device_train_batch_size=4
--per_device_eval_batch_size=4
--predict_with_generate

test.json

{"text": "I'm sitting here in a boring room. It's just another rainy Sunday afternoon. I'm wasting my time I got nothing to do. I'm hanging around I'm waiting for you. But nothing ever happens. And I wonder", "summary": "I'm sitting in a room where I'm waiting for something to happen"}
{"text": "I see trees so green, red roses too. I see them bloom for me and you. And I think to myself what a wonderful world. I see skies so blue and clouds so white. The bright blessed day, the dark sacred night. And I think to myself what a wonderful world.", "summary": "I'm a gardener and I'm a big fan of flowers."}
{"text": "Christmas time is here. Happiness and cheer. Fun for all that children call. Their favorite time of the year. Snowflakes in the air. Carols everywhere. Olden times and ancient rhymes. Of love and dreams to share", "summary": "It's that time of year again."}

Thanks,

Laura

The problem is that since the inputs are passed in the trainer by default (based on my PR):

As I said above, this needs to be controlled by a flag (for instance include_inputs_for_metrics) in TrainingArguments which would be False by default to avoid any breaking change.

Thanks for the clarification, that sounds better. I've submitted an additional commit in the PR. Is the first time I update a PR, let me know if I've missed something.

src/transformers/trainer.py

I thought about the OOM issue, but should I add:

if self.include_inputs_for_metrics:

Everywhere in the code? I can't think of a more elegant way :)

And for the new changes, is it ok just another commit in the pull request just like I did the last time? Just checking, so I don't do a mess :P

should I add if self.include_inputs_for_metrics: Everywhere in the code? I can't think of a more elegant way :)

You should make sure that the inputs are left as None like the labels/losses by only setting some inputs when the flag is False. I've left a comment to show you where.

And for the new changes, is it ok just another commit in the pull request just like I did the last time? Just checking, so I don't do a mess :P

That's completely ok, we will squash everything when merging.

Changes done, let me know any further feedback :)

I've added the requested changes. Also, I've run make fixup, so there are also some automatic indentation changes as well.

Thanks! There is one last issue with a docstring badly formatted. Could you run make style on your branch?

Sure, no problem. I'm having issues with the dependencies to run make style, could you please let me know what's the full command for installing all the suite?
pip install .[quality]

This is my log:

% make style
black examples tests src utils
Skipping .ipynb files as Jupyter dependencies are not installed.
You can fix this by running ``pip install black[jupyter]``
All done! ✨ 🍰 ✨
1519 files left unchanged.
isort examples tests src utils
WARNING: Unable to parse file examples due to [Errno 21] Is a directory: './transformers/examples'
WARNING: Unable to parse file tests due to [Errno 21] Is a directory: './transformers/tests'
WARNING: Unable to parse file src due to [Errno 21] Is a directory: './transformers/src'
WARNING: Unable to parse file utils due to [Errno 21] Is a directory: './transformers/utils'
/Library/Developer/CommandLineTools/usr/bin/make autogenerate_code
running deps_table_update
updating src/transformers/dependency_versions_table.py
/Library/Developer/CommandLineTools/usr/bin/make extra_style_checks
python utils/custom_init_isort.py
doc-builder style src/transformers docs/source --max_len 119 --path_to_docs docs/source
make[1]: doc-builder: No such file or directory
make[1]: *** [extra_style_checks] Error 1
make: *** [style] Error 2

It looks like your branch is not up to par with the main branch for the setup. Can you manually install pip install hf-doc-builder? It should be in the quality extras but maybe you don't have it because it was added recently-ish.

Now is working :) Thanks for that one. I've committed the updated file.

% make style
black examples tests src utils
Skipping .ipynb files as Jupyter dependencies are not installed.
You can fix this by running ``pip install black[jupyter]``
All done! ✨ 🍰 ✨
1519 files left unchanged.
isort examples tests src utils
/Library/Developer/CommandLineTools/usr/bin/make autogenerate_code
running deps_table_update
updating src/transformers/dependency_versions_table.py
/Library/Developer/CommandLineTools/usr/bin/make extra_style_checks
python utils/custom_init_isort.py
doc-builder style src/transformers docs/source --max_len 119 --path_to_docs docs/source
Overwriting content of src/transformers/training_args.py.
Cleaned 1 files

Yes, it's all good now. Thanks again for your contribution!

Awesome! :) Thanks for leading this effort, good job :D