KEMBAR78
Fix label smoothing incompatibility with multi-label classification by avchauzov · Pull Request #40296 · huggingface/transformers · GitHub
Skip to content

Conversation

@avchauzov
Copy link
Contributor

What does this PR do?

Fixes a RuntimeError when using label_smoothing_factor > 0 with multi-label classification.

Previously, users would get a confusing error:
RuntimeError: gather(): Expected dtype int32/int64 for index

This PR adds a clear validation check in LabelSmoother that detects this incompatible combination and raises a helpful error message instead:

ValueError: Label smoothing is not compatible with multi-label classification.
Multi-label classification uses one-hot encoded labels (float tensors),
but label smoothing requires integer class indices.
Please set label_smoothing_factor=0.0 in your TrainingArguments.

Changes:

  • Add validation in LabelSmoother.__call__() to detect multi-label classification
  • Raise clear ValueError with actionable solution
  • Add test case to prevent regression

Testing:

  • ✅ All existing tests pass
  • ✅ New test covers the fix
  • ✅ Original reproduction case now shows clear error

Fixes #40258

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

@zach-huggingface @SunMarc @qgallouedec - trainer related changes

Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks ! just a nit

- Move check from LabelSmoother to Trainer.__init__() for better architecture
- Use model.config.problem_type instead of tensor inference for robustness
- Warn and disable smoothing instead of raising error for better UX
- Update test to verify warning behavior
@andreabosisio
Copy link

thank you @avchauzov @SunMarc for your work on this! I was wondering if it would actually be possible to implement label smoothing for multi-label classification tasks (#40301)

Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks !

@SunMarc SunMarc enabled auto-merge (squash) August 25, 2025 14:13
@SunMarc
Copy link
Member

SunMarc commented Aug 25, 2025

thank you @avchauzov @SunMarc for your work on this! I was wondering if it would actually be possible to implement label smoothing for multi-label classification tasks (#40301)

I won't have the time but hopefully the community picks it !

@SunMarc SunMarc merged commit 04c2bae into huggingface:main Aug 25, 2025
24 checks passed
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@avchauzov avchauzov deleted the fix-label-smoothing-multi-label-incompatibility branch August 25, 2025 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RuntimeError in loss computation with label_smoothing_factor > 0 for multi-label classification

4 participants