🚨 Remove Contrastive Search decoding strategy #40428
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
manueldeprada
changed the title
| ) | ||
| generation_config.cache_implementation = None | ||
|
|
||
| # assisted decoding and contrastive search need to roll-back the Cache, which is not supported if |
There was a problem hiding this comment.
Two things:
- related to our conversation today, here we force non-sliding windows for speculative decoding
- Contrastive search should pass
cache_implementation="dynamic_full", and we should create anotherifbelow. Otherwise, when we removeGenerationMode.CONTRASTIVE_SEARCH, we won't be able to instantiate the right type of cache.
There was a problem hiding this comment.
added dynamic_full as suggested and a warning on the hub code: https://huggingface.co/transformers-community/contrastive-search/commit/1e37df04e0e3f4c0a36cadeb09b65eef59d884f1
Since dynamic_full is quite niche, we probably don't want to document it further for now right?
cc @Cyrilvallez as well as this is related to cache refactors
There was a problem hiding this comment.
Since dynamic_full is quite niche, we probably don't want to document it further for now right?
Yeah, I'm okay with it being an internal option for now 👍
|
build-doc |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: bart, csm, gemma, gpt2, gpt_bigcode, gptj, idefics, idefics2, idefics3, kosmos2_5, lfm2, llama, mistral, opt, paligemma2, smolvlm |
3 participants
Removes Contrastive Search generation strategy from the codebase. Directs users to the
transformers-community/contrastive-searchrepository.It has been a warning for a few releases, but now
trust_remote_code=Trueis required to run contrastive search.