Fix broken image inference for Fuyu model #39915
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Thanks! Just noticed that Fuyu has no multimodal tests and thus we are more likely to break it. If you can add multimodal test class, would be nice
In this file similar to llava tests where dummy image inputs are also prepared in self.prepare_config_and_inputs
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
| inputs = processor(images=image, text=text_prompt_coco_captioning, return_tensors="pt").to( | ||
| torch_device, torch.float16 | ||
| ) |
There was a problem hiding this comment.
Seems there has been a generation test for Fuyu, but I found it was running on CPU and also failing on main branch:
> self.assertEqual(generated_text, "A blue bus parked on the side of a road.")
E AssertionError: 'image shows what \n\n\n' != 'A blue bus parked on the side of a road.'
E + A blue bus parked on the side of a road.
E - image shows what
E -
E -
E -
tests/models/fuyu/test_modeling_fuyu.py:254: AssertionError
So I just move this test to run on GPU.
There was a problem hiding this comment.
Yep, we have slow Integration tests. I meant fast test with dummy inputs, which are run always under each PR
|
run-slow: fuyu |
zucchini-nlp
added
the
for patch
Signed-off-by: Isotr0py <2037008807@qq.com>
|
[For maintainers] Suggested jobs to run (before merge) run-slow: fuyu |
* fix fuyu Signed-off-by: Isotr0py <2037008807@qq.com> * oops Signed-off-by: Isotr0py <2037008807@qq.com> * run test on GPU Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> * clean unused Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> * revert Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> * add fuyu multimodal test Signed-off-by: Isotr0py <2037008807@qq.com> * fix Signed-off-by: Isotr0py <2037008807@qq.com> --------- Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
3 participants
What does this PR do?
When updating Transformers version in vLLM CI, we found fuyu's image inference is broken with gibberish:
https://buildkite.com/vllm/ci/builds/25473#01985c4d-d267-408b-87f5-5d77ae09d90c
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.