[video processors] decode only sampled videos -> less RAM and faster processing #39600
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: aria, aya_vision, blip, bridgetower, chameleon, clip, colpali, deepseek_vl, deepseek_vl_hybrid, emu3, eomt, flava, gemma3, gemma3n, glm4v |
|
This comment contains run-slow, running the specified jobs: models: ['models/aria', 'models/aya_vision', 'models/blip', 'models/bridgetower', 'models/chameleon', 'models/clip', 'models/colpali', 'models/deepseek_vl', 'models/deepseek_vl_hybrid', 'models/emu3', 'models/eomt', 'models/flava', 'models/gemma3', 'models/gemma3n', 'models/glm4v'] |
|
@qubvel I merged the related PR and rebased |
|
run-slow: aria, aya_vision, blip, bridgetower, chameleon, clip, colpali, deepseek_vl, deepseek_vl_hybrid, emu3, eomt, flava, gemma3, gemma3n, glm4v |
|
This comment contains run-slow, running the specified jobs: models: ['models/aria', 'models/aya_vision', 'models/blip', 'models/bridgetower', 'models/chameleon', 'models/clip', 'models/colpali', 'models/deepseek_vl', 'models/deepseek_vl_hybrid', 'models/emu3', 'models/eomt', 'models/flava', 'models/gemma3', 'models/gemma3n', 'models/glm4v'] |
|
run-slow: qwen2_vl, qwen2_5_vl, qwen2_5_omni, smolvlm, llava_onevision, llava_next_video, perception_lm |
|
This comment contains run-slow, running the specified jobs: models: ['models/llava_next_video', 'models/llava_onevision', 'models/perception_lm', 'models/qwen2_5_omni', 'models/qwen2_5_vl', 'models/qwen2_vl', 'models/smolvlm'] |
|
On no, new torch release doesn't work well with Bytes objects 😓 (fails only in CI, still figuring out why) |
|
run-slow: qwen2_vl, qwen2_5_vl, qwen2_5_omni, smolvlm, llava_onevision, llava_next_video, perception_lm |
1 similar comment
|
run-slow: qwen2_vl, qwen2_5_vl, qwen2_5_omni, smolvlm, llava_onevision, llava_next_video, perception_lm |
|
This comment contains run-slow, running the specified jobs: models: ['models/llava_next_video', 'models/llava_onevision', 'models/perception_lm', 'models/qwen2_5_omni', 'models/qwen2_5_vl', 'models/qwen2_vl', 'models/smolvlm'] |
|
The CI is impossible to pass 🙃 |
|
@bot /style |
|
Style bot fixed some files and pushed the changes. |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, aya_vision, blip, bridgetower, chameleon, clip, colpali, deepseek_vl, deepseek_vl_hybrid, emu3, eomt, flava, gemma3, gemma3n, glm4v |
3 participants
What does this PR do?
This PR moves the video decoding code entirely into video processors, so that we can load only necessary video frames into memory. To be consistent with video processors, I also updated image processors to accept
strin inputs and optionally load images.The docs for video processors are also updated explaining how frames are sampled and what users need to do to turn it on/off. Note that we'll be using by default
torchcodecand fallback totorchvision, and we won't support any arbitrary video decoders within video processor class. Otherwise we'd need to introduce morekwargsand handle differences between decoders, which bloats up the code even more