tensorrt_llm/inputs/multimodal.py (3)

482-484: Return type/docstring mismatch, long line (E501), and avoid unnecessary PIL conversions.

• find_mm_token_lengths now returns a dict but the annotation and comment still say List[int].
• Break the long f-string to satisfy E501.
• When items are tensors, compute H/W from shape instead of converting to PIL; validate non-empty video list.

-def find_mm_token_lengths(mm_data: Dict[str, Any],
-                          input_processor: Any) -> List[int]:
-    """Get multimodal token lengths from multimodal data items. """
+def find_mm_token_lengths(mm_data: Dict[str, Any],
+                          input_processor: Any) -> Dict[str, List[int]]:
+    """Get per-modality multimodal token lengths from multimodal data items."""
@@
-        if not hasattr(input_processor, f"get_num_tokens_per_{modality}"):
-            raise AttributeError(
-                f"Input processor {type(input_processor).__name__} does not have 'get_num_tokens_per_{modality}' method required for multimodal hashing."
-            )
+        if not hasattr(input_processor, f"get_num_tokens_per_{modality}"):
+            raise AttributeError(
+                f"Input processor {type(input_processor).__name__} does not have "
+                f\"get_num_tokens_per_{modality}\" required for multimodal hashing."
+            )
@@
-            if modality == "image":
-                if isinstance(item, torch.Tensor):
-                    item = ToPILImage()(item)
-                num_tokens = input_processor.get_num_tokens_per_image(
-                    image_width=item.width,
-                    image_height=item.height,
-                )
+            if modality == "image":
+                if isinstance(item, torch.Tensor):
+                    h, w = int(item.shape[-2]), int(item.shape[-1])
+                else:
+                    w, h = item.width, item.height
+                num_tokens = input_processor.get_num_tokens_per_image(
+                    image_width=w,
+                    image_height=h,
+                )
                 modality_token_lengths.append(num_tokens)
             elif modality == "video":
-                assert isinstance(item, list), "Video must be a list of frames"
-                if isinstance(item[0], torch.Tensor):
-                    item = [ToPILImage()(frame) for frame in item]
-                num_tokens = input_processor.get_num_tokens_per_video(
-                    video_width=item[0].width,
-                    video_height=item[0].height,
-                    num_frames=len(item),
-                )
+                assert isinstance(item, list), "Video must be a list of frames"
+                if not item:
+                    raise ValueError("Video frame list is empty")
+                if isinstance(item[0], torch.Tensor):
+                    h, w = int(item[0].shape[-2]), int(item[0].shape[-1])
+                else:
+                    w, h = item[0].width, item[0].height
+                num_tokens = input_processor.get_num_tokens_per_video(
+                    video_width=w,
+                    video_height=h,
+                    num_frames=len(item),
+                )
                 modality_token_lengths.append(num_tokens)
@@
-    return num_mm_tokens  # flatten all mm instances to a single list
+    return num_mm_tokens  # mapping: modality -> list of lengths

Also applies to: 493-496, 501-518, 521-521

---

423-430: Harden tensor serialization in serialize_item.

Calling .numpy() on non-CPU tensors raises; also ensure contiguity. Update the tensor branch accordingly.

def serialize_item(obj: object) -> bytes:
    ...
    if isinstance(obj, torch.Tensor):
        t = obj.detach()
        if t.is_sparse:
            t = t.coalesce().to_dense()
        if t.device.type != "cpu":
            t = t.to("cpu")
        return t.contiguous().numpy().tobytes()

---

482-484: Update find_mm_token_lengths signature and docstring

• Change the return annotation in tensorrt_llm/inputs/multimodal.py from -> List[int] to -> Dict[str, List[int]] and revise its docstring to describe the mapping of modality names to token-length lists.
• Only one call site remains (in tensorrt_llm/inputs/registry.py at lines 424–426), which already unpacks the dict via next(iter(...)).

tensorrt_llm/inputs/registry.py (1)

311-331: Fix return types: methods promise Tuple[str, ...] but return generators.

Wrap the generator expressions with tuple().

     def get_registered_image_model_types(self) -> Tuple[str, ...]:
-        return (
+        return tuple(
             model_type
             for model_type in self._multimodal_placeholder_by_model_type
             if "image" in self.
             _multimodal_placeholder_by_model_type[model_type].placeholder_map)
@@
     def get_registered_video_model_types(self) -> Tuple[str, ...]:
-        return (
+        return tuple(
             model_type
             for model_type in self._multimodal_placeholder_by_model_type
             if "video" in self.
             _multimodal_placeholder_by_model_type[model_type].placeholder_map)
@@
     def get_registered_audio_model_types(self) -> Tuple[str, ...]:
-        return (
+        return tuple(
             model_type
             for model_type in self._multimodal_placeholder_by_model_type
             if "audio" in self.
             _multimodal_placeholder_by_model_type[model_type].placeholder_map)

tensorrt_llm/_torch/models/modeling_qwen2vl.py (6)

1-1: Add the required NVIDIA 2025 Apache-2.0 header block.

Per repository guidelines, prepend the license header to this Python source file.

Apply this diff:

+ # Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+ # Licensed under the Apache License, Version 2.0 (the "License");
+ # you may not use this file except in compliance with the License.
+ # You may obtain a copy of the License at
+ #     http://www.apache.org/licenses/LICENSE-2.0
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.

---

224-226: Fix Python 3.8 typing: use typing.Dict/Any rather than PEP 585 builtins; also correct any to Any.

dict[str, any] is invalid on 3.8 and any is a function.

Apply:

-    def _preprocess(self, text: dict[str, any], mm_data: dict[str, any],
+    def _preprocess(self, text: Dict[str, Any], mm_data: Dict[str, Any],
                     mm_processor_kwargs: Dict[str, Any]):

---

252-258: Fix Python 3.8 typing in return annotation.

Replace PEP 585 dict[...] with Dict[...].

Apply:

-    def get_mrope_config(
+    def get_mrope_config(
             self,
             input_ids: torch.IntTensor,
             image_grid_thw: torch.LongTensor,
             video_grid_thw: torch.LongTensor,
             attention_mask: torch.Tensor,
-            second_per_grid_ts: torch.Tensor = None) -> dict[str, torch.Tensor]:
+            second_per_grid_ts: torch.Tensor = None) -> Dict[str, torch.Tensor]:

---

319-321: Fix Python 3.8 typing: use Type[...] instead of type[...].

Also import Type from typing.

Apply:

-from typing import Any, Dict, List, Optional, Tuple, Union
+from typing import Any, Dict, List, Optional, Tuple, Union, Type

-    def __init__(self, model_config: ModelConfig[PretrainedConfig],
-                 model_class: type[PreTrainedModel]):
+    def __init__(self, model_config: ModelConfig[PretrainedConfig],
+                 model_class: Type[PreTrainedModel]):

---

123-131: Avoid shadowing input_ids parameter; it leaks into later usage.

Inner loop reuses input_ids, then Line 221 uses .device on the (now 1D) shadowed tensor. Rename local vars and reference the 2D tensor for device.

Apply:

-        for i, input_ids in enumerate(total_input_ids):
-            input_ids = input_ids[attention_mask[i] == 1]
+        for i, seq_input_ids in enumerate(total_input_ids):
+            seq_input_ids = seq_input_ids[attention_mask[i] == 1]
             image_nums, video_nums = 0, 0
             vision_start_indices = torch.argwhere(
-                input_ids == vision_start_token_id).squeeze(1)
-            vision_tokens = input_ids[vision_start_indices + 1]
+                seq_input_ids == vision_start_token_id).squeeze(1)
+            vision_tokens = seq_input_ids[vision_start_indices + 1]
             image_nums = (vision_tokens == image_token_id).sum()
             video_nums = (vision_tokens == video_token_id).sum()
-            input_tokens = input_ids.tolist()
+            input_tokens = seq_input_ids.tolist()

-        mrope_position_deltas = torch.tensor(
-            mrope_position_deltas, device=input_ids.device).unsqueeze(1)
+        mrope_position_deltas = torch.tensor(
+            mrope_position_deltas, device=total_input_ids.device).unsqueeze(1)

Also applies to: 220-221

---

246-250: Guard against missing vision_token_id; avoid AttributeError.

Use getattr and build the mask robustly.

Apply:

-        masks = (input_ids == self.model_config.image_token_id) | (
-            input_ids == self.model_config.vision_token_id) | (
-                input_ids == self.model_config.video_token_id)
+        masks = (input_ids == self.model_config.image_token_id) | (
+            input_ids == self.model_config.video_token_id)
+        vision_token_id = getattr(self.model_config, "vision_token_id", None)
+        if vision_token_id is not None:
+            masks = masks | (input_ids == vision_token_id)