tensorrt_llm/_torch/auto_deploy/shim/demollm.py (2)

124-129: Include end_id safely (handle None and duplicates).

Guard against None and avoid duplicating the single-token stop sequence.

-        stop_tokens.append([sampling_params.end_id])
+        # Add EOS as a stop sequence if provided and not already present.
+        if sampling_params.end_id is not None and [sampling_params.end_id] not in stop_tokens:
+            stop_tokens.append([int(sampling_params.end_id)])

---

1-1: Missing NVIDIA Apache-2.0 header (2025).

Per repo guidelines, prepend the license header.

+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

tensorrt_llm/_torch/auto_deploy/models/patches/pixtral.py (2)

56-57: Typo in meta signature: max_widht.

Use max_width to mirror the op’s schema.

-    max_widht: int,
+    max_width: int,

---

1-7: Missing NVIDIA Apache-2.0 header (2025).

Add the license header at the top.

+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

tensorrt_llm/_torch/auto_deploy/models/factory.py (1)

1-1: Missing NVIDIA Apache-2.0 header (2025).

Per repo standard, add the header.

+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

tensorrt_llm/_torch/auto_deploy/models/__init__.py (1)

1-2: Avoid wildcard export and add license header.

• Using from .factory import * makes the public surface implicit and brittle.
• Per repo guidelines, prepend the NVIDIA Apache-2.0 header.

Apply header addition (example):

+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from . import hf, mistral3, patches
-from .factory import *
+from .factory import (  # TODO: list explicit symbols instead of wildcard
+    *
+)

tests/unittest/_torch/auto_deploy/unit/singlegpu/models/test_mistral3.py (1)

14-15: Avoid relying on dict for Dim equivalence; compare stable representations.

__dict__ is brittle (e.g., __slots__ or internal changes). Compare repr instead, or specific attributes if available.

Apply:

-    # Unfortunately, direct object comparisons do not work.
-    assert pixel_values_dynamic_shape[0].__dict__ == image_sizes_dynamic_shape[0].__dict__
+    # Direct object comparisons may not work; compare stable representations.
+    assert repr(pixel_values_dynamic_shape[0]) == repr(image_sizes_dynamic_shape[0])

tensorrt_llm/_torch/auto_deploy/llm_args.py (1)

217-221: Sort registry entries for deterministic error message.

Stable ordering aids testing and readability.

Apply:

-        if not ModelFactoryRegistry.has(value):
-            raise ValueError(
-                f"'{value}' does not exist in the model factory registry. Available values: "
-                f"{ModelFactoryRegistry.entries()}."
-            )
+        if not ModelFactoryRegistry.has(value):
+            available = ", ".join(sorted(ModelFactoryRegistry.entries()))
+            raise ValueError(
+                f"'{value}' does not exist in the model factory registry. "
+                f"Available values: [{available}]."
+            )

tensorrt_llm/_torch/auto_deploy/models/mistral3.py (1)

24-29: Guard against missing pixel_values entry

If a child overrides get_extra_inputs and renames/removes pixel_values, accessing it here will KeyError. Add a defensive check.

-        batch_dim = extra_inputs["pixel_values"][1]()[0]
+        assert "pixel_values" in extra_inputs, "pixel_values extra input is required"
+        batch_dim = extra_inputs["pixel_values"][1]()[0]

tensorrt_llm/_torch/auto_deploy/models/hf.py (3)

376-393: Regex remap: precompile patterns and guard overlapping substitutions.

Repeated re.sub per-key per-pattern can be expensive and order-dependent. Precompile once and skip if no match.

Apply:

-        conversion_mapping = self._checkpoint_conversion_mapping
+        conversion_mapping = self._checkpoint_conversion_mapping
         if conversion_mapping:
+            compiled = [(re.compile(p), r) for p, r in conversion_mapping.items()]
             keys_to_process = list(state_dict.keys())
             for key in keys_to_process:
-                new_key = key
-                for pattern, replacement in conversion_mapping.items():
-                    new_key = re.sub(pattern, replacement, new_key)
+                new_key = key
+                for pat, replacement in compiled:
+                    new_key_new, n = pat.subn(replacement, new_key)
+                    if n:
+                        new_key = new_key_new
             if new_key != key:
                 state_dict[new_key] = state_dict.pop(key)

---

395-432: Reverse-mapping is lossy; add sanity checks and safer reversal.

lstrip("^") and stripping (...) can mis-reverse patterns; enforce bijection and only rewrite when exactly one rule matches.

Apply:

-        reverse_key_mapping = {v: k for k, v in conversion_mapping.items()}
-        self._mapping = reverse_key_mapping
+        reverse_key_mapping = {v: k for k, v in conversion_mapping.items()}
+        # Only keep unique reversals
+        self._mapping = {k: v for k, v in reverse_key_mapping.items() if list(reverse_key_mapping.values()).count(v) == 1}

And in __call__:

-                for pattern, replacement in self._mapping.items():
-                    replacement = replacement.lstrip("^")
-                    replacement = re.sub(r"\(.*\)", "", replacement)
-                    new_key, n_replace = re.subn(pattern, replacement, key)
+                matches = []
+                for pattern, replacement in self._mapping.items():
+                    new_key_trial, n_replace = re.subn(pattern, replacement, key)
+                    if n_replace:
+                        matches.append((new_key_trial, n_replace))
+                if len(matches) == 1:
+                    new_key = matches[0][0]

---

563-565: Dynamic shape min(32) vs example image dims default(16).

The dynamic shape min is 32 while _example_image_dims defaults to (16, 16). Align to avoid shape-constraint contradictions in export.

Apply:

-        none_pixel_values = torch.zeros(0, 3, 336, 336)
+        none_pixel_values = torch.zeros(0, 3, 336, 336)
         return {"pixel_values": (none_pixel_values, _get_dynamic_shape)}

And in _get_dynamic_shape:

-                2: Dim("img_height", min=32, max=2048),
-                3: Dim("img_width", min=32, max=2048),
+                2: Dim("img_height", min=self._example_image_dims[0], max=2048),
+                3: Dim("img_width", min=self._example_image_dims[1], max=2048),

tensorrt_llm/_torch/auto_deploy/models/patches/mistral3.py (2)

77-87: Validity check is correct but non-obvious; consider readability.

Optional: replace XOR with equality check for clarity.

Apply:

-    if (input_ids is None) ^ (inputs_embeds is not None):
+    if (input_ids is None) == (inputs_embeds is None):
         raise ValueError("You must specify exactly one of input_ids or inputs_embeds")

---

103-118: Pixel dtype cast: prefer model/vision dtype over hardcoded bfloat16.

Unconditional bfloat16 can break on hardware or models not using bf16. Use the vision tower param dtype.

Apply:

-        pixel_values = pixel_values.to(torch.bfloat16)
+        target_dtype = next(self.vision_tower.parameters()).dtype if hasattr(self.vision_tower, "parameters") else pixel_values.dtype
+        pixel_values = pixel_values.to(target_dtype)

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: samuellees
PR: NVIDIA/TensorRT-LLM#6974
File: tensorrt_llm/serve/scripts/benchmark_dataset.py:558-566
Timestamp: 2025-08-18T08:42:02.640Z
Learning: In TensorRT-LLM's RandomDataset (tensorrt_llm/serve/scripts/benchmark_dataset.py), when using --random-token-ids option, sequence length accuracy is prioritized over semantic correctness for benchmarking purposes. The encode/decode operations should use skip_special_tokens=True and add_special_tokens=False to ensure exact target token lengths.

Learnt from: venkywonka
PR: NVIDIA/TensorRT-LLM#6650
File: tests/integration/test_lists/qa/llm_perf_cluster.yml:33-37
Timestamp: 2025-08-06T03:47:16.802Z
Learning: Ministral is a valid model name from Mistral AI, distinct from the regular Mistral models. In TensorRT-LLM test configurations, "ministral_8b" and "ministral_8b_fp8" are correct model identifiers and should not be changed to "mistral_8b".