tests/integration/test_lists/qa/README.md (1)

35-35: Clarify requirements install path or add a note about TensorRT-LLM_PATH

Using ${TensorRT-LLM_PATH} is fine, but many users will run from repo root. Either change to a relative path or add a note to export the variable before running to avoid confusion.

If you prefer to keep the env var, consider this minimal tweak for clarity:

-pip3 install -r ${TensorRT-LLM_PATH}/requirements-dev.txt
+# from repo root:
+# pip3 install -r requirements-dev.txt
+# or, if using an env var:
+pip3 install -r ${TensorRT-LLM_PATH}/requirements-dev.txt

tests/integration/test_lists/qa/llm_function_l20.txt (4)

1-1: Nit: capitalize and hyphenate “single-GPU” in the header

Minor readability/consistency improvement with the README entry.

-# only covers single gpu cases
+# Only covers single-GPU cases

---

35-36: Verify feasibility of Qwen3-235B tests on L20

The A22B/235B variants are extremely large; even with fp8/throughput/latency params, these usually exceed single L20 memory. If these rely on mocked data or reduced configs, add a comment stating that. Otherwise, please move or guard them.

Option A (comment them out here, keep in a larger-GPU list):

-accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_fp8[throughput_latency]
-accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_fp8[latency]
+# (Commented out for L20 single-GPU; move to multi-GPU list if needed)
+# accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_fp8[throughput_latency]
+# accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_fp8[latency]

Option B: keep them but add explicit skip handling in the tests based on device/GPU memory.

---

19-34: Confirm attn backends and guided decoding feature support on L20

FLASHINFER and TRT-LLM backends plus guided decoding (xgrammar/llguidance), EAGLE3, and N-gram features can have backend- or arch-specific constraints. If any are not supported on L20, mark them skipped or move to an appropriate list to avoid red CI.

I can help generate a pytest skip marker strategy keyed by GPU type or capability if you want to keep everything in one place.

---

48-69: Sanity-check multimodal quickstart selections for L20 memory/perf

Video/image variants and larger models (e.g., Gemma-3-27B) may not fit single L20 without special configs. If these rely on quantization, small prompts, or mock assets, consider adding a brief header comment to this file to note that, or move heavy cases to a separate list.

If needed, I can split this list into “L20-light” and “L20-heavy” to control CI scheduling.

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: yibinl-nvidia
PR: NVIDIA/TensorRT-LLM#6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.

tests/integration/test_lists/qa/llm_function_l20.txt (1)

28-29: Remove 4-GPU cases from a single-GPU L20 list

The two entries explicitly require 4 GPUs and contradict this file’s scope. They’ll fail on single-GPU L20 runners and may cause mis-scheduling.

Apply the diff to drop them:

-accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[xgrammar]
-accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[llguidance]

tests/integration/test_lists/qa/llm_function_l20.txt (2)

1-1: Use a structured header to aid CI mapping and linting

The free-form comment is human-friendly but hard to validate programmatically. Recommend a parseable header (kept as a comment to avoid breaking consumers) so mapping/lint tools can enforce constraints.

-# only covers single gpu cases
+# test-list: llm_function_l20
+# single_gpu: true
+# gpu_arch: L20
+# note: only covers single-GPU cases

I can add a small validator in scripts/test_to_stage_mapping.py to enforce single_gpu: true lists contain no entries with “4gpus”.

---

45-53: Verify multimodal video cases have runtime deps on L20 runners

The NVILA/LLava/Qwen-VL video cases often require ffmpeg/NVDEC and larger host RAM/disk for sample assets. Ensure L20 CI images include these; otherwise, gate video parametrizations or replace with image-only variants for this list.

If needed, I can split video cases into a dedicated llm_function_l20_multimedia.txt and keep this file image-only to reduce flakiness.

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.