tests/integration/defs/accuracy/references/cnn_dailymail.yaml (1)

48-55: Optional: Specify kv_cache_quant_algo for FP8 entries for parity

Many entries in this file include kv_cache_quant_algo alongside quant_algo for FP8. For consistency and to avoid ambiguity in future, consider adding kv_cache_quant_algo: FP8 to these two new FP8 entries.

Suggested edit:

 bigcode/starcoder2-7b:
   - accuracy: 26.611
   - quant_algo: FP8
-    accuracy: 26.611
+    kv_cache_quant_algo: FP8
+    accuracy: 26.611
 mistralai/Codestral-22B-v0.1:
   - accuracy: 30.316
   - quant_algo: FP8
-    accuracy: 30.316
+    kv_cache_quant_algo: FP8
+    accuracy: 30.316

tests/integration/defs/accuracy/references/mmlu.yaml (1)

232-239: Optional: Add kv_cache_quant_algo to FP8 entries for consistency

To match the style used by several other models (and to clarify whether KV cache is quantized), consider specifying kv_cache_quant_algo: FP8 under the FP8 entries.

Suggested edit:

 bigcode/starcoder2-7b:
   - accuracy: 41.35
   - quant_algo: FP8
-    accuracy: 41.35
+    kv_cache_quant_algo: FP8
+    accuracy: 41.35
 mistralai/Codestral-22B-v0.1:
   - accuracy: 61.72
   - quant_algo: FP8
-    accuracy: 61.72
+    kv_cache_quant_algo: FP8
+    accuracy: 61.72

tests/integration/defs/accuracy/test_llm_api_pytorch.py (1)

2427-2441: Nit: Hoist kv_cache_config to a class attribute for consistency

Other tests in this suite often define kv_cache_config on the class. Minor stylistic alignment and future-proofing if you add more methods.

Suggested edit:

 class TestCodestral_22B_V01(LlmapiAccuracyTestHarness):
     MODEL_NAME = "mistralai/Codestral-22B-v0.1"
     MODEL_PATH = f"{llm_models_root()}/Codestral-22B-v0.1"
+    kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.6)
 
     @pytest.mark.timeout(2400)
     @pytest.mark.skip_less_device_memory(80000)
     def test_auto_dtype(self):
-        kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.6)
-        with LLM(self.MODEL_PATH, kv_cache_config=kv_cache_config) as llm:
+        with LLM(self.MODEL_PATH, kv_cache_config=self.kv_cache_config) as llm:
             task = CnnDailymail(self.MODEL_NAME)
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
             task = GSM8K(self.MODEL_NAME)
             task.evaluate(llm)

tests/integration/defs/accuracy/test_llm_api.py (3)

443-461: Optional: factor out repeated evaluation into a helper to reduce duplication

Both test methods evaluate the same two tasks; make it a private helper for readability and future extensions.

Apply this diff within the class to DRY up the task execution:

 class TestStarCoder2_7B(LlmapiAccuracyTestHarness):
@@
     @pytest.mark.skip_less_device_memory(70000)
     def test_auto_dtype(self):
         with LLM(self.MODEL_PATH, kv_cache_config=self.kv_cache_config) as llm:
-            task = CnnDailymail(self.MODEL_NAME)
-            task.evaluate(llm)
-            task = MMLU(self.MODEL_NAME)
-            task.evaluate(llm)
+            self._eval_cnn_and_mmlu(llm)
@@
     def test_fp8(self):
         quant_config = QuantConfig(QuantAlgo.FP8)
         with LLM(self.MODEL_PATH,
                  quant_config=quant_config,
                  kv_cache_config=self.kv_cache_config) as llm:
-            task = CnnDailymail(self.MODEL_NAME)
-            task.evaluate(llm)
-            task = MMLU(self.MODEL_NAME)
-            task.evaluate(llm)
+            self._eval_cnn_and_mmlu(llm)
+
+    def _eval_cnn_and_mmlu(self, llm):
+        task = CnnDailymail(self.MODEL_NAME)
+        task.evaluate(llm)
+        task = MMLU(self.MODEL_NAME)
+        task.evaluate(llm)

---

451-462: Optional: consider FP8 KV-cache to align with weight quantization (if references expect it)

If the FP8 references for this model were generated with FP8 KV-cache, set kv_cache_quant_algo accordingly; otherwise keep as-is.

Apply if needed:

     def test_fp8(self):
         quant_config = QuantConfig(QuantAlgo.FP8)
-        with LLM(self.MODEL_PATH,
-                 quant_config=quant_config,
-                 kv_cache_config=self.kv_cache_config) as llm:
+        kv_cache_config = KvCacheConfig(
+            quant_algo=QuantAlgo.FP8,
+            free_gpu_memory_fraction=self.kv_cache_config.free_gpu_memory_fraction
+        )
+        with LLM(self.MODEL_PATH,
+                 quant_config=quant_config,
+                 kv_cache_config=kv_cache_config) as llm:
             task = CnnDailymail(self.MODEL_NAME)
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)

---

469-487: Optional: include GSM8K for Codestral to match available references and PyTorch parity

GSM8K accuracy refs for Codestral were added (and the PyTorch suite runs it). Consider adding it here for cross-backend parity.

Apply this diff to run GSM8K in both methods:

     @pytest.mark.skip_less_device_memory(80000)
     def test_auto_dtype(self):
         with LLM(self.MODEL_PATH, kv_cache_config=self.kv_cache_config) as llm:
             task = CnnDailymail(self.MODEL_NAME)
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
@@
     def test_fp8(self):
         quant_config = QuantConfig(QuantAlgo.FP8)
         with LLM(self.MODEL_PATH,
                  quant_config=quant_config,
                  kv_cache_config=self.kv_cache_config) as llm:
             task = CnnDailymail(self.MODEL_NAME)
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.