tests/integration/defs/accuracy/accuracy_core.py (4)

70-80: Add a concise Google‑style docstring to HypothesisTestingParams.

Improves API clarity and aligns with the repo’s Python docstring guidelines.

 @dataclass(slots=True)
 class HypothesisTestingParams:
+    """Hypothesis testing configuration for accuracy checks.
+
+    Attributes:
+        ref_accuracy (float): Reference accuracy for threshold computation.
+            Typically in [0, 100] for percentage-like metrics; may be raw for
+            lower-is-better metrics (e.g., perplexity).
+        num_samples (int): Number of samples to evaluate.
+        alpha (float): Type I error rate in (0, 1). Defaults to 0.05.
+        beta (float): Type II error rate in (0, 1). Defaults to 0.2.
+        sigma (float): Per-sample score standard deviation scaled to the metric
+            range (e.g., 50.0 for Bernoulli accuracy in 0–100 scale).
+        higher_is_better (bool): True if larger metric is better; False otherwise.
+        theta (float): Computed minimum detectable effect (post-init).
+        threshold (float): Computed decision threshold (post-init).
+    """

---

112-119: Add failure margin to the assertion message for faster debugging.

Including how far the result is from the threshold speeds triage.

     def assert_passing(self, accuracy: float) -> None:
         compare_op = ">=" if self.higher_is_better else "<="
-        err_msg = f"Reference accuracy is {self.ref_accuracy:.3f}, threshold is {self.threshold:.3f}. Expected accuracy {compare_op} threshold, but got {accuracy:.3f}. Please see hypothesis testing report:\n{self.report(accuracy)}"
+        margin = (accuracy - self.threshold) if self.higher_is_better else (self.threshold - accuracy)
+        err_msg = (
+            f"Reference accuracy is {self.ref_accuracy:.3f}, threshold is {self.threshold:.3f}. "
+            f"Expected accuracy {compare_op} threshold (margin: {margin:.3f}), but got {accuracy:.3f}. "
+            f"Please see hypothesis testing report:\n{self.report(accuracy)}"
+        )
         if self.higher_is_better:
             assert accuracy >= self.threshold, err_msg
         else:
             assert accuracy <= self.threshold, err_msg

---

148-150: Complete the docstring with Returns and Raises sections (Google style).

Aligns with repo docs and clarifies contract.

-    def get_hypothesis_testing_params(self,
-                                      **acc_specs) -> HypothesisTestingParams:
-        """Get hypothesis testing parameters via accuracy specifications.
+    def get_hypothesis_testing_params(self,
+                                      **acc_specs) -> HypothesisTestingParams:
+        """Get hypothesis testing parameters via accuracy specifications.
 
-        Args:
-            acc_specs: Accuracy specifications, currently including:
-                dtype (str): Model data type. Defaults to 'auto'.
-                quant_algo (str): Quantizaion algorithm. Defaults to None.
-                kv_cache_quant_algo (str): KV cache quantizaion algorithm. Defaults to None.
-                spec_dec_algo (str): Speculative decoding algorithm. Defaults to None.
-                extra_acc_spec (str): Extra accuracy specifications. Defaults to None.
-        """
+        Args:
+            acc_specs: Accuracy specifications, currently including:
+                dtype (str): Model data type. Defaults to 'auto'.
+                quant_algo (str): Quantization algorithm. Defaults to None.
+                kv_cache_quant_algo (str): KV cache quantization algorithm. Defaults to None.
+                spec_dec_algo (str): Speculative decoding algorithm. Defaults to None.
+                extra_acc_spec (str): Extra accuracy specification tag. Defaults to None.
+
+        Returns:
+            HypothesisTestingParams: Filled hypothesis testing parameters for this task/model/spec.
+
+        Raises:
+            ValueError: If no matching reference is registered and TRTLLM_ACCURACY_NO_REFERENCE != "1".
+        """

Also applies to: 152-159

---

204-210: Clarify the integration-test log message.

It does not truly “skip verification”; it uses a permissive threshold. Make the message precise.

         if is_integration_test:
             logger.info(
-                "Running in INTEGRATION_TEST mode: using only 1 sample and skipping accuracy verification"
+                "Running in INTEGRATION_TEST mode: using 1 sample and a permissive threshold (effectively skipping strict accuracy verification)"
             )
             hypothesis_testing_params = HypothesisTestingParams(ref_accuracy=0,
                                                                 num_samples=1)

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro