KEMBAR78
[None][test] add test-model-suites option in integration conftest.py by ruodil · Pull Request #8016 · NVIDIA/TensorRT-LLM · GitHub
Skip to content

Conversation

@ruodil
Copy link
Collaborator

@ruodil ruodil commented Sep 26, 2025

Summary by CodeRabbit

  • Tests
    • Added a new CLI option --test-model-suites to run only tests matching specified suite names.
    • Supports multiple suite names separated by spaces or semicolons.
    • Applies after existing regex and waiver-based filters for precise selection.
    • Ensures consistent deselection and reporting during test collection, improving targeted test runs.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 26, 2025

📝 Walkthrough

Walkthrough

Adds a new pytest option (--test-model-suites) and integrates suite-based test filtering into collection. Introduces a helper to parse suite names and deselect non-matching tests. Applies filtering after existing regex and waiver-based filters within pytest_generate_tests and pytest_collection_modifyitems.

Changes

Cohort / File(s) Summary of Changes
Test selection and filtering
tests/integration/defs/conftest.py
Added --test-model-suites option; introduced deselect_by_test_model_suites(test_model_suites, items, test_prefix, config); parses semicolon/space-delimited suite names; matches against derived test names (with optional prefix); triggers pytest_deselected; integrates filtering in pytest_generate_tests and pytest_collection_modifyitems after existing regex/waiver filters.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant User as CLI
  participant Pytest as pytest
  participant Conftest as conftest.py
  participant Helper as deselect_by_test_model_suites

  User->>Pytest: run pytest [--test-model-suites "..."]
  Pytest->>Conftest: pytest_generate_tests(items, config)
  Note over Conftest: Existing regex/waiver filtering occurs
  Conftest->>Helper: deselect_by_test_model_suites(opts, items, prefix, config)
  Helper-->>Conftest: selected_items, deselected_items
  Conftest-->>Pytest: call pytest_deselected(deselected_items)

  Pytest->>Conftest: pytest_collection_modifyitems(items, config)
  Note over Conftest: Apply suite filtering after prior filters
  Conftest->>Helper: deselect_by_test_model_suites(opts, items, prefix, config)
  Helper-->>Conftest: updated items list
  Conftest-->>Pytest: final items to run
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
Description Check ⚠️ Warning The pull request description includes the template scaffolding but no actual title, description text, or test coverage details outside of HTML comments so the required template sections are effectively empty. Please replace the commented template lines with a valid title following the repository format, add substantive content under “## Description” and “## Test Coverage,” and complete or adjust the PR checklist items as needed.
✅ Passed checks (1 passed)
Check name Status Explanation
Title Check ✅ Passed The title concisely describes the primary change by indicating that a test-model-suites option is being added in the integration conftest.py file and follows the repository’s required ticket and type format.
✨ Finishing touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
tests/integration/defs/conftest.py (2)

2062-2069: Improve CLI UX: allow multiple --test-model-suites flags and clarify help

Let users pass the flag multiple times instead of space/semicolon splitting in a single string. This is more standard for pytest and avoids parsing pitfalls.

Apply this diff:

-    parser.addoption(
-        "--test-model-suites",
-        action="store",
-        default=None,
-        help=
-        "Specify test model suites separated by semicolons or spaces. Each suite can contain special characters. "
-        "Example: --test-model-suites=suite1;suite2;suite3 or --test-model-suites=suite1 suite2 suite3",
-    )
+    parser.addoption(
+        "--test-model-suites",
+        action="append",
+        default=[],
+        help=(
+            "Specify test model suites. May be provided multiple times, or as a single value "
+            "with names separated by semicolons or spaces. "
+            "Examples: --test-model-suites suite1 --test-model-suites suite2 "
+            "or --test-model-suites='suite1;suite2'"
+        ),
+    )

2169-2212: Make matching case-insensitive and handle list inputs; simplify loop

  • Accept list inputs (if switching to action="append").
  • Case-insensitive substring match to reduce surprises.
  • Minor simplification and micro-optimizations.

Apply this diff:

-def deselect_by_test_model_suites(test_model_suites, items, test_prefix, config):
-    """Filter tests based on the test model suites specified.
-    If a test matches any of the test model suite names, it is considered selected.
-    
-    Args:
-        test_model_suites: String containing test model suite names separated by semicolons
-        items: List of pytest items to filter
-        test_prefix: Test prefix if any
-        config: Pytest config object
-    """
-    if not test_model_suites:
-        return
-    
-    # Split by semicolon or space and strip whitespace
-    suite_names = [suite.strip() for suite in test_model_suites.replace(';', ' ').split() if suite.strip()]
-    
-    if not suite_names:
-        return
-    
-    selected = []
-    deselected = []
-    
-    for item in items:
-        # Get the test name without prefix for comparison
-        test_name = item.nodeid
-        if test_prefix and test_name.startswith(f"{test_prefix}/"):
-            test_name = test_name[len(f"{test_prefix}/"):]
-        
-        # Check if any suite name matches the test name
-        found = False
-        for suite_name in suite_names:
-            if suite_name in test_name or test_name.endswith(suite_name):
-                found = True
-                break
-        
-        if found:
-            selected.append(item)
-        else:
-            deselected.append(item)
-    
-    if deselected:
-        config.hook.pytest_deselected(items=deselected)
-    items[:] = selected
+def deselect_by_test_model_suites(test_model_suites, items, test_prefix, config):
+    """Filter tests based on the test model suites specified.
+    A test is selected if any suite name is a substring of the test's nodeid (case-insensitive).
+
+    Args:
+        test_model_suites: String or list of strings with suite names. Strings may contain semicolon/space-separated names.
+        items: List of pytest items to filter.
+        test_prefix: Optional test prefix to strip from nodeid during matching.
+        config: Pytest config object.
+    """
+    if not test_model_suites:
+        return
+
+    # Normalize input to a list of suite tokens
+    if isinstance(test_model_suites, str):
+        raw_tokens = test_model_suites.replace(";", " ").split()
+    else:
+        raw_tokens = []
+        for entry in test_model_suites:
+            if entry:
+                raw_tokens.extend(str(entry).replace(";", " ").split())
+
+    suite_names = [t.strip().lower() for t in raw_tokens if t and t.strip()]
+    if not suite_names:
+        return
+
+    selected, deselected = [], []
+    for item in items:
+        # Get the test name without prefix for comparison
+        test_name = item.nodeid
+        if test_prefix and test_name.startswith(f"{test_prefix}/"):
+            test_name = test_name[len(f"{test_prefix}/"):]
+        test_name_lower = test_name.lower()
+
+        if any(suite in test_name_lower for suite in suite_names):
+            selected.append(item)
+        else:
+            deselected.append(item)
+
+    if deselected:
+        config.hook.pytest_deselected(items=deselected)
+    items[:] = selected

Optional: If you want to support quoted suite names (with spaces), switch to shlex-based parsing and add import shlex at the top:

  • Parsing idea: for chunk in test_model_suites.split(';'): suite_names.extend(shlex.split(chunk))
📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2db22fb and 128fd66.

📒 Files selected for processing (1)
  • tests/integration/defs/conftest.py (4 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{h,hpp,hh,hxx,cpp,cxx,cc,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Use only spaces, no tabs; indent with 4 spaces.

Files:

  • tests/integration/defs/conftest.py
**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Python code must target Python 3.8+.
Indent Python code with 4 spaces; do not use tabs.
Maintain module namespace when importing; prefer 'from package.subpackage import foo' then 'foo.SomeClass()' instead of importing the class directly.
Python filenames should be snake_case (e.g., some_file.py).
Python classes use PascalCase names.
Functions and methods use snake_case names.
Local variables use snake_case; prefix 'k' for variables that start with a number (e.g., k_99th_percentile).
Global variables use upper SNAKE_CASE prefixed with 'G' (e.g., G_MY_GLOBAL).
Constants use upper SNAKE_CASE (e.g., MY_CONSTANT).
Avoid shadowing variables from an outer scope.
Initialize all externally visible members of a class in the constructor.
Prefer docstrings for interfaces that may be used outside a file; comments for in-function or file-local interfaces.
Use Google-style docstrings for classes and functions (Sphinx-parsable).
Document attributes and variables inline so they render under the class/function docstring.
Avoid reflection when a simpler, explicit approach suffices (e.g., avoid dict(**locals()) patterns).
In try/except, catch the most specific exceptions possible.
For duck-typing try/except, keep the try body minimal and use else for the main logic.

Files:

  • tests/integration/defs/conftest.py
**/*.{cpp,cxx,cc,h,hpp,hh,hxx,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Prepend the NVIDIA Apache-2.0 copyright header with current year to the top of all source files (e.g., .cpp, .h, .cu, .py).

Files:

  • tests/integration/defs/conftest.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Pre-commit Check
🔇 Additional comments (1)
tests/integration/defs/conftest.py (1)

2116-2116: Suite filtering integration looks correct; confirm precedence

Running suite filtering after regex and test-list filtering, and before waives, is sensible. Please confirm this precedence is intentional (i.e., waives only apply to the already suite-filtered set).

Also applies to: 2147-2149

Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com>
@ruodil ruodil force-pushed the user/ruodil/org_cases branch from 128fd66 to 7d07564 Compare September 26, 2025 07:28
@ruodil
Copy link
Collaborator Author

ruodil commented Sep 26, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #20079 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #20079 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #15127 completed with status: 'FAILURE'

@ruodil
Copy link
Collaborator Author

ruodil commented Oct 7, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #20696 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #20696 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #15634 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

@ruodil ruodil merged commit 971610e into NVIDIA:main Oct 8, 2025
5 checks passed
kris1025 pushed a commit to kris1025/TensorRT-LLM that referenced this pull request Oct 14, 2025
…VIDIA#8016)

Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com>
Co-authored-by: Ruodi Lu <ruodil@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants