[None][fix] Disable torch.compile for CapturableGuidedDecoder #7871

syuoni · 2025-09-19T10:56:23Z

Description

torch.compile kernels are called with GIL being held (pytorch/pytorch#163061). As described here, this could deadlock with CUDA callback to Python code.

This PR disables torch.compile globally when CapturableGuidedDecoder is used.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

syuoni · 2025-09-19T10:56:37Z

/bot run

tensorrt-cicd · 2025-09-19T11:02:04Z

PR_Github #19349 [ run ] triggered by Bot

coderabbitai · 2025-09-19T11:05:05Z

📝 Walkthrough

Walkthrough

Adds torch.compiler.set_stance("force_eager") in GuidedDecoder.init and CapturableGuidedDecoder.init to enforce eager execution when the GIL is held, referencing PyTorch issue 163061. No public APIs were changed.

Changes

Cohort / File(s)	Summary
Torch compile stance safeguard `tensorrt_llm/_torch/pyexecutor/guided_decoder.py`	Inserted torch.compiler.set_stance("force_eager") with a comment in GuidedDecoder.init and CapturableGuidedDecoder.init; no signature changes; reference to PyTorch issue 163061.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant App as Application
  participant GD as GuidedDecoder
  participant TC as torch.compiler

  App->>GD: __init__(...)
  activate GD
  GD->>TC: set_stance("force_eager")
  note right of TC: Configure kernels to run eagerly
  deactivate GD

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description Check	⚠️ Warning	The PR description contains only the repository template and checklist with no substantive "Description" or "Test Coverage" content; it does not explain what was changed, why the change was needed, or which tests cover the change, even though the raw summary and objectives indicate the code inserts torch.compiler.set_stance("force_eager") in GuidedDecoder.init and CapturableGuidedDecoder.init to avoid deadlocks.	Please complete the PR description by adding a concise "Description" that explains the change and rationale (mention the guided_decoder.py edits and reference the related PyTorch issue if applicable), a "Test Coverage" section listing unit/integration tests or manual verification steps that exercise the new behavior, and any relevant notes for reviewers (compatibility, docs, CODEOWNERS) before merging.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The PR title "[None][fix] Disable torch.compile for CapturableGuidedDecoder" is concise and accurately identifies a real change in the changeset (disabling torch.compile behavior for CapturableGuidedDecoder), so it meaningfully summarizes the primary intent; however, the diff also adds the same defensive set_stance call in GuidedDecoder.init, so the title omits a related change and could be broadened for completeness.

✨ Finishing touches

📝 Generate Docstrings

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7d28acd and 94f24ed.

📒 Files selected for processing (1)

tensorrt_llm/_torch/pyexecutor/guided_decoder.py (1 hunks)

🧰 Additional context used

📓 Path-based instructions (3)

**/*.{h,hpp,hh,hxx,cpp,cxx,cc,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Use only spaces, no tabs; indent with 4 spaces.

Files:

tensorrt_llm/_torch/pyexecutor/guided_decoder.py

**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Python code must target Python 3.8+.
Indent Python code with 4 spaces; do not use tabs.
Maintain module namespace when importing; prefer 'from package.subpackage import foo' then 'foo.SomeClass()' instead of importing the class directly.
Python filenames should be snake_case (e.g., some_file.py).
Python classes use PascalCase names.
Functions and methods use snake_case names.
Local variables use snake_case; prefix 'k' for variables that start with a number (e.g., k_99th_percentile).
Global variables use upper SNAKE_CASE prefixed with 'G' (e.g., G_MY_GLOBAL).
Constants use upper SNAKE_CASE (e.g., MY_CONSTANT).
Avoid shadowing variables from an outer scope.
Initialize all externally visible members of a class in the constructor.
Prefer docstrings for interfaces that may be used outside a file; comments for in-function or file-local interfaces.
Use Google-style docstrings for classes and functions (Sphinx-parsable).
Document attributes and variables inline so they render under the class/function docstring.
Avoid reflection when a simpler, explicit approach suffices (e.g., avoid dict(**locals()) patterns).
In try/except, catch the most specific exceptions possible.
For duck-typing try/except, keep the try body minimal and use else for the main logic.

Files:

tensorrt_llm/_torch/pyexecutor/guided_decoder.py

**/*.{cpp,cxx,cc,h,hpp,hh,hxx,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Prepend the NVIDIA Apache-2.0 copyright header with current year to the top of all source files (e.g., .cpp, .h, .cu, .py).

Files:

tensorrt_llm/_torch/pyexecutor/guided_decoder.py

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

GitHub Check: Pre-commit Check

🔇 Additional comments (1)

tensorrt_llm/_torch/pyexecutor/guided_decoder.py (1)

427-430: Approve — keep fix but make stance scope explicit

Fix in CapturableGuidedDecoder prevents torch.compile/GIL deadlocks. Because torch.compiler.set_stance("force_eager") is process-global, either also call it from the base GuidedDecoder (tensorrt_llm/_torch/pyexecutor/guided_decoder.py — class GuidedDecoder) for consistency, or document why it should be restricted to CapturableGuidedDecoder.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

syuoni · 2025-09-20T05:38:33Z

/bot run

tensorrt-cicd · 2025-09-20T05:44:13Z

PR_Github #19393 [ run ] triggered by Bot

tensorrt-cicd · 2025-09-20T10:37:33Z

PR_Github #19393 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #14568 completed with status: 'SUCCESS'

tensorrt_llm/_torch/pyexecutor/guided_decoder.py

QiJune

LGTM

…#7871) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

fix

94f24ed

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

syuoni requested review from QiJune and juney-nvidia September 19, 2025 10:56

syuoni self-assigned this Sep 19, 2025

syuoni requested a review from a team as a code owner September 19, 2025 10:56

coderabbitai bot mentioned this pull request Sep 20, 2025

[None][doc] Update tech blog12 #7884

Merged

1 task

QiJune requested a review from leslie-fang25 September 22, 2025 01:39

QiJune reviewed Sep 22, 2025

View reviewed changes

tensorrt_llm/_torch/pyexecutor/guided_decoder.py Show resolved Hide resolved

QiJune approved these changes Sep 22, 2025

View reviewed changes

syuoni merged commit 639d410 into NVIDIA:main Sep 22, 2025
7 of 9 checks passed

syuoni deleted the fix-deadlock branch September 22, 2025 02:04

JunyiXu-nv pushed a commit to JunyiXu-nv/TensorRT-LLM that referenced this pull request Sep 22, 2025

[None][fix] Disable torch.compile for CapturableGuidedDecoder (NVIDIA…

0f422aa

…#7871) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

nv-lschneider pushed a commit to nv-lschneider/TensorRT-LLM that referenced this pull request Sep 22, 2025

[None][fix] Disable torch.compile for CapturableGuidedDecoder (NVIDIA…

9829e39

…#7871) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[None][fix] Disable torch.compile for CapturableGuidedDecoder #7871

[None][fix] Disable torch.compile for CapturableGuidedDecoder #7871

Uh oh!

syuoni commented Sep 19, 2025 •

edited

Loading

Uh oh!

syuoni commented Sep 19, 2025

Uh oh!

tensorrt-cicd commented Sep 19, 2025

Uh oh!

coderabbitai bot commented Sep 19, 2025

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Uh oh!

syuoni commented Sep 20, 2025

Uh oh!

tensorrt-cicd commented Sep 20, 2025

Uh oh!

tensorrt-cicd commented Sep 20, 2025

Uh oh!

Uh oh!

QiJune left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[None][fix] Disable torch.compile for CapturableGuidedDecoder #7871

[None][fix] Disable torch.compile for CapturableGuidedDecoder #7871

Uh oh!

Conversation

syuoni commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

syuoni commented Sep 19, 2025

Uh oh!

tensorrt-cicd commented Sep 19, 2025

Uh oh!

coderabbitai bot commented Sep 19, 2025

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Pre-merge checks and finishing touches

Uh oh!

syuoni commented Sep 20, 2025

Uh oh!

tensorrt-cicd commented Sep 20, 2025

Uh oh!

tensorrt-cicd commented Sep 20, 2025

Uh oh!

Uh oh!

QiJune left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

syuoni commented Sep 19, 2025 •

edited

Loading