KEMBAR78
Fix GEMM+AR fusion on blackwell by xavier-nvidia · Pull Request #5563 · NVIDIA/TensorRT-LLM · GitHub
Skip to content

Fix GEMM+AR fusion on blackwell #5563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 9, 2025
Merged

Fix GEMM+AR fusion on blackwell #5563

merged 1 commit into from
Jul 9, 2025

Conversation

@xavier-nvidia
Copy link
Contributor

@xavier-nvidia xavier-nvidia commented Jun 28, 2025 •
edited
Loading

Description

  • Fix 'Assertion failed: file.is_open()' in GemmAllReducePlugin that occurs when TRT-LLM runs inside different container than used for building the engine - GemmAllReducePlugin would create kernel heuristics file in /tmp/, which won't exist in a different container. There are two solutions implemented 1) use static heuristics (default) which doesn't require heuristics file, and 2) specify directory of heuristics file via GEMM_AR_PLUGIN_ENGINE_DIR env variable.
  • Adds GB200 support

Test Coverage

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch 2 times, most recently from 953021c to 2ace5f5 Compare June 29, 2025 05:22
@xavier-nvidia
Copy link
Contributor Author

/bot run --only-multi-gpu-test

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch from 2ace5f5 to 56b0dc0 Compare June 29, 2025 19:44
@litaotju litaotju requested review from Tabrizian and yuxianq June 30, 2025 03:56
@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch 2 times, most recently from 542b552 to 1dab03c Compare June 30, 2025 18:27
@xavier-nvidia
Copy link
Contributor Author

/bot run --only-multi-gpu-test

1 similar comment
@niukuo
Copy link
Collaborator

niukuo commented Jul 1, 2025

/bot run --only-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10420 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10420 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #7704 (Partly Tested) completed with status: 'FAILURE'

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch from 1dab03c to 27f4a8b Compare July 1, 2025 18:29
@xavier-nvidia
Copy link
Contributor Author

/bot run --only-multi-gpu-test

@xavier-nvidia
Copy link
Contributor Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10702 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10702 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #7911 completed with status: 'FAILURE'

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch from 27f4a8b to 180e24d Compare July 3, 2025 00:06
@xavier-nvidia
Copy link
Contributor Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10717 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10717 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #7915 completed with status: 'FAILURE'

@xavier-nvidia
Copy link
Contributor Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10758 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10758 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #7946 completed with status: 'FAILURE'

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch from 180e24d to 8452c84 Compare July 3, 2025 22:38
@xavier-nvidia
Copy link
Contributor Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10877 [ run ] triggered by Bot

@xavier-nvidia
Copy link
Contributor Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10885 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10877 [ run ] completed with state ABORTED
/LLM/main/L0_MergeRequest_PR pipeline #8040 completed with status: 'FAILURE'

@xavier-nvidia
Copy link
Contributor Author

/bot run --post-merge --only-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11180 [ run ] triggered by Bot

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch 2 times, most recently from 0f6a77e to 51b2b53 Compare July 7, 2025 23:42
@xavier-nvidia
Copy link
Contributor Author

/bot run --only-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11194 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11180 [ run ] completed with state ABORTED

@nvzhihanj nvzhihanj requested a review from kaiyux July 8, 2025 00:29
@tensorrt-cicd
Copy link
Collaborator

PR_Github #11194 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8283 (Partly Tested) completed with status: 'FAILURE'

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch from 51b2b53 to 2ae2dc8 Compare July 8, 2025 04:54
@xavier-nvidia
Copy link
Contributor Author

/bot run --only-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11212 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11212 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #8295 (Partly Tested) completed with status: 'FAILURE'

@xavier-nvidia
Copy link
Contributor Author

/bot run --only-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11218 [ run ] triggered by Bot

@kaiyux kaiyux enabled auto-merge (squash) July 8, 2025 06:43
@tensorrt-cicd
Copy link
Collaborator

PR_Github #11218 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8298 (Partly Tested) completed with status: 'FAILURE'

@kaiyux
Copy link
Member

kaiyux commented Jul 8, 2025

/bot run --only-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11306 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11306 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #8361 (Partly Tested) completed with status: 'FAILURE'

auto-merge was automatically disabled July 8, 2025 16:17

Head branch was pushed to by a user without write access

@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch from 2ae2dc8 to c26e37b Compare July 8, 2025 16:17
@xavier-nvidia
Fix GEMM+AR nvbugs 5219533,5127801,5072306
adb586f 
Update NVLS bootstrap to support MNNVL

Signed-off-by: xsimmons <xsimmons@nvidia.com>
@xavier-nvidia xavier-nvidia force-pushed the feat/gemm_AR_blackwell_trt branch from c26e37b to adb586f Compare July 8, 2025 16:19
@xavier-nvidia
Copy link
Contributor Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11331 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11331 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8382 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

@kaiyux kaiyux merged commit b6013da into NVIDIA:main Jul 9, 2025
3 checks passed
zhou-yuxin pushed a commit to zhou-yuxin/TensorRT-LLM that referenced this pull request Jul 15, 2025
@xavier-nvidia @zhou-yuxin
Fix GEMM+AR fusion on blackwell (NVIDIA#5563)
135b7a2 
Signed-off-by: xsimmons <xsimmons@nvidia.com>
Signed-off-by: Yuxin <yuxinz@nvidia.com>
@MartinMarciniszyn MartinMarciniszyn mentioned this pull request Jul 16, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@symphonylyh symphonylyh symphonylyh approved these changes

@yuxianq yuxianq Awaiting requested review from yuxianq

@Tabrizian Tabrizian Awaiting requested review from Tabrizian

@kaiyux kaiyux Awaiting requested review from kaiyux

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xavier-nvidia @niukuo @tensorrt-cicd @kaiyux @symphonylyh