KEMBAR78
[nvbug/5337601][fix] Fix disagg + speculative decoding by Tabrizian · Pull Request #5558 · NVIDIA/TensorRT-LLM · GitHub
Skip to content

[nvbug/5337601][fix] Fix disagg + speculative decoding #5558

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 4, 2025
Merged

[nvbug/5337601][fix] Fix disagg + speculative decoding #5558

merged 2 commits into from
Jul 4, 2025

Conversation

@Tabrizian
Copy link
Member

@Tabrizian Tabrizian commented Jun 27, 2025 •
edited
Loading

[nvbug/5337601][fix] Fix disagg + speculative decoding

@Tabrizian Tabrizian requested a review from a team as a code owner June 27, 2025 23:09
@Tabrizian Tabrizian requested a review from lfr-0531 June 27, 2025 23:09
QiJune
QiJune reviewed Jun 28, 2025
View reviewed changes
@Tabrizian Tabrizian changed the base branch from main to release/0.21 June 30, 2025 16:42
@Tabrizian Tabrizian requested a review from a team as a code owner June 30, 2025 16:42
@Tabrizian Tabrizian changed the base branch from release/0.21 to main July 1, 2025 17:17
@Tabrizian Tabrizian changed the base branch from main to release/0.21 July 1, 2025 17:23
@Tabrizian Tabrizian requested a review from IzzyPutterman July 1, 2025 17:24
@Tabrizian
Copy link
Member Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10520 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10520 [ run ] completed with state FAILURE
/LLM/release-0.21/L0_MergeRequest_PR pipeline #93 completed with status: 'FAILURE'

@Tabrizian Tabrizian force-pushed the pr-5525 branch 2 times, most recently from d6e4460 to daf47ab Compare July 1, 2025 18:28
@Tabrizian
Copy link
Member Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10525 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10525 [ run ] completed with state FAILURE
/LLM/release-0.21/L0_MergeRequest_PR pipeline #95 completed with status: 'FAILURE'

@Tabrizian
Copy link
Member Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10528 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10528 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #96 completed with status: 'FAILURE'

@Tabrizian
Copy link
Member Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10535 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10694 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10689 [ run ] completed with state ABORTED

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10694 [ run ] completed with state FAILURE
/LLM/release-0.21/L0_MergeRequest_PR pipeline #126 completed with status: 'FAILURE'

@Tabrizian
Copy link
Member Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10707 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10707 [ run ] completed with state FAILURE
/LLM/release-0.21/L0_MergeRequest_PR pipeline #129 completed with status: 'FAILURE'

@Tabrizian
Copy link
Member Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10713 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10713 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #134 completed with status: 'FAILURE'

mikeiovine and others added 2 commits July 3, 2025 09:05
@mikeiovine @Tabrizian
[nvbug/5337601][fix] Fix disagg + speculative decoding
fc5b559 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
@Tabrizian
Minor edit
a628337 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

Fix accuracy for disagg + eagle

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
@Tabrizian
Copy link
Member Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10852 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10852 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #151 completed with status: 'ABORTED'

@Tabrizian
Copy link
Member Author

/bot run --add-multi-gpu-test

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10869 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #10869 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #152 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

lfr-0531
lfr-0531 reviewed Jul 4, 2025
View reviewed changes
@pcastonguay pcastonguay merged commit 518915b into NVIDIA:release/0.21 Jul 4, 2025
3 checks passed
@pcastonguay pcastonguay mentioned this pull request Jul 4, 2025
Merged
dc3671 pushed a commit to dc3671/TensorRT-LLM that referenced this pull request Jul 10, 2025
@Tabrizian @mikeiovine @dc3671
[nvbug/5337601][fix] Fix disagg + speculative decoding (NVIDIA#5558)
09a4359 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
dc3671 pushed a commit to dc3671/TensorRT-LLM that referenced this pull request Jul 10, 2025
@Tabrizian @mikeiovine @dc3671
[nvbug/5337601][fix] Fix disagg + speculative decoding (NVIDIA#5558)
7bdbe12 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
dc3671 pushed a commit to dc3671/TensorRT-LLM that referenced this pull request Jul 11, 2025
@Tabrizian @mikeiovine @dc3671
[nvbug/5337601][fix] Fix disagg + speculative decoding (NVIDIA#5558)
8684cb0 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
dc3671 pushed a commit to dc3671/TensorRT-LLM that referenced this pull request Jul 14, 2025
@Tabrizian @mikeiovine @dc3671
[nvbug/5337601][fix] Fix disagg + speculative decoding (NVIDIA#5558)
81a4b33 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
dc3671 pushed a commit to dc3671/TensorRT-LLM that referenced this pull request Jul 14, 2025
@Tabrizian @mikeiovine @dc3671
[nvbug/5337601][fix] Fix disagg + speculative decoding (NVIDIA#5558)
27c9b04 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
dc3671 pushed a commit to dc3671/TensorRT-LLM that referenced this pull request Jul 14, 2025
@Tabrizian @mikeiovine @dc3671
[nvbug/5337601][fix] Fix disagg + speculative decoding (NVIDIA#5558)
4b6af20 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
dc3671 pushed a commit that referenced this pull request Jul 14, 2025
@Tabrizian @mikeiovine @dc3671
[nvbug/5337601][fix] Fix disagg + speculative decoding (#5558)
c8874a7 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
@jhaotingc jhaotingc mentioned this pull request Jul 22, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@QiJune QiJune QiJune left review comments

@lfr-0531 lfr-0531 lfr-0531 left review comments

@pcastonguay pcastonguay pcastonguay approved these changes

@MartinMarciniszyn MartinMarciniszyn MartinMarciniszyn approved these changes

@IzzyPutterman IzzyPutterman Awaiting requested review from IzzyPutterman

@schetlur-nv schetlur-nv Awaiting requested review from schetlur-nv

+1 more reviewer

@raayandhar raayandhar raayandhar left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@Tabrizian @tensorrt-cicd @MartinMarciniszyn @QiJune @lfr-0531 @pcastonguay @raayandhar @mikeiovine