-
Notifications
You must be signed in to change notification settings - Fork 1.8k
[TRTLLM-6991][chore] add DeepSeek-R1 FP8 accuracy tests on Blackwell #6710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TRTLLM-6991][chore] add DeepSeek-R1 FP8 accuracy tests on Blackwell #6710
Conversation
📝 WalkthroughWalkthroughTest adjusts MOE and KV-cache initialization in TestDeepSeekR1.test_fp8_blockscale based on SM version; accuracy YAMLs add FP8 kv_cache entries for DeepSeek-R1; the test is added to QA lists and GB200 multi-node timeout; Jenkins multi-node splits increased by one. Changes
Sequence Diagram(s)sequenceDiagram
participant Runner
participant TestDeepSeekR1
participant SMChecker
Runner->>TestDeepSeekR1: run test_fp8_blockscale
TestDeepSeekR1->>SMChecker: get_sm_version()
SMChecker-->>TestDeepSeekR1: sm_version
alt sm_version == 100
TestDeepSeekR1->>TestDeepSeekR1: set moe_config(backend="DEEPGEMM", max_num_tokens=16384)\nset kv_cache_config(free_gpu_memory_fraction=0.6)
else
TestDeepSeekR1->>TestDeepSeekR1: set moe_config(default)\nset kv_cache_config(free_gpu_memory_fraction=0.9)
end
TestDeepSeekR1->>Runner: proceed with selected configs
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15 minutes Possibly related PRs
Suggested labels
Suggested reviewers
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tests/integration/defs/accuracy/test_llm_api_pytorch.py(1 hunks)tests/integration/test_lists/test-db/l0_dgx_b200.yml(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.py: Python code should conform to Python 3.8+.
Indent Python code with 4 spaces. Do not use tabs.
Always maintain the namespace when importing in Python, even if only one class or function from a module is used.
Python filenames should use snake_case (e.g., some_file.py).
Python classes should use PascalCase (e.g., class SomeClass).
Python functions and methods should use snake_case (e.g., def my_awesome_function():).
Python local variables should use snake_case. Prefix k for variable names that start with a number (e.g., k_99th_percentile).
Python global variables should use upper snake_case and prefix G (e.g., G_MY_GLOBAL).
Python constants should use upper snake_case (e.g., MY_CONSTANT).
Avoid shadowing variables declared in an outer scope in Python.
Initialize all externally visible members of a Python class in the constructor.
For interfaces that may be used outside a Python file, prefer docstrings over comments.
Comments in Python should be reserved for code within a function, or interfaces that are local to a file.
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx.
Attributes and variables in Python can be documented inline; attribute docstrings will be rendered under the class docstring.
Avoid using reflection in Python when functionality can be easily achieved without it.
When using try-except blocks in Python, limit the except to the smallest set of errors possible.
When using try-except blocks to handle multiple possible variable types in Python, keep the body of the try as small as possible, using the else block to implement the logic.
Files:
tests/integration/defs/accuracy/test_llm_api_pytorch.py
**/*.{cpp,h,hpp,cc,cxx,cu,py}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the current year. This includes .cpp, .h, .cu, .py, and any other source files which are compiled or interpreted.
Files:
tests/integration/defs/accuracy/test_llm_api_pytorch.py
🧠 Learnings (3)
📓 Common learnings
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.
📚 Learning: in tensorrt-llm testing, it's common to have both cli flow tests (test_cli_flow.py) and pytorch api ...
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
Applied to files:
tests/integration/test_lists/test-db/l0_dgx_b200.ymltests/integration/defs/accuracy/test_llm_api_pytorch.py
📚 Learning: in tensorrt-llm, test files (files under tests/ directories) do not require nvidia copyright headers...
Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.
Applied to files:
tests/integration/test_lists/test-db/l0_dgx_b200.yml
🧬 Code Graph Analysis (1)
tests/integration/defs/accuracy/test_llm_api_pytorch.py (3)
tensorrt_llm/_utils.py (1)
get_sm_version(681-683)tests/integration/defs/conftest.py (1)
get_sm_version(1857-1860)tensorrt_llm/llmapi/llm_args.py (1)
MoeConfig(166-188)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (1)
tests/integration/defs/accuracy/test_llm_api_pytorch.py (1)
1634-1642: LGTM: Appropriate hardware-specific MoE backend selection.The conditional logic correctly selects the DEEPGEMM backend for Blackwell B200 GPUs (SM version 100) while falling back to the default MoeConfig for other hardware. This aligns with the PR objective of adding DeepSeek-R1 FP8 accuracy tests specifically for Blackwell platform.
|
/bot run --post-merge |
|
PR_Github #14470 [ run ] triggered by Bot |
|
/bot kill |
|
PR_Github #14475 [ kill ] triggered by Bot |
|
PR_Github #14470 [ run ] completed with state |
|
PR_Github #14475 [ kill ] completed with state |
cda210a to
3563e05
Compare
|
/bot run --post-merge |
|
PR_Github #14490 [ run ] triggered by Bot |
|
PR_Github #14490 [ run ] completed with state |
|
/bot run --post-merge --disable-fail-fast |
|
PR_Github #14545 [ run ] triggered by Bot |
|
PR_Github #14545 [ run ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #14688 [ run ] triggered by Bot |
|
PR_Github #14688 [ run ] completed with state |
|
/bot run --disable-fail-fast --add-multi-gpu-test |
|
PR_Github #14709 [ run ] triggered by Bot |
|
PR_Github #14709 [ run ] completed with state |
|
/bot run --disable-fail-fast --post-merge --only-multi-gpu-test |
|
PR_Github #14740 [ run ] triggered by Bot |
|
/bot kill |
|
PR_Github #14770 [ kill ] triggered by Bot |
|
PR_Github #15375 [ run ] triggered by Bot |
|
PR_Github #15375 [ run ] completed with state |
|
/bot run |
|
PR_Github #15453 [ run ] triggered by Bot |
|
PR_Github #15453 [ run ] completed with state |
|
/bot run |
|
PR_Github #15501 [ run ] triggered by Bot |
|
PR_Github #15501 [ run ] completed with state |
|
/bot run |
|
PR_Github #15507 [ run ] triggered by Bot |
|
PR_Github #15507 [ run ] completed with state |
|
/bot run |
|
PR_Github #15533 [ run ] triggered by Bot |
|
PR_Github #15533 [ run ] completed with state |
|
/bot run |
|
PR_Github #15572 [ run ] triggered by Bot |
|
PR_Github #15572 [ run ] completed with state |
|
/bot run |
|
PR_Github #15603 [ run ] triggered by Bot |
|
PR_Github #15603 [ run ] completed with state |
|
/bot run |
|
PR_Github #15630 [ run ] triggered by Bot |
|
PR_Github #15630 [ run ] completed with state |
Description
Add DeepSeek-R1 FP8 accuracy tests on Blackwell.
Test Coverage
accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_fp8_blockscale[throughput]
Summary by CodeRabbit
Tests
Documentation
Chores