[None][doc] Add deployment guide section to the official doc website #6669
Conversation
📝 WalkthroughWalkthroughA new "Deployment Guide" section was added to the documentation index, introducing three quick-start markdown guides for deploying Llama4 Scout, DeepSeek R1, and Llama3.3-70B models on TensorRT-LLM. The Llama3.3-70B guide is newly authored, and minor updates were made to the Llama4 Scout guide title and reference links. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Documentation Index
participant Deployment Guide (Llama3.3-70B)
participant Deployment Guide (Llama4 Scout)
participant Deployment Guide (DeepSeek R1)
User->>Documentation Index: Accesses index.rst
Documentation Index->>User: Displays new "Deployment Guide" section
User->>Deployment Guide (Llama3.3-70B): Reads Llama3.3-70B quick-start guide
User->>Deployment Guide (Llama4 Scout): Reads Llama4 Scout quick-start guide
User->>Deployment Guide (DeepSeek R1): Reads DeepSeek R1 quick-start guide
Estimated code review effort🎯 2 (Simple) | ⏱️ ~7 minutes Possibly related PRs
Suggested labels
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
docs/source/deployment-guide/quick-start-recipe-for-trttllm-llama4-scout.md (4)
1-1: Inconsistent model naming – “Llama4” vs “Llama 4”Throughout TensorRT-LLM docs we use the space-separated form “Llama 4”. Changing only the title to “Llama4” introduces a mixed style that hurts searchability and indexing.
Prefer “Llama 4 Scout 17B …” for consistency with Meta’s official model name and the license reference on Line 11.-# Quick Start Recipe for Llama4 Scout 17B FP8 and NVFP4 on TensorRT-LLM - Blackwell & Hopper Hardware +# Quick Start Recipe for Llama 4 Scout 17B FP8 and NVFP4 on TensorRT-LLM – Blackwell & Hopper Hardware
23-26: Model label / link mismatch (NVFP4 ≠ FP4)The bullet states NVFP4 model but links to a file that ends in -FP4.
If the checkpoint is indeed “NVFP4”, the URL or filename should reflect that; otherwise drop the “NV” prefix to avoid user confusion when searching on Hugging Face.-* NVFP4 model: [Llama-4-Scout-17B-16E-Instruct-FP4](https://huggingface.co/nvidia/Llama-4-Scout-17B-16E-Instruct-FP4) +* FP4 model (NVFP4): [Llama-4-Scout-17B-16E-Instruct-NVFP4](https://huggingface.co/nvidia/Llama-4-Scout-17B-16E-Instruct-NVFP4)(or update the link if the checkpoint is still “-FP4”)
95-117: Avoid HTML entities for indentation – breaks in some Markdown renderersThe
 entities used to indent option descriptions are rendered literally in several Sphinx/Markdown pipelines, resulting in “ Description:” in the final docs.
Use standard Markdown indent (two spaces) or definition lists instead.Example:
#### `--tp_size` **Description:** Sets the **tensor-parallel size**…
58-69: YAML snippet – missing root key comment & quoting$EXTRA_LLM_API_FILE
- The snippet writes top-level keys directly; calling out that this is the entire file avoids users appending to an existing config.
- Wrap the path in quotes to handle spaces.
-EXTRA_LLM_API_FILE=/tmp/config.yml +EXTRA_LLM_API_FILE="/tmp/config.yml"Consider adding a comment:
# /tmp/config.yml – overwrite if it exists enable_attention_dp: false …
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
docs/source/deployment-guide/quick-start-recipe-for-trttllm-llama4-scout.md(1 hunks)docs/source/index.rst(1 hunks)
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: yibinl-nvidia
PR: NVIDIA/TensorRT-LLM#6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.
Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.
📚 Learning: in tensorrt-llm, test files (files under tests/ directories) do not require nvidia copyright headers...
Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.
Applied to files:
docs/source/deployment-guide/quick-start-recipe-for-trttllm-llama4-scout.mddocs/source/index.rst
📚 Learning: in tensorrt-llm, examples directory can have different dependency versions than the root requirement...
Learnt from: yibinl-nvidia
PR: NVIDIA/TensorRT-LLM#6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.
Applied to files:
docs/source/deployment-guide/quick-start-recipe-for-trttllm-llama4-scout.mddocs/source/index.rst
📚 Learning: in tensorrt-llm testing, it's common to have both cli flow tests (test_cli_flow.py) and pytorch api ...
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
Applied to files:
docs/source/deployment-guide/quick-start-recipe-for-trttllm-llama4-scout.mddocs/source/index.rst
🔇 Additional comments (1)
docs/source/index.rst (1)
38-39: Path casing / hyphen checkVerify that the companion file actually lives at
docs/source/deployment-guide/quick-start-recipe-for-deepseek-r1-on-trt-llm.md
(the repo previously used “DeepSeek-R1” with capital “R”). A mismatched path will break the link and fail the docs build.
|
depends on #6543 |
nv-guomingz
force-pushed
the
user/guomingz/vdr_doc
branch
from
7ae3c48 to
a6312db
Compare
nv-guomingz
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (5)
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md (5)
1-5: Model name looks non-standard—verify official naming before publishingMeta’s public releases are branded “Llama 3 70B”. The repeated “3.3” variant in the title and throughout the guide may confuse users and break Hugging Face links if the repo is actually
nvidia/Llama-3-70B-….-# Quick Start Recipe for Llama3.3 70B on TensorRT-LLM - Blackwell & Hopper Hardware +# Quick-Start Recipe for Llama 3 70B on TensorRT-LLM – Blackwell & Hopper GPUsPlease double-check the HF model IDs and rename consistently (title, links, commands, YAML, benchmark script).
60-70: YAML heredoc contains hard-tab indentation—copy-paste riskUsing tab characters (or inconsistent spaces) inside a heredoc can silently break YAML parsing. Convert to two-space indentation to match the rest of the guide and avoid surprises.
-kv_cache_config: - dtype: fp8 +kv_cache_config: + dtype: fp8
238-243: Add a language identifier to fenced code block (markdownlint MD040)The linter flagged this block; specifying
shellkeeps syntax highlighting consistent with earlier snippets.-``` +```shell MODEL_PATH=nvidia/Llama-3.3-70B-Instruct-FP8 ...
246-251: Missing language identifier on result blockFor fixed-width output, mark the fence as
textto silence MD040 and improve readability.-``` +```text |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| ...
324-333: Same lint issue on benchmark sample outputAdd an identifier (
text) so editors don’t treat it as generic code.-``` +```text ============ Serving Benchmark Result ============ Successful requests: 16 ...
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md(1 hunks)docs/source/deployment-guide/quick-start-recipe-for-llama4-scout-on-trtllm.md(2 hunks)docs/source/index.rst(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- docs/source/deployment-guide/quick-start-recipe-for-llama4-scout-on-trtllm.md
🚧 Files skipped from review as they are similar to previous changes (1)
- docs/source/index.rst
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: yibinl-nvidia
PR: NVIDIA/TensorRT-LLM#6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.
Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
📚 Learning: in tensorrt-llm, test files (files under tests/ directories) do not require nvidia copyright headers...
Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.
Applied to files:
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md
📚 Learning: in tensorrt-llm, examples directory can have different dependency versions than the root requirement...
Learnt from: yibinl-nvidia
PR: NVIDIA/TensorRT-LLM#6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.
Applied to files:
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md
📚 Learning: in tensorrt-llm testing, it's common to have both cli flow tests (test_cli_flow.py) and pytorch api ...
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
Applied to files:
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md
🪛 LanguageTool
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md
[style] ~115-~115: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ch** backend. #### --max_batch_size Description: The maximum number of ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~119-~119: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...r processing. #### --max_num_tokens Description: The maximum total numb...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~123-~123: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...scheduled batch. #### --max_seq_len Description: The maximum possible s...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~127-~127: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ut tokens. #### --trust_remote_code Description: Allows TensorRT-LLM to...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~142-~142: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...: Sets the data type for the KV cache. Default: auto (uses the data ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~146-~146: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...checkpoint). #### cuda_graph_config Description: A section for configur...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~148-~148: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...g CUDA graphs to optimize performance. Options: enable_paddi...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~156-~156: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...or which a CUDA graph will be created. Default: 0 **Rec...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~160-~160: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...-max_batch_size command-line option. batch_sizes: A specific list of ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~162-~162: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...batch sizes to create CUDA graphs for. Default: None #### `moe_conf...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~168-~168: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...n for Mixture-of-Experts (MoE) models. Options: backend: The ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~172-~172: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...The backend to use for MoE operations. Default: CUTLASS #### `atten...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~178-~178: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...end to use for attention calculations. Default: TRTLLM See the [TorchLlmA...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~303-~303: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...l,e2el" done EOF chmod +x bench.sh ``` To benchmark the FP4 model, replace --mod...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🪛 markdownlint-cli2 (0.17.2)
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md
238-238: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
246-246: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
324-324: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (1)
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md (1)
126-129: Flag the security implications of--trust_remote_code
--trust_remote_codeexecutes arbitrary Python from the model repository at runtime. This is fine for demos but may violate internal security policies in production environments. Please add a one-liner warning and, if possible, a pointer to offline/“no-remote-code” workflows.
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md
Show resolved
Hide resolved
|
/bot run --stage-list "A10-Build-docs" |
|
PR_Github #14412 [ run ] triggered by Bot |
|
PR_Github #14412 [ run ] completed with state |
nv-guomingz
force-pushed
the
user/guomingz/vdr_doc
branch
from
a6312db to
2ca5ad5
Compare
|
/bot run --stage-list "A10-Build_Docs" |
|
PR_Github #14425 [ run ] triggered by Bot |
|
PR_Github #14425 [ run ] completed with state |
nv-guomingz
force-pushed
the
user/guomingz/vdr_doc
branch
from
2ca5ad5 to
1cf7ef1
Compare
|
/bot skip --comment "docs build phase already pass" |
|
PR_Github #14473 [ skip ] triggered by Bot |
|
PR_Github #14473 [ skip ] completed with state |
docs/source/deployment-guide/quick-start-recipe-for-llama3.3-70b-on-trtllm.md
Outdated
Show resolved
Hide resolved
nv-guomingz
force-pushed
the
user/guomingz/vdr_doc
branch
from
1cf7ef1 to
09e4542
Compare
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
nv-guomingz
force-pushed
the
user/guomingz/vdr_doc
branch
from
09e4542 to
428a4ab
Compare
|
/bot skip --comment "docs build pass" |
|
PR_Github #14482 [ skip ] triggered by Bot |
|
PR_Github #14482 [ skip ] completed with state |
litaotju
changed the title
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
5 participants
Preview:
Summary by CodeRabbit