[TRTLLM-6761][refactor] Replace LogitBiasLogitsProcessor with embedding bias tensor system #6464
Conversation
📝 WalkthroughWalkthroughThis update refactors logit bias handling across the codebase, replacing the previous processor-based approach with direct embedding bias tensor application in both server and sampler logic. Supporting changes include new utility functions for bias conversion, expanded and parameterized tests, and modularized test utilities. Integration tests were updated to add new logit bias test entries. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant OpenAIServer
participant OpenAIProtocol
participant Sampler
participant LlmRequest
Client->>OpenAIServer: Sends request (with logit_bias)
OpenAIServer->>OpenAIProtocol: to_sampling_params(vocab_size)
OpenAIProtocol->>OpenAIProtocol: _logit_bias_to_embedding_bias(logit_bias, vocab_size)
OpenAIProtocol-->>OpenAIServer: SamplingParams(embedding_bias)
OpenAIServer->>Sampler: sample(SamplingParams)
Sampler->>LlmRequest: Initialize (embedding_bias)
Sampler->>Sampler: _apply_embedding_bias(logits, requests)
Sampler-->>OpenAIServer: Sampled tokens
OpenAIServer-->>Client: Response
Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Suggested labels
Suggested reviewers
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
coderabbitai
bot
requested review from
FrankD412,
amitz-nv,
litaotju,
nv-guomingz,
shaharmor98 and
symphonylyh
|
@coderabbitai pause |
coderabbitai
bot
changed the title
coderabbitai
bot
added
the
Doc
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tests/unittest/llmapi/apps/utils.py (2)
131-133: Fix docstring formatting issues.The docstring needs formatting improvements to follow Python standards.
Apply this diff to fix the docstring formatting:
-def make_server_with_custom_sampler_fixture(api_type: str) -> Callable: - '''Factory for a pytest fixture that launches a server with a custom sampler config. - api_type: 'chat' or 'completions' (for error messages only) - ''' +def make_server_with_custom_sampler_fixture(api_type: str) -> Callable: + '''Factory for a pytest fixture that launches a server with a custom sampler config. + + Args: + api_type: 'chat' or 'completions' (for error messages only). + '''
165-167: Fix docstring formatting.The docstring needs a blank line between summary and description.
Apply this diff:
-def make_common_fixtures(): - '''Factory to create common fixtures used across multiple test files. - Returns a dictionary of fixture functions that can be assigned in test files. - ''' +def make_common_fixtures(): + '''Factory to create common fixtures used across multiple test files. + + Returns a dictionary of fixture functions that can be assigned in test files. + '''
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
tensorrt_llm/_torch/auto_deploy/shim/demollm.py(1 hunks)tensorrt_llm/_torch/pyexecutor/llm_request.py(2 hunks)tensorrt_llm/_torch/pyexecutor/sampler.py(3 hunks)tensorrt_llm/sampling_params.py(1 hunks)tensorrt_llm/serve/openai_protocol.py(6 hunks)tensorrt_llm/serve/openai_server.py(2 hunks)tests/unittest/llmapi/apps/_test_openai_chat.py(2 hunks)tests/unittest/llmapi/apps/_test_openai_completions.py(2 hunks)tests/unittest/llmapi/apps/utils.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{cpp,h,hpp,cc,cxx,cu,py}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
All code must contain an NVIDIA copyright header that includes the current year. This includes .cpp, .h, .cu, .py, and any other source files which are compiled or interpreted.
Files:
tensorrt_llm/_torch/auto_deploy/shim/demollm.pytensorrt_llm/serve/openai_server.pytensorrt_llm/sampling_params.pytensorrt_llm/_torch/pyexecutor/sampler.pytensorrt_llm/serve/openai_protocol.pytests/unittest/llmapi/apps/utils.pytests/unittest/llmapi/apps/_test_openai_chat.pytests/unittest/llmapi/apps/_test_openai_completions.pytensorrt_llm/_torch/pyexecutor/llm_request.py
**/*.py
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.py: The code developed for TensorRT-LLM should conform to Python 3.8+.
Indent Python code with 4 spaces. Do not use tabs.
Always maintain the namespace when importing in Python, even if only one class or function from a module is used.
Python filenames should use snake_case (e.g., some_file.py).
Python classes should use PascalCase (e.g., class SomeClass).
Python functions and methods should use snake_case (e.g., def my_awesome_function():).
Python local variables should use snake_case. Prefix k for variable names that start with a number (e.g., k_99th_percentile).
Python global variables should use upper snake_case and prefix G (e.g., G_MY_GLOBAL).
Python constants should use upper snake_case (e.g., MY_CONSTANT).
Avoid shadowing variables declared in an outer scope in Python.
Initialize all externally visible members of a class in the constructor in Python.
For interfaces that may be used outside a file, prefer docstrings over comments in Python.
Comments in Python should be reserved for code within a function, or interfaces that are local to a file.
Use Google style docstrings for classes and functions in Python, which can be parsed by Sphinx.
Attributes and variables in Python can be documented inline; attribute docstrings will be rendered under the docstring for the class.
Avoid using reflection in Python when functionality can be easily achieved without it.
When using try-except blocks in Python, limit the except to the smallest set of errors possible.
When using try-except blocks to handle multiple possible variable types in Python, keep the body of the try as small as possible, using the else block to implement the logic.
Files:
tensorrt_llm/_torch/auto_deploy/shim/demollm.pytensorrt_llm/serve/openai_server.pytensorrt_llm/sampling_params.pytensorrt_llm/_torch/pyexecutor/sampler.pytensorrt_llm/serve/openai_protocol.pytests/unittest/llmapi/apps/utils.pytests/unittest/llmapi/apps/_test_openai_chat.pytests/unittest/llmapi/apps/_test_openai_completions.pytensorrt_llm/_torch/pyexecutor/llm_request.py
🧠 Learnings (5)
📓 Common learnings
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tests/unittest/llmapi/apps/utils.py (1)
Learnt from: moraxu
PR: #6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tests/unittest/llmapi/apps/_test_openai_chat.py (1)
Learnt from: moraxu
PR: #6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tests/unittest/llmapi/apps/_test_openai_completions.py (1)
Learnt from: moraxu
PR: #6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tensorrt_llm/_torch/pyexecutor/llm_request.py (1)
Learnt from: yechank-nvidia
PR: #6254
File: tensorrt_llm/_torch/pyexecutor/model_engine.py:1201-1204
Timestamp: 2025-07-22T09:22:14.726Z
Learning: In TensorRT-LLM's multimodal processing pipeline, shared tensor recovery using from_shared_tensor() is only needed during the context phase. Generation requests reuse the already-recovered tensor data and only need to call strip_for_generation() to remove unnecessary multimodal data while preserving the recovered tensors. This avoids redundant tensor recovery operations during generation.
🪛 Ruff (0.12.2)
tests/unittest/llmapi/apps/utils.py
131-133: 1 blank line required between summary line and description
(D205)
131-133: First line should end with a period, question mark, or exclamation point
Add closing punctuation
(D415)
165-167: 1 blank line required between summary line and description
(D205)
🔇 Additional comments (27)
tensorrt_llm/sampling_params.py (1)
5-5: LGTM! Import cleanup after LogitBiasLogitsProcessor removal.The removal of
Dictfrom the imports is consistent with the removal of theLogitBiasLogitsProcessorclass that previously used this type.tensorrt_llm/serve/openai_server.py (2)
255-258: LGTM! Correctly passes vocabulary size for logit bias conversion.The addition of
vocab_sizeparameter enables the conversion of OpenAI-stylelogit_biasdictionaries to embedding bias tensors in the sampling parameters.
407-410: LGTM! Consistent vocab_size handling for completions.The completion endpoint correctly mirrors the chat endpoint's approach by passing the tokenizer's vocabulary size for logit bias conversion.
tensorrt_llm/_torch/auto_deploy/shim/demollm.py (1)
205-209: LGTM! Clean implementation of embedding bias application.The code correctly applies the embedding bias to logits during sampling, with proper device handling to avoid tensor device mismatches.
tensorrt_llm/_torch/pyexecutor/llm_request.py (2)
331-338: LGTM! Proper preprocessing of embedding bias tensor.The code correctly prepares a 1D version of the embedding bias tensor for efficient downstream use, handling both 1D and multi-dimensional inputs appropriately.
438-442: LGTM! Robust handling of embedding bias conversion.The code properly handles different input types (tensor vs. non-tensor) and ensures tensor detachment to avoid gradient tracking issues. The float32 dtype is appropriate for bias values.
tensorrt_llm/serve/openai_protocol.py (6)
8-8: LGTM! Required import for tensor operations.The torch import is necessary for creating the embedding bias tensor in the new helper function.
21-48: Well-implemented conversion function with proper validation.The
_logit_bias_to_embedding_biasfunction correctly:
- Validates token IDs as integers with clear error messages
- Checks vocabulary bounds to prevent out-of-range access
- Creates a properly sized embedding bias tensor
- Uses float32 dtype consistent with the rest of the codebase
256-277: Comprehensive documentation for vocab_size parameter.The docstring thoroughly explains the purpose and usage of the
vocab_sizeparameter, including the rationale for the 32000 default value. This will help API users understand the parameter's role in logit bias conversion.
309-310: LGTM! Clean replacement of logits processor with embedding bias.The code correctly uses the new embedding bias approach instead of the removed
LogitBiasLogitsProcessor.
572-578: LGTM! Consistent parameter handling in chat completions.The chat completion method mirrors the completion method's approach with appropriate documentation.
610-611: LGTM! Consistent embedding bias usage.The chat completion request correctly uses the same embedding bias conversion approach as the completion request.
tensorrt_llm/_torch/pyexecutor/sampler.py (4)
208-213: LGTM! Well-designed helper function.The function provides a clean interface for accessing the embedding bias tensor with appropriate defensive programming using
hasattr. The docstring clearly explains its purpose.
419-444: LGTM! Efficient fast path implementation.The fast path correctly handles embedding bias application by:
- Creating a boolean mask to identify requests with bias
- Stacking bias values for vectorized operations
- Cloning logits before modification to avoid in-place changes on shared tensors
- Applying biases selectively using the mask
The approach is efficient and handles the common case well.
456-495: Well-implemented batched strategy with vectorized bias application.The batched strategy implementation correctly handles embedding bias by:
- Collecting biases per request along with their step counts
- Using
repeat_interleaveto expand biases for draft tokens (multiple steps per request)- Building a comprehensive mask for all steps that need bias application
- Applying all biases in a single vectorized operation
The logic for handling draft tokens and the offset calculation for the bias mask is correct. The vectorized approach should provide good performance.
501-529: Correct per-request bias application.The per-request loop properly handles embedding bias by:
- Retrieving bias for each individual request
- Applying bias directly when batched results aren't available
- Using the batched results when available to avoid redundant computation
- Ensuring device compatibility with
non_blocking=TrueThe conditional logic correctly determines when to apply bias vs. use batched results.
tests/unittest/llmapi/apps/utils.py (5)
11-17: LGTM! Robust error handling for tokenizer interactions.The function properly handles various failure modes when interacting with tokenizers and uses
pytest.skipappropriately. The exception types covered (IndexError, AttributeError, TypeError) are comprehensive for tokenizer edge cases.
19-94: Well-designed test helper with comprehensive coverage.The helper function effectively tests logit bias functionality by:
- Testing both positive and negative bias effects with strong values (±80)
- Supporting both completions and chat API endpoints
- Using deterministic temperature (0.0) for reliable testing
- Handling tokenizer dependencies gracefully with fallback token ID
- Using clear assertions that verify expected behavior
The approach of testing "Paris" as the capital of France provides a good semantic test case where bias effects should be clearly observable.
96-128: LGTM! Proper error handling validation.The helper function correctly validates that invalid logit bias inputs raise
BadRequestErrorby:
- Testing non-integer keys which should be rejected
- Supporting both API endpoints consistently
- Using appropriate pytest exception handling
- Providing clear test inputs that should fail validation
The test case with
"invalid_token"as a key is a good choice since logit bias expects integer token IDs.
130-162: Excellent fixture factory design for parameterized testing.The fixture factory provides a clean way to create customizable server fixtures by:
- Allowing parameterization of sampler backend (
use_trtllm_sampler)- Managing temporary YAML configuration files properly
- Using appropriate fixture scope ('function') for test isolation
- Providing proper server lifecycle management with context manager
This enables comprehensive testing across different sampler implementations.
164-198: Well-structured common fixtures factory.The factory creates reusable fixtures with appropriate configurations:
- Uses module scope for expensive operations (server setup)
- Provides parameterized fixtures for comprehensive testing
- Returns a dictionary for easy assignment in test files
- Follows pytest best practices for fixture organization
This promotes code reuse and consistency across test files.
tests/unittest/llmapi/apps/_test_openai_chat.py (3)
14-16: LGTM! Proper imports for modular test design.The imports correctly bring in the shared test utilities, enabling code reuse and consistent testing patterns across different API endpoints.
529-549: Excellent parameterized test design for comprehensive coverage.The new test effectively validates logit bias functionality by:
- Testing both torch and TRT LLM sampler backends through parameterization
- Using function-scoped fixtures for proper test isolation
- Delegating core logic to the shared helper for consistency
- Following good naming conventions and documentation
This ensures the embedding bias mechanism works correctly across different sampler implementations.
552-556: LGTM! Good refactoring to use shared test logic.The modification correctly delegates to the shared helper function while maintaining the same test coverage. This promotes consistency and reduces code duplication across test files.
tests/unittest/llmapi/apps/_test_openai_completions.py (3)
11-13: LGTM! Consistent imports for modular testing.The imports properly bring in the shared utilities, maintaining consistency with the chat test refactoring.
377-400: Excellent parameterized test implementation.The new test provides comprehensive coverage by:
- Testing both sampler backends (torch and TRT LLM) through parameterization
- Using the completions-specific API endpoint
- Delegating to the shared helper for consistent test logic
- Following the same pattern as the chat tests for maintainability
This ensures the embedding bias mechanism works correctly for the completions API across different sampler implementations.
403-408: LGTM! Proper refactoring to shared test logic.The modification correctly delegates to the shared helper function, maintaining test coverage while reducing code duplication and ensuring consistency with the chat API tests.
✅ Actions performedReviews paused. |
venkywonka
removed request for
FrankD412,
litaotju,
nv-guomingz,
suyoggupta and
symphonylyh
venkywonka
removed
the
Doc
venkywonka
force-pushed
the
user/venky/logit-bias-refactor
branch
from
e8964e0 to
929ce01
Compare
venkywonka
force-pushed
the
user/venky/logit-bias-refactor
branch
from
929ce01 to
2b046cc
Compare
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
venkywonka
force-pushed
the
user/venky/logit-bias-refactor
branch
from
6e5f920 to
426811e
Compare
|
/bot run |
|
/bot run |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tests/unittest/llmapi/apps/utils.py (2)
83-108: Consider improving assertion specificity.The negative bias test only checks that "Paris" is not in the output, but doesn't verify that an alternative reasonable answer appears. This could potentially pass even if the output is empty or nonsensical.
Consider enhancing the assertion to also verify that some reasonable alternative appears:
- assert 'Paris' not in output, f"Did not expect 'Paris' in output with negative logit bias, got: {output}" + assert 'Paris' not in output, f"Did not expect 'Paris' in output with negative logit bias, got: {output}" + assert len(output.strip()) > 0, f"Expected non-empty output with negative logit bias, got: {output}"
145-177: Fix docstring formatting issues.The static analysis correctly identified docstring formatting problems that should be addressed for consistency with project standards.
Apply this fix for the docstring formatting:
-def make_server_with_custom_sampler_fixture(api_type: str) -> Callable: - '''Factory for a pytest fixture that launches a server with a custom sampler config. - api_type: 'chat' or 'completions' (for error messages only) - ''' +def make_server_with_custom_sampler_fixture(api_type: str) -> Callable: + '''Factory for a pytest fixture that launches a server with a custom sampler config. + + Args: + api_type: 'chat' or 'completions' (for error messages only). + '''
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (11)
tensorrt_llm/_torch/pyexecutor/llm_request.py(2 hunks)tensorrt_llm/_torch/pyexecutor/sampler.py(3 hunks)tensorrt_llm/sampling_params.py(2 hunks)tensorrt_llm/serve/openai_protocol.py(6 hunks)tensorrt_llm/serve/openai_server.py(2 hunks)tests/integration/defs/test_e2e.py(2 hunks)tests/integration/test_lists/test-db/l0_l40s.yml(1 hunks)tests/unittest/llmapi/apps/_test_openai_chat.py(2 hunks)tests/unittest/llmapi/apps/_test_openai_completions.py(2 hunks)tests/unittest/llmapi/apps/utils.py(1 hunks)tests/unittest/llmapi/test_llm_pytorch.py(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (9)
- tests/integration/test_lists/test-db/l0_l40s.yml
- tensorrt_llm/serve/openai_server.py
- tests/integration/defs/test_e2e.py
- tensorrt_llm/sampling_params.py
- tensorrt_llm/_torch/pyexecutor/llm_request.py
- tensorrt_llm/_torch/pyexecutor/sampler.py
- tests/unittest/llmapi/apps/_test_openai_completions.py
- tests/unittest/llmapi/apps/_test_openai_chat.py
- tensorrt_llm/serve/openai_protocol.py
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: in tensorrt-llm testing, it's common to have both cli flow tests (test_cli_flow.py) and pytorch api ...
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
Applied to files:
tests/unittest/llmapi/apps/utils.pytests/unittest/llmapi/test_llm_pytorch.py
📚 Learning: in tensorrt-llm, examples directory can have different dependency versions than the root requirement...
Learnt from: yibinl-nvidia
PR: NVIDIA/TensorRT-LLM#6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.
Applied to files:
tests/unittest/llmapi/apps/utils.py
🪛 Ruff (0.12.2)
tests/unittest/llmapi/apps/utils.py
146-148: 1 blank line required between summary line and description
(D205)
146-148: First line should end with a period, question mark, or exclamation point
Add closing punctuation
(D415)
tests/unittest/llmapi/test_llm_pytorch.py
160-160: First line should end with a period, question mark, or exclamation point
Add closing punctuation
(D415)
🔇 Additional comments (4)
tests/unittest/llmapi/apps/utils.py (3)
26-32: LGTM - Good error handling for tokenizer edge cases.The function correctly handles various tokenizer exceptions and uses
pytest.skipappropriately when token extraction fails. The graceful fallback ensures tests don't fail due to tokenizer issues.
34-82: LGTM - Comprehensive logit bias effect testing with good fallback strategy.The helper function effectively tests both positive and negative logit bias effects across different API types. The fallback token ID (3681) for "Paris" when tokenizer import fails is a practical approach to maintain test coverage.
111-143: LGTM - Proper error validation for invalid logit bias.The function correctly tests that invalid logit bias inputs (non-integer keys) raise the expected
BadRequestError. The test coverage for both API types is comprehensive.tests/unittest/llmapi/test_llm_pytorch.py (1)
148-186: Excellent comprehensive test for embedding bias across TorchSampler strategies.This test effectively validates the new embedding bias tensor system across all three TorchSampler execution paths:
- Fast path (no mixed sampler, no logprobs)
- Batched strategy (mixed sampler enabled, no logprobs)
- Per-request path (no mixed sampler, logprobs enabled)
The use of maximum float32 bias value ensures deterministic "Z" token generation, and disabling TRT-LLM sampler properly exercises the TorchSampler code paths.
Minor docstring formatting fix for consistency:
- """Test embedding bias application in all 3 TorchSampler paths: fast, batched strategy, and per-request""" + """Test embedding bias application in all 3 TorchSampler paths: fast, batched strategy, and per-request."""
|
PR_Github #14090 [ run ] triggered by Bot |
|
PR_Github #14074 [ run ] completed with state |
|
PR_Github #14091 [ run ] triggered by Bot |
|
PR_Github #14090 [ run ] completed with state |
|
PR_Github #14091 [ run ] completed with state |
…ng bias tensor system (NVIDIA#6464) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com> Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>
…ng bias tensor system (NVIDIA#6464) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
7 participants
Summary by CodeRabbit
New Features
Bug Fixes
Tests
Chores
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.