[TRTLLM-5195][feat] Multimodal Disagg Support in TRTLLM #5000
Conversation
Initial commit to add standalone encoder engine Add encoder server Add ForkingPickler to enable sharemem transfer Add cudaIPC support and intermed to disagg e2e Enable E2E in disagg mode [1/N] Refactor: Relocate mm_encoder and MultiModalParams [2/N] Refactor: move MM request/response/result to dedicated files [3/N] Refactor: move shared IPC tensor to a dedicated place [4/N] Refactor: Enable multigpu on llm server + Diable sharedtensor pool [5/N] Cleanup: Remove unnecessary authkey sync Genai-perf script + Delay decre sharetensor ref + Port image load fix from gh-main
chang-l
requested review from
amukkara,
jaedeok-nvidia,
rakib-hasan and
yechank-nvidia
chang-l
changed the title
| tensor_pool = get_handle_buffer() | ||
| tensor_pool.add_handle(str(request.py_request_id), shared_tensor) | ||
| multimodal_embedding.copy_(shared_tensor) | ||
| self.mm_emb_dist.broadcast(multimodal_embedding) |
There was a problem hiding this comment.
Not sure this is the right position to broadcast. Should we broadcast on _fetch_new_requests?
I got a comment that adding action on prepare_tp_inputs could decrease the performance of sheculder_overlap.
There was a problem hiding this comment.
I see.. In fact, this is nccl bcast and should not block cpu.
One concern is that since we need to bcast mm_embedding cuda tensor for every request, moving to _fetch_new_requests would require us to loop all requests (in batch) to bcast their mm_embedding in one place Not sure about the perf implications compared to the current flow, i.e., each forward pass broadcasts mm_embed and consumes it immediately.
|
let me add @Shunkangz for viz. |
There was a problem hiding this comment.
The MR is quite large (>3000 new lines). Can it be broken into multiple smaller MRs with unit tests to make it easier to review? Also, right now we only have 1 e2e test, and 1 unit test for shared tensor. This is not sufficient for the 3000+ new lines of code.
There was a problem hiding this comment.
this multimodal executor is to executor the standalone vision encoder, right?
I think replicate/inherit most of the code from PyExecutor might not be ideal, as it's harder to maintain one more PyExecutor class.
Should we consider reusing the no-KV path in the PyExecutor (e.g., this path can run BERT which is also an encoder model, previous PR https://gitlab-master.nvidia.com/ftp/tekit/-/merge_requests/8280) and extend PyExecutor w/o duplicating the class?
|
Thanks @pcastonguay. |
5 participants
Multimodal Disagg Support in TRTLLM
This is a POC of enabling disagg support for multimodal inputs in pytorch flow.
To-do before merging:
Add example/doc/benchmark scriptAdd functional test for torch.Tensor cuda IPC/shared memory utilityAdd e2e testSome preliminary results
Model:
llava-hf/llava-v1.6-mistral-7b-hfTool:
genai-perfSetup:
1 LLM server (TP1) and/or 1 MM server (TP1)Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]Launch build/test pipelines. All previously running jobs will be killed.
--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-[Post-Merge]-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.