jenkins/scripts/slurm_run.sh (1)

3-5: Strengthen ERR trap and avoid noisy xtrace

Good addition. Two tweaks will make failures clearer and propagate traps into functions/subshells.

Apply this diff:

-# Set up error handling
-set -exuo pipefail
-trap 'echo "Script failed on line $LINENO"; exit 1' ERR
+# Set up error handling
+set -Eeuo pipefail
+trap 'rc=$?; echo "Error on line $LINENO: $BASH_COMMAND (exit $rc)"; exit $rc' ERR

Optional: temporarily disable set -x around any env | sort or secrets-related commands to reduce credential leakage in logs.

jenkins/L0_Test.groovy (5)

108-108: SSH resilience: add a connect timeout

New keepalive settings look good. Add an explicit connect timeout to fail fast on dead frontends.

-COMMON_SSH_OPTIONS = "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o TCPKeepAlive=no -o ServerAliveInterval=30 -o ServerAliveCountMax=20"
+COMMON_SSH_OPTIONS = "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o TCPKeepAlive=no -o ServerAliveInterval=30 -o ServerAliveCountMax=20 -o ConnectTimeout=30"

---

114-118: Avoid per-call random login-node selection to reduce flakiness

Using SlurmConfig.getRandomLoginNode(cluster.host) at every call can produce different frontends within a single job. If home dirs or Slurm state are not perfectly shared, result fetch/cleanup can sporadically fail.

Apply these in-function changes:

• Compute once and reuse:

// Near where `cluster` is defined in each function
def loginHost = SlurmConfig.getRandomLoginNode(cluster.host)

• Replace the map entries in the shown ranges:

- host         : SlurmConfig.getRandomLoginNode(cluster.host),
+ host         : loginHost,

Confirm GB/ DGX login nodes share the same homedir and Slurm ctlview; if not, this refactor is recommended.

Also applies to: 146-152, 206-214, 354-361, 471-478

---

392-401: Simplify --gpus quoting to avoid shell/inside quoting edge cases

The current echo with nested quotes can surface as literal quotes depending on shell and Jenkins docker.inside wrapping. Docker accepts unquoted device lists.

-                        if [ -n "$NV_GPU" ]; then
-                            echo "--gpus '\"device=$NV_GPU\"'"
-                        elif [ -n "$CUDA_VISIBLE_DEVICES" ]; then
-                            echo "--gpus '\"device=$CUDA_VISIBLE_DEVICES\"'"
+                        if [ -n "$NV_GPU" ]; then
+                            echo "--gpus device=$NV_GPU"
+                        elif [ -n "$CUDA_VISIBLE_DEVICES" ]; then
+                            echo "--gpus device=$CUDA_VISIBLE_DEVICES"
                         else
                             echo "--gpus ${gpuCount}"
                         fi

Please sanity-check one run where NV_GPU is a comma list and another where it's unset to ensure docker.inside() accepts the resulting arg unchanged.

---

403-418: Docker args: small hardening nits

Looks good. Two small suggestions:

• Ensure volume paths are quoted-safe if envs ever contain spaces (unlikely here).
• Consider adding --pull=missing or keeping the explicit pull you already have; both are fine.

No code change required if you prefer current style.

---

543-545: Mirror bash strict mode improvements in the multi-node launcher

Same recommendation as slurm_run.sh for better diagnostics and trap propagation.

-    set -exuo pipefail
-    trap 'echo "Script failed on line $LINENO"; exit 1' ERR
+    set -Eeuo pipefail
+    trap 'rc=$?; echo "Error on line $LINENO: $BASH_COMMAND (exit $rc)"; exit $rc' ERR

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

jenkins/scripts/slurm_run.sh (7)

3-6: Good hardening; fix ShellCheck SC2154 and minor trap robustness.

Predeclare rc (or disable the warning) to satisfy ShellCheck with set -u, and keep the trap minimal.

Apply this diff:

 # Set up error handling
-set -Exeuo pipefail
-trap 'rc=$?; echo "Error on line $LINENO: $BASH_COMMAND (exit $rc)"; exit $rc' ERR
+set -Exeuo pipefail
+# shellcheck disable=SC2154 # rc is set from $? inside the trap
+rc=0  # predeclare for nounset + shellcheck
+trap 'rc=$?; echo "Error on line $LINENO: $BASH_COMMAND (exit '"$rc"')"; exit "$rc"' ERR

---

7-12: Quote path variables to avoid word-splitting/globbing.

Paths from CI envs can contain spaces. Quote cd and file paths.

Apply this diff:

-cd $resourcePathNode
-llmSrcNode=$resourcePathNode/TensorRT-LLM/src
+cd "$resourcePathNode"
+llmSrcNode="$resourcePathNode/TensorRT-LLM/src"
@@
-coverageConfigFile="$jobWorkspace/.coveragerc"
+coverageConfigFile="$jobWorkspace/.coveragerc"

---

24-35: Ensure apt-get is available; avoid flakiness.

Running apt-get on shared nodes can fail without update or permissions. Gate it and update indexes.

Apply this diff:

-    apt-get install -y libffi-dev
+    if command -v apt-get >/dev/null 2>&1; then
+        apt-get update -y && apt-get install -y --no-install-recommends libffi-dev
+    else
+        echo "apt-get not found; skipping libffi-dev install"
+    fi

---

35-41: Host resolution: quote and add a more reliable fallback.

getent hosts is safer across distros than relying on hostname -f only.

Apply this diff:

-    if [ -n "$HOST_NODE_NAME" ]; then
-        hostNodeName="$HOST_NODE_NAME"
-    else
-        hostNodeName=$(hostname -f || hostname)
-    fi
-    echo "HOST_NODE_NAME = $hostNodeName ; GPU_UUIDS = $gpuUuids ; STAGE_NAME = $stageName"
+    if [ -n "${HOST_NODE_NAME:-}" ]; then
+        hostNodeName="$HOST_NODE_NAME"
+    else
+        hostNodeName="$(getent hosts "$(hostname -f 2>/dev/null || hostname)" 2>/dev/null | awk '{print $2}' | head -1)"
+        hostNodeName="${hostNodeName:-$(hostname -f 2>/dev/null || hostname)}"
+    fi
+    echo "HOST_NODE_NAME = $hostNodeName ; GPU_UUIDS = $gpuUuids ; STAGE_NAME = $stageName"

---

47-47: Unused or unclear variable recomposition.

testList is recomputed but not used below (the script uses testListPathNode). Remove or use consistently.

Apply this diff:

-testList="$testList_$splitId"
+# testList is unused; remove if not required elsewhere
+# testList="${testList}_${splitId}"

---

25-34: General quoting of command arguments.

Please quote variable expansions in wget, tar, pip, and cd to avoid surprises.

Apply this diff:

-    wget -nv $llmTarfile
-    tar -zxf $tarName
+    wget -nv "$llmTarfile"
+    tar -zxf "$tarName"
@@
-    cd $llmSrcNode && pip3 install --retries 1 -r requirements-dev.txt
-    cd $resourcePathNode &&  pip3 install --force-reinstall --no-deps TensorRT-LLM/tensorrt_llm-*.whl
+    cd "$llmSrcNode" && pip3 install --retries 1 -r requirements-dev.txt
+    cd "$resourcePathNode" && pip3 install --force-reinstall --no-deps TensorRT-LLM/tensorrt_llm-*.whl

---

27-28: Prefer command -v over which.

which can be a shell builtin or external; command -v is more portable.

Apply this diff:

-    which python3
+    command -v python3

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: venkywonka
PR: NVIDIA/TensorRT-LLM#6029
File: .github/pull_request_template.md:45-53
Timestamp: 2025-08-27T17:50:13.264Z
Learning: For PR templates in TensorRT-LLM, avoid suggesting changes that would increase developer overhead, such as converting plain bullets to mandatory checkboxes. The team prefers guidance-style bullets that don't require explicit interaction to reduce friction in the PR creation process.

jenkins/scripts/slurm_run.sh (1)

33-33: git safe.directory scope is overly broad

Avoid *; restrict to the repo path to reduce attack surface.

-    git config --global --add safe.directory "*"
+    if git -C "$resourcePathNode/TensorRT-LLM" rev-parse --show-toplevel >/dev/null 2>&1; then
+      git config --global --add safe.directory "$resourcePathNode/TensorRT-LLM"
+    fi

jenkins/L0_Test.groovy (1)

2149-2157: Split-count mismatch for GB200 2-node stages (pre-merge vs YAML)

Stage 1–4 use split_count=4; Stage 5 introduces a 5th split, which will be empty if the YAML still defines 4 pre_merge tests. Align counts or extend YAML.

Run a quick check locally to confirm actual splits before merging:

#!/bin/bash
python3 scripts/test_to_stage_mapping.py --list-stages | grep 'GB200-8_GPUs-2_Nodes-PyTorch-' || true

tensorrt_llm/llmapi/trtllm-llmapi-launch (1)

2-2: Optional: strengthen error handling

Consider set -Eeo pipefail to improve propagation through functions/pipelines without enabling -u (which would require guarding env reads).

-set -e
+set -Eeo pipefail

jenkins/scripts/slurm_run.sh (1)

3-6: Silence SC2154 and keep robust ERR trap

ShellCheck warns rc may be unassigned in the trap. Predeclare rc (no functional change) to satisfy static analysis.

-# Set up error handling
-set -Exeuo pipefail
-trap 'rc=$?; echo "Error in file ${BASH_SOURCE[0]} on line $LINENO: $BASH_COMMAND (exit $rc)"; exit $rc' ERR
+# Set up error handling
+rc=0  # for ShellCheck; overwritten in trap
+set -Exeuo pipefail
+trap 'rc=$?; echo "Error in file ${BASH_SOURCE[0]} on line $LINENO: $BASH_COMMAND (exit $rc)"; exit $rc' ERR

jenkins/L0_Test.groovy (2)

112-116: Stabilize login-node selection across steps

SlurmConfig.getRandomLoginNode(...) is called repeatedly; if nodes differ across steps, paths under /home/svc_tensorrt/bloom/scripts/... may not exist on the new frontend (depending on FS layout). Prefer selecting once and reusing for the whole job (and pass into helpers).

Would you like me to draft a patch that threads a single loginNode through uploadResults, cleanUpNodeResources*, and the Slurm run functions?

Also applies to: 153-160, 216-223, 305-313, 367-374, 485-492

---

558-560: Shell trap in generated slurm_launch.sh: SC2154 nit

Mirror the earlier ShellCheck fix to predeclare rc in the generated script.

-                    set -Exeuo pipefail
-                    trap 'rc=\$?; echo "Error in file \${BASH_SOURCE[0]} on line \$LINENO: \$BASH_COMMAND (exit \$rc)"; exit \$rc' ERR
+                    rc=0
+                    set -Exeuo pipefail
+                    trap 'rc=\$?; echo "Error in file \${BASH_SOURCE[0]} on line \$LINENO: \$BASH_COMMAND (exit \$rc)"; exit \$rc' ERR

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: venkywonka
PR: NVIDIA/TensorRT-LLM#6029
File: .github/pull_request_template.md:45-53
Timestamp: 2025-08-27T17:50:13.264Z
Learning: For PR templates in TensorRT-LLM, avoid suggesting changes that would increase developer overhead, such as converting plain bullets to mandatory checkboxes. The team prefers guidance-style bullets that don't require explicit interaction to reduce friction in the PR creation process.

Learnt from: fredricz-20070104
PR: NVIDIA/TensorRT-LLM#7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

jenkins/scripts/slurm_run.sh (1)

33-33: Don’t mark all repos as safe.directory

Wildcarding “*” weakens git safety in shared runners. Restrict to the actual repo root.

-    git config --global --add safe.directory "*"
+    repo_root="$(cd "$resourcePathNode/TensorRT-LLM/src" && git rev-parse --show-toplevel 2>/dev/null || true)"
+    if [ -n "$repo_root" ]; then
+      git config --global --add safe.directory "$repo_root"
+    fi

jenkins/L0_Test.groovy (1)

2147-2158: Pre-merge split-count mismatch: stage 5 likely empty

You added “GB200-8_GPUs-2_Nodes-PyTorch-5” with split_count=5, but the test-db pre_merge section has 4 tests; split 5 will be empty. Drop stage 5 or add a 5th pre_merge test.

-        "GB200-8_GPUs-2_Nodes-PyTorch-5": ["gb200-multi-node", "l0_gb200_multi_nodes", 5, 5, 8, 2],
+        // Pre-merge currently has 4 splits in l0_gb200_multi_nodes.yml
+        // "GB200-8_GPUs-2_Nodes-PyTorch-5": ["gb200-multi-node", "l0_gb200_multi_nodes", 5, 5, 8, 2],

To verify quickly:

#!/bin/bash
# Expect 4 pre-merge splits to have tests; 5th should be empty today.
rg -n "GB200-8_GPUs-2_Nodes-PyTorch-[1-5]" jenkins/L0_Test.groovy
python3 scripts/test_to_stage_mapping.py --list-stages | grep 'GB200-8_GPUs-2_Nodes-PyTorch-' | wc -l || true

jenkins/scripts/slurm_run.sh (2)

3-5: Good: proactive error diagnostics; minor caution on logging commands

Trap prints $BASH_COMMAND which may include secrets. Consider gating it behind a DEBUG flag or redacting known tokens.

---

49-49: Ensure launcher is executable before use

Tarball perms can vary. Safer to chmod the new launcher.

-    "$llmSrcNode/jenkins/scripts/trtllm-llmapi-launch"
+    "chmod +x $llmSrcNode/jenkins/scripts/trtllm-llmapi-launch" \
+    "$llmSrcNode/jenkins/scripts/trtllm-llmapi-launch"

jenkins/scripts/trtllm-llmapi-launch (3)

2-2: Enable -u for unset vars

Keep consistency with other scripts and fail fast on typos/unset env.

-set -Exeo pipefail
+set -Eeuo pipefail

---

26-41: Port/HMAC setup: add fallbacks and reduce race window

• Port probe can race; prefer ephemeral ports or keep the socket bound until export completes.
• Openssl might be missing; fall back to Python secrets.

-function export_free_tcp_addr_for_spawn_proxy_process {
-    # find free port starting from 10012
-    local free_port=$(python -c 'import socket; s=socket.socket();
-port = 10012
-while True:
-    try:
-        s.bind(("", port))
-        break
-    except OSError:
-        port += 1
-print(port); s.close()')
-    export TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR="tcp://127.0.0.1:${free_port}"
-    log_stderr "TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR: $TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR"
-
-    export TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY=$(openssl rand -hex 32)
-}
+function export_free_tcp_addr_for_spawn_proxy_process {
+    # Prefer ephemeral port to minimize races; fall back to scan from 10012
+    local free_port
+    free_port=$(python3 - <<'PY'
+import socket
+s=socket.socket()
+try:
+    s.bind(("127.0.0.1", 0))
+    print(s.getsockname()[1])
+finally:
+    s.close()
+PY
+    ) || free_port=""
+    if [ -z "$free_port" ]; then
+        free_port=$(python3 - <<'PY'
+import socket
+s=socket.socket(); port=10012
+while True:
+    try:
+        s.bind(("127.0.0.1", port)); print(port); break
+    except OSError:
+        port += 1
+s.close()
+PY
+        )
+    fi
+    export TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR="tcp://127.0.0.1:${free_port}"
+    log_stderr "TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR: $TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR"
+    if command -v openssl >/dev/null 2>&1; then
+        export TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY="$(openssl rand -hex 32)"
+    else
+        export TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY="$(python3 - <<'PY'
+import secrets; print(secrets.token_hex(32))
+PY
+        )"
+    fi
+}

---

5-12: Remove unused vars

native_mpi_rank and pid are never used.

-native_mpi_rank=$OMPI_COMM_WORLD_RANK
...
-pid=$(ps -o pid= -p $$ | tr -d ' ')

jenkins/L0_Test.groovy (1)

326-327: Nit: fix log grammar

“Sleeping to before …” → “Sleeping before …”.

-Utils.exec(pipeline, script: "echo Sleeping to before Slurm job submission; sleep \$((RANDOM % 29 + 1))")
+Utils.exec(pipeline, script: "echo Sleeping before Slurm job submission; sleep \$((RANDOM % 29 + 1))")

Also applies to: 585-586

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: fredricz-20070104
PR: NVIDIA/TensorRT-LLM#7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: galagam
PR: NVIDIA/TensorRT-LLM#6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

jenkins/scripts/slurm_run.sh (2)

33-33: git safe.directory scope is overly broad.

-    git config --global --add safe.directory "*"
+    repoRoot="$(git -C "$resourcePathNode/TensorRT-LLM" rev-parse --show-toplevel 2>/dev/null || true)"
+    if [[ -n "$repoRoot" ]]; then
+      git config --global --add safe.directory "$repoRoot"
+    fi

---

98-108: Drop eval; execute argv array and preserve exit code under set -e.

Current eval loses quoting and can mis-handle args; also brittle with set -e.

-fullCmd="${testCmdLines[*]}"
-echo "Full Command: $fullCmd"
-
-# Turn off "exit on error" so the following lines always run
-set +e
-trap - ERR
-
-eval $fullCmd
-exitCode=$?
-echo "Pytest exit code: $exitCode"
-exit $exitCode
+echo "Full Command: ${testCmdLines[*]}"
+# Turn off "exit on error" so the following lines always run
+set +e
+trap - ERR
+"${testCmdLines[@]}"
+exitCode=$?
+set -e
+echo "Pytest exit code: $exitCode"
+exit "$exitCode"

Companion fix: convert options currently encoded as a single token with a space to either “--opt=value” or two separate tokens, otherwise the array form will pass them as one argument.

-    "--rootdir $llmSrcNode/tests/integration/defs"
-    "--splits $splits"
-    "--group $splitId"
-    "--junit-xml $resultsPath/results.xml"
+    "--rootdir=$llmSrcNode/tests/integration/defs"
+    "--splits=$splits"
+    "--group=$splitId"
+    "--junit-xml=$resultsPath/results.xml"

If perf mode is used:

-        "--perf-log-formats csv"
-        "--perf-log-formats yaml"
+        "--perf-log-formats=csv"
+        "--perf-log-formats=yaml"

jenkins/scripts/trtllm-llmapi-launch (5)

2-2: Enable nounset; current script references unset envs.

Add -u; we already guard mpi_rank via default expansions.

-set -Eeo pipefail
+set -Eeuo pipefail

---

8-9: Harden logger; avoid echo -e and unquoted $@.

-log_stderr() { echo -e "\033[33m$@\033[0m" >&2; }
+log_stderr() { printf '\033[33m%s\033[0m\n' "$*" >&2; }

---

5-5: Remove unused variable.

-native_mpi_rank=$OMPI_COMM_WORLD_RANK

---

11-11: Remove unused PID capture.

-pid=$(ps -o pid= -p $$ | tr -d ' ')

---

49-52: Possible race: task may start before leader is ready.

You spawn the task, then start the leader. If the client connects early, it may fail/flap. Prefer starting the leader, waiting until ready, then running the task; or add client-side retry.

I can provide a small “wait-ready” loop around the leader if desired.

jenkins/scripts/slurm_run.sh (5)

48-49: Verify launcher path before chmod.

-llmapiLaunchScript="$llmSrcNode/jenkins/scripts/trtllm-llmapi-launch"
-chmod +x $llmapiLaunchScript
+llmapiLaunchScript="$llmSrcNode/jenkins/scripts/trtllm-llmapi-launch"
+if [[ ! -x "$llmapiLaunchScript" ]]; then
+  [[ -f "$llmapiLaunchScript" ]] || { echo "Missing: $llmapiLaunchScript"; exit 2; }
+  chmod +x "$llmapiLaunchScript"
+fi

---

86-94: Handle empty LD_LIBRARY_PATH under -u; drop no-op sed.

Under set -u, an unset LD_LIBRARY_PATH aborts the script. Also, sed '[[:space:]]+/_/g' doesn’t match without -E and isn’t needed.

-containerLDLibPath=$LD_LIBRARY_PATH
-containerLDLibPath=$(echo "$containerLDLibPath" | sed 's/[[:space:]]+/_/g')
+containerLDLibPath="${LD_LIBRARY_PATH-}"

---

5-5: ShellCheck warning about rc is benign; optional tidy.

-trap 'rc=$?; echo "Error in file ${BASH_SOURCE[0]} on line $LINENO: $BASH_COMMAND (exit $rc)"; exit $rc' ERR
+trap 'st=$?; echo "Error in file ${BASH_SOURCE[0]} on line $LINENO: $BASH_COMMAND (exit ${st})"; exit ${st}' ERR

---

59-66: Minor: prefer --rootdir= over split tokens for clarity.

Already covered in the array-exec change; keeping a single token with a space is fragile.

---

43-43: Unset envs with -u can abort the script.

CPP_TEST_TIMEOUT_OVERRIDDEN reads pytestTestTimeout, which may be unset depending on the Jenkins job.

Consider defaults:

-export CPP_TEST_TIMEOUT_OVERRIDDEN=$pytestTestTimeout
+export CPP_TEST_TIMEOUT_OVERRIDDEN="${pytestTestTimeout:-600}"

Repeat this pattern for MODEL_CACHE_DIR, STAGE_NAME, etc., or fail fast with a clear error if they’re required.

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: fredricz-20070104
PR: NVIDIA/TensorRT-LLM#7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.