[https://nvbugs/6463987][fix] Pin NCCL to 2.29.7 for GB300 GLM-5-NVFP4 perf

[chenfeiz0326]

Summary

• Pins libnccl2 to 2.29.7 in docker/common/install_cuda_libs.sh and caps nvidia-nccl-cu13<=2.29.7 in requirements.txt to work around a NCCL 2.30.4 small-message MoE all-to-all regression on GB300 (nvbugs/6463987).
• All other 26.05 stack components stay: cuBLAS 13.4.1.2, cuDNN 9.22, torch 2.11.0, base container.

Verified on main HEAD

5 unpinned reps of perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL] on aws-cmh GB300 for each config. Both against main HEAD a9544e0ded. 10 distinct nodes across the two sets:

Config	Runtime NCCL	reps (sorted)	median	mean	CV	fast/slow
main tot (unpatched)	2.30.4	2.54, 2.57, 2.58, 4.04, 4.16	2.58	3.18	26.52% ⚠	2/3 bimodal
main tot + this PR	2.29.7	4.09, 4.12, 4.12, 4.17, 4.18	4.12	4.14	0.91% ✅	5/0 stable

• Median improvement: +59.7% (2.58 → 4.12 tok/s)
• CV drops 29×: 27% (bimodal) → 0.9% (dead stable)
• All 5 tot+PR reps landed on the fast branch across 5 different nodes (nvl72d164-T12, nvl72d089-T17, nvl72d182-T18, nvl72d156-T07, nvl72d055-T06)
• Every slurm-*.out confirms the intended NCCL version was loaded at runtime (NCCL version 2.30.4+cuda13.2 for tot; NCCL version 2.29.7+cuda13.2 for tot+PR)

Root cause

On GB300 with GLM-5-NVFP4 con1 (dep=2, tep=4, 75 MoE layers, MTP-3 speculative, tiny top-k=8 payloads), NCCL 2.30.4 exhibits bimodal per-process algorithm selection in the MoE all-to-all dispatch kernel. Same-node isolation A/B on aws-cmh, everything else identical, singles out NCCL as the sole variable:

Config	n	median tok/s	CV	fast/slow
b1 (pre-PR#15087, native NCCL 2.29.7)	6	4.03	1.94%	6/0 ✅
b2 + LD_PRELOAD NCCL 2.29.7	5	4.15	1.33%	5/0 ✅
b2 (PR#15087, native NCCL 2.30.4)	6	3.38	27.0%	3/3 ⚠
bad (position 28, native NCCL 2.30.4)	6	2.59	25.4%	2/4 ⚠

Swapping only NCCL to 2.29.7 on the b2 wheel (same cuBLAS 13.4.1.2, same cuDNN 9.22, same torch 2.11.0, same container) restores b1-equivalent stable perf across 5 different aws-cmh GB300 nodes.

Mechanism

Nsys GPU-time analysis of b2 slow-branch reps shows 87% of GPU time in the MoE a2a kernels:

• moeA2ADispatchKernel<BlockPolicy, 8>: 780 calls × 4.85 ms = 3.78 s
• moeA2ASanitizeExpertIdsKernel: 780 calls × 4.84 ms = 3.77 s

On the fast branch (one rep out of 5 caught a partial fast-a2a state), the same kernels run at 17.3 µs/call — a 280× swing between algorithm picks. All non-a2a kernels (Cutlass DSL, cuBLAS, MLA-DSA attention) are unchanged (within 1-2 %) between branches.

GLM-5's 75-MoE-layer stack multiplies the per-a2a delta into the observed 1.5-2× end-to-end regression.

This is an NCCL 2.30 internal-tactic selection issue, not a TRT-LLM code change. Upstream NCCL should be notified; this PR is the workaround while that is investigated.

Alternatives considered

• PR [https://nvbugs/6463987][perf] Revert NGC PyTorch 26.05 stack upgrade for GB300 GLM-5-fp4 ctx_only NVFP4 regression #16601 (revert whole 26.05 stack) — correct in spirit but wider than needed; also reverts cuBLAS, cuDNN, torch, container. Closed.
• PR [https://nvbugs/6463987][perf] Pin cuBLAS to 13.4.0.1 to work around GB300 NVFP4 GEMM heuristic regression #16623 (cuBLAS-only downgrade 13.4.1.2 → 13.4.0.1) — ruled out by nsys: cuBLAS kernels are byte-identical between fast and slow branches. Closed.

This PR is the surgical minimum: one package downgrade in two files.

Container rebuild requirement

The pyxis container image (LLM_SBSA_DOCKER_IMAGE from jenkins/current_image_tags.properties) bakes libnccl2 from the apt-installed package. This PR changes install_cuda_libs.sh, so the image tag must be re-generated by the CI build for the fix to take effect at runtime. Otherwise the container's baked-in NCCL 2.30.4 will still be loaded ahead of the pip wheel and the fix will not apply.

The main-HEAD verification above used LD_PRELOAD to force-load the pip-installed libnccl.so.2 (2.29.7) over the container's baked-in libnccl2 (2.30.4) — this is how the same numbers this PR is expected to produce were reproduced without rebuilding the container. Once the container rebuild lands from CI, the runtime NCCL will be 2.29.7 natively and the LD_PRELOAD workaround will not be needed.

Test plan

• Same-node A/B on aws-cmh GB300 con1: b2+NCCL2.29.7 median 4.15 tok/s CV 1.3% (5/0 fast) vs b2+native NCCL 2.30.4 median 3.38 tok/s CV 27% (3/3 bimodal)
• main HEAD verification (unpinned, 10 different nodes): main tot median 2.58 tok/s CV 27% (2/3 bimodal) vs main tot + this PR median 4.12 tok/s CV 0.9% (5/0 stable) — see above table
• Runtime NCCL identity confirmed in every slurm-*.out
• CI perf-sanity across the full GLM-5 test matrix once the container rebuilds
• Post-merge: 5-rep same-node reproduce on main HEAD to confirm sticky

Links

• Bug: https://nvbugs/6463987
• Main-HEAD fix-verification data (10 reps, 10 different aws-cmh GB300 nodes): computelab:/home/scratch.chenfeiz_gpu/repo/20260801-nccl29-fix-verify/
• Prior triage (nsys traces, kernel CSVs, best/worst reproduce logs): computelab:/home/scratch.chenfeiz_gpu/repo/20260730xxx/

[coderabbitai]

Walkthrough

The pull request updates the default NCCL package version in the CUDA installation script and lowers the nvidia-nccl-cu13 dependency upper bound in requirements.txt to 2.29.7.

Changes

NCCL Version Alignment

Layer / File(s)	Summary
NCCL version constraints docker/common/install_cuda_libs.sh, requirements.txt	The default NCCL version and Python dependency upper bound change from 2.30.4 to 2.29.7.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: atrifex

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the targeted NCCL downgrade and its GB300 GLM-5-NVFP4 performance purpose.
Description check	✅ Passed	The description explains the issue, root cause, solution, test evidence, pending validation, and container rebuild requirement.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@requirements.txt`:
- Line 29: Regenerate the dependency metadata from requirements.txt so
security_scanning/pyproject.toml and its lock files consistently cap
nvidia-nccl-cu13 at 2.29.7, matching the declared requirement and Docker
installer.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4377c4a1-6abe-459d-8db6-f96fca003e76

📥 Commits

Reviewing files that changed from the base of the PR and between a9544e0 and 8b91f32.

📒 Files selected for processing (2)

• docker/common/install_cuda_libs.sh
• requirements.txt

[coderabbitai]

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n -C 3 \
  'nvidia-nccl-cu13|NCCL_VER|2\.29\.7|2\.30\.4' \
  requirements.txt \
  security_scanning/pyproject.toml \
  security_scanning/examples/auto_deploy/poetry.lock \
  docker/common/install_cuda_libs.sh

rg -n -C 4 \
  'security_scanning|requirements\.txt|install_cuda_libs\.sh' \
  . || true

Repository: NVIDIA/TensorRT-LLM

Length of output: 50377

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- security_scanning/pyproject.toml ---'
cat -n security_scanning/pyproject.toml | sed -n '1,45p'

printf '%s\n' '--- lock metadata ---'
cat -n security_scanning/examples/auto_deploy/poetry.lock | sed -n '2378,2400p;4122,4140p'

printf '%s\n' '--- lock generator ---'
cat -n scripts/generate_lock_file.py | sed -n '40,175p'

printf '%s\n' '--- NCCL references in dependency/config files ---'
rg -n -i \
  'nvidia[-_]nccl[-_]cu13|NCCL_VER|libnccl|nccl4py' \
  --glob 'requirements*.txt' \
  --glob 'pyproject.toml' \
  --glob 'poetry.lock' \
  --glob '*.sh' \
  --glob 'Dockerfile*' \
  . | head -n 300

Repository: NVIDIA/TensorRT-LLM

Length of output: 17256

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- tracked dependency declarations ---'
python3 - <<'PY'
from pathlib import Path
import re

pattern = re.compile(r'nvidia[-_]nccl[-_]cu13|NCCL_VER|libnccl', re.I)
for path in sorted(Path('.').rglob('*')):
    if not path.is_file() or any(part in {'.git', 'node_modules', 'build', 'dist'} for part in path.parts):
        continue
    if path.name not in {'requirements.txt', 'requirements-dev.txt', 'requirements-windows.txt',
                         'pyproject.toml', 'poetry.lock'} and not path.name.startswith('Dockerfile') \
       and path.suffix not in {'.sh', '.groovy'}:
        continue
    try:
        lines = path.read_text(errors='replace').splitlines()
    except OSError:
        continue
    matches = [(i, line) for i, line in enumerate(lines, 1) if pattern.search(line)]
    if matches:
        print(path)
        for i, line in matches:
            print(f'  {i}: {line}')
PY

printf '%s\n' '--- generation and security-scan integration ---'
rg -n -C 5 \
  'generate_lock_file|security_scanning|poetry (install|lock)|black duck|Black Duck' \
  jenkins scripts security_scanning \
  --glob '*.groovy' --glob '*.py' --glob '*.toml' --glob '*.md' --glob '*.yml' --glob '*.yaml' \
  | head -n 240

printf '%s\n' '--- changed paths ---'
git status --short
git diff --stat

Repository: NVIDIA/TensorRT-LLM

Length of output: 7955

---

Regenerate the security-scanning dependency files.

security_scanning/pyproject.toml still permits nvidia-nccl-cu13 through 2.30.4, while requirements.txt and the Docker installer cap it at 2.29.7. Regenerate security_scanning/pyproject.toml and its lock files from requirements.txt.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@requirements.txt` at line 29, Regenerate the dependency metadata from
requirements.txt so security_scanning/pyproject.toml and its lock files
consistently cap nvidia-nccl-cu13 at 2.29.7, matching the declared requirement
and Docker installer.