[cherry-pick][release/2.13] fix(distributed): subgroup NCCL regression + nccl_utils TRT 11 cleanup (#4407)#4408
Merged
Merged
Conversation
test_native_nccl.py: - Fix test_distributed_mode_subgroup and test_distributed_mode_context_switch regressions introduced by PR #4222 (Python runtime rework): add dist.barrier(group=subgroup) before first TRT forward so PyTorch's lazy ncclComm_t is initialized and bind_nccl_comm() sees a non-zero getCommPtr(). - Remove all use_python_runtime=True flags (deprecated no-op in 2.13). - Add _cpp_runtime_available() guard to TestPythonRuntimePickle so it skips in C++ builds where _TRTEngine's custom op is never registered. - Fix _multirank_scatter_correctness / _multirank_gather_correctness in run_multirank_tests(): wrap with lambdas that loop over root ranks to match their (root, rank, world_size, device) signature (broken since PR #4321). - Remove _multirank_distributed_mode_subgroup_python and _multirank_distributed_mode_context_switch_python (cannot work when C++ extension is loaded); collapse _multirank_pg_migration to cpp-only. _nccl_utils.py: - setup_nccl_for_torch_tensorrt(): drop ensure_nccl_symlink() call and pre-load libnccl.so.2 directly instead of via the unversioned symlink. TRT 11.0+ (ncclWrapper.cpp) tries dlopen("libnccl.so.2") first, matching PyTorch's soname exactly; glibc reuses the already-loaded handle so no symlink is needed. ensure_nccl_symlink() is kept for TRT < 11.0 users. - Update module docstring to reflect confirmed TRT 11.0+ behavior.
github-actions
Bot
added
component: tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #4407 onto
release/2.13. Applied cleanly (single commit, no conflicts);_nccl_utils.pyandtest_native_nccl.pyare byte-identical to #4407's head.What it does:
test_distributed_mode_subgroup,test_distributed_mode_context_switch) introduced by Python runtime rework #4222 (Python runtime rework): addsdist.barrier(group=subgroup)before the first TRT forward so PyTorch's lazyncclComm_tis initialized andbind_nccl_comm()sees a non-zerogetCommPtr().use_python_runtime=True(deprecated no-op in 2.13); collapses the python-runtime-only multirank variants to cpp-only._multirank_scatter/gather_correctnessdispatch inrun_multirank_tests()(broken since TRT 11 MD Ops #4321)._cpp_runtime_available()guard toTestPythonRuntimePickle.setup_nccl_for_torch_tensorrt()drops theensure_nccl_symlink()call and pre-loadslibnccl.so.2directly. TRT 11.0+ (ncclWrapper.cpp)dlopenslibnccl.so.2first, matching PyTorch's soname, so glibc reuses the already-loaded handle — no symlink needed.ensure_nccl_symlink()retained for TRT < 11.0.Original PR: #4407
🤖 Generated with Claude Code