Skip to content

[https://nvbugs/6541322][fix] Remove stale waiver for TestQwen3NextInstruct::test_bf16_4gpu[dep4] - #17150

Open
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6541322
Open

[https://nvbugs/6541322][fix] Remove stale waiver for TestQwen3NextInstruct::test_bf16_4gpu[dep4]#17150
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6541322

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Root cause: The Qwen3-Next-80B bf16 TP4/EP4 test aborted during LLM startup on the V2 KV-cache Mamba SSM slot check, which computed only 75 available SSM slots against the 129 required by the configured batch size, raising before any model weights were loaded. The underlying defect was the V2 SSM pool being rounded below the minimum-slot floor implied by max_num_requests; that floor fix has since landed in HEAD via commit 7f7dccf, so the failing code path no longer exists. The waiver entry for this test therefore outlived the defect it was tracking.
  • Fix: Verification on current HEAD confirmed the reported failure signature is gone — the run cleared init completely (weights loaded, 19 CUDA-graph sizes captured, KV cache sized, 3786 MMLU requests submitted), which is structurally incompatible with the reported abort that produced zero model-load lines. The only remaining crash was unrelated GPU co-tenancy (a peer process holding ~121 GB/GPU, visible only to in-container nvidia-smi, leaving 53 MB free at a cublas allocation), so no product change was warranted. The change is waiver-only: drop the dep4 SKIP line so the test runs again in CI, while leaving the sibling tp1_block_reuse-cutlass waiver under the same bug in place since it was not shown to be fixed.
  • Automated fix generated by repair-bot

Test plan

  • Verify fix on the same GPU type as the original failure
  • Check for regressions in related tests

Links

Dev Engineer Review

  • Removed the stale dep4 waiver for TestQwen3NextInstruct::test_bf16_4gpu[dep4].
  • The waiver removal matches the fix for the V2 Mamba SSM slot-sizing defect.
  • The sibling tp1_block_reuse-cutlass waiver remains unchanged.
  • The test-list change is limited to one valid waiver entry.

QA Engineer Review

  • Modified tests/integration/test_lists/waives.txt.
  • Removed one waiver entry.
  • No test code changed.
  • CBTS coverage data is unavailable. Verdict: needs follow-up.

The V2 Mamba state pool ValueError was fixed by 7f7dccf
(KVCacheManagerV2 C++ translation, NVIDIA#14047), which added an SSM
min-slot floor in _build_cache_config using the same expression as
the live/dummy-slot check it guards, so the pool can no longer be
sized below that floor.

Remove the now-stale waiver for test_bf16_4gpu[dep4]. The sibling
test_nvfp4[tp1_block_reuse-cutlass] waiver is left in place: it has
an independent root cause tracked separately under nvbugs/6535767.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 652d6793-daad-475d-9837-5674bd3488b2

📥 Commits

Reviewing files that changed from the base of the PR and between 946f9c5 and 672b815.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Walkthrough

The change removes the skip entry for TestQwen3NextInstruct::test_bf16_4gpu[dep4], associated with NVBug 6535767, from the integration waiver list.

Changes

Cohort / File(s) Summary
Test waiver removal
tests/integration/test_lists/waives.txt
Removes the waiver for the Qwen3Next BF16 four-GPU accuracy test.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: bowenfu, yingguo-trt, mzweilz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the NVBugs issue, fix type, stale waiver, and affected test.
Description check ✅ Passed The description explains the root cause and waiver-only fix, and includes relevant verification details and the bug link.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants