[https://nvbugs/6480621][test] Revert to 60-second KV transfer timeout for GB300 DeepSeek V4 Pro disaggregated perf-sanity

[chienchunhung]

Summary

• Revert kv_transfer_timeout_ms from 600000 ms to 60000 ms for both the GEN and CTX workers in the targeted GB300 DeepSeek V4 Pro disaggregated perf-sanity configuration.
• Keep the Python NIXL transceiver and every other workload parameter unchanged.

Motivation

NVBug 6480621 reported KV-transfer request failures after the 60-second timeout under a high-concurrency GB300 DeepSeek V4 Pro disaggregated E2E workload.

The same CI target previously passed in post-merge build #2873 with an explicit 600-second timeout. This PR restores the target to 60 seconds to verify whether current main can pass without the relaxed timeout.

Validation evidence

Targeted CI run: L0_MergeRequest_PR #51197

• Pipeline: SUCCESS
• Selected stage: SUCCESS
• Slurm job 2756069: COMPLETED on the first attempt, with 2,130 seconds of Slurm runtime
• Tests: 1 passed, 0 failed, 0 skipped
• Exact target: PASSED in 1,348.092 seconds (about 22m28s)
• Commit tested: 38ca970e96e22bb5653bb340430429d35fde9c6f
• Test configuration: Python NIXL transceiver with kv_transfer_timeout_ms: 60000 on both GEN and CTX
• Local/static checks: YAML parsing, git diff --check, DCO, and pre-commit passed

Target test:

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_deepseek-v4-pro-fp4_8k1k_con180_ctx3_dep4_gen1_dep32_eplb384_mtp3_ccb-NIXL]

Target stage:

GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2

Interpretation and remaining scope

This result demonstrates that the targeted CI regression proxy passes at current main with the original 60-second KV-transfer timeout. It supports restoring the stricter timeout in this test.

It does not, by itself, prove that the full workload from NVBug 6480621 is resolved:

• Original report: agentperf E2E, 8 CTX workers × 4 GPUs, 1 GEN worker × 32 GPUs, concurrency 1760
• CI proxy: gen-only, 3 CTX workers × 4 GPUs, 1 GEN worker × 32 GPUs, concurrency 180

The result is cumulative at current main; it does not isolate which merged change removed the symptom. Before closing NVBug 6480621, rerun the original or an equivalently stressful E2E reproducer with the 60-second timeout, preferably more than once to establish stability.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

[chienchunhung]

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

[tensorrt-cicd]

PR_Github #63105 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

[tensorrt-cicd]

PR_Github #63106 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

[tensorrt-cicd]

PR_Github #63105 [ run ] completed with state ABORTED. Commit: 38ca970

Link to invocation

[tensorrt-cicd]

PR_Github #63106 [ run ] completed with state SUCCESS. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51197 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

[tensorrt-cicd]

PR_Github #63144 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

[tensorrt-cicd]

PR_Github #63144 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51231 (Partly Tested) completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

[tensorrt-cicd]

PR_Github #63150 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

[tensorrt-cicd]

PR_Github #63150 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51234 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

[tensorrt-cicd]

PR_Github #63165 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

[tensorrt-cicd]

PR_Github #63165 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51249 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation