[NVBUG-6448152][test] replay pre-PR15356 native admission

[chienchunhung]

Purpose

TEST ONLY — DO NOT REVIEW OR MERGE.

This draft performs the requested complete pre-PR #15356 replay of the native GUARANTEED_NO_EVICT first admission gate on the exact frozen slow baseline. It tests every transfer-related first-gate accounting change together while leaving the PR #15356 Python transport-budget gate enabled at multiplier 1.

The causal treatment is the signed first parent:

• frozen slow baseline: dd4a7ac2992f2b02dce381ce4e630c4ba4199436
• treatment head: 33833104032ad6157e4aba6150fe0cc9552de9d6
• treatment tree: 4025abdfa3881e695f1565c5b11808e18a74bd08
• stable patch-id: 3cd8c43e4c5a0b219fe8754c08e3a2afe08a9f3c
• binary diff SHA256: 0d791d1001942c3b0ea98dcc04bbf574682958b4235ac8ead89080ac516acdad

The current CI head 1c39d556886c93a4646a0679faa3559b64ddbcb8 adds latest main 946f9c5fbf0f30855533b9b3baa2a2a621b1742c as an ancestry-only second parent on preserved prior CI head 6cc12437167c941222231d37c2e25d6adbb07c9c. Its tree remains byte-identical to the treatment tree. Review and interpret only the baseline-to-treatment causal diff. This ancestry vehicle must never merge.

Single treatment

For GUARANTEED_NO_EVICT, a DISAGG_GENERATION_TRANS_IN_PROGRESS request is skipped before all four pre-PR #15356 accounting decisions:

1. logical maxNumRequests slots;
2. self-KV reservation;
3. cross-KV reservation;
4. PEFT reservation.

All non-transfer requests retain the original scheduler order and checks. STATIC and MAX_UTILIZATION are unchanged. Transfer-only requests remain non-compute-scheduled.

The downstream Python DisaggTransferAdmissionController is behaviorally unchanged at exact multiplier 1 / max_tokens_in_buffer=131104: normal selection, reverts, and wait decisions remain active. Added Python code observes its decisions only.

Exercised discriminator

The treatment path is primary. A read-only current-behavior shadow runs only on transfer-active iterations and only until the first exact ordered candidate divergence on each scheduling rank; it is absent on idle iterations and permanently disabled after divergence. Immediate ID-free markers and shutdown summaries split:

• pre_only_disagg_init;
• pre_only_ready_generation;
• pre_only_other.

The shadow result must be an ordered subset of the treatment result. This bounds instrumentation overhead while proving the treatment changed a reachable decision.

Frozen workload and acceptance

The exact historical C++/NIXL DeepSeek-R1 128K/8K workload remains fixed: GB300, three nodes/twelve GPUs, CTX PP4, GEN DEP8, 512 total requests, original selector/image/runtime/topology, mode2/cancellation-off, and multiplier-1 second gate.

A valid result requires:

• 512/512, zero failed, b_is_valid=true, clean CTX/GEN/disaggregated shutdown;
• treatment startup on every relevant GEN scheduling rank;
• transfer_active > 0 and an exact event=exercised old-versus-treatment divergence on every such rank;
• ordered-subset evidence and per-path shutdown counts;
• observer evidence from the unchanged second gate.

Recovery toward the exact pre-PR parent result (1515.84 output tok/s; slow child 799.73) with exercised evidence implicates the first-gate transfer-accounting bundle. Near 800 tok/s with a ready-generation divergence rules the bundle down as sufficient. Init-only divergence followed by unchanged Python deferral proves second-gate masking. Missing per-rank reachability or required functional evidence censors the result.

Formatting, configured hooks, DCO, diff checks, Python compilation, Python 3.13 test-list checks, and independent causal review passed. Focused pytest is blocked only by unavailable transformers; no local C++ build tree exists, so automatic compile gates must pass before one exact trigger.

The scarce GB300 trigger is intentionally withheld until the higher-priority PR #16645 versus rebased-main comparison has secured its matched runs.

[chienchunhung]

The old CI head is preserved on codex/backup-pr17141-pre-main-ancestry-refresh-20260731-6cc1243. The draft now points to signed ancestry-only refresh 1c39d556886c93a4646a0679faa3559b64ddbcb8 over latest main 946f9c5fbf0f30855533b9b3baa2a2a621b1742c. The treatment tree remains exactly 4025abdfa3881e695f1565c5b11808e18a74bd08; the frozen baseline-to-treatment patch-id and binary diff are unchanged. One exact GB300 trigger will follow renewed gates and mergeability verification.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #63160 [ run ] triggered by Bot. Commit: 1c39d55 Link to invocation

[tensorrt-cicd]

PR_Github #63160 [ run ] completed with state FAILURE. Commit: 1c39d55
/LLM/main/L0_MergeRequest_PR pipeline #51243 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Terminal result for the exact native first-admission replay:

• Exact workload completed 512/512 requests with 0 failures and a valid benchmark result.
• Output throughput: 799.18 tok/s (13,586.13 total tok/s), versus the exact PR [TRTLLM-12721][fix] Bound disagg transfer polling and admission #15356 baseline of 799.73 output tok/s: -0.069%. No throughput recovery was observed.
• The treatment was active and transfer-in-progress activity was observed on every GEN scheduling rank (0-7).
• No old-versus-treatment candidate divergence was observed: no event=exercised marker fired, and the observed pre/post candidate counts were identical.
• The unchanged transport-budget gate admitted all candidates it received; it reported no deferral, limiting, or wait.

Strict interpretation: this run confirms no recovery, but the replayed first-admission decision was nonbinding for this measured request sequence, so it neither implicates nor rules out that decision under a workload that forces divergence. The CI performance-threshold failure is the expected consequence of remaining near the known slow baseline, not a functional failure.

Closing this TEST ONLY draft unmerged. The diagnostic branch is preserved.