[DO NOT MERGE][Test] DSV4 FP4 B300 SGLang AgentX DEP8 c512 on nightly (HiCache + MegaMoE FP4-act) - #2680
[DO NOT MERGE][Test] DSV4 FP4 B300 SGLang AgentX DEP8 c512 on nightly (HiCache + MegaMoE FP4-act)#2680yhyang201 wants to merge 2 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Not approving — the inline finding is a real correctness bug: the new CONC>=512 branch is gated only on DP_ATTENTION/CONC, so it also fires for the pre-existing production dsv4-fp4-b300-sglang-agentic-hicache-mtp config's c512 point (pinned to v0.5.17-cu130), contradicting the PR description's claim that existing entries are untouched.
Extended reasoning...
Overview
This PR adds a new test-only nightly config (dsv4-fp4-b300-sglang-agentic-hicache-mtp-nightly) and modifies the shared launcher script benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh to add a MegaMoE FP4-activation A2A path, a larger chunked-prefill size, a lowered mem-fraction, a new indexer flag, and router retry/circuit-breaker hardening, all gated on DP_ATTENTION=true && CONC>=512.
Security risks
None of the changes touch auth, crypto, or permissions. The changes are confined to benchmark launch scripts and YAML sweep configs, so there is no meaningful security surface here.
Level of scrutiny
This warrants a normal level of scrutiny despite the 'DO NOT MERGE / test-only' framing, because the script being modified is shared: I confirmed in configs/nvidia-master.yaml that the pre-existing dsv4-fp4-b300-sglang-agentic-hicache-mtp config (image v0.5.17-cu130) has a DEP8 arm with dp-attn: true and 512 in its conc-list, which runs through the exact same script. The new gating logic added at lines 127-146 and 157-159 checks only DP_ATTENTION/CONC, with no config-specific or image-specific discriminator, so it will also apply nightly-only flags (megamoe A2A, the fp4-indexer flag, mem-fraction 0.85, chunked-prefill 131072) to that production entry's c512 point — directly contradicting the PR description's 'Existing entries untouched' claim. This could either break the pinned-image launch (if the image's argparse rejects the new flags) or silently change a production baseline's recipe.
Other factors
The PR is explicitly labeled DO NOT MERGE / test-only and is expected to be closed after the author gets the measurement they want, which lowers the blast radius somewhat, but the shared-script gating bug is real and should be fixed (e.g. gate on KV_OFFLOADING=dram, which does differ between the two configs) before this pattern is reused or the PR is left open longer than intended.
| ) | ||
| MEM_FRACTION_STATIC=0.95 | ||
| if [ "$CONC" -ge 512 ]; then | ||
| # Leave room for FlashInfer's transient MoE workspace at the DEP8 tail. | ||
| MEM_FRACTION_STATIC=0.94 | ||
| # DEP8 c512: MegaMoE FP4-activation A2A (same DeepGEMM path as the | ||
| # vLLM amxf4 recipe) plus a larger per-step prefill budget so the | ||
| # 5,677-request warmup finishes inside the job budget. The DSV4 | ||
| # indexer's fp32 MQA logits transient scales with the per-rank chunk | ||
| # and has no budget cap in this image, so static memory must stay at | ||
| # 0.85 (the only value that has completed c512 end to end). | ||
| PARALLEL_ARGS+=(--moe-a2a-backend megamoe) | ||
| export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1 | ||
| export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1 | ||
| export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=16384 | ||
| MEM_FRACTION_STATIC=0.85 | ||
| CHUNKED_PREFILL_SIZE=131072 | ||
| else | ||
| CHUNKED_PREFILL_SIZE=16384 | ||
| fi | ||
| CHUNKED_PREFILL_SIZE=16384 | ||
| else | ||
| PARALLEL_ARGS+=( |
There was a problem hiding this comment.
🔴 The new CONC>=512 branch in dsv4_fp4_b300_sglang_mtp.sh (megamoe A2A backend, SGLANG_OPT_DEEPGEMM_MEGA_MOE_* env vars, --enable-deepseek-v4-fp4-indexer, mem-fraction 0.85, chunked-prefill 131072) is gated only on DP_ATTENTION and CONC, with no check on image or config key. The existing production config dsv4-fp4-b300-sglang-agentic-hicache-mtp (pinned to v0.5.17-cu130) already has a DEP8 arm with dp-attn: true and conc-list including 512, so it will also hit this nightly-only branch — contradicting the PR description's claim that "Existing entries untouched" and risking a startup failure or silently altered baseline on that production c512 point.
Extended reasoning...
The bug: benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh is the single shared launcher for both the pre-existing dsv4-fp4-b300-sglang-agentic-hicache-mtp config (image lmsysorg/sglang:v0.5.17-cu130) and the new dsv4-fp4-b300-sglang-agentic-hicache-mtp-nightly config (image lmsysorg/sglang:nightly-dev-cu13-20260818-c0b6474b) added in this PR. Both keys resolve to this script because they share model-prefix: dsv4, precision: fp4, runner: b300-nv, framework: sglang, and the agentic-coding scenario.
Code path: In configs/nvidia-master.yaml, the existing dsv4-fp4-b300-sglang-agentic-hicache-mtp config's last arm is { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [52, 72, 100, 128, 144, 196, 512], router: {...} } — note dp-attn: true and 512 already in its conc-list. When this arm's c512 point runs, the script receives DP_ATTENTION=true and CONC=512. The new branch added at lines 129-146 (if [ "$CONC" -ge 512 ]) and the indexer flag at lines 157-159 (if [ "$DP_ATTENTION" = "true" ] && [ "$CONC" -ge 512 ]) check nothing about which config or image is invoking the script — they fire purely on the env vars DP_ATTENTION/CONC set by whatever caller runs them.
Why nothing prevents this: The script has no notion of "which config key called me" or "which image am I running under" — it's purely parameterized by env vars (TP, CONC, DP_ATTENTION, KV_OFFLOADING, etc.), and both the old and new config's DEP8/c512 arms produce the identical DP_ATTENTION=true, CONC=512 combination. There's no image-version check, no dedicated flag (e.g. checking KV_OFFLOADING=dram — which does differ between the two configs, since the existing arm uses kv-offloading: none while the nightly arm uses kv-offloading: dram), so the gate is broader than intended.
Impact: Before this PR, the existing v0.5.17 c512 point ran with MEM_FRACTION_STATIC=0.94, CHUNKED_PREFILL_SIZE=16384, no --moe-a2a-backend megamoe, and no --enable-deepseek-v4-fp4-indexer. After this PR merges, that same production entry's c512 point will additionally receive --moe-a2a-backend megamoe, three SGLANG_OPT_DEEPGEMM_MEGA_MOE_* env vars, --enable-deepseek-v4-fp4-indexer, MEM_FRACTION_STATIC dropped to 0.85, and CHUNKED_PREFILL_SIZE raised to 131072. The PR description itself frames megamoe/the fp4-indexer flag as nightly-only capabilities being validated on the new nightly image — on the pinned v0.5.17-cu130 image these flags may not be recognized by sglang.launch_server's argparse, causing an immediate startup failure, or if they happen to exist, they'd silently change the production baseline's launch recipe. Either way this directly contradicts the PR description's explicit claim "Existing entries untouched."
Step-by-step proof:
configs/nvidia-master.yaml(pre-existing, line ~1174):dsv4-fp4-b300-sglang-agentic-hicache-mtp→ arm{ tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [..., 512] }.- The sweep driver launches the shared script with
DP_ATTENTION=true,CONC=512,KV_OFFLOADING=none. - Script line 127:
if [ "$DP_ATTENTION" = "true" ]; then→ true, enters DP-attn branch, setsMEM_FRACTION_STATIC=0.95. - Script line ~133 (new):
if [ "$CONC" -ge 512 ]; then→ true (512 >= 512) — this is new logic added by this PR with no image/config check. - Sets
PARALLEL_ARGS+=(--moe-a2a-backend megamoe), exports the threeSGLANG_OPT_DEEPGEMM_MEGA_MOE_*vars, setsMEM_FRACTION_STATIC=0.85,CHUNKED_PREFILL_SIZE=131072. - Script line ~157 (new):
if [ "$DP_ATTENTION" = "true" ] && [ "$CONC" -ge 512 ]; then MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer); fi→ true, adds the indexer flag. sglang.launch_serveris invoked on imagev0.5.17-cu130with these nightly-only flags/env vars it was never validated against — the production c512 point's recipe has changed from what it was before this PR, or fails to launch outright.
Fix: Scope the new branch to the nightly config specifically — e.g., gate on KV_OFFLOADING = "dram" (which does differ: the existing arm uses kv-offloading: none, the nightly arm uses kv-offloading: dram) or introduce an explicit env var/flag set only by the new nightly config key, rather than keying purely off CONC -ge 512.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32286791553 |
1 participant
DO NOT MERGE — test-only run, will be closed after measurement / 仅测试用，测完即关，不合并
Summary / 概要
Measures current sglang nightly on B300 AgentX DEP8 at concurrency 512. The existing sglang c512 point has never completed in CI; this PR tests whether HiCache host offload plus MegaMoE FP4-activation A2A and a larger prefill chunk lets it finish and sets a baseline.
dsv4-fp4-b300-sglang-agentic-hicache-mtp-nightlyonlmsysorg/sglang:nightly-dev-cu13-20260818-c0b6474b, single arm: DEP8 + HiCache DRAM offload + MTP, conc-list [512]. Existing entries untouched.DP_ATTENTION && CONC >= 512only:--moe-a2a-backend megamoe,SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1,SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1,SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=16384,--enable-deepseek-v4-fp4-indexerCHUNKED_PREFILL_SIZE=131072(16384 tokens per DP rank)MEM_FRACTION_STATIC=0.85(the DSV4 indexer fp32 logits transient scales with the per-rank chunk and has no cap in this image)--disable-retries.Planned follow-up commits on this PR (one sweep each): mem-fraction 0.88 probe,
SGLANG_RUST_SERVER=1.测量 sglang nightly 在 B300 AgentX DEP8 c512 的当前上限。新增独立 nightly config key（单点 c512 + HiCache 主机卸载 + MegaMoE FP4 激活 A2A + 16384/rank prefill chunk + mem 0.85），现有 entry 零改动，recipe 改动全部 gate 在 CONC>=512。后续 commit 阶梯：mem 0.88 试探、
SGLANG_RUST_SERVER=1。🤖 Generated with Claude Code