[AMD][MI35X] Serve Qwen3.5 MXFP4-AttnFP8-V2 on the MI355X SGLang single-node arms - #2695
[AMD][MI35X] Serve Qwen3.5 MXFP4-AttnFP8-V2 on the MI355X SGLang single-node arms#2695yichiche wants to merge 1 commit into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
8b94398 to
5e9a889
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
yichiche
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32339406541 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32340034806 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32340034806 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32340034806 |
…le-node arms Switch qwen3.5-fp4-mi355x-sglang and qwen3.5-fp4-mi355x-sglang-mtp from amd/Qwen3.5-397B-A17B-MXFP4 to amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2. The image, search space, runner, precision, and launch scripts are unchanged.
5e9a889 to
453dae8
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32340034806 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32390324525 |
1 participant
Motivation
amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2is a newer AMD checkpoint of Qwen3.5-397B-A17B that keeps MXFP4 weights and additionally runs attention in FP8. It supersedesamd/Qwen3.5-397B-A17B-MXFP4as the checkpoint we want the MI355X SGLang single-node submission to represent.Modifications
In
configs/amd-master.yaml, changemodelonqwen3.5-fp4-mi355x-sglangandqwen3.5-fp4-mi355x-sglang-mtp:amd/Qwen3.5-397B-A17B-MXFP4→amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2That is the whole change. The
imagestays atlmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260730, and the TP and concurrency search space,isl/osl, runner, precision, and framework are untouched. No benchmark script is modified —runners/launch_mi355x-amds.shresolves the launch script frommodel-prefix,precision, andframework, so both arms keep usingbenchmarks/single_node/fixed_seq_len/qwen3.5_fp4_mi355x.shandqwen3.5_fp4_mi355x_mtp.sh, with the checkpoint arriving through$MODEL.The
qwen3.5-fp4-mi355x-atomarm still servesamd/Qwen3.5-397B-A17B-MXFP4and is deliberately left alone.Append the corresponding
perf-changelog.yamltrigger covering both config keys.Accuracy Tests
No accuracy-affecting logic changes in this repo. The submission now measures a different published checkpoint; correctness of the checkpoint itself is upstream of this repo. The generated matrix keeps its usual eval rows for both arms.
Benchmarking
Repo validation was run locally:
python -m pytest utils/matrix_logic/ -q→ 232 passed.python utils/matrix_logic/generate_sweep_configs.py full-sweep --config-files configs/amd-master.yaml --model-prefix qwen3.5 --precision fp4 --runner-type mi355x --scenario-type fixed-seq-len→ 29 configs: 19 onamd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2at the unchangedv0.5.16-rocm720-mi35x-20260730image, plus the 10 unchanged ATOM configs still onamd/Qwen3.5-397B-A17B-MXFP4.End-to-end MI355X numbers will come from the sweep triggered on this PR (
full-sweep-fail-fast). Because the checkpoint changes, the resulting numbers are not a like-for-like continuation of the previous MXFP4 series on these two config keys.Conflicts
#2640 touches these same two config keys: it bumps the image to
v0.5.17-rocm720-mi35x-20260818and adds--kv-cache-dtype fp8_e4m3plus--disable-shared-experts-fusionto the launch scripts. The two PRs change different lines of the same config blocks, so whichever lands first the other needs a rebase, but the changes are complementary rather than contradictory.