feat(config): add GLM-5.2 GB300 Dynamo-TRT AgentX / 添加 GLM-5.2 GB300 Dynamo-TRT AgentX 配置 - #2657
feat(config): add GLM-5.2 GB300 Dynamo-TRT AgentX / 添加 GLM-5.2 GB300 Dynamo-TRT AgentX 配置#2657RohitNagraj wants to merge 12 commits into
Conversation
添加 GLM-5.2 GB300 Dynamo-TRT AgentX 配置和多节点 srt-slurm 配方。
同步 origin/main，并将本配置的性能变更日志条目保留在文件末尾。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
在变更日志中补充 PR #2657 链接。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| check_env_vars \ | ||
| MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION \ | ||
| SRT_FRONTEND_HOST SRT_FRONTEND_PORT | ||
|
|
||
| export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" | ||
| echo "Using srt-slurm frontend endpoint: $AIPERF_SERVER_URL" |
There was a problem hiding this comment.
🔴 agentic_srt.sh now hard-requires SRT_FRONTEND_HOST/SRT_FRONTEND_PORT via check_env_vars and drops the previous http://localhost:$PORT fallback (both in AIPERF_SERVER_URL and wait_for_agentic_servers_idle's metrics URL). This script is shared by ~70 pre-existing agentic recipes (e.g. trtllm/qwen3.5 disagg agentx, sglang/deepseek-v4 agentic) that never set the new client_placement/orchestrator_placement keys introduced only for the 7 new GLM-5.2 recipes, and that pin older srt-slurm versions (v1.0.25/v1.0.29/v1.0.36) than the v1.0.38 used here — if those older versions don't inject SRT_FRONTEND_HOST/PORT, every one of those recipes hard-fails immediately at check_env_vars.
Extended reasoning...
What the bug is: agentic_srt.sh (benchmarks/multi_node/agentic_srt.sh:12-17) is a shared client-side benchmark script referenced by benchmark.command in dozens of pre-existing srt-slurm recipes, not just the 7 new GLM-5.2 recipes this PR adds. The PR adds SRT_FRONTEND_HOST SRT_FRONTEND_PORT to the check_env_vars call and removes the previous http://localhost:$PORT fallback, both in AIPERF_SERVER_URL's default (benchmark_lib.sh's build_replay_cmd) and directly in wait_for_agentic_servers_idle's metrics URL. check_env_vars calls exit 1 the instant any listed variable is unset, so this turns an optional convenience path into a hard requirement for every caller of the script.\n\nCode path / proof: I grepped the repo for every recipe whose benchmark.command points at agentic_srt.sh and cross-checked which ones set the new client_placement/orchestrator_placement keys. Only the 7 new GLM-5.2 recipes added in this PR set those keys (grep -rl 'client_placement\|orchestrator_placement' srt-slurm-recipes/ returns only the 7 new glm5.2 files). Directly inspecting a pre-existing recipe confirms it: trtllm/qwen3.5/gb300-fp4/disagg/agentx/disagg-gb300-1p1d-dep1-tep2-c44-b8-mtp-kvoffload.yaml has a bare benchmark: block with no placement keys, calling the exact same agentic_srt.sh. That recipe (and the deepseek-v4 sglang/vllm agentic recipes, also callers of this script) pin older srt-slurm releases in launch_gb300-nv.sh (v1.0.29 for qwen3.5-trt, v1.0.36 for the generic IS_AGENTIC fp8 vllm branch) — strictly older than the v1.0.38 checkout this PR's new glm5.2+dynamo-trt branch uses. SRT_FRONTEND_HOST/SRT_FRONTEND_PORT appear nowhere else in the repo, so they must be injected by srt-slurm itself at runtime, and the injection is plausibly tied to the same multi-node client/orchestrator placement mechanism that is new to v1.0.38 and only used by the new recipes. The original hardcoded http://localhost:$PORT is itself evidence that frontend-address injection was not relied upon (and possibly not available) when this script was first written for the older recipes, which instead colocate the benchmark client with the frontend node.\n\nWhy nothing currently prevents this: check_env_vars is a blunt fail-fast helper with no per-recipe scoping — once these two vars are added to the shared script's required list, every recipe that reaches this line is affected equally, whether or not it uses the new placement mechanism. There is no in-repo evidence (and no test) that srt-slurm injects SRT_FRONTEND_HOST/SRT_FRONTEND_PORT for the older, non-placement-aware recipe paths.\n\nStep-by-step manifestation: (1) A workflow run picks the pre-existing qwen3.5-fp4-gb300-dynamo-trt-agentx config (or any of the deepseek-v4/minimax-m3/kimi-k2.5 agentic configs) that still routes through agentic_srt.sh. (2) The launcher clones srt-slurm at its pinned older version (v1.0.29/v1.0.36) and submits the job unmodified — nothing about this PR touches those recipes or their pinned versions. (3) On the benchmark-client node, agentic_srt.sh sources benchmark_lib.sh and hits check_env_vars ... SRT_FRONTEND_HOST SRT_FRONTEND_PORT. (4) If the older srt-slurm release never sets those two env vars (because the older release predates the placement-based frontend-address injection), the check fails and the script does before running any benchmark traffic — a hard, silent-until-CI-fails regression across every affected recipe, despite this PR's intended scope being only the 7 new GLM-5.2 recipes.\n\nSuggested fix: Restore a localhost fallback when the new vars are unset, e.g. AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST:-localhost}:${SRT_FRONTEND_PORT:-$PORT}" and drop SRT_FRONTEND_HOST/SRT_FRONTEND_PORT from the hard check_env_vars list (or gate the requirement on the presence of a placement key), so pre-existing recipes keep colocating on localhost while new placement-aware recipes get the injected frontend address. Alternatively, confirm with srt-slurm's changelog/source that all pinned versions used by every consuming recipe (v1.0.25, v1.0.29, v1.0.36, sa-submission-q2-2026) inject these vars unconditionally, in which case this can be a documented invariant rather than a silent risk.
| sparse_attention_config: | ||
| algorithm: dsa | ||
| enable_heuristic_topk: true | ||
| use_cute_dsl_paged_mqa_logits: true | ||
| speculative_config: | ||
| decoding_type: MTP |
There was a problem hiding this comment.
🟡 The decode sparse_attention_config in dynamo-disagg-gb300-6p1d-dep8-c227-b16-mtp3.yaml sets use_cute_dsl_paged_mqa_logits: true but omits use_cute_dsl_topk: true, unlike all 6 other new recipes in this PR (including the structurally similar 8p1d and 5p1d) which set both flags together. This looks like a copy/paste omission that leaves 6p1d on the non-CuteDSL top-k path; recommend adding use_cute_dsl_topk: true for consistency with the other topology points.
Extended reasoning...
The decode sparse_attention_config block in dynamo-disagg-gb300-6p1d-dep8-c227-b16-mtp3.yaml (lines 136-141) is:
sparse_attention_config:
algorithm: dsa
enable_heuristic_topk: true
use_cute_dsl_paged_mqa_logits: true
speculative_config:It is missing use_cute_dsl_topk: true. Every other one of the 7 new GLM-5.2 recipes added in this PR sets both use_cute_dsl_paged_mqa_logits: true and use_cute_dsl_topk: true together in the decode block: 1p1d, 1p3d, 1p4d, 3p4d, 5p1d, and 8p1d all pair the two flags. 6p1d is the sole outlier across all 7 files.
The closest analog is 8p1d, whose decode block is otherwise near-identical to 6p1d's: both use moe_config.backend: CUTEDSL, use_low_precision_moe_combine: true, enable_attention_dp: true, enable_lm_head_tp_in_adp: false, mtp3, and max_batch_size: 16 — they differ only in scale (dep8/tp8 for 6p1d vs dep16/tp16 for 8p1d). 8p1d sets use_cute_dsl_topk: true; 6p1d does not. Given how mechanically similar the two recipes are (likely generated by copying one and adjusting scale parameters), the missing flag in 6p1d reads as an unintentional omission rather than deliberate per-topology tuning.
Nothing in the existing YAML schema or downstream tooling (benchmark_lib.sh, agentic_srt.sh, nvidia-master.yaml) validates that these two DSA flags are paired, so nothing catches or prevents this inconsistency — the recipe is syntactically valid and the benchmark will run to completion either way.
Impact: this is not a crash or correctness bug — the engine still starts and serves requests. With use_cute_dsl_topk unset, TensorRT-LLM's DSA sparse-attention kernel falls back to its non-CuteDSL top-k selection path for that one topology point (dep8/tp8, conc=227), while every other recipe in this PR benchmarks the CuteDSL top-k kernel. That makes the 6p1d benchmark number not directly comparable to its siblings and likely represents a missed optimization for that specific point (CuteDSL kernels are typically the faster path when available on this hardware/precision combination).
Proof by comparison (decode sparse_attention_config block, all 7 recipes):
- 1p1d, 1p3d, 1p4d, 3p4d:
algorithm: dsa,enable_heuristic_topk: true,use_cute_dsl_paged_mqa_logits: true,use_cute_dsl_topk: true - 5p1d: same as above (despite decode
moe_config.backend: TRTLLM, not CUTEDSL — showing the pairing holds regardless of MoE backend) - 8p1d: same as above (decode
moe_config.backend: CUTEDSL, the closest structural match to 6p1d) - 6p1d:
algorithm: dsa,enable_heuristic_topk: true,use_cute_dsl_paged_mqa_logits: true—use_cute_dsl_topkmissing
Fix: add use_cute_dsl_topk: true immediately after use_cute_dsl_paged_mqa_logits: true in the 6p1d decode sparse_attention_config block, matching the pattern used in all 6 sibling recipes.
All four independent verifiers confirmed this factual pattern (with one verifier noting a minor, non-material correction: 5p1d's decode MoE backend is TRTLLM rather than CUTEDSL as originally stated, though this does not change the core finding since 5p1d still pairs both cute_dsl flags). No verifier disputed the inconsistency itself; all converged on nit severity since the config remains valid and the benchmark still runs — only the resulting numbers for that one topology point are affected.
| KV_OFFLOADING: none | ||
| MODEL: nvidia/GLM-5.2-NVFP4 | ||
| MODEL_PREFIX: glm5.2 | ||
| FRAMEWORK: dynamo-trt | ||
| PRECISION: fp4 | ||
| CONC: '1' | ||
| DURATION: '3600' |
There was a problem hiding this comment.
🟡 Two of the seven new GLM-5.2 recipes (dynamo-disagg-gb300-1p3d-tep8-c1-b1-mtp7.yaml lines 187-193, and dynamo-disagg-gb300-8p1d-dep16-c259-b16-mtp3.yaml) append MODEL/MODEL_PREFIX/FRAMEWORK/PRECISION/CONC/DURATION to benchmark.env, while the other 5 siblings in this PR (and every qwen3.5/deepseek-v4 agentic recipe) omit them since the harness already injects these values. Recommend dropping the 6 redundant lines from these two files to match the other 5 and avoid CONC/DURATION silently drifting from the master-config conc-list.
Extended reasoning...
What the bug is: In this PR's set of seven new GLM-5.2 GB300 Dynamo-TRT recipes, five (1p1d, 1p4d, 3p4d, 5p1d, 6p1d) end their benchmark.env block at KV_OFFLOADING: none, while two — dynamo-disagg-gb300-1p3d-tep8-c1-b1-mtp7.yaml (lines 187-193) and dynamo-disagg-gb300-8p1d-dep16-c259-b16-mtp3.yaml (lines 188-194) — additionally set MODEL: nvidia/GLM-5.2-NVFP4, MODEL_PREFIX: glm5.2, FRAMEWORK: dynamo-trt, PRECISION: fp4, CONC: '1'/'259', and DURATION: '3600'.\n\nWhy these are redundant, not required: The benchmark command for every one of these recipes is bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh, whose check_env_vars call (agentic_srt.sh:12-13) hard-requires MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION SRT_FRONTEND_HOST SRT_FRONTEND_PORT. None of RESULT_FILENAME, SRT_FRONTEND_HOST, or SRT_FRONTEND_PORT appear in any of the seven recipes' benchmark.env, which proves the harness/srt-slurm launcher injects the full required set into the job environment before invoking this script — the recipe's own benchmark.env is additive convenience, not the sole source. Since the five sibling files (and the pre-existing qwen3.5 gb300-fp4/disagg/agentx/*.yaml recipes and the deepseek-v4 vllm agentic recipes, all driving the same script) run successfully while omitting all six of these keys, the two files' copies are provably dead weight left over from copy-pasting a different template.\n\nConcrete proof-by-example: Take dynamo-disagg-gb300-8p1d-dep16-c259-b16-mtp3.yaml. Its benchmark.env hardcodes CONC: '259'. The corresponding configs/nvidia-master.yaml search-space entry for this topology already sets conc-list: [259], which the harness translates into the CONC env var it exports before running agentic_srt.sh. If a maintainer later retunes that conc-list entry to, say, [300] without noticing the hardcoded duplicate in the YAML's benchmark.env, nothing forces the two to match — whichever one the harness actually honors, the other becomes a misleading, stale artifact sitting in the checked-in recipe file. Since the harness-injected value is what agentic_srt.sh's check_env_vars sees (env exports from the launcher take effect regardless of what's also declared in the YAML's static block, and the five other siblings prove the recipe-level copy isn't needed at all), the duplicate is pure drift risk with no functional benefit.\n\nImpact: Purely cosmetic/maintenance risk today — the recipes still run correctly because the harness supplies the authoritative values regardless. But it makes these two files look 'special' compared to their five siblings for no reason, and creates a silent-drift trap if conc-list/duration are ever changed in nvidia-master.yaml without updating the recipe file (or vice versa).\n\nFix: Delete the six lines (MODEL, MODEL_PREFIX, FRAMEWORK, PRECISION, CONC, DURATION) from both dynamo-disagg-gb300-1p3d-tep8-c1-b1-mtp7.yaml and dynamo-disagg-gb300-8p1d-dep16-c259-b16-mtp3.yaml's benchmark.env blocks so all seven GLM-5.2 recipes are structurally consistent.
| else | ||
| echo "WARNING: RUN_EVAL=true but no eval results found at $EVAL_DIR" | ||
| fi | ||
|
|
||
| # srt-slurm stages eval artifacts but does not write the metadata file | ||
| # consumed by score validation. Reuse the canonical metadata writer so | ||
| # topology and recipe identity stay aligned with the workflow inputs. | ||
| eval_conc_value="${EVAL_CONC:-${CONC:-1}}" | ||
| ( | ||
| export IS_MULTINODE=true | ||
| # shellcheck source=benchmarks/benchmark_lib.sh | ||
| source "$GITHUB_WORKSPACE/benchmarks/benchmark_lib.sh" | ||
| _write_lm_eval_meta_json \ | ||
| "$GITHUB_WORKSPACE/meta_env.json" "" "$eval_conc_value" |
There was a problem hiding this comment.
🟡 In runners/launch_gb300-nv.sh:664 (new eval meta_env.json block), eval_conc_value="${EVAL_CONC:-${CONC:-1}}" is passed straight to _write_lm_eval_meta_json as metadata_conc, which embeds it unquoted as "conc": ${metadata_conc}, in the JSON template (benchmarks/benchmark_lib.sh). EVAL_CONC is documented as possibly a space-separated list (e.g. eval-all-concs runs), which would produce invalid JSON like "conc": 10 20 30,. The other caller, append_lm_eval_summary, avoids this by extracting only the first value via metadata_conc="${batch_concs%% *}" — this new call site should do the same, e.g. eval_conc_value="${eval_conc_value%% *}".
Extended reasoning...
What the bug is: The new eval-metadata block added by this PR at runners/launch_gb300-nv.sh:661-670 computes eval_conc_value="${EVAL_CONC:-${CONC:-1}}" and passes it verbatim as the third positional argument (metadata_conc) to _write_lm_eval_meta_json in benchmarks/benchmark_lib.sh. That function embeds the value unquoted directly into a heredoc JSON template: "conc": ${metadata_conc},. A JSON number field cannot legally hold multiple space-separated tokens, so if EVAL_CONC ever contains more than one value, the resulting meta_env.json is syntactically invalid JSON.
Why this can actually happen: EVAL_CONC is not always a single value. .github/workflows/benchmark-multinode-tmpl.yml documents the eval-conc workflow input as 'Concurrency value or space-separated list for eval requests' and maps it directly to EVAL_CONC. utils/evals/EVALS.md states explicitly: 'For multi-node all-evals, EVAL_CONC is a space-separated list.' run-sweep.yml/e2e-tests.yml populate this with join(matrix.config.conc, ' ') when eval-all-concs is enabled. So this is a documented, supported, first-class flow elsewhere in the same pipeline — just one this new call site doesn't handle.
Why existing code doesn't already protect against it: The codebase already has a proven pattern for this exact hazard. The other caller of _write_lm_eval_meta_json, append_lm_eval_summary (also in benchmark_lib.sh), handles a multi-value concurrency list (EVAL_BATCHED_CONCS) by extracting only the first token before calling the writer: metadata_conc="${batch_concs%% *}". The full list is preserved separately in the eval_concs/completed_eval_concs/failed_eval_concs JSON array fields, keeping the scalar conc field a single integer. The new call site in launch_gb300-nv.sh skips this extraction entirely and passes the raw value through.
Step-by-step proof:
- A multi-node sweep sets
eval-all-concs: true, producing e.g.EVAL_CONC="10 20 30"via the workflow'sjoin(..., ' '). launch_gb300-nv.shruns withRUN_EVAL=true, reaches line 664:eval_conc_value="10 20 30".- It calls
_write_lm_eval_meta_json "/meta_env.json" "" "10 20 30". - Inside the heredoc, this renders as:
"conc": 10 20 30,— which is not valid JSON (bare token sequence in a value position). - Any downstream consumer that does
json.load()on this file (the block's own comment says this feeds score validation;utils/evals/validate_scores.pyandutils/validate_reusable_sweep_artifacts.pyboth parsemeta_env.json) throws a JSON decode error, failing score/artifact validation for the run.
Fix: Extract just the first token before passing it through, matching the existing convention: eval_conc_value="${eval_conc_value%% *}" right after the current assignment, or pass "${eval_conc_value%% *}" directly into the _write_lm_eval_meta_json call.
Severity note: This is a real correctness defect in newly-added code, but it is latent for this PR specifically — every GLM-5.2 recipe/scenario added here uses a single-value conc-list (e.g. [1], [30], [152]), so EVAL_CONC will never be multi-valued for any config this PR ships. It only manifests if this launcher is later exercised with the documented multi-value eval-conc/eval-all-concs flow, which no current recipe routed through launch_gb300-nv.sh uses. Given it doesn't block or break anything at merge time and is a one-line, low-risk hardening fix, this is a nit rather than a merge-blocker.
| elif [[ "$IS_AGENTIC" == "1" && $FRAMEWORK == "dynamo-trt" && $MODEL_PREFIX == "glm5.2" ]]; then | ||
| git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" | ||
| cd "$SRT_REPO_DIR" | ||
| git checkout v1.0.38 | ||
| TRTLLM_RECIPES_DIR="benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2" | ||
| mkdir -p "$TRTLLM_RECIPES_DIR" | ||
| cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2" \ | ||
| "$TRTLLM_RECIPES_DIR" | ||
| if [[ "${EVAL_ONLY:-false}" == "true" ]]; then | ||
| find "$TRTLLM_RECIPES_DIR" -name "*.yaml" \ | ||
| -exec sed -i '/TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS/d' {} + | ||
| fi |
There was a problem hiding this comment.
🟡 runners/launch_gb300-nv.sh's dcgm-power preflight (lines 145-146) assumes CONFIG_FILE is recipes/-relative and strips that prefix to find the mirrored recipe file, but the new glm5.2 dynamo-trt CONFIG_FILE values (e.g. benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2/gb300-fp4/agentic/....yaml in configs/nvidia-master.yaml) start with benchmarks/ instead, so the strip is a no-op and _RECIPE_SRC becomes a doubled, nonexistent path. The -f check then silently fails and the telemetry scan never runs, leaving USES_DCGM_POWER=0 unconditionally for this whole recipe family — currently harmless since none of the 7 recipes enable dcgm-power, but it would silently defeat detection for a future glm5.2 dynamo-trt power recipe.
Extended reasoning...
What the bug is. runners/launch_gb300-nv.sh has a preflight block (lines 140-156) that decides whether a job is a "dcgm-power" lane by reading the recipe YAML referenced by CONFIG_FILE and scanning it for an enabled telemetry: {provider: dcgm-power, enabled: true} block:
USES_DCGM_POWER=0
_RECIPE_REL="${CONFIG_FILE%%:*}"
_RECIPE_SRC="$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/${_RECIPE_REL#recipes/}"
if [[ -n "$CONFIG_FILE" && -f "$_RECIPE_SRC" ]] && awk '...' "$_RECIPE_SRC"; then
USES_DCGM_POWER=1
fiThe ${_RECIPE_REL#recipes/} strip assumes every CONFIG_FILE value is recipes/-relative — which was true for every existing entry in configs/nvidia-master.yaml before this PR (e.g. qwen3.5's recipes/trtllm/qwen3.5/gb300-fp4/disagg/agentx/..., for which the strip correctly yields the workspace-relative mirror path under benchmarks/multi_node/srt-slurm-recipes/).
The code path that triggers it. This PR's 7 new glm5.2-fp4-gb300-dynamo-trt-agentic-mtp entries instead set CONFIG_FILE=benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2/gb300-fp4/agentic/....yaml (matching where runners/launch_gb300-nv.sh:209-220's new elif branch mirrors the recipes to, via TRTLLM_RECIPES_DIR="benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2"). Because this value does not start with recipes/, ${_RECIPE_REL#recipes/} is a no-op, so _RECIPE_SRC becomes:
$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2/gb300-fp4/agentic/....yaml
a doubled, nonexistent path.
Why existing code doesn't prevent it. The -f "$_RECIPE_SRC" guard silently swallows this: it just evaluates to false, so the whole if short-circuits and USES_DCGM_POWER stays at its initialized value of 0 — no error, no warning, nothing visibly wrong in the launcher output.
Step-by-step proof.
CONFIG_FILE="benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2/gb300-fp4/agentic/dynamo-disagg-gb300-1p1d-tep8-c20-b5-mtp5.yaml"(fromconfigs/nvidia-master.yamlline ~8497, additional-settings for theagentic-codingscenario)._RECIPE_REL="${CONFIG_FILE%%:*}"→ same string (no:zip_overridesuffix here)._RECIPE_REL#recipes/→ unchanged, since the string starts withbenchmarks/, notrecipes/._RECIPE_SRC="$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5.2/gb300-fp4/agentic/dynamo-disagg-gb300-1p1d-tep8-c20-b5-mtp5.yaml"— doubled and nonexistent.[[ -f "$_RECIPE_SRC" ]]is false → theawktelemetry scan never executes.USES_DCGM_POWERremains0, regardless of what the actual recipe YAML'stelemetry:block says.
Impact. Today this is inert: none of the 7 new glm5.2 dynamo-trt recipe YAMLs in this PR contain a telemetry: {provider: dcgm-power, enabled: true} block, so USES_DCGM_POWER=0 happens to be the correct answer regardless. But the detection mechanism itself is defeated for the entire glm5.2-fp4-gb300-dynamo-trt-agentic-mtp family — if a future recipe in this family enabled dcgm-power telemetry, the launcher would silently skip the power-lane branch (producer pin override, DCGM exporter squash import, the fp8-only guard at line ~163, and the exporter provenance snapshot in the EXIT trap), rather than either enabling the power lane or failing loudly.
How to fix. Either (a) make the new glm5.2 dynamo-trt CONFIG_FILE values recipes/-relative like every other entry (aligning with the existing convention and the qwen3.5 precedent), or (b) make the preflight's path resolution convention-agnostic — e.g. strip whichever of recipes/ or benchmarks/multi_node/srt-slurm-recipes/ prefix is actually present, or resolve directly from the checked-in benchmarks/multi_node/srt-slurm-recipes/... workspace path when CONFIG_FILE already points there.
更新 GLM-5.2 GB300 Pareto01 配方，并移除重复的可选身份信息。
合并 main，并按仅追加规则解决性能变更日志冲突。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32171457338 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32192472154 |
更新 GLM-5.2 GB300 TensorRT-LLM AgentX Pareto 配置，并关闭所有点的性能指标响应。
合并 origin/main 并解决性能变更日志冲突。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32298789181 |
将六个 GLM-5.2 解耦配方的前端放置在头节点，以匹配评测路由，并同步 main。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32320897226 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32320897226 |
将 GB300 Slurm 启动器切换到 batch_3 分区，并同步 main。
同步 main 分支的最新变更。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32338308177 |
仅为 GLM-5.2 GB300 Dynamo-TRT AgentX 配置启用 batch_3 分区。
使 srtctl 提交的作业使用启动脚本解析后的 Slurm 分区。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32340541338 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32346724519 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32346724519 |
2 participants
Description / 说明
Adds the GLM-5.2 NVFP4 GB300 AgentX configuration and seven srt-slurm recipes for Dynamo with TensorRT-LLM disaggregated serving. Updates the launcher and AgentX frontend routing for the new recipe set.
添加 GLM-5.2 NVFP4 GB300 AgentX 配置及七个 srt-slurm 配方，用于 Dynamo 与 TensorRT-LLM 解耦推理，并更新新配方所需的启动脚本和 AgentX 前端路由。