feat(config): add GLM-5.2 GB300 Dynamo-TRT AgentX / 添加 GLM-5.2 GB300 Dynamo-TRT AgentX 配置 #2657
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| name: dynamo-agg-gb300-tp8-c1-b2-mtp8 | ||
|
|
||
| model: | ||
| path: nvidia/GLM-5.2-NVFP4 | ||
| container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22.post1 | ||
| precision: fp4 | ||
|
|
||
| dynamo: | ||
| install: true | ||
| wheel: 1.4.0.dev20260807 | ||
| request_plane: tcp | ||
|
|
||
| health_check: | ||
| max_attempts: 270 | ||
| interval_seconds: 10 | ||
|
|
||
| resources: | ||
| gpu_type: gb300 | ||
| gpus_per_node: 4 | ||
| agg_nodes: 2 | ||
| agg_workers: 1 | ||
| gpus_per_agg: 8 | ||
|
|
||
| backend: | ||
| type: trtllm | ||
| aggregated_environment: | ||
| HF_HUB_OFFLINE: "1" | ||
| TRANSFORMERS_OFFLINE: "1" | ||
| TQDM_DISABLE: "1" | ||
| HF_HUB_DISABLE_PROGRESS_BARS: "1" | ||
| TLLM_LOG_LEVEL: INFO | ||
| TRTLLM_SERVER_DISABLE_GC: "1" | ||
| TRTLLM_WORKER_DISABLE_GC: "1" | ||
| TRTLLM_ENABLE_PDL: "1" | ||
| NCCL_GRAPH_MIXING_SUPPORT: "0" | ||
| MIMALLOC_PURGE_DELAY: "0" | ||
| PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True | ||
| TRTLLM_SERVE_ENABLE_MSGSPEC: "1" | ||
| TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD: "0.10" | ||
| UCX_CUDA_IPC_ENABLE_MNNVL: "y" | ||
| UCX_RNDV_SCHEME: put_zcopy | ||
| TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: "3.06" | ||
| TRTLLM_DISAGG_IDLE_CHECK: pr17324 | ||
| DYN_TRTLLM_ENABLE_ATTENTION_DP: "1" | ||
| DYN_ENGINE_CONV_AFFINITY: "1" | ||
| DYN_TOKENIZER: fastokens | ||
| DYN_PUBLISH_KV_EVENTS: "0" | ||
| trtllm_config: | ||
| aggregated: | ||
| cuda_graph_config: | ||
| batch_sizes: [1, 2] | ||
| enable_padding: false | ||
| disable_overlap_scheduler: false | ||
| enable_attention_dp: false | ||
| enable_chunked_prefill: true | ||
| enable_lm_head_tp_in_adp: false | ||
| trust_remote_code: true | ||
| kv_cache_config: | ||
| dtype: fp8 | ||
| enable_block_reuse: true | ||
| event_buffer_max_size: 0 | ||
| free_gpu_memory_fraction: 0.85 | ||
| tokens_per_block: 64 | ||
| max_batch_size: 2 | ||
| max_num_tokens: 8192 | ||
| max_seq_len: 1048576 | ||
| moe_config: | ||
| backend: TRTLLM | ||
| use_low_precision_moe_combine: true | ||
| moe_expert_parallel_size: 1 | ||
| num_postprocess_workers: 8 | ||
| perf_metrics_max_requests: 100000 | ||
| pipeline_parallel_size: 1 | ||
| print_iter_log: true | ||
| return_perf_metrics: false | ||
| sparse_attention_config: | ||
| algorithm: dsa | ||
| enable_heuristic_topk: true | ||
| use_cute_dsl_paged_mqa_logits: true | ||
| speculative_config: | ||
| decoding_type: MTP | ||
| max_draft_len: 8 | ||
| num_nextn_predict_layers: 8 | ||
| stream_interval: 20 | ||
| tensor_parallel_size: 8 | ||
| publish_events_and_metrics: false | ||
|
|
||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: | ||
| ETCD_LEASE_TTL: '120' | ||
| DYN_ROUTER_QUEUE_THRESHOLD: None | ||
| DYN_ROUTER_TEMPERATURE: '0' | ||
| DYN_TOKENIZER_CACHE: '1' | ||
| DYN_TOKENIZER_CACHE_BYTES: '8000000000' | ||
| DYN_TOKENIZER: fastokens | ||
| DYN_TCP_REQUEST_TIMEOUT: '30' | ||
| DYN_LOG: warn | ||
| DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400' | ||
| args: | ||
| router-mode: kv | ||
| no-kv-events: true | ||
| active-decode-blocks-threshold: None | ||
| active-prefill-tokens-threshold: None | ||
| active-prefill-tokens-threshold-frac: None | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: /infmax-workspace | ||
| RESULT_DIR: /logs/agentic | ||
| PORT: '8000' | ||
| IS_MULTINODE: 'true' | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400' | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache | ||
| HF_HUB_CACHE: /hf_hub_cache | ||
| WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126 | ||
| AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES: "0" | ||
| SERVED_MODEL_NAME: GLM-5.2-NVFP4 | ||
| MAX_MODEL_LEN: '1048576' | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true' | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0' | ||
| OPENAI_API_KEY: EMPTY | ||
| KV_OFFLOADING: none | ||
| MODEL: nvidia/GLM-5.2-NVFP4 | ||
| MODEL_PREFIX: glm5.2 | ||
| FRAMEWORK: dynamo-trt | ||
| PRECISION: fp4 | ||
| CONC: '1' | ||
| DURATION: '3600' | ||
|
Comment on lines
+126
to
+132
Contributor
There was a problem hiding this comment. 🟡 Two of the seven new GLM-5.2 recipes ( Extended reasoning...What the bug is: In this PR's set of seven new GLM-5.2 GB300 Dynamo-TRT recipes, five ( |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| name: dynamo-disagg-gb300-1p1d-tep8-c20-b5-mtp5 | ||
|
|
||
| model: | ||
| path: nvidia/GLM-5.2-NVFP4 | ||
| container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22.post1 | ||
| precision: fp4 | ||
|
|
||
| dynamo: | ||
| install: true | ||
| wheel: 1.4.0.dev20260807 | ||
| request_plane: tcp | ||
|
|
||
| health_check: | ||
| max_attempts: 270 | ||
| interval_seconds: 10 | ||
|
|
||
| resources: | ||
| gpu_type: gb300 | ||
| gpus_per_node: 4 | ||
| prefill_nodes: 1 | ||
| prefill_workers: 1 | ||
| gpus_per_prefill: 4 | ||
| decode_nodes: 2 | ||
| decode_workers: 1 | ||
| gpus_per_decode: 8 | ||
|
|
||
| backend: | ||
| type: trtllm | ||
| prefill_environment: &server_environment | ||
| HF_HUB_OFFLINE: "1" | ||
| TRANSFORMERS_OFFLINE: "1" | ||
| TQDM_DISABLE: "1" | ||
| HF_HUB_DISABLE_PROGRESS_BARS: "1" | ||
| TLLM_LOG_LEVEL: INFO | ||
| TRTLLM_SERVER_DISABLE_GC: "1" | ||
| TRTLLM_WORKER_DISABLE_GC: "1" | ||
| TRTLLM_ENABLE_PDL: "1" | ||
| NCCL_GRAPH_MIXING_SUPPORT: "0" | ||
| MIMALLOC_PURGE_DELAY: "0" | ||
| PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True | ||
| TRTLLM_SERVE_ENABLE_MSGSPEC: "1" | ||
| TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD: "0.10" | ||
| TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC: "600" | ||
| UCX_CUDA_IPC_ENABLE_MNNVL: "y" | ||
| UCX_RNDV_SCHEME: put_zcopy | ||
| TRTLLM_KVCACHE_SEND_BUFFER_COUNT: "1" | ||
| TRTLLM_KVCACHE_RECV_BUFFER_COUNT: "1" | ||
| TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: "2.61" | ||
| DYN_TRTLLM_ENABLE_ATTENTION_DP: "1" | ||
| DYN_ENGINE_CONV_AFFINITY: "1" | ||
| DYN_TOKENIZER: fastokens | ||
| DYN_PUBLISH_KV_EVENTS: "0" | ||
| decode_environment: *server_environment | ||
| trtllm_config: | ||
| prefill: | ||
| attention_dp_config: | ||
| enable_kv_cache_aware_routing: false | ||
| kv_cache_routing_conversation_affinity: true | ||
| kv_cache_routing_max_sessions: 65536 | ||
| cache_transceiver_config: | ||
| backend: NIXL | ||
| transceiver_runtime: PYTHON | ||
| kv_cache_bounce_size_mb: 5120 | ||
| max_tokens_in_buffer: 1048576 | ||
| kv_transfer_timeout_ms: 600000 | ||
| cuda_graph_config: null | ||
| disable_overlap_scheduler: false | ||
| enable_attention_dp: true | ||
| enable_chunked_prefill: true | ||
| trust_remote_code: true | ||
| kv_cache_config: | ||
| dtype: fp8 | ||
| enable_block_reuse: true | ||
| event_buffer_max_size: 0 | ||
| free_gpu_memory_fraction: 0.75 | ||
| host_cache_size: 137438953472 | ||
| tokens_per_block: 64 | ||
| max_batch_size: 256 | ||
| max_num_tokens: 8192 | ||
| max_seq_len: 1048576 | ||
| moe_config: | ||
| backend: CUTEDSL | ||
| moe_expert_parallel_size: 4 | ||
| num_postprocess_workers: 8 | ||
| pipeline_parallel_size: 1 | ||
| print_iter_log: true | ||
| return_perf_metrics: false | ||
| scheduler_config: | ||
| capacity_scheduler_policy: MAX_UTILIZATION | ||
| sparse_attention_config: | ||
| algorithm: dsa | ||
| enable_heuristic_topk: true | ||
| speculative_config: | ||
| decoding_type: MTP | ||
| max_draft_len: 5 | ||
| tensor_parallel_size: 4 | ||
| decode: | ||
| cache_transceiver_config: | ||
| backend: NIXL | ||
| transceiver_runtime: PYTHON | ||
| kv_cache_bounce_size_mb: 5120 | ||
| max_tokens_in_buffer: 1048576 | ||
| kv_transfer_timeout_ms: 600000 | ||
| cuda_graph_config: | ||
| batch_sizes: [1, 2, 4, 5] | ||
| enable_padding: true | ||
| enable_attention_dp: false | ||
| trust_remote_code: true | ||
| kv_cache_config: | ||
| dtype: fp8 | ||
| enable_block_reuse: false | ||
| event_buffer_max_size: 0 | ||
| free_gpu_memory_fraction: 0.8 | ||
| tokens_per_block: 64 | ||
| max_batch_size: 5 | ||
| max_num_tokens: 128 | ||
| max_seq_len: 1048576 | ||
| moe_config: | ||
| backend: TRTLLM | ||
| use_low_precision_moe_combine: true | ||
| moe_expert_parallel_size: 8 | ||
| num_postprocess_workers: 4 | ||
| pipeline_parallel_size: 1 | ||
| print_iter_log: true | ||
| return_perf_metrics: false | ||
| sparse_attention_config: | ||
| algorithm: dsa | ||
| enable_heuristic_topk: true | ||
| use_cute_dsl_paged_mqa_logits: true | ||
| use_cute_dsl_topk: true | ||
| speculative_config: | ||
| decoding_type: MTP | ||
| max_draft_len: 5 | ||
| stream_interval: 20 | ||
| tensor_parallel_size: 8 | ||
| publish_events_and_metrics: false | ||
|
|
||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: | ||
| ETCD_LEASE_TTL: '120' | ||
| DYN_ROUTER_QUEUE_THRESHOLD: None | ||
| DYN_ROUTER_TEMPERATURE: '0' | ||
| DYN_TOKENIZER_CACHE: '1' | ||
| DYN_TOKENIZER_CACHE_BYTES: '8000000000' | ||
| DYN_TOKENIZER: fastokens | ||
| DYN_TCP_REQUEST_TIMEOUT: '30' | ||
| DYN_LOG: warn | ||
| DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400' | ||
| args: | ||
| router-mode: kv | ||
| no-kv-events: true | ||
| active-decode-blocks-threshold: None | ||
| active-prefill-tokens-threshold: None | ||
| active-prefill-tokens-threshold-frac: None | ||
| orchestrator_placement: head | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| client_placement: last_decode | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: /infmax-workspace | ||
| RESULT_DIR: /logs/agentic | ||
| PORT: '8000' | ||
| IS_MULTINODE: 'true' | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400' | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache | ||
| HF_HUB_CACHE: /hf_hub_cache | ||
| WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126 | ||
| AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES: "0" | ||
| SERVED_MODEL_NAME: GLM-5.2-NVFP4 | ||
| MAX_MODEL_LEN: '1048576' | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true' | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0' | ||
| OPENAI_API_KEY: EMPTY | ||
| KV_OFFLOADING: none |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 agentic_srt.sh now hard-requires SRT_FRONTEND_HOST/SRT_FRONTEND_PORT via check_env_vars and drops the previous http://localhost:$PORT fallback (both in AIPERF_SERVER_URL and wait_for_agentic_servers_idle's metrics URL). This script is shared by ~70 pre-existing agentic recipes (e.g. trtllm/qwen3.5 disagg agentx, sglang/deepseek-v4 agentic) that never set the new client_placement/orchestrator_placement keys introduced only for the 7 new GLM-5.2 recipes, and that pin older srt-slurm versions (v1.0.25/v1.0.29/v1.0.36) than the v1.0.38 used here — if those older versions don't inject SRT_FRONTEND_HOST/PORT, every one of those recipes hard-fails immediately at check_env_vars.
Extended reasoning...
What the bug is:
agentic_srt.sh(benchmarks/multi_node/agentic_srt.sh:12-17) is a shared client-side benchmark script referenced bybenchmark.commandin dozens of pre-existing srt-slurm recipes, not just the 7 new GLM-5.2 recipes this PR adds. The PR addsSRT_FRONTEND_HOST SRT_FRONTEND_PORTto thecheck_env_varscall and removes the previoushttp://localhost:$PORTfallback, both inAIPERF_SERVER_URL's default (benchmark_lib.sh'sbuild_replay_cmd) and directly inwait_for_agentic_servers_idle's metrics URL.check_env_varscallsexit 1the instant any listed variable is unset, so this turns an optional convenience path into a hard requirement for every caller of the script.\n\nCode path / proof: I grepped the repo for every recipe whosebenchmark.commandpoints atagentic_srt.shand cross-checked which ones set the newclient_placement/orchestrator_placementkeys. Only the 7 new GLM-5.2 recipes added in this PR set those keys (grep -rl 'client_placement\|orchestrator_placement' srt-slurm-recipes/returns only the 7 new glm5.2 files). Directly inspecting a pre-existing recipe confirms it:trtllm/qwen3.5/gb300-fp4/disagg/agentx/disagg-gb300-1p1d-dep1-tep2-c44-b8-mtp-kvoffload.yamlhas a barebenchmark:block with no placement keys, calling the exact sameagentic_srt.sh. That recipe (and the deepseek-v4 sglang/vllm agentic recipes, also callers of this script) pin older srt-slurm releases inlaunch_gb300-nv.sh(v1.0.29 for qwen3.5-trt, v1.0.36 for the genericIS_AGENTICfp8 vllm branch) — strictly older than the v1.0.38 checkout this PR's newglm5.2+dynamo-trtbranch uses.SRT_FRONTEND_HOST/SRT_FRONTEND_PORTappear nowhere else in the repo, so they must be injected by srt-slurm itself at runtime, and the injection is plausibly tied to the same multi-node client/orchestrator placement mechanism that is new to v1.0.38 and only used by the new recipes. The original hardcodedhttp://localhost:$PORTis itself evidence that frontend-address injection was not relied upon (and possibly not available) when this script was first written for the older recipes, which instead colocate the benchmark client with the frontend node.\n\nWhy nothing currently prevents this:check_env_varsis a blunt fail-fast helper with no per-recipe scoping — once these two vars are added to the shared script's required list, every recipe that reaches this line is affected equally, whether or not it uses the new placement mechanism. There is no in-repo evidence (and no test) that srt-slurm injectsSRT_FRONTEND_HOST/SRT_FRONTEND_PORTfor the older, non-placement-aware recipe paths.\n\nStep-by-step manifestation: (1) A workflow run picks the pre-existingqwen3.5-fp4-gb300-dynamo-trt-agentxconfig (or any of the deepseek-v4/minimax-m3/kimi-k2.5 agentic configs) that still routes throughagentic_srt.sh. (2) The launcher clones srt-slurm at its pinned older version (v1.0.29/v1.0.36) and submits the job unmodified — nothing about this PR touches those recipes or their pinned versions. (3) On the benchmark-client node,agentic_srt.shsourcesbenchmark_lib.shand hitscheck_env_vars ... SRT_FRONTEND_HOST SRT_FRONTEND_PORT. (4) If the older srt-slurm release never sets those two env vars (because the older release predates the placement-based frontend-address injection), the check fails and the script does before running any benchmark traffic — a hard, silent-until-CI-fails regression across every affected recipe, despite this PR's intended scope being only the 7 new GLM-5.2 recipes.\n\nSuggested fix: Restore a localhost fallback when the new vars are unset, e.g.AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST:-localhost}:${SRT_FRONTEND_PORT:-$PORT}"and dropSRT_FRONTEND_HOST/SRT_FRONTEND_PORTfrom the hardcheck_env_varslist (or gate the requirement on the presence of a placement key), so pre-existing recipes keep colocating on localhost while new placement-aware recipes get the injected frontend address. Alternatively, confirm with srt-slurm's changelog/source that all pinned versions used by every consuming recipe (v1.0.25, v1.0.29, v1.0.36, sa-submission-q2-2026) inject these vars unconditionally, in which case this can be a documented invariant rather than a silent risk.