-
Notifications
You must be signed in to change notification settings - Fork 264
feat(glm52-agentx): Add B300 Dynamo+TRT-LLM AgentX recipes #2666
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
RohitNagraj
wants to merge
9
commits into
main
Choose a base branch
from
glm5.2-fp4-b300-dynamo-trt-agentic-mtp
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
987c64d
feat(config): add GLM-5.2 B300 Dynamo-TRT AgentX MTP
RohitNagraj d12be31
chore(changelog): link GLM-5.2 B300 submission
RohitNagraj 39f082a
Merge branch 'main' into glm5.2-fp4-b300-dynamo-trt-agentic-mtp
RohitNagraj 82ad667
fix(b300): align GLM AgentX transport and result naming
RohitNagraj 7d95009
fix(b300): configure GLM AgentX UCX transports
RohitNagraj b27c990
feat(glm52-agentx): Add B300 Dynamo+TRT-LLM AgentX recipes
hshrivastava-droid 5996c81
Merge branch 'main' into glm5.2-fp4-b300-dynamo-trt-agentic-mtp
hshrivastava-droid 5e24052
fix(b300): restore CUDA UCX transports for GLM AgentX
RohitNagraj 7c4160b
Merge branch 'main' into glm5.2-fp4-b300-dynamo-trt-agentic-mtp
hshrivastava-droid File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
204 changes: 204 additions & 0 deletions
204
...lurm-recipes/trtllm/glm5.2/b300-fp4/agentic/dynamo-disagg-b300-1p1d-tep8-c20-b5-mtp5.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,204 @@ | ||
| name: dynamo-disagg-b300-1p1d-tep8-c20-b5-mtp5 | ||
|
|
||
| model: | ||
| path: nvidia/GLM-5.2-NVFP4 | ||
| container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22.post1 | ||
| precision: fp4 | ||
|
|
||
| identity: | ||
| model: | ||
| repo: "nvidia/GLM-5.2-NVFP4" | ||
| revision: "aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa" | ||
| container: | ||
| image: "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22.post1" | ||
| frameworks: | ||
| dynamo: "1.4.0.dev20260807" | ||
| tensorrt_llm: "1.3.0rc22.post1" | ||
|
|
||
| dynamo: | ||
| install: true | ||
| wheel: 1.4.0.dev20260807 | ||
| request_plane: tcp | ||
|
|
||
| slurm: | ||
| time_limit: "4:00:00" | ||
|
|
||
| health_check: | ||
| max_attempts: 270 | ||
| interval_seconds: 10 | ||
|
|
||
| resources: | ||
| gpu_type: b300 | ||
| gpus_per_node: 8 | ||
| spread_workers: true | ||
| prefill_nodes: 1 | ||
| prefill_workers: 1 | ||
| gpus_per_prefill: 4 | ||
| decode_nodes: 1 | ||
| decode_workers: 1 | ||
| gpus_per_decode: 8 | ||
|
|
||
| backend: | ||
| type: trtllm | ||
| prefill_environment: &server_environment | ||
| HF_HUB_OFFLINE: "1" | ||
| TRANSFORMERS_OFFLINE: "1" | ||
| TQDM_DISABLE: "1" | ||
| HF_HUB_DISABLE_PROGRESS_BARS: "1" | ||
| TLLM_LOG_LEVEL: INFO | ||
| TRTLLM_SERVER_DISABLE_GC: "1" | ||
| TRTLLM_WORKER_DISABLE_GC: "1" | ||
| TRTLLM_ENABLE_PDL: "1" | ||
| NCCL_GRAPH_MIXING_SUPPORT: "0" | ||
| MIMALLOC_PURGE_DELAY: "0" | ||
| PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True | ||
| TRTLLM_SERVE_ENABLE_MSGSPEC: "1" | ||
| TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD: "0.10" | ||
| TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC: "600" | ||
| UCX_TLS: rc,cuda_ipc,cuda_copy,sm,self,tcp | ||
| UCX_MAX_RNDV_RAILS: "2" | ||
| UCX_MAX_HCA_PER_GPU: inf | ||
| UCX_LOG_LEVEL: info | ||
| TRTLLM_CTX_LOCAL_HCA_PIN: "1" | ||
| UCX_RNDV_SCHEME: put_zcopy | ||
| TRTLLM_KVCACHE_SEND_BUFFER_COUNT: "1" | ||
| TRTLLM_KVCACHE_RECV_BUFFER_COUNT: "1" | ||
| TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: "2.61" | ||
| DYN_TRTLLM_ENABLE_ATTENTION_DP: "1" | ||
| DYN_ENGINE_CONV_AFFINITY: "1" | ||
| DYN_TOKENIZER: fastokens | ||
| DYN_PUBLISH_KV_EVENTS: "0" | ||
| decode_environment: *server_environment | ||
| trtllm_config: | ||
| prefill: | ||
| attention_dp_config: | ||
| enable_kv_cache_aware_routing: false | ||
| kv_cache_routing_conversation_affinity: true | ||
| kv_cache_routing_max_sessions: 65536 | ||
| cache_transceiver_config: | ||
| backend: NIXL | ||
| transceiver_runtime: PYTHON | ||
| kv_cache_bounce_size_mb: 5120 | ||
| max_tokens_in_buffer: 1048576 | ||
| kv_transfer_timeout_ms: 600000 | ||
| cuda_graph_config: null | ||
| disable_overlap_scheduler: true | ||
| enable_attention_dp: true | ||
| enable_chunked_prefill: true | ||
| trust_remote_code: true | ||
| kv_cache_config: | ||
| dtype: fp8 | ||
| enable_block_reuse: true | ||
| event_buffer_max_size: 0 | ||
| free_gpu_memory_fraction: 0.75 | ||
| host_cache_size: 412316860416 | ||
| tokens_per_block: 64 | ||
| max_batch_size: 256 | ||
| max_num_tokens: 8192 | ||
| max_seq_len: 1048576 | ||
| moe_config: | ||
| backend: CUTEDSL | ||
| moe_expert_parallel_size: 4 | ||
| num_postprocess_workers: 8 | ||
| pipeline_parallel_size: 1 | ||
| print_iter_log: true | ||
| return_perf_metrics: false | ||
| scheduler_config: | ||
| capacity_scheduler_policy: MAX_UTILIZATION | ||
| context_chunking_policy: EQUAL_PROGRESS | ||
| sparse_attention_config: | ||
| algorithm: dsa | ||
| enable_heuristic_topk: true | ||
| speculative_config: | ||
| decoding_type: MTP | ||
| num_nextn_predict_layers: 5 | ||
| tensor_parallel_size: 4 | ||
| decode: | ||
| cache_transceiver_config: | ||
| backend: NIXL | ||
| transceiver_runtime: PYTHON | ||
| kv_cache_bounce_size_mb: 5120 | ||
| max_tokens_in_buffer: 1048576 | ||
| kv_transfer_timeout_ms: 600000 | ||
| cuda_graph_config: | ||
| batch_sizes: [1, 2, 4, 5] | ||
| enable_padding: true | ||
| enable_attention_dp: false | ||
| trust_remote_code: true | ||
| kv_cache_config: | ||
| dtype: fp8 | ||
| enable_block_reuse: false | ||
| event_buffer_max_size: 0 | ||
| host_cache_size: 0 | ||
| free_gpu_memory_fraction: 0.8 | ||
| tokens_per_block: 64 | ||
| max_batch_size: 5 | ||
| max_num_tokens: 128 | ||
| max_seq_len: 1048576 | ||
| moe_config: | ||
| backend: TRTLLM | ||
| use_low_precision_moe_combine: true | ||
| moe_expert_parallel_size: 8 | ||
| num_postprocess_workers: 4 | ||
| pipeline_parallel_size: 1 | ||
| print_iter_log: true | ||
| return_perf_metrics: false | ||
| sparse_attention_config: | ||
| algorithm: dsa | ||
| enable_heuristic_topk: true | ||
| use_cute_dsl_paged_mqa_logits: true | ||
| speculative_config: | ||
| decoding_type: MTP | ||
| max_draft_len: 5 | ||
| stream_interval: 20 | ||
| tensor_parallel_size: 8 | ||
| publish_events_and_metrics: false | ||
|
|
||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: | ||
| ETCD_LEASE_TTL: '120' | ||
| DYN_ROUTER_QUEUE_THRESHOLD: None | ||
| DYN_ROUTER_TEMPERATURE: '0' | ||
| DYN_TOKENIZER_CACHE: '1' | ||
| DYN_TOKENIZER_CACHE_BYTES: '8000000000' | ||
| DYN_TOKENIZER: fastokens | ||
| DYN_TCP_REQUEST_TIMEOUT: '30' | ||
| DYN_LOG: warn | ||
| DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400' | ||
| args: | ||
| router-mode: kv | ||
| no-kv-events: true | ||
| active-decode-blocks-threshold: None | ||
| active-prefill-tokens-threshold: None | ||
| active-prefill-tokens-threshold-frac: None | ||
| orchestrator_placement: first_decode | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| client_placement: last_decode | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: /infmax-workspace | ||
| RESULT_DIR: /logs/agentic | ||
| PORT: '8000' | ||
| IS_MULTINODE: 'true' | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400' | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache | ||
| HF_HUB_CACHE: /hf_hub_cache | ||
| WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126 | ||
| AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES: "0" | ||
| SERVED_MODEL_NAME: GLM-5.2-NVFP4 | ||
| MAX_MODEL_LEN: '1048576' | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true' | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0' | ||
| OPENAI_API_KEY: EMPTY | ||
| KV_OFFLOADING: none | ||
| MODEL: nvidia/GLM-5.2-NVFP4 | ||
| MODEL_PREFIX: glm5.2 | ||
| FRAMEWORK: dynamo-trt | ||
| PRECISION: fp4 | ||
| CONC: '20' | ||
| DURATION: '3600' | ||
| RESULT_FILENAME: glm52_fp4_dynamo-trt_1p1d-tep8_c20 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 benchmarks/multi_node/agentic_srt.sh is the shared benchmark.type=custom client sourced by ~80 srt-slurm multi-node agentic recipes (dsv4, qwen3.5, minimax-m3, kimi-k3, glm5.2-sglang, etc.), each pinning a different srt-slurm branch/fork/commit. This PR makes SRT_FRONTEND_HOST/SRT_FRONTEND_PORT a hard check_env_vars requirement with no MODEL_PREFIX/FRAMEWORK guard, and switches AIPERF_SERVER_URL/wait_for_agentic_servers_idle to depend on them instead of the previous http://localhost:$PORT. These two vars appear nowhere else in the repo, so they must come from srt-slurm at runtime, and only the new glm5.2 dynamo-trt path (v1.0.36) was validated here — if any of the other pinned srt-slurm versions do not export them into the custom command env, every other agentic recipe fails immediately at check_env_vars on its next sweep, and this PR is not the one that would surface it since its own CI only exercises the new glm5.2 configs.
Extended reasoning...
What the bug is.
benchmarks/multi_node/agentic_srt.sh(lines 12-16) is the sharedbenchmark.type=customclient invoked by roughly 80 srt-slurm multi-node agentic recipe YAMLs across dsv4, qwen3.5, minimax-m3, kimi-k3, and glm5.2-sglang, not just the new glm5.2 dynamo-trt recipes added in this PR. Before this change the script required onlyMODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATIONand talked tohttp://localhost:$PORT, relying on the client being co-located with the frontend (the script's own header comment says it runs 'against the already-ready frontend on the head node'). This PR appendsSRT_FRONTEND_HOST SRT_FRONTEND_PORTtocheck_env_varsunconditionally and rewiresAIPERF_SERVER_URL/wait_for_agentic_servers_idle's metrics URL to depend on them, with noMODEL_PREFIX/FRAMEWORKguard around the new requirement.\n\nWhy it can regress unrelated recipes.SRT_FRONTEND_HOSTandSRT_FRONTEND_PORTdo not appear anywhere else in the repository — they are not exported by any launcher script or recipe YAML env block — so the only place they can come from is the srt-slurm tool injecting them into the custom benchmark command's environment at runtime. Different agentic recipes pin different srt-slurm branches/forks/commits (main,sa-submission-q2-2026,cam/sa-submission-q2-2026, various pinned SHAs, and nowv1.0.36for glm5.2 dynamo-trt). Only the new glm5.2 dynamo-trt path was dry-run validated in this PR's checklist; nothing establishes that the older/other srt-slurm versions also inject these two vars into the custom command environment.\n\nStep-by-step proof of the failure mode. (1) A pre-existing recipe, e.g. a dsv4 dynamo-sglang agentic YAML, still hasbenchmark.type: customwithcommand: bash .../agentic_srt.sh. (2) Its launcher pins an srt-slurm branch other than v1.0.36 (e.g.mainorsa-submission-q2-2026). (3) On the next scheduled sweep, srt-slurm starts the frontend/workers and then execs the custom benchmark command. (4)agentic_srt.shsourcesbenchmark_lib.shand immediately callscheck_env_vars ... SRT_FRONTEND_HOST SRT_FRONTEND_PORT. (5) If that srt-slurm version's custom-command environment does not set those two vars (plausible, since the pre-PR script never needed or used them and instead worked purely vialocalhost:$PORTco-location),check_env_varsprints the missing-vars error andexit 1fires before any benchmark work happens — the whole job fails instantly regardless of whether the server itself came up healthy.\n\nWhy nothing else in the PR catches this. The PR's own CI/validation (YAML/bash syntax checks, master/recipe parity, and the v1.0.36 dry-run) only exercises the seven new glm5.2 dynamo-trt configs added here. None of the ~80 other agentic recipes sharing this script are re-run as part of this PR's validation, so a break in them would only surface on their own regularly scheduled sweep, after merge — exactly the kind of latent regression code review is supposed to catch.\n\nTelling asymmetry with the sibling fix. The companion change inbenchmark_lib.sh'sbuild_replay_cmdis defensive:--url ${AIPERF_SERVER_URL:-http://localhost:$PORT}, i.e. it falls back to the old behavior ifAIPERF_SERVER_URLisn't set.agentic_srt.shdoes the opposite — it hard-fails viacheck_env_varsbefore that fallback could ever apply, even thoughAIPERF_SERVER_URLis itself derived from the two new required vars just above it.\n\nSuggested fix. Scope the new requirement to the glm5.2 dynamo-trt case (mirroring the$FRAMEWORK/$MODEL_PREFIXguards already used elsewhere inlaunch_b300-nv.shfor this same feature), or make the two vars optional with a fallback consistent with thebenchmark_lib.shchange:export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST:-localhost}:${SRT_FRONTEND_PORT:-$PORT}", and dropSRT_FRONTEND_HOST/SRT_FRONTEND_PORTfrom the unconditionalcheck_env_varslist.