Skip to content
Open
2 changes: 1 addition & 1 deletion benchmarks/benchmark_lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1996,7 +1996,7 @@ build_replay_cmd() {
# rolling TTFT/ITL/throughput block and emit it every 30 seconds.
export AIPERF_UI_REALTIME_METRICS_ENABLED=true
REPLAY_CMD="$AIPERF_CLI profile --scenario inferencex-agentx-mvp"
REPLAY_CMD+=" --url http://localhost:$PORT"
REPLAY_CMD+=" --url ${AIPERF_SERVER_URL:-http://localhost:$PORT}"
REPLAY_CMD+=" --endpoint /v1/chat/completions"
REPLAY_CMD+=" --endpoint-type chat"
REPLAY_CMD+=" --streaming"
Expand Down
9 changes: 7 additions & 2 deletions benchmarks/multi_node/agentic_srt.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,12 @@ set -x
INFMAX_CONTAINER_WORKSPACE="${INFMAX_CONTAINER_WORKSPACE:-/infmax-workspace}"
source "$INFMAX_CONTAINER_WORKSPACE/benchmarks/benchmark_lib.sh"

check_env_vars MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION
check_env_vars \
MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION \
SRT_FRONTEND_HOST SRT_FRONTEND_PORT

export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}"
Comment on lines +12 to +16

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 benchmarks/multi_node/agentic_srt.sh is the shared benchmark.type=custom client sourced by ~80 srt-slurm multi-node agentic recipes (dsv4, qwen3.5, minimax-m3, kimi-k3, glm5.2-sglang, etc.), each pinning a different srt-slurm branch/fork/commit. This PR makes SRT_FRONTEND_HOST/SRT_FRONTEND_PORT a hard check_env_vars requirement with no MODEL_PREFIX/FRAMEWORK guard, and switches AIPERF_SERVER_URL/wait_for_agentic_servers_idle to depend on them instead of the previous http://localhost:$PORT. These two vars appear nowhere else in the repo, so they must come from srt-slurm at runtime, and only the new glm5.2 dynamo-trt path (v1.0.36) was validated here — if any of the other pinned srt-slurm versions do not export them into the custom command env, every other agentic recipe fails immediately at check_env_vars on its next sweep, and this PR is not the one that would surface it since its own CI only exercises the new glm5.2 configs.

Extended reasoning...

What the bug is. benchmarks/multi_node/agentic_srt.sh (lines 12-16) is the shared benchmark.type=custom client invoked by roughly 80 srt-slurm multi-node agentic recipe YAMLs across dsv4, qwen3.5, minimax-m3, kimi-k3, and glm5.2-sglang, not just the new glm5.2 dynamo-trt recipes added in this PR. Before this change the script required only MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION and talked to http://localhost:$PORT, relying on the client being co-located with the frontend (the script's own header comment says it runs 'against the already-ready frontend on the head node'). This PR appends SRT_FRONTEND_HOST SRT_FRONTEND_PORT to check_env_vars unconditionally and rewires AIPERF_SERVER_URL / wait_for_agentic_servers_idle's metrics URL to depend on them, with no MODEL_PREFIX/FRAMEWORK guard around the new requirement.\n\nWhy it can regress unrelated recipes. SRT_FRONTEND_HOST and SRT_FRONTEND_PORT do not appear anywhere else in the repository — they are not exported by any launcher script or recipe YAML env block — so the only place they can come from is the srt-slurm tool injecting them into the custom benchmark command's environment at runtime. Different agentic recipes pin different srt-slurm branches/forks/commits (main, sa-submission-q2-2026, cam/sa-submission-q2-2026, various pinned SHAs, and now v1.0.36 for glm5.2 dynamo-trt). Only the new glm5.2 dynamo-trt path was dry-run validated in this PR's checklist; nothing establishes that the older/other srt-slurm versions also inject these two vars into the custom command environment.\n\nStep-by-step proof of the failure mode. (1) A pre-existing recipe, e.g. a dsv4 dynamo-sglang agentic YAML, still has benchmark.type: custom with command: bash .../agentic_srt.sh. (2) Its launcher pins an srt-slurm branch other than v1.0.36 (e.g. main or sa-submission-q2-2026). (3) On the next scheduled sweep, srt-slurm starts the frontend/workers and then execs the custom benchmark command. (4) agentic_srt.sh sources benchmark_lib.sh and immediately calls check_env_vars ... SRT_FRONTEND_HOST SRT_FRONTEND_PORT. (5) If that srt-slurm version's custom-command environment does not set those two vars (plausible, since the pre-PR script never needed or used them and instead worked purely via localhost:$PORT co-location), check_env_vars prints the missing-vars error and exit 1 fires before any benchmark work happens — the whole job fails instantly regardless of whether the server itself came up healthy.\n\nWhy nothing else in the PR catches this. The PR's own CI/validation (YAML/bash syntax checks, master/recipe parity, and the v1.0.36 dry-run) only exercises the seven new glm5.2 dynamo-trt configs added here. None of the ~80 other agentic recipes sharing this script are re-run as part of this PR's validation, so a break in them would only surface on their own regularly scheduled sweep, after merge — exactly the kind of latent regression code review is supposed to catch.\n\nTelling asymmetry with the sibling fix. The companion change in benchmark_lib.sh's build_replay_cmd is defensive: --url ${AIPERF_SERVER_URL:-http://localhost:$PORT}, i.e. it falls back to the old behavior if AIPERF_SERVER_URL isn't set. agentic_srt.sh does the opposite — it hard-fails via check_env_vars before that fallback could ever apply, even though AIPERF_SERVER_URL is itself derived from the two new required vars just above it.\n\nSuggested fix. Scope the new requirement to the glm5.2 dynamo-trt case (mirroring the $FRAMEWORK/$MODEL_PREFIX guards already used elsewhere in launch_b300-nv.sh for this same feature), or make the two vars optional with a fallback consistent with the benchmark_lib.sh change: export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST:-localhost}:${SRT_FRONTEND_PORT:-$PORT}", and drop SRT_FRONTEND_HOST/SRT_FRONTEND_PORT from the unconditional check_env_vars list.

echo "Using srt-slurm frontend endpoint: $AIPERF_SERVER_URL"

BASE_RESULT_DIR="${RESULT_DIR:-/logs/agentic}"
BASE_RESULT_FILENAME="$RESULT_FILENAME"
Expand All @@ -32,7 +37,7 @@ install_agentic_deps
wait_for_agentic_servers_idle() {
local timeout_seconds="${AIPERF_DRAIN_TIMEOUT_SECONDS:-1800}"
local poll_seconds="${AIPERF_DRAIN_POLL_SECONDS:-10}"
local frontend_metrics_url="http://localhost:${PORT}/metrics"
local frontend_metrics_url="${AIPERF_SERVER_URL%/}/metrics"

"$AIPERF_PYTHON" - \
"$timeout_seconds" \
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
name: dynamo-disagg-b300-1p1d-tep8-c20-b5-mtp5

model:
path: nvidia/GLM-5.2-NVFP4
container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22.post1
precision: fp4

identity:
model:
repo: "nvidia/GLM-5.2-NVFP4"
revision: "aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa"
container:
image: "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22.post1"
frameworks:
dynamo: "1.4.0.dev20260807"
tensorrt_llm: "1.3.0rc22.post1"

dynamo:
install: true
wheel: 1.4.0.dev20260807
request_plane: tcp

slurm:
time_limit: "4:00:00"

health_check:
max_attempts: 270
interval_seconds: 10

resources:
gpu_type: b300
gpus_per_node: 8
spread_workers: true
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 4
decode_nodes: 1
decode_workers: 1
gpus_per_decode: 8

backend:
type: trtllm
prefill_environment: &server_environment
HF_HUB_OFFLINE: "1"
TRANSFORMERS_OFFLINE: "1"
TQDM_DISABLE: "1"
HF_HUB_DISABLE_PROGRESS_BARS: "1"
TLLM_LOG_LEVEL: INFO
TRTLLM_SERVER_DISABLE_GC: "1"
TRTLLM_WORKER_DISABLE_GC: "1"
TRTLLM_ENABLE_PDL: "1"
NCCL_GRAPH_MIXING_SUPPORT: "0"
MIMALLOC_PURGE_DELAY: "0"
PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
TRTLLM_SERVE_ENABLE_MSGSPEC: "1"
TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD: "0.10"
TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC: "600"
UCX_TLS: rc,cuda_ipc,cuda_copy,sm,self,tcp
UCX_MAX_RNDV_RAILS: "2"
UCX_MAX_HCA_PER_GPU: inf
UCX_LOG_LEVEL: info
TRTLLM_CTX_LOCAL_HCA_PIN: "1"
UCX_RNDV_SCHEME: put_zcopy
TRTLLM_KVCACHE_SEND_BUFFER_COUNT: "1"
TRTLLM_KVCACHE_RECV_BUFFER_COUNT: "1"
TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: "2.61"
DYN_TRTLLM_ENABLE_ATTENTION_DP: "1"
DYN_ENGINE_CONV_AFFINITY: "1"
DYN_TOKENIZER: fastokens
DYN_PUBLISH_KV_EVENTS: "0"
decode_environment: *server_environment
trtllm_config:
prefill:
attention_dp_config:
enable_kv_cache_aware_routing: false
kv_cache_routing_conversation_affinity: true
kv_cache_routing_max_sessions: 65536
cache_transceiver_config:
backend: NIXL
transceiver_runtime: PYTHON
kv_cache_bounce_size_mb: 5120
max_tokens_in_buffer: 1048576
kv_transfer_timeout_ms: 600000
cuda_graph_config: null
disable_overlap_scheduler: true
enable_attention_dp: true
enable_chunked_prefill: true
trust_remote_code: true
kv_cache_config:
dtype: fp8
enable_block_reuse: true
event_buffer_max_size: 0
free_gpu_memory_fraction: 0.75
host_cache_size: 412316860416
tokens_per_block: 64
max_batch_size: 256
max_num_tokens: 8192
max_seq_len: 1048576
moe_config:
backend: CUTEDSL
moe_expert_parallel_size: 4
num_postprocess_workers: 8
pipeline_parallel_size: 1
print_iter_log: true
return_perf_metrics: false
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
context_chunking_policy: EQUAL_PROGRESS
sparse_attention_config:
algorithm: dsa
enable_heuristic_topk: true
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 5
tensor_parallel_size: 4
decode:
cache_transceiver_config:
backend: NIXL
transceiver_runtime: PYTHON
kv_cache_bounce_size_mb: 5120
max_tokens_in_buffer: 1048576
kv_transfer_timeout_ms: 600000
cuda_graph_config:
batch_sizes: [1, 2, 4, 5]
enable_padding: true
enable_attention_dp: false
trust_remote_code: true
kv_cache_config:
dtype: fp8
enable_block_reuse: false
event_buffer_max_size: 0
host_cache_size: 0
free_gpu_memory_fraction: 0.8
tokens_per_block: 64
max_batch_size: 5
max_num_tokens: 128
max_seq_len: 1048576
moe_config:
backend: TRTLLM
use_low_precision_moe_combine: true
moe_expert_parallel_size: 8
num_postprocess_workers: 4
pipeline_parallel_size: 1
print_iter_log: true
return_perf_metrics: false
sparse_attention_config:
algorithm: dsa
enable_heuristic_topk: true
use_cute_dsl_paged_mqa_logits: true
speculative_config:
decoding_type: MTP
max_draft_len: 5
stream_interval: 20
tensor_parallel_size: 8
publish_events_and_metrics: false

frontend:
type: dynamo
enable_multiple_frontends: false
env:
ETCD_LEASE_TTL: '120'
DYN_ROUTER_QUEUE_THRESHOLD: None
DYN_ROUTER_TEMPERATURE: '0'
DYN_TOKENIZER_CACHE: '1'
DYN_TOKENIZER_CACHE_BYTES: '8000000000'
DYN_TOKENIZER: fastokens
DYN_TCP_REQUEST_TIMEOUT: '30'
DYN_LOG: warn
DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400'
args:
router-mode: kv
no-kv-events: true
active-decode-blocks-threshold: None
active-prefill-tokens-threshold: None
active-prefill-tokens-threshold-frac: None
orchestrator_placement: first_decode

benchmark:
type: custom
client_placement: last_decode
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
RESULT_DIR: /logs/agentic
PORT: '8000'
IS_MULTINODE: 'true'
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400'
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
HF_HUB_CACHE: /hf_hub_cache
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES: "0"
SERVED_MODEL_NAME: GLM-5.2-NVFP4
MAX_MODEL_LEN: '1048576'
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true'
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0'
OPENAI_API_KEY: EMPTY
KV_OFFLOADING: none
MODEL: nvidia/GLM-5.2-NVFP4
MODEL_PREFIX: glm5.2
FRAMEWORK: dynamo-trt
PRECISION: fp4
CONC: '20'
DURATION: '3600'
RESULT_FILENAME: glm52_fp4_dynamo-trt_1p1d-tep8_c20
Loading
Loading