[https://nvbugs/6525059][fix] Drop 8 from FP8BlockScaleMoeRunner::mSupportedTileN, covering both the… - #17135
Open
trtllm-agent wants to merge 1 commit into
Open
[https://nvbugs/6525059][fix] Drop 8 from FP8BlockScaleMoeRunner::mSupportedTileN, covering both the…#17135trtllm-agent wants to merge 1 commit into
8 from FP8BlockScaleMoeRunner::mSupportedTileN, covering both the…#17135trtllm-agent wants to merge 1 commit into
Conversation
… MoE
The TRTLLM-Gen FP8 block-scale MoE picks tile_tokens_dim=8 whenever
num_tokens*top_k/local_num_experts <= 8, i.e. for warmup and tiny decode
batches. On sm100f/sm103 every DeepSeek-FP8 GEMM2 cubin at that tile
(bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128{,u2}_..._dsFp8_{schedS,schPd4x2x2x3}_bN_...)
faults with cudaErrorIllegalAddress in its TMA-OOB epilogue store, so the
warmup forward of Qwen3-30B-A3B-FP8 dies. Both the Static and the
Persistent variant fault, so the tile size is the discriminator rather
than a single bad cubin.
Remove 8 from the runner's supported tile list. This covers the autotuner
tactic list and the tileN == -1 fallback (which clamps on front()) in one
place, and avoids ever constructing the tileN=8 runner -- filtering the
cubins in TrtllmGenBatchedGemmRunner::skipQuirks instead would leave that
runner with no passing config and throw during construction. FP8
block-scale MoE stays enabled; small batches now run at tileN=16.
Unwaive the GB200/GB300 TestQwen3_30B_A3B::test_dummy_load_format entries.
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
trtllm-agent
requested review from
QiJune,
crazydemo,
mlefeb01,
mzweilz,
schetlur-nv,
sunnyqgg and
xxi-nv
July 31, 2026 19:01
Contributor
WalkthroughThe FP8 MoE runner no longer supports tileN=8. Its supported tile sizes are 16, 32, 64, and 128. Two GB200 and GB300 Qwen3 integration-test waivers are removed. ChangesFP8 MoE support
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp (1)
367-373: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winName the supported tile sizes.
Line 373 uses non-exempt numeric literals directly in
mSupportedTileN. Replace them withk-prefixedconstexprconstants.Proposed change
- : mSupportedTileN{16, 32, 64, 128} + : mSupportedTileN{kSupportedTileN16, kSupportedTileN32, kSupportedTileN64, kSupportedTileN128} private: + static constexpr int32_t kSupportedTileN16{16}; + static constexpr int32_t kSupportedTileN32{32}; + static constexpr int32_t kSupportedTileN64{64}; + static constexpr int32_t kSupportedTileN128{128};As per coding guidelines, avoid magic literals except
0,nullptr,true, andfalse, and usek-prefixed camelCase constants.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp` around lines 367 - 373, Define k-prefixed constexpr constants for the supported tile sizes near the relevant configuration declarations, then initialize mSupportedTileN with those named constants instead of the numeric literals 16, 32, 64, and 128. Preserve the existing supported-size set and ordering.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp`:
- Around line 367-373: Validate cached and requested tile values against
mSupportedTileN before runner lookup in the relevant run() path, so unsupported
tileN=8 configurations are rejected or invalidated instead of reaching
mRunners.at(). Preserve valid cached configurations and the existing supported
tile set.
---
Nitpick comments:
In `@cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp`:
- Around line 367-373: Define k-prefixed constexpr constants for the supported
tile sizes near the relevant configuration declarations, then initialize
mSupportedTileN with those named constants instead of the numeric literals 16,
32, 64, and 128. Preserve the existing supported-size set and ordering.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3cdeff4c-99e0-47d4-91f8-d7821162500c
📒 Files selected for processing (2)
cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpptests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
- tests/integration/test_lists/waives.txt
Comment on lines
+367
to
+373
| // tileN=8 is excluded: every DeepSeek-FP8 GEMM2 cubin at that tile faults with | ||
| // cudaErrorIllegalAddress in its epilogue store on sm100f/sm103, so the tile size is | ||
| // broken rather than a single cubin (https://nvbugs/6525059). Dropping it here rather | ||
| // than in TrtllmGenBatchedGemmRunner::skipQuirks both raises the fallback clamp floor | ||
| // below and avoids constructing a tileN=8 runner, which would throw on an empty | ||
| // passing-config list. | ||
| : mSupportedTileN{16, 32, 64, 128} |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 8 -i \
'tile_config_pair|mRunners\.at|tileN\s*==\s*-1|getValidConfigs|autotuner|tactic.*cache|cache.*tactic' .Repository: NVIDIA/TensorRT-LLM
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
target="cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp"
printf '%s\n' '--- target symbols ---'
rg -n -C 12 'mSupportedTileN|mRunners|tileN|run\(|getValidConfigs|tile_config_pair' "$target"
printf '%s\n' '--- related definitions and call sites ---'
rg -n -C 6 'tile_config_pair|mRunners\.at|AutoTuner|autotune|Tactic' cpp/tensorrt_llm --glob '*.{cpp,cc,cxx,h,hpp}' | head -n 1200Repository: NVIDIA/TensorRT-LLM
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- autotuner files ---'
fd -i 'autotuner' . | head -n 100
printf '%s\n' '--- exact cache and replay symbols ---'
rg -n -C 10 \
'class AutoTuner|def (autotune|replay|capture|clear_cache)|cache_path|torch_workflow|tile_config_pair|run_moe|get_valid_configs' \
tensorrt_llm tests cpp --glob '*.{py,cpp,cc,cxx,h,hpp}' \
| head -n 1800Repository: NVIDIA/TensorRT-LLM
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="tensorrt_llm/_torch/autotuner.py"
printf '%s\n' '--- cache implementation ---'
sed -n '420,760p' "$file"
printf '%s\n' '--- autotuner operation selection ---'
rg -n -C 14 \
'get_valid_configs|profiling_cache|cache.*get|cache.*miss|tactic|replay|run_with|selected' \
"$file" | head -n 2200Repository: NVIDIA/TensorRT-LLM
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- FP8BlockScaleMoERunner references ---'
rg -n -C 16 \
'FP8BlockScaleMoERunner|fp8_block_scale|run_moe|tile_config_pair|get_valid_configs' \
tensorrt_llm tests cpp --glob '*.{py,cpp,cc,cxx,h,hpp}' \
| head -n 2400
printf '%s\n' '--- choose_one continuation ---'
sed -n '1039,1285p' tensorrt_llm/_torch/autotuner.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- exact get_valid_configs callers ---'
rg -l 'get_valid_configs' tensorrt_llm --glob '*.py' | while read -r f; do
echo "### $f"
rg -n -C 12 'get_valid_configs' "$f"
done
printf '%s\n' '--- exact custom class call sites ---'
rg -n -C 12 \
'FP8BlockScaleMoERunner|run_moe\(|tile_config_pair' \
tensorrt_llm/_torch --glob '*.py' | head -n 1800Repository: NVIDIA/TensorRT-LLM
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import ast
import re
cpp = Path("cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp").read_text()
py = Path("tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py").read_text()
autotuner = Path("tensorrt_llm/_torch/autotuner.py").read_text()
assert "mSupportedTileN{16, 32, 64, 128}" in cpp
assert "mRunners.at(tileN)" in cpp
unique_id = re.search(
r"class FP8BlockScaleMoERunner\(TunableRunner\):.*?"
r"def unique_id\(self\):\s*return \((.*?)\)",
py,
re.S,
)
assert unique_id, "FP8BlockScaleMoERunner.unique_id() not found"
unique_fields = ast.literal_eval("(" + unique_id.group(1).strip() + ",)")
assert unique_fields == (
"self.top_k",
"self.intermediate_size",
"self.local_num_experts",
"self.act_type",
)
assert "cached_runner_id, tactic, min_time = self.cache[cache_key]" in autotuner
assert "return True, idx, tactic, min_time" in autotuner
assert "kernel_runner.run_moe(" in py
# Model the relevant cache behavior: the current runner matches the old key,
# and the stored tactic is returned unchanged.
current_runner_key = ("FP8BlockScaleMoERunner", "(2, Intermediate, 8, 0)", "profile")
persisted_key = current_runner_key
persisted_value = (0, [8, 17], 1.0)
cache = {persisted_key: persisted_value}
current_runners = [current_runner_key]
hits = [
(idx, cache[key][1])
for idx, key in enumerate(current_runners)
if key in cache
]
assert hits == [(0, [8, 17])]
print("stale tactic reaches the current runner unchanged: [8, 17]")
print("unique_id omits the supported tile set")
print("run() performs an unchecked mRunners.at(tileN) lookup")
PYRepository: NVIDIA/TensorRT-LLM
Length of output: 936
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
cpp = Path("cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp").read_text()
py = Path("tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py").read_text()
autotuner = Path("tensorrt_llm/_torch/autotuner.py").read_text()
assert "mSupportedTileN{16, 32, 64, 128}" in cpp
assert "mRunners.at(tileN)" in cpp
match = re.search(
r"class FP8BlockScaleMoERunner\(TunableRunner\):.*?"
r"def unique_id\(self\):\s*return \((.*?)\)",
py,
re.S,
)
assert match, "FP8BlockScaleMoERunner.unique_id() not found"
assert match.group(1).replace(" ", "").replace("\n", "") == (
"self.top_k,self.intermediate_size,self.local_num_experts,self.act_type"
)
assert "cached_runner_id, tactic, min_time = self.cache[cache_key]" in autotuner
assert "return True, idx, tactic, min_time" in autotuner
# Relevant cache model: the current runner has the same cache identity as the
# old runner, so the old tactic is returned without tile validation.
cache = {
("FP8BlockScaleMoERunner", "(2, Intermediate, 8, 0)", "profile"):
(0, [8, 17], 1.0)
}
current_runner_keys = [
("FP8BlockScaleMoERunner", "(2, Intermediate, 8, 0)", "profile")
]
hits = [
cache[key][1] for key in current_runner_keys if key in cache
]
assert hits == [[8, 17]]
print("stale tactic reaches the current runner unchanged: [8, 17]")
print("unique_id omits the supported tile set")
print("run() performs an unchecked mRunners.at(tileN) lookup")
PYRepository: NVIDIA/TensorRT-LLM
Length of output: 309
Reject unsupported tile values before runner lookup.
unique_id() omits the supported tile set, so persisted caches can replay [8, config]. run() then calls mRunners.at(8) and throws std::out_of_range. Reject unsupported tiles or invalidate affected cache entries.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp` around lines 367 - 373, Validate
cached and requested tile values against mSupportedTileN before runner lookup in
the relevant run() path, so unsupported tileN=8 configurations are rejected or
invalidated instead of reaching mRunners.at(). Preserve valid cached
configurations and the existing supported tile set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tile_tokens_dim=8faults withcudaErrorIllegalAddressin its TMA-OOB epilogue store, and tileN=8 is selected for warmup/tiny batches.8fromFP8BlockScaleMoeRunner::mSupportedTileN, covering both the autotuner tactic list and thetileN == -1fallback without constructing a config-less tileN=8 runner; unwaive the GB200/GB300 entries.Test plan
Links
Dev Engineer Review
tileN=8fromFP8BlockScaleMoeRunner::mSupportedTileN.tileN=16.tileN == -1fallback.tileN=8runner.QA Engineer Review
SKIPentries forTestQwen3_30B_A3B::test_dummy_load_format.Verdict: needs follow-up