[#17126][fix] Align FP8 block-scales Python gates with C++ SM121 aliasing - #17166
Draft
DhineshPonnarasan wants to merge 3 commits into
Draft
[#17126][fix] Align FP8 block-scales Python gates with C++ SM121 aliasing#17166DhineshPonnarasan wants to merge 3 commits into
DhineshPonnarasan wants to merge 3 commits into
Conversation
…15503 Add a debug-only instrumentation gated on the TRTLLM_DIFF_TRACE environment variable. When enabled, the Attention.forward method saves the last-token slice of six priority tensors at layer 0 during context (prefill) forwards and compares each new save against the first context forward (Run 1). Saved tensors in priority order: A_qkv - self.qkv_proj output B_q, B_k, B_v - q,k,v immediately before the attention backend C_attn_output - attention backend output D_o_proj - o_proj output When the first differing tensor is found, later-priority tensors are skipped to minimize trace overhead. When TRTLLM_DIFF_TRACE is not set, the hooks early-exit with a single bool check and no GPU or disk I/O is performed. Signed-off-by: Dhinesh Ponnarasan <dhineshponnarasan@gmail.com>
4 tasks
DhineshPonnarasan
changed the title
DhineshPonnarasan
changed the title
DhineshPonnarasan
force-pushed
the
fix/sm121-fp8-block-scales-gate-alignment
branch
from
August 1, 2026 21:59
7c2b6e7 to
41a1e85
Compare
On GB10 (SM121), C++ getSMVersion() aliases 121 to 120 so the C++ kernel runs the SM120 code path, but Python get_sm_version() returns 121, causing Python to select the wrong branch. This produces a scale-layout mismatch (Scale dtype must be Int32. on dense/MLA paths). Change all Python FP8 block-scales gates from get_sm_version() == 120 to get_sm_version() in (120, 121) so Python and C++ agree on which code path to use. Fixes: NVIDIA#17126 Signed-off-by: Dhinesh Ponnarasan <dhineshponnarasan@gmail.com>
DhineshPonnarasan
force-pushed
the
fix/sm121-fp8-block-scales-gate-alignment
branch
from
August 1, 2026 22:04
41a1e85 to
7c9ab2c
Compare
Signed-off-by: Dhinesh Ponnarasan <160256912+DhineshPonnarasan@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
On SM121 (GB10/DGX Spark), the C++ getSMVersion() in cudaUtils.h aliases SM121→120, so the C++ kernel correctly runs the SM120 code path. However, Python get_sm_version() returns 121, causing Python to select the wrong branch. This produces a scale-layout mismatch error (Scale dtype must be Int32.) on dense/MLA FP8 block-scales paths.
Changes
Changed all Python FP8 block-scales gates from get_sm_version() == 120 to get_sm_version() in (120, 121) across 7 files:
No C++ changes are needed since the aliasing already exists in cpp/include/tensorrt_llm/common/cudaUtils.h.
Verification
Requires an SM121 machine to confirm the forward pass no longer raises Scale dtype must be Int32. and that get_sm_version() returns 121 while the SM120 code path is correctly selected.