Wrong encoder output for NeMo Parakeet (FastConformer) on RTX 5090 (sm_120a) — all precisions, all opt levels, TRT 10.14 and 11.1

[cribbinsb]

Description

The encoder of NVIDIA's own parakeet-tdt-0.6b-v3 (NeMo FastConformer, standard ONNX export)
produces numerically wrong output on an RTX 5090 (sm_120a) with every TensorRT configuration
we tried, while onnxruntime-CUDA on the same GPU matches the fp32 CPU reference to ~7e-7. The
corruption is deterministic (bit-identical across runs) and severe enough that downstream TDT
decoding emits an empty transcript. The same ONNX + same TensorRT version produced correct
engines on an RTX 3090 (sm_86) — only the target architecture changed.

Cosine similarity of the encoder output vs the fp32 onnxruntime reference (1.0 = correct;
values below computed on identical input, ref_mel_jw16k.npy, shapes (1,128,1201)):

build	cosine
TRT 10.14.1.48, bf16	0.826
TRT 10.14.1.48, fp32	0.351
TRT 10.14.1.48, fp32 strongly-typed	0.351
TRT 10.14.1.48, bf16, --builderOptimizationLevel=1	0.826
TRT 11.1.0.106 (pip), default	0.3518
TRT 11.1.0.106, opt level 0 / 1 / 2	0.335 / 0.351 / 0.352
onnxruntime CUDA EP, same GPU	1.0000 (7e-7 vs CPU fp32)

Additional signatures:

• 114 of 151 output frames diverge; 37 are correct — a partial-frame pattern suggestive of an
  attention-mask/window handling fault rather than accumulated numeric error.
• The 10.14 engines also report encoded_lengths = 149 instead of the correct 151
  (11.1 gets 151, output still wrong).
• Fresh timing caches / --dumpOptimizationProfile variations do not change the result;
  outputs are bit-identical run to run.
• Verified through three independent I/O paths (two independently written Python harnesses and
  trtexec --loadInputs/--exportOutput): identical results per engine, so measurement error
  is excluded.

Environment

• GPU: NVIDIA GeForce RTX 5090 (sm_120a)
• Driver: 595.71.05 (open kernel modules)
• CUDA: 13.0 (nvcc release 13.0)
• TensorRT: 10.14.1.48 (apt, +cuda13.0) and 11.1.0.106 (pip wheel) — both affected
• OS: Ubuntu 24.04 x86_64
• Model: parakeet-tdt-0.6b-v3 encoder, ONNX export via NeMo (fp32, external weights;
  input audio_signal (1,128,T) fp32 + length int64; outputs outputs (1,1024,T/8) fp32 +
  encoded_lengths int64)

Reproduction

Attached: ref_mel_jw16k.npy (reference input), ref_encout_ort.npy (fp32 ORT reference
output), build_and_compare_trt.py (builds the engine from the ONNX with the TRT Python API and
prints the cosine; TRT_OPT_LEVEL env selects builder optimization level), compare_ort.py
(the onnxruntime side). The encoder ONNX itself is ~2.4 GB and is reproducible with NeMo's
standard export of nvidia/parakeet-tdt-0.6b-v3; we can supply our exact file on request.

trtexec-only repro (no Python in the execution path):

trtexec --onnx=encoder-parakeet-tdt-0.6b-v3.onnx \
        --saveEngine=enc.plan --shapes=audio_signal:1x128x1201,length:1
trtexec --loadEngine=enc.plan --shapes=audio_signal:1x128x1201,length:1 \
        --loadInputs=audio_signal:audio_signal.bin,length:length_i64.bin \
        --exportOutput=out.json --iterations=1 --warmUp=0 --duration=0
# cosine(out.json vs ref_encout_ort.npy) => 0.35-0.83 depending on precision; expected ~1.0

Possibly related

• TRT 11.1 strongly-typed FP32 engine silently produces wrong results for a DETR-style detection model (correct in ONNXRuntime and TRT 10.13) #4813 — TRT 11.1 strongly-typed fp32 silently wrong vs ONNXRuntime on sm_120 (RTX 5070 Ti),
  deformable-attention (DETR-family) model; reported correct on TRT 10.13. Same
  silent-accuracy-on-Blackwell class; our case additionally reproduces on 10.14.
• TensorRT 10.16 Myelin crash on Blackwell (sm_120) — Broadcast stride mismatch in SoVITS decoder #4743 / TRT 10.15.1 Myelin failure on RTX PRO 6000 (sm_120) with Fast-FoundationStereo post_runner #4715 — sm_120 Myelin faults (crashes rather than silent corruption).

Expected behavior

Encoder output numerically equivalent to the onnxruntime fp32 reference (as it is when the same
ONNX is built for sm_86, and as onnxruntime-CUDA achieves on the same sm_120a GPU).

[cribbinsb]

Follow-up: we bisected this further and it resolves into two distinct bugs, both localized. Same setup as the original report (RTX 5090 / sm_120a, driver 595.71.05, repro'd on both TRT 10.14.1.48 and 11.1.0.106; ORT-CUDA on the same GPU used as reference, matches fp32 within 6.7e-7). The same ONNX deployed on Jetson Orin (sm_87) and previously on RTX 3090 (sm_86) is correct — this is sm_120a-specific.

Bug 1: encoded_lengths integer arithmetic is evaluated in reduced-precision float

The model computes output lengths from the length input via a small integer chain (Cast/Add/Sub/Div ×3, one per stride-2 subsampling stage). In a --bf16 build the engine returns wrong encoded_lengths at almost every input length (ORT ground truth in parens):

T (input)	correct	engine (bf16 build)
40	5	5 ✔
1201	151	149
4000	500	500 ✔
11999	1500	1496
12000 (opt shape)	1500	1504
12001	1501	1504
20000	2500	2496
24000 (max shape)	3000	3008

The wrong values match the length arithmetic being computed in bfloat16: bf16(12000)=12032 → chain → 1504; bf16(20000)=19968 → 2496; bf16(11999)=11968 → 1496; bf16(24000)=24064 → 3008. The two correct rows (40, 4000) are exactly the values that survive bf16 rounding through the chain. (1201's off-by-two suggests intermediates are also rounded, not just the input.)

fp32 builds hit the same bug class through TF32: default fp32 build returns wrong lengths; adding --noTF32 makes encoded_lengths exact at every T we tested. So shape/length integer arithmetic is being folded into reduced-precision float math. This presumably affects any model that computes masks/lengths in-graph and is silent — no warning at build time.

Bug 2: fusion-dependent miscompile of the masked-conv subsampling region (precision-independent)

With lengths fixed (--noTF32, full fp32), the encoder output is still wrong: cosine 0.316 vs reference. Bisection via intermediate-tensor taps (adding internal tensors as extra graph outputs; the final-output cosine was unchanged by tapping, 0.3158 in both, so tapping did not perturb the failure):

• First divergence is in pre_encode (the ConvSubsampling front-end): its output is at cosine 0.126 before a single conformer layer runs. All 24 conformer layers downstream are garbage-in-garbage-out.
• Op-level taps inside pre_encode: the first three Convs are exact (cosine 1.000000), then the tensor right after conv.3 (mask-multiply + ReLU) is at 0.375. The region between is NeMo's length-derived time-mask machinery (Range/Less/Equal/Where/Expand → Mul).
• The extracted 515-node pre_encode subgraph reproduces the wrong output exactly (0.126104, bit-identical cosine to full graph) — a small standalone repro.
• The bug is fusion-context dependent: the same subgraph with the mask-chain tensors additionally marked as outputs compiles correctly (mask values bit-exact, masked product cosine 1.0000006). Bypassing the (batch-1 no-op) mask Muls and dead-code-pruning to a plain 27-node Conv/ReLU/MatMul chain also compiles exactly (1.0000000). Meanwhile the 40-node length subgraph alone compiles correctly too — every piece is fine in isolation; the fault appears only in specific fusion groupings.
• The 24-layer conformer stack extracted as its own model (inputs = pre_encode output + length) builds exactly correct in fp32 (--noTF32: cosine 1.0000001) and correct-within-rounding in bf16 (0.991) — so the miscompile is confined to the masked-conv region.

Workarounds that do not help (each just produces a different wrong output, suggesting the bad codegen moves with fusion grouping): --builderOptimizationLevel 0–3 (0.335–0.352), --stronglyTyped (0.351), --noTF32 (0.316), --precisionConstraints=obey --layerPrecisions='/pre_encode/*:fp32' (0.256), --markDebug on the conv outputs (0.826, unchanged), dummy extra outputs on the mask tensors in the full graph (0.826), removing the mask multiplies from the full graph (0.826).

Working workaround (for anyone else hitting this): split at the pre_encode boundary into two engines — mask-free pre_encode (fp32) + conformer stack (bf16, taking pre_encode output and length as inputs). End-to-end cosine 0.982 vs fp32 reference at T=24000 (consistent with pure bf16 rounding), correct lengths, and performance parity with the single-engine build.

Happy to provide the extracted subgraph repros (515-node failing pre_encode, tap scripts, length-sweep script) in addition to the original bundle if useful.