Question about minimum CUDA version for vector/block scaling modes in cublasLtMatmul

[Shmilkky]

Hi,

I would like to ask about the minimum CUDA version required for using the following scaling modes in cublasLtMatmul:

• CUBLASLT_MATMUL_DESC_A_SCALE_POINTER set with
  CUBLASLT_MATMUL_MATRIX_SCALE_VEC128_32F
• CUBLASLT_MATMUL_DESC_B_SCALE_POINTER set with
  CUBLASLT_MATMUL_MATRIX_SCALE_BLK128x128_32F

When using these scale modes in cublasLtMatmul, what is the minimum supported CUDA version?

If there are also specific requirements on GPU architecture or cuBLASLt version, I would appreciate it if you could clarify that as well.

Thanks!

[mkukula-nvidia]

Hi @Shmilkky,
Both CUBLASLT_MATMUL_MATRIX_SCALE_VEC128_32F and CUBLASLT_MATMUL_MATRIX_SCALE_BLK128x128_32F were introduced in cuBLAS 12.9 / CUDA Toolkit 12.9.0 (May 2025) - that's your minimum required version.
GPU architecture requirement: Hopper (sm_90) only, i.e., H100 or H200.
One thing to note: cuBLAS 12.9.1 brought a targeted performance improvement for the 128×128 2D block scaling path on Hopper specifically, so 12.9.1 or later is recommended if performance matters to you.

[Shmilkky]

Hi @Shmilkky, Both CUBLASLT_MATMUL_MATRIX_SCALE_VEC128_32F and CUBLASLT_MATMUL_MATRIX_SCALE_BLK128x128_32F were introduced in cuBLAS 12.9 / CUDA Toolkit 12.9.0 (May 2025) - that's your minimum required version. GPU architecture requirement: Hopper (sm_90) only, i.e., H100 or H200. One thing to note: cuBLAS 12.9.1 brought a targeted performance improvement for the 128×128 2D block scaling path on Hopper specifically, so 12.9.1 or later is recommended if performance matters to you.

Thanks for the clarification.

So if I am using B200, does that mean I cannot use CUBLASLT_MATMUL_MATRIX_SCALE_VEC128_32F or CUBLASLT_MATMUL_MATRIX_SCALE_BLK128x128_32F?

If so, does B200 support CUBLASLT_MATMUL_MATRIX_SCALE_VEC16_UE4M3 and CUBLASLT_MATMUL_MATRIX_SCALE_VEC32_UE8M0 instead?

[mkukula-nvidia]

Yes, CUBLASLT_MATMUL_MATRIX_SCALE_VEC16_UE4M3 and CUBLASLT_MATMUL_MATRIX_SCALE_VEC32_UE8M0 are the correct and recommended scaling modes for Blackwell (sm_100).

These correspond to the native 1D block-scaled data types that Blackwell's Tensor Cores accelerate directly in hardware:

• VEC16_UE4M3: 16-element blocks with UE4M3 scale type, used with CUDA_R_4F_E2M1 (FP4) inputs
• VEC32_UE8M0: 32-element blocks with UE8M0 scale type, used with MX-FP8 (CUDA_R_8F_E4M3/CUDA_R_8F_E5M2) inputs

Both were introduced in CUDA 12.8.0. Unlike the Hopper modes (which are software-implemented using FP32 scale factors), these are natively supported by Blackwell hardware and will be significantly faster on Blackwell-series GPUs.

See the LtNvfp4Matmul and LtMxfp8Matmul samples for reference usage.

[Shmilkky]

Yes, CUBLASLT_MATMUL_MATRIX_SCALE_VEC16_UE4M3 and CUBLASLT_MATMUL_MATRIX_SCALE_VEC32_UE8M0 are the correct and recommended scaling modes for Blackwell (sm_100).

These correspond to the native 1D block-scaled data types that Blackwell's Tensor Cores accelerate directly in hardware:

• VEC16_UE4M3: 16-element blocks with UE4M3 scale type, used with CUDA_R_4F_E2M1 (FP4) inputs
• VEC32_UE8M0: 32-element blocks with UE8M0 scale type, used with MX-FP8 (CUDA_R_8F_E4M3/CUDA_R_8F_E5M2) inputs

Both were introduced in CUDA 12.8.0. Unlike the Hopper modes (which are software-implemented using FP32 scale factors), these are natively supported by Blackwell hardware and will be significantly faster on Blackwell-series GPUs.

See the LtNvfp4Matmul and LtMxfp8Matmul samples for reference usage.

Thank you very much for the detailed clarification. This is very helpful.