Skip to content

fix(dispatch): Make VNNI its own ISA level and lower AVX512 to skylake-avx512 - #375

Draft
ahuber21 wants to merge 1 commit into
dispatch/03-validate-surfacefrom
dispatch/05-vnni-level
Draft

fix(dispatch): Make VNNI its own ISA level and lower AVX512 to skylake-avx512#375
ahuber21 wants to merge 1 commit into
dispatch/03-validate-surfacefrom
dispatch/05-vnni-level

Conversation

@ahuber21

Copy link
Copy Markdown
Contributor

The AVX512 level's translation unit was compiled at -march=cascadelake, which
enables AVX512-VNNI. That level promises AVX-512 F/BW/DQ about the host and
nothing more, so every kernel in that object file was compiled with permission to
use instructions a Skylake-SP does not have. The VNNI kernels that existed
guarded themselves with a runtime check, but the guard only covered the calls
written by hand; the compiler was free to emit vpdpwssd anywhere in the
translation unit on its own initiative.

AVX_AVAILABILITY::AVX512_VNNI moves the check to the one place a level is
chosen -- the entry point -- and lets each translation unit be compiled at
exactly what its level promises. The int8/int8 and uint8/uint8 kernels move to
the new level; every other pair promotes to float before doing arithmetic, where
VNNI has nothing to offer, so those pairs have no kernel at this level. That is
what keeps a fourth level from costing a fourth of everything: 54 new
instantiations rather than 432.

Two consequences worth naming:

  • The pairs that move need an AVX512-level kernel to fall back to, and it has to
    live outside #if SVS_AVX512_VNNI. Inside, it would be absent from the AVX512
    translation unit -- now compiled where that macro is 0 -- and silently replaced
    by the generic template. This is why the two halves cannot land separately.
  • The entry points must not dispatch to a level with no kernel for the pair in
    hand, for the same reason. svs::distance::has_vnni_kernel answers that,
    generated from the same list the kernels are, and it is if constexpr, so it
    compiles away for the pairs that do not move.

The generated header now also defines SVS_ISA_LEVEL_<enumerator> per level, so
a surface that leaves a level out is visible to the code that dispatches on it.
Dropping the VNNI level degrades correctly -- those pairs stay on AVX512, and the
probe reports 864 kernels instead of 918. Dropping AVX2 or AVX512 is an #error
rather than a silent fall back to unvectorized code. ISA levels are not
configuration the way the extent list is: a level exists because kernels, a
translation unit and a CPUID check exist for it.

Verified on the default surface: 918 kernels declared, instantiated and reachable
with none instantiated by the consumer; all 456 vpdpwssd encodings in
vnni.cpp.o and zero in avx512.cpp.o and avx2.cpp.o, where before all 456 sat
in the AVX512 level's object file; the AVX2 object identical in symbol names and
sizes to before; the new L2Impl<128,int8,int8,AVX512> vectorized 16-wide float,
not scalar. [distance] passes with the same 134402115 assertions as before, and
ctest is unchanged. Also verified with SVS_NO_AVX512=YES (both translation
units compile generic, zero AVX-512 encodings, all 918 still linked) and against
the reduced surface (306 kernels).

Part 5 of 5 of the ISA dispatching v2 milestone.

@ahuber21 ahuber21 added this to the ISA dispatching v2 milestone Aug 24, 2026
@ahuber21 ahuber21 changed the title Make VNNI its own ISA level, and lower the AVX512 level to skylake-avx512 fix(dispatch): Make VNNI its own ISA level and lower AVX512 to skylake-avx512 Aug 24, 2026
@ahuber21
ahuber21 force-pushed the dispatch/04-link-probe branch from f2849e6 to bdd2303 Compare August 24, 2026 06:57
@ahuber21
ahuber21 force-pushed the dispatch/05-vnni-level branch 2 times, most recently from 03f7bdd to 9b93baa Compare August 24, 2026 09:44
@ahuber21
ahuber21 changed the base branch from dispatch/04-link-probe to dispatch/03-validate-surface August 24, 2026 09:44
@ahuber21

Copy link
Copy Markdown
Contributor Author

The dispatch checks from #372 pick the change up on their own, which is what they
were written for. dispatch_instructions_avx512 now judges avx512.cpp.o at
skylake-avx512 and so forbids VNNI there -- it fails on the old object file, and
passes on this one. The cascadelake row gains vnni as a requirement, because
a VNNI level whose object file contains no VNNI is 54 instantiations of dead
weight; vnni.cpp.o has all 456. Three mechanical follow-ons: the per-level
object libraries are named after the level rather than the -march, since two
levels no longer imply two budgets; dispatch_surface_execution breaks on
int8/int8 rather than float/float, as a float-promoting pair has no kernel at
the top level and would legitimately route one lower; and the VNNI predicate joins
the other two in tests/multi-arch/x86/host_levels.h.

…x512

The AVX512 level's translation unit was compiled at -march=cascadelake, which
enables AVX512-VNNI. That level promises AVX-512 F/BW/DQ about the host and
nothing more, so every kernel in that object file was compiled with permission
to use instructions a Skylake-SP does not have. The VNNI kernels that existed
guarded themselves with a runtime check, but the guard only covered the calls
that were written by hand; the compiler was free to emit vpdpwssd anywhere in
the TU on its own initiative.

Adding AVX_AVAILABILITY::AVX512_VNNI as a fourth level moves the check to the
one place a level is chosen -- the entry point -- and lets each TU be compiled
at exactly what its level promises. The int8/int8 and uint8/uint8 kernels move
to the new level; every other pair promotes to float before doing arithmetic,
where VNNI has nothing to offer, so those pairs have no kernel at this level.
That is what keeps a fourth level from costing a fourth of everything: 54 new
instantiations rather than 432.

Two consequences worth naming:

  - The pairs that move need an AVX512-level kernel to fall back to, and it has
    to live outside `#if SVS_AVX512_VNNI`. Inside, it would be absent from the
    AVX512 TU -- which is now compiled where that macro is 0 -- and silently
    replaced by the generic template. This is why the two halves of the change
    cannot land separately.

  - The entry points must not dispatch to a level that has no kernel for the
    pair in hand, for the same reason. `svs::distance::has_vnni_kernel` answers
    that, generated from the same list the kernels are, and it is `if constexpr`
    so it compiles away for the pairs that do not move.

The generated header now also defines SVS_ISA_LEVEL_<enumerator> per level, so
a surface that leaves a level out is visible to the code that dispatches on it.
Dropping the VNNI level degrades correctly -- those pairs stay on AVX512, and
the probe reports 864 kernels instead of 918. Dropping AVX2 or AVX512 is an
`#error` instead, because the entry points reach those two for every type pair.
ISA levels are not configuration the way the extent list is: a level exists
because kernels, a TU and a CPUID check exist for it.

The dispatch checks pick the change up on their own, which is what they were
written for. dispatch_instructions_avx512 now judges avx512.cpp.o at
skylake-avx512 and so forbids VNNI there, and it fails on the old object file;
the cascadelake row gains `vnni` as a requirement, because a VNNI level whose
object file has no VNNI in it is 54 instantiations of dead weight. Three
mechanical follow-ons: the per-level object libraries are named after the level
rather than the -march, since two levels now share neither; the execution check
breaks on int8/int8 rather than float/float, as a float-promoting pair has no
kernel at the top level and would route one lower; and the VNNI predicate joins
the other two in tests/multi-arch/x86/host_levels.h.

Verified on the default surface: 918 kernels declared, instantiated and
reachable with none instantiated by the consumer; all 456 vpdpwssd encodings in
vnni.cpp.o, zero in avx512.cpp.o and avx2.cpp.o, where before all 456 sat in
the AVX512 level's object file; the AVX2 object identical in symbol names and
sizes to before; the new L2Impl<128,int8,int8,AVX512> vectorized 16-wide float,
not scalar. `[distance]` passes with the same 134402115 assertions as before,
all eight dispatch tests pass, and ctest is otherwise unchanged. Also verified
with SVS_NO_AVX512=YES (both TUs compile generic, zero AVX-512 encodings, all
918 still linked) and with the reduced surface (306 kernels).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ahuber21
ahuber21 force-pushed the dispatch/03-validate-surface branch from 0a0a6e7 to 135d3cc Compare August 24, 2026 10:28
@ahuber21
ahuber21 force-pushed the dispatch/05-vnni-level branch from 9b93baa to ed3dbc9 Compare August 24, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant