From 408454546ee1589ae6cefcd0cf42bc6889da9a4a Mon Sep 17 00:00:00 2001 From: Chris Rackauckas Date: Sat, 8 Aug 2026 05:11:57 -0400 Subject: [PATCH] Inline the backward panel kernels into the driver The last measurable backward-vs-forward difference was a fixed ~5 ns (~20 cycle) cost per kernel call: the compiled bodies contain identical vector work (every FP opcode count matches) but the backward one carries +15 integer bookkeeping instructions and two more callee-saved registers, of which llvm-mca models only about half the observed cost; source-level variants (sinking the pre-offset pointers under the loop guard, absolute-index loops, store-order changes) did not move it. Inlining uldiv_solve_W_u! and uldiv_solve_W! into _ldiv_U! removes the call boundary altogether and lets LLVM hoist stride-invariant setup out of the driver loop. The R-remainder and small kernels keep their calls (they sit behind @nif dispatch). Left-upper vs left-lower ldiv! at equal UNIT (min times, EPYC 7502): n=36 1.09x, n=48 1.06x, n=128 1.02x, n>=256 1.00-1.01x. Co-Authored-By: Chris Rackauckas --- src/TriangularSolve.jl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/TriangularSolve.jl b/src/TriangularSolve.jl index 153f940..f0198e9 100644 --- a/src/TriangularSolve.jl +++ b/src/TriangularSolve.jl @@ -1819,6 +1819,7 @@ end ) where {W,U,UNIT} z = static(0) q = quote + $(Expr(:meta, :inline)) # C = A / L in the transposed (row-major) layout; equivalently, C' = L' \ A' # with L' upper triangular — backward substitution for the left-upper solve. # Solves the block C_{j+[0,W), n+[0,W*U)}; columns [n+W*U, nend) are already @@ -1936,6 +1937,7 @@ end ) where {W,UNIT} z = static(0) quote + $(Expr(:meta, :inline)) # Like `uldiv_solve_W_u!`, except no unrolling, just a `W`x`W` block A11 = getfield(vload(spa, $(Unroll{1,1,W,2,W,zero(UInt),1})(($z, n))), :data)