Optimized RVV q1_0 dot

[pl752]

Continuation of #10 for risc-v V extension

Implemented two fixed vlen kernels loosely inspired by AVX2 implementation
VLA causes severe overhead and task only have two realistic VL combinations (in simple form)

Benchmarks were performed with:
OrangePI RV2 sbc (Ky X1 / spacemit k1) 8gb
Armbian Debian trixie rolling release at 6.18.26-current-spacemit kernel
Built with official Spacemit toolchain, but IME wasn't used.
Command: llama-bench -m Bonsai-1.7B.gguf -p 64 -n 16 -t 8 -r 3 -fa 1 -mmp 0
Perplexity for 5x512 chunks: Mean KLD 0.00027, PPL 21.09, Same top p 99,22%

Flow	pp 64 t/s	tg 16 t/s	Speedup
Scalar	1.19	0.94	1.0x / 1.0x
VL128*	6.08	4.65	5.1x / 4.9x
VL256	10.56	7.84	8.9x / 8.3x

• * forced VLEN 128 kernel with LMUL=2, for VLEN >= 256: LMUL=1

As always, I would appreciate your feedback

[khosravipasha]

Thanks, that's impressive speed on such device :)

Do people need a special setup to build and run this, or the llama.cpp build tools work?

Would be happy to merge it to our fork, don't have a similar device to test it myself though. Will review more closely later this week.

For some reason stopped getting email notifications from Github.

[Copilot]

Pull request overview

Adds a RISC-V RVV-specific implementation for the q1_0 × q8_0 dot product in the CPU backend, continuing the codebase’s architecture-specific quantized dot-product optimizations.

Changes:

• Added two fixed-width RVV kernels for ggml_vec_dot_q1_0_q8_0 targeting 128-bit and 256-bit vector configurations.
• Added RVV runtime dispatch in the RISC-V quantized dot-product path.
• Updated the RISC-V fallback aliasing so this path can call the true generic implementation when needed.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
ggml/src/ggml-cpu/arch/riscv/quants.c	Adds the new RVV q1_0×q8_0 kernels, helper tables, and runtime dispatch logic.
ggml/src/ggml-cpu/arch-fallback.h	Removes the RISC-V alias for the q1_0 generic dot product so the arch-specific implementation can fall back correctly.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pl752]

I don't actually know if llama.cpp accounts for Zvl64b, it seems it's for embedded or 32 bit cores

[khosravipasha]

yeah copilot might be confused.

Saw a similar PR in main llama.cpp ggml-org#22500
is that related?

[pl752]

@khosravipasha No this PR is independent from my, but yes it's about this dot prod op too, though implementation there is not very efficient due to it not using vla nor fixed vlen specializzed kernels and relying on LMUL==4, which forces hardware to do 4 op per intruction, also it uses slightly different logic for gathering and masking, I think I need to open my PR too and also try similar approaches.

[pl752]

Turned out I have slept on some free performance, by not noticing better intruction for mask loading:

vlen	run	old	vlm	delta
128	pp64	6.08 t/s	10.14 t/s	+66.77%
128	tg16	4.65 t/s	7.50 t/s	+61.29%
256	pp64	10.56 t/s	13.36 t/s	+26.52%
256	tg16	7.84 t/s	9.71 t/s	+23.85%

Perplexity is alright, will prepare PR for mainline

[pl752]

@khosravipasha The thingy got merged, but one question about vneg(qy) vs vsub(0,qy) remained, but now I am busy on x86 repack PR.

[pl752]

Question has resolved itself as there was a confusion about the last comment in the main PR