Check freed debug block fill patterns 32 bytes at a time

[janrysavy]

Summary

This changes CheckFreedDebugBlockFillPatternIntact to validate 32 bytes per main-loop iteration instead of one 8-byte word. At 4 KiB, throughput improved by 40.18% on AMD Win64 and 14.45% on AMD Win32; Intel improved by 38.31% on Win64 but only 3.30% on Win32. The change adds four UInt64 comparisons before the existing 8-byte remainder loop, while preserving the freed-object prefix check, 4/2/1-byte remainder handling, corruption result, and logging path. The diff adds 16 lines and changes one comment.

This is complementary to #92: that PR can disable fill-pattern diagnostics for selected configurations, while this change speeds up the unchanged diagnostic path when freed-block validation remains enabled.

Tracking issue: #99

Benchmark

The benchmark repeatedly allocates and frees one block in FastMM5 debug mode with stack-trace collection disabled. Results are medians of 25 alternating baseline/candidate process pairs after three warmups. Throughput gain is (baseline time / candidate time - 1) * 100; the 95% interval is a deterministic 10,000-resample bootstrap interval for the paired median.

AMD Ryzen 9 7950X:

Target	User size	Operations	Baseline median	Candidate median	Throughput gain (95% CI)
Win64	64 B	4,000,000	190.79 ms	178.72 ms	6.58% (6.12% to 8.03%)
Win32	64 B	4,000,000	173.38 ms	171.36 ms	1.28% (0.24% to 1.69%)
Win64	4 KiB	750,000	336.83 ms	240.14 ms	40.18% (40.11% to 40.43%)
Win32	4 KiB	750,000	358.24 ms	312.89 ms	14.45% (14.34% to 14.66%)
Win64	64 KiB	50,000	310.49 ms	217.46 ms	42.91% (42.68% to 43.23%)
Win32	64 KiB	50,000	331.14 ms	284.16 ms	16.34% (14.86% to 17.69%)

Intel Core i7-8750H:

Target	User size	Operations	Baseline median	Candidate median	Throughput gain (95% CI)
Win64	64 B	2,100,000	203.97 ms	192.86 ms	5.15% (3.94% to 7.62%)
Win32	64 B	2,000,000	188.55 ms	179.00 ms	4.18% (3.00% to 5.04%)
Win64	4 KiB	268,000	311.63 ms	224.39 ms	38.31% (36.36% to 39.89%)
Win32	4 KiB	286,000	364.46 ms	353.56 ms	3.30% (2.68% to 3.45%)
Win64	64 KiB	17,000	291.12 ms	203.33 ms	43.69% (42.19% to 44.60%)
Win32	64 KiB	18,000	340.18 ms	325.99 ms	4.29% (3.85% to 4.82%)

The 4 KiB and 64 KiB gains reproduced on AMD Win32 and Win64 and on Intel Win64, but Intel Win32 improved by only 3.30% to 4.29%, so the result depends strongly on architecture and CPU. Additional 25-pair tests cover 15, 39, 40, and 41 bytes. The 39-byte case exercises the new failed 32-byte loop guard before the existing 8-byte path; the 40/41-byte cases enter the new loop.

The worst AMD result was -2.12% at 39 bytes on Win32, with all Win64 controls positive. The worst Intel result was -1.58% at 15 bytes on Win32 (CI -2.93% to -0.33%), again with all Win64 controls positive. Avoiding those boundary losses would require an additional fallback path, which does not seem justified for debug mode.

Tests ran on an AMD Ryzen 9 7950X and an Intel Core i7-8750H under Windows 11 build 26200, using Delphi Win32/Win64 37.0.59082.6021 with release-style -O+ builds. The Intel runs use fewer operations to keep process duration similar; both builds still perform identical work within each pair, and the pairing and analysis are unchanged.

Correctness

FastMMDebugPatternTest.dpr passes for the DCC 37 baseline and candidate on Win32 and Win64 on both CPUs. Additional compatibility runs pass with DCC 35 and 36 Win32/Win64 and with DCC 37 PurePascal. The test verifies every allocated fill byte for sizes 1 through 1025, then corrupts and restores every byte for freed-block sizes 1 through 65 and 257. It accepts only the expected EInvalidPointer and requires a clean scan afterward, covering the object marker and every 32/8/4/2/1-byte validation remainder.

Full benchmark source, raw AMD and Intel samples, summaries, and correctness tests: https://github.com/janrysavy/FastMM5/tree/ed3229e678bbe4b2455b8435168442d51f430996/perf-review

Scope

The change keeps the current comparison strategy and only reduces loop overhead. It adds no SIMD requirement and does not change the public API or diagnostic contract.

[pleriche]

Thanks for the suggestion. I've further improved the routine by coaxing the compiler into reducing the number of branches. The code isn't pretty though.

This routine seems like a good candidate for a SIMD assembly implementation. I'll put it on my to-do list.

[TetzkatLipHoka]

This routine seems like a good candidate for a SIMD assembly implementation. I'll put it on my to-do list.

I had a go at it out of curiosity, and it turned out well enough to be worth handing over rather than sitting on. No PR - you said you would rather ask for one, and this is on your list, so this is just the idea plus the numbers. Take it, change it, or ignore it.

The idea

pcmpeqb against a broadcast fill byte, and the results accumulated with pand instead of being branched on, so the loop keeps the single exit test and the data independent run time the scalar version has. One pmovmskb at the end: $FFFF means every byte matched. 64 bytes per iteration:

@Loop64Bytes:
  movdqu xmm2, [rcx + rdx]
  movdqu xmm3, [rcx + rdx + 16]
  movdqu xmm4, [rcx + rdx + 32]
  movdqu xmm5, [rcx + rdx + 48]
  pcmpeqb xmm2, xmm1        //xmm1 = 16 copies of the fill byte
  pcmpeqb xmm3, xmm1
  pcmpeqb xmm4, xmm1
  pcmpeqb xmm5, xmm1
  pand xmm2, xmm3
  pand xmm4, xmm5
  pand xmm2, xmm4
  pand xmm0, xmm2           //xmm0 = accumulator
  add rdx, 64
  cmp rdx, -64
  jle @Loop64Bytes

Two choices that keep it low risk:

• Only the whole 16 byte units go to the vector helper. The freed object marker and the trailing bytes stay on your existing scalar path, so the diff adds 140 lines and removes none.
• The vector path only engages from 128 whole vector bytes upward. Below that the executed code is exactly what it is today, so "no regression for small blocks" is a property of the code rather than something I have to prove with a benchmark.

SSE2 is enough, so it fits what the file already does for the move routines: guaranteed under 64-bit, and gated on System.TestSSE and 2 <> 0 for 32-bit.

Numbers

Your alloc/free workload in debug mode with stack traces off, so the routine runs once per iteration on the reuse path. Median of 25 alternating pairs, each sample in a fresh process, Delphi 13.1, one machine (Ryzen 9 7950X):

User size	Win64	Win32
64 B (control, path unchanged)	-1.0%	+1.9%
128 B (control, just below the threshold)	-0.7%	+2.4%
136 B (first size that engages)	+0.3%	+4.5%
256 B	+3.2%	+10.7%
1 KiB	+18.9%	+36.0%
4 KiB	+65.2%	+106.5%
16 KiB	+87.5%	+158.1%
64 KiB	+41.6%	+80.2%

The part I did not expect: Win32 gains the most, which is exactly where #97 came out weakest (+3.30% on Intel Win32 at 4 KiB). So this lands on the side that the scalar widening could not reach, rather than overlapping with it.

Correctness

In the real allocator: 12 medium block sizes chosen to cover exact multiples of 64, 16/32/48 byte vector remainders and 1/2/3/7/15 byte scalar tails, with every one of 34,949 byte positions corrupted in turn - all detected, on Win32 and Win64, and the unmodified build gives identical results. Separately, the routine in isolation against a plain byte loop for sizes 1 to 600, clean and with every byte position corrupted, deliberately unaligned since a user area is only pointer aligned. PurePascal still compiles and passes, and the four relevant programs from #103 pass.

Two caveats

One CPU and one compiler version, so this is not the cross architecture picture janrysavy produced - the direction was consistent everywhere I looked, but the magnitudes will move.

Also a methodology warning, in case anyone benchmarks this routine: measuring both variants inside one executable is useless here. Below the threshold both run identical code, and that null control still swung by up to 30% from code layout interaction alone. Separate executables and a fresh process per sample brought the noise floor down to about 3%, which is why the table above is worth anything at all.

Happy to send the patch, or a PR if you would prefer one after all - otherwise the snippet above is the whole trick and the rest is bookkeeping.

[TetzkatLipHoka]

Correction to the table above: the CPU is an Intel Core i9-12900K, not a Ryzen 9 7950X. I took the AMD designation from the tables in #99 instead of stating the machine I actually measured on. The numbers themselves are unchanged - only the hardware attribution was wrong.

It does change how one line should be read, and in a useful direction. I presented the Win32 gains as landing where #97 came out weakest, and that weak case (+3.30% at 4 KiB) was Intel Win32. Since my measurements are Intel too, that is a closer comparison than I implied, not a cross-vendor one: different generation (Coffee Lake versus Alder Lake), same vendor.

One methodology detail that belongs with the corrected hardware: the 12900K is a hybrid design, and the benchmark pins itself to one logical processor, which on this part is a P-core. E-cores would give different figures. The pinning is identical for both builds and the pairs alternate, so it does not bias the comparison, but the absolute times are P-core times.

Apologies for the noise - better to have it right in the thread than to leave a wrong CPU attached to the numbers.

[janrysavy]

Thanks for sharing this. I reconstructed @TetzkatLipHoka's SIMD approach from the published description and assembly loop, then measured it on an AMD Ryzen 9 7950X and an Intel Core i7-8750H, with both Win32 and Win64. This is not a byte-for-byte rerun of the original Core i9-12900K candidate because the complete patch was not posted.

Both builds are based on upstream commit 823ba351842a69977c509ff74d68acf08b3a1bc1, which includes Pierre's 32-byte scalar implementation. I used Delphi 13.1 with optimization enabled, separate upstream and SIMD executables, FastMM5 debug mode with stack traces and event output disabled, one worker pinned to logical processor 0, three warmups, and 25 alternating fresh-process pairs per size. The reported value is the median paired 100 * (upstream time / SIMD time - 1), so +100% means twice the throughput.

The straightforward reconstructed Win64 integration added a SIMD helper call inside the Pascal checker. Although the helper was not called for the control sizes, its presence made the compiler save five nonvolatile registers on every check and moved the following allocator routines. That produced a repeatable Intel regression of 3.58% at 64 B, with a bootstrap 95% interval reaching -3.92%.

I tried a different Win64 integration to remove that cost. The scalar checker keeps the upstream instruction flow, tests the 128 B crossover once, and tail-jumps to an out-of-line SSE2 checker only for larger blocks. Corruption logging is also an out-of-line cold path. CheckFreedDebugBlockFillPatternIntact, CheckFreeDebugBlockIntact, and all following FastMM5 allocator routines remain at the same addresses as in the upstream executable. Win32 retains the reconstructed helper integration and its original 136 B crossover.

User size	Ryzen 9 7950X Win64	Ryzen 9 7950X Win32	Core i7-8750H Win64	Core i7-8750H Win32
64 B control	-0.35%	-1.76%	+1.08%	-0.08%
128 B - Win64 SIMD, Win32 control	+2.89%	-0.36%	+7.72%	+0.19%
136 B - SIMD on both targets	+2.41%	+2.05%	+4.60%	+6.86%
256 B	+5.74%	+8.09%	+8.36%	+14.75%
1 KiB	+34.28%	+44.50%	+23.07%	+58.19%
4 KiB	+79.12%	+101.17%	+60.72%	+116.85%
16 KiB	+98.06%	+130.98%	+85.27%	+175.11%
64 KiB	+102.53%	+140.29%	+63.36%	+132.01%

The refined Win64 integration removes the control regression. The AMD 64 B result is statistically neutral, with a bootstrap 95% interval of [-0.89%, +0.44%], while the Intel 64 B result is +1.08% with [0.57%, 1.12%]. The material gains begin at 128 B on Win64 and continue to grow through the KiB sizes. The Win32 values in the table are the earlier measurements because the Win32 implementation did not change. The Core i7-8750H results also reproduce the shape of the corrected Core i9-12900K results, although the exact magnitudes differ by generation.

For correctness, I used 12 real medium-block sizes totaling exactly 34,949 byte positions, covering the 0/16/32/48-byte vector remainders and 1/2/3/7/15-byte scalar tails. Every position was corrupted individually and every corruption was detected by the final binaries on AMD and Intel, Win32 and Win64. The existing debug-pattern test passes, the upstream PurePascal path remains unchanged, and the current FastMM5 quick test suite passes all 18 Win32/Win64 runs with the refined source.

The refined patch is available as janrysavy/FastMM5@9e0ce1a on the agent/refined-debug-fill-simd branch.

This suggests that the SSE2 loop is worthwhile for debug-mode freed-block validation. The layout-preserving Win64 implementation has a larger assembly surface than the straightforward helper call, but it avoids charging small blocks for a SIMD path they do not use.