Runtime debug mode: medium block spans are never returned to the OS - unbounded committed memory growth under multithreaded load

[TetzkatLipHoka]

Summary

When runtime debug mode is active (FastMM_EnterDebugMode), freed medium blocks are intentionally not coalesced (FastMM_FreeMem_InternalFreeMediumBlock_ManagerAlreadyLocked, DebugModeCounter > 0 branch — only SetBlockIsFreeFlag is called). As a side effect, the "is the entire span free?" check in the same function compares the individual block size against SpanSize - CMediumBlockSpanHeaderSize, a condition that can practically never become true for a span that ever held more than one block. Consequently OS_FreeVirtualMemory is never called for medium block spans while debug mode is active.

Single-threaded this is benign: the free-block bins reach a steady state and committed memory stays flat. Multithreaded it becomes an unbounded ratchet: whenever a thread misses the bins because the arena holding a suitable free block is momentarily locked (attempt 1 in FastMM_GetMem_GetMediumBlock skips locked arenas; attempt 3 locks the first unlocked arena and allocates a new sequential-feed span if that arena's bins have no fit), a fresh DefaultMediumBlockSpanSize span (3 MB) is committed. In normal mode such transient over-allocation self-heals — coalescing eventually makes spans fully free and they are returned to the OS. In debug mode every such span is permanent, and its contents degrade irreversibly: exact-fit allocation (FastMM_GetMem requests GetMediumBlock(ASize, ASize, ASize)) splits binned blocks on every reuse, remainders below CMinimumMediumBlockSize (2880) are not binned at all, and merging never happens, so free fragments only ever get smaller and more numerous.

FastMM_ProcessAllPendingFrees does not help — it merely moves pending blocks into the bins; binned is not decommitted.

Reproduction

8 worker threads, each doing 30,000 iterations of GetMem(random size 1..70,000) + FillChar + FreeMem — no cross-thread frees, no leaks (final AllocatedBytes ≈ 4 KB):

program DebugModeSpanGrowth;
{$APPTYPE CONSOLE}
uses FastMM5, Windows, SysUtils;

var
  GIters, GMaxSize: Integer;
  GDone: Integer;

function StressThread(AParam: Pointer): Integer; stdcall;
var
  LSeed: Cardinal;
  LIter, LSize: Integer;
  P: Pointer;
begin
  LSeed := Cardinal(AParam) * $9E3779B9 + 1;
  for LIter := 1 to GIters do
  begin
    LSeed := LSeed xor (LSeed shl 13);
    LSeed := LSeed xor (LSeed shr 17);
    LSeed := LSeed xor (LSeed shl 5);
    LSize := Integer(LSeed mod Cardinal(GMaxSize)) + 1;
    GetMem(P, LSize);
    FillChar(P^, LSize, Byte(LSize));
    FreeMem(P);
  end;
  InterlockedIncrement(GDone);
  Result := 0;
end;

var
  LThreads: array[0..7] of THandle;
  LId: Cardinal;
  I: Integer;
begin
  GIters := 30000;
  GMaxSize := 70000;
  if not FastMM_EnterDebugMode then Halt(2);
  for I := 0 to 7 do
    LThreads[I] := CreateThread(nil, 0, @StressThread, Pointer(I + 1), 0, LId);
  for I := 0 to 7 do
  begin
    WaitForSingleObject(LThreads[I], INFINITE);
    CloseHandle(LThreads[I]);
  end;
  FastMM_ProcessAllPendingFrees;
  with FastMM_GetUsageSummary do
    WriteLn('Allocated=', AllocatedBytes, ' Overhead=', OverheadBytes);
  FastMM_ExitDebugMode;
end.

Observed on current master (13976ed), Win32, Delphi 10 Seattle, default optimization strategy (4 medium block arenas); thread-count scaling cross-checked with Delphi 7 and 13.1 builds of the same test against a source-compatible branch (same numbers ±10%):

Configuration	Committed medium spans afterwards
1 thread × 240,000 iterations, debug mode	6 spans, 18 MB — stable (identical at 480,000 iterations)
2 threads × 120,000, debug mode	73 spans, 219 MB
4 threads × 60,000, debug mode	128 spans, 384 MB
8 threads × 30,000, debug mode	355 spans, 1,065 MB
8 threads × 60,000, debug mode	crash (2 GB address space exhausted)
8 threads × 30,000, debug mode, +12% cross-thread frees	same growth — cross-thread frees make no difference
8 threads × 30,000, normal mode	5 spans, 15 MB

FastMM_MediumBlockThreadContentionCount stayed 0 in the 2- and 4-thread runs — the growth is driven by ordinary lock-skipping in the allocation attempts, not by full back-off contention.

Block-level statistics for the 8×30,000 debug run (via FastMM_WalkBlocks): 104,354 free medium blocks totalling 1,063 MB stuck in never-released spans; 46,964 of them (45%) are smaller than CMinimumMediumBlockSize and therefore not even binned (split remainders — permanently unusable without coalescing); largest free block 70,208 bytes, i.e. large requests increasingly cannot be served from the bins and keep forcing new spans.

Why this matters

Runtime debug mode is one of FastMM5's headline features. Any long-running multithreaded 32-bit application that enables it under load will exhaust its address space; 64-bit applications grow without bound (committed memory). The effect is easy to misdiagnose as an application leak because AllocatedBytes stays flat while OverheadBytes grows.

Suggested fix

Keep the no-coalescing behavior (it preserves the freed-block fill pattern, the debug header with allocation/free stack traces, and the CheckFreeDebugBlockIntact after-free-write detection), but restore the span-release path in debug mode by checking explicitly whether all blocks in the span are free: walk the span's block headers on free (early exit on the first in-use block; exclude the current sequential-feed span while it still has an unfed region), and if the span is fully free, un-bin its free blocks and release it, honoring the existing keep-a-span heuristic.

The walk is O(blocks in span) only on the free path, which in debug mode is already dominated by stack-trace capture. Use-after-free detection is not weakened: dangling writes into a released span fault immediately, which is a strictly louder signal than a fill-pattern mismatch. Pending free blocks still carry the in-use flag until they are processed, so a span can never be released underneath a pending-free chain. (A per-span used-block counter, like BlocksInUse for small block spans, would make the check O(1) at the cost of maintaining the counter in the allocation/split paths.)

I have implemented and tested the walk-based variant — see the accompanying PR. Results with the fix, same 8×30,000 debug-mode scenario on master: committed medium spans drop from 1,065 MB to 72 MB, and the amount is a fluctuating steady state instead of monotone growth (8×120,000 iterations: 60 MB — no longer crashes). Normal-mode behavior is byte-identical (OverheadBytes unchanged).

Workarounds until fixed

• Build 32-bit executables with {$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE} (delays exhaustion, does not prevent it).
• Keep runtime debug-mode sessions short; leave and re-enter around suspect scenarios rather than running under load for long periods.
• Prefer 64-bit builds for debug-mode soak testing.

[pleriche]

Hi, thanks for the report. I am busy looking into this, and the other AI-assisted reports. Feedback to follow.

[TetzkatLipHoka]

Take your time. Will test the D7 backport some more then I'll push D7/FPC Support aswell.
Besides iam looking into Profiling a little. Gonna spam some more pulls soon ;)

[pleriche]

I have made two changes that I think will go a long way to alleviate the issues raised here:

1. I have changed the behaviour to always merge free medium blocks with adjacent block fragments that are too small to put in a size bin. This will allow the space for such blocks to be reused.
2. I have added the FastMM_GetDebugModeOptions and FastMM_SetDebugModeOptions calls. These allow the application to tune the behaviour in debug mode, in particular with reference to block freeing behaviour. It is now possible to restore the address space reclamation behaviour of non-debug mode, while still getting the benefit of the other debug features.

I've added this line to the test case above:
FastMM_SetDebugModeOptions(FastMM_GetDebugModeOptions - [dmoNeverMergeFreeMediumBlocks]);

With this line added memory usage remains fairly stable.

[TetzkatLipHoka]

Thank you — verified against master b182cbe with the reproduction from this issue (8 threads, 70KB max block size, debug mode, 32-bit): with default options the overhead is still ~1,195 MB at 8x30k iterations, but with FastMM_SetDebugModeOptions(FastMM_GetDebugModeOptions - [dmoNeverMergeFreeMediumBlocks]); it drops to a stable 15.7 MB at both 8x30k and 8x120k (which previously exhausted the 32-bit address space) — matching non-debug-mode levels. That resolves the issue for us, and is a better result than the span-walk approach in #83, so I'll close that PR. One note for other readers: the merging sacrifices the freed-block fill patterns on merged medium blocks, so keep the default if you rely on use-after-free detection there and have address space to spare.