Two independently-linked mimalloc v3 instances (CPython 3.14 vendored + static in a Python extension) SIGSEGV in _mi_theap_collect_retired at process exit when Arrow's v2 copy is also loaded (macOS arm64)

[kkollsga]

On macOS arm64 a Python 3.14 process can end up hosting three independent mimalloc instances, and a specific pairing crashes at interpreter teardown:

• CPython 3.14.3 vendors mimalloc v3.3.2 (initializes even in python -c pass)
• our Rust extension (kglite, PyO3/abi3 cdylib) statically links mimalloc v3.3.1 as the Rust #[global_allocator] (via libmimalloc-sys, no override feature — we register no malloc zone and export no symbols beyond PyInit_*)
• pyarrow 24.0.0's bundled libarrow carries its own mimalloc (v2-line, per MIMALLOC_VERBOSE banner)

Repro (deterministic, 100+ runs):

python3.14 -c "import pyarrow; import kglite; kglite.KnowledgeGraph()"
# exit 139 — SIGSEGV at teardown; both import orders; import alone suffices

Crash frame (lldb): EXC_BAD_ACCESS (address=0x17) in libarrow.2400.dylib at _mi_theap_collect_retired + 124, single-frame backtrace (fires from teardown machinery; unwind unavailable), register holding the page pointer contains 0x1 — a corrupted/foreign thread-heap page list. os._exit(0) avoids it entirely (teardown-only). Enabling faulthandler perturbs timing enough to sometimes mask it.

Discriminating matrix (all verified 5x, fresh venvs):

extension's mimalloc	pyarrow	result
v3.3.1	24.0.0	SIGSEGV
v3.3.1	23.0.1 / 22.0.0	clean
v2.3.01 (only change)	24.0.0	clean
none (allocator removed)	24.0.0	clean
v3.3.1	absent	clean (CPython-v3 + ext-v3 alone coexist)

So the fatal combination requires two v3-line instances (CPython's + ours) plus pyarrow 24's presence — its v2 instance appears to change teardown ordering/registration such that one v3 instance's thread-heap cleanup walks state poisoned by the other. Not TLS-model dependent (local_dynamic_tls unchanged), not zone-related (no instance registers a zone in our builds).

Question for maintainers: are multiple independently-linked mimalloc copies in one process expected to be teardown-safe? If yes, this looks like a v3 regression in instance isolation (thread-done callbacks / reserved-slot sharing?); if no, that constraint deserves loud documentation — Rust wheels commonly ship #[global_allocator] mimalloc (v3 default since libmimalloc-sys 0.1.47), and CPython ≥3.13 vendors v3, so the collision surface is the whole PyO3 ecosystem × modern Python.

Our workaround: pinning our copy to the v2 line (mimalloc crate features = ["v2"]), shipping in kglite 0.12.12. Happy to provide the minimal reproducer wheel pair + full MIMALLOC_VERBOSE census logs.

[kkollsga]

Cross-filed the Arrow-side regression angle (24.0.0 vs ≤23.0.1): apache/arrow#50428

[pitrou]

For the record, Arrow makes the mimalloc symbols private in libarrow.so.

[kkollsga]

Found the channel. On macOS, mimalloc ≥ v3.2.6 keeps the thread-heap pointer in fixed pthread TCB slots:

// include/mimalloc/prim.h
#elif defined(__APPLE__)   // macOS
  #define MI_TLS_MODEL_FIXED_SLOT           1
  #define MI_TLS_MODEL_FIXED_SLOT_DEFAULT   108
  #define MI_TLS_MODEL_FIXED_SLOT_CACHED    109

The TCB is process-global per thread, so every co-resident v3 instance shares these two cells regardless of symbol visibility. Each instance can load a foreign instance's mi_theap_t* and walk it with its own struct offsets. _mi_theap_cached_set also decrefs the previous slot value (init.c), so one instance can mutate a refcount inside another instance's heap struct.

This explains the full matrix: 3.3.1 × 3.1.5 clean (3.1.5 pre-dates the slot model), 3.3.1 × 3.2.7 crashes (layouts differ), 3.3.1 × 3.3.1 clean only because layouts coincide, v2 × anything clean (v2 uses the slot only under MI_MALLOC_OVERRIDE).

[pitrou]

v2 × anything clean (v2 uses the slot only under MI_MALLOC_OVERRIDE).

Hmm, so why does the crash occur in libarrow here?

[pitrou]

Ah, I guess your assumption that libarrow was using mimalloc 2.x was mistaken, then?

[pitrou]

The TCB is process-global per thread, so every co-resident v3 instance shares these two cells regardless of symbol visibility.

That's clearly a concern, because mimalloc is getting increased usage (especially in Rust-space apparently?).

Is there a way to make sure that each mimalloc version uses different TLS keys, or is the TLS key space too restricted?

[pitrou]

I see this snippet in prim.h:

- MI_TLS_MODEL_FIXED_SLOT: use a fixed slot in the TLS block (default on macOS)
    This reserves an unused and fixed TLS slot. This is fast and avoids the problem
    where the underlying TLS implementation (or the loader) will call itself `malloc`
    on a first access to a thread local (and recurse in the MI_TLS_MODEL_THREAD_LOCAL).
    This goes wrong though if the OS or a library uses the same fixed slot.

The malloc recursion problem only exists if MI_OVERRIDE is enabled, right?
Would it be safe to define MI_TLS_MODEL_THREAD_LOCAL when compiling Arrow's private mimalloc since we also define MI_OVERRIDE to OFF? @daanx

[kkollsga]

A few verified data points on the open questions.

On the earlier version question: yes, my v2 attribution for libarrow was mistaken. Arrow's cpp/thirdparty/versions.txt is authoritative: 23.0.0 bundles mimalloc v3.1.5, 24.0.0 bundles v3.2.7, main bundles v3.3.1. The matrix in my mechanism comment uses the corrected numbers.

On whether the TLS key space is restricted: slots 108/109 are not free. Apple's libpthread (private/pthread/tsd_private.h) reserves them for the Swift runtime:

/* Keys 100-109 are for the Swift runtime */
#define __PTK_FRAMEWORK_SWIFT_KEY8   108
#define __PTK_FRAMEWORK_SWIFT_KEY9   109
/* Keys 110-115 for libmalloc */

The prim.h comment already records one eviction ("we used before __PTK_FRAMEWORK_OLDGC_KEY9 (89) but that seems used now"), so squatting in this range has collided before, independent of the multi-instance case.

On whether MI_TLS_MODEL_THREAD_LOCAL is safe with MI_OVERRIDE=OFF: tested empirically. Same extension (Rust cdylib, mimalloc as global allocator, no override), built four ways, co-resident with pyarrow 24.0.0 (bundled 3.2.7, fixed-slot) on macOS arm64, py3.14:

build	model (verified via nm -u)	coexist ×7	8-thread stress ×3
v3.3.1 stock	fixed slot 108/109	SIGSEGV all	SIGSEGV all
v3.3.1 -DMI_HAS_TLS_SLOT=0	DYNAMIC_PTHREADS (pthread_getspecific import appears)	clean	clean
v3.3.1 patched	THREAD_LOCAL + MI_TLS_RECURSE_GUARD (__tlv_bootstrap appears)	clean	clean
v2.3.01	__thread	clean	clean

So both non-slot models are collision-safe without override, including thread churn. Two caveats from the same runs: THREAD_LOCAL cannot currently be selected externally on macOS (the __APPLE__ branch needs a source edit; DYNAMIC_PTHREADS is reachable via -DMI_HAS_TLS_SLOT=0), and the fixed slot is measurably why it exists: on allocation-hot benchmarks both safe models cost ~7-10% versus the fixed slot (isolated to the TLS fast path; the v3 allocator itself matched v2 in the same harness, and __tlv_get_addr was not faster than pthread_getspecific).

Impact note: import pandas transitively loads pyarrow.lib when pyarrow is installed, so the co-residence crash triggers through an ordinary pandas dependency chain with no direct pyarrow use.

[pitrou]

Two caveats from the same runs: THREAD_LOCAL cannot currently be selected externally on macOS

Are you sure? Based on this snippet it should be possible to pass -DMI_TLS_MODEL_THREAD_LOCAL when compiling mimalloc.

mimalloc/include/mimalloc/prim.h

Lines 381 to 398 in 0dfdcbc

	// Default TLS model
	#if !defined(MI_TLS_MODEL_THREAD_LOCAL) && !defined(MI_TLS_MODEL_DYNAMIC_PTHREADS)
	#if defined(_WIN32)
	#define MI_TLS_MODEL_DYNAMIC_WIN32 1
	#elif defined(__APPLE__) && MI_HAS_TLS_SLOT && !defined(__POWERPC__) // macOS on arm64 or x64
	// #define MI_TLS_MODEL_DYNAMIC_PTHREADS 1 // also works but a bit slower
	#define MI_TLS_MODEL_FIXED_SLOT 1
	#define MI_TLS_MODEL_FIXED_SLOT_DEFAULT 108 // seems unused. @apple: it would be great to get 2 official slots for custom allocators :-)
	#define MI_TLS_MODEL_FIXED_SLOT_CACHED 109
	// we used before __PTK_FRAMEWORK_OLDGC_KEY9 (89) but that seems used now.
	// see <https://github.com/rweichler/substrate/blob/master/include/pthread_machdep.h>
	#elif defined(__APPLE__) || defined(__OpenBSD__) || defined(__ANDROID__)
	#define MI_TLS_MODEL_DYNAMIC_PTHREADS 1
	// #define MI_TLS_MODEL_DYNAMIC_PTHREADS_DEFAULT_ENTRY_IS_NULL 1
	#else
	#define MI_TLS_MODEL_THREAD_LOCAL 1
	#endif
	#endif

(FTR, are you using AI for your messages?)

[pitrou]

And actually, there's even a CMake option for it:

mimalloc/CMakeLists.txt

Lines 59 to 61 in 0dfdcbc

	option(MI_TLS_MODEL_LOCAL "Force TLS model based on thread local variables" OFF)
	option(MI_TLS_MODEL_PTHREADS "Force TLS model based on pthreads" OFF)
	option(MI_TLS_RECURSE_GUARD "Force a recursion check on thread local access (MI_TLS_MODEL_LOCAL only)" OFF)

[daanx]

Just to add: for macOS, using the cmake option -DMI_TLS_MODEL_PTHREADS=ON is recommended in this case (using -DMI_TLS_MODEL_LOCAL=ON is also good (but on my M1 just a tad slower)).

In particular, when linking multiple mimalloc instances in the same program on macOS we cannot use the direct TLS slots and need to fall back to pthread locals or regular __thread locals. Now, these have improved over the last few years so performance should still be fine. Hope this fixes the issues.