Optimize anonymous mmap and page-fault paths

[Max042004]

Anonymous memory is reworked on a lazy foundation: sys_mmap only records the region, while page tables, zeroing and host-memory commit are deferred to first touch. Two optimization layers sit on top.

mmap: the common shape (addr == 0, MAP_PRIVATE|MAP_ANONYMOUS, RW) is served entirely at EL1 with no vCPU exit and no lock. The shim stays stateless: it bump-allocates from a host-carved per-vCPU VA arena and reports each grant through a per-vCPU SPSC ring that the host drains whenever it takes mmap_lock, so no cross-vCPU synchronization exists. Arenas recycle freed address space through the gap allocator, refill automatically when a request does not fit (the new arena sized to cover the triggering request, with an ARENA_MAX guard so oversized requests fall back without churning the arena), and decay back to workload scale based on served-length history.

Page fault: a per-2MiB-block dirty map lets materialization skip zeroing blocks whose slab bytes are known zero, moving the host page commit off the fault and out of mmap_lock onto the guest's own first writes. The TLBI protocol emits a single range op per materialized block instead of per-page invalidations, and a repeat fault on an already-valid block returns without re-zeroing.

Host-side access to untouched lazy memory (I/O buffers, futex words, signal frames, shmat/shmdt copies) materializes on demand with lock-order-safe pre-faulting, and PROT_NONE stays a pure faulting reservation. tests/test-mmap-lazy.c and tests/test-mmap-fastpath.c pin the contract: zero-on-reuse, huge sparse reservations, host access to untouched memory, PROT_NONE round trips, fork, racing first touch, and arena refill/revocation.

---

Summary by cubic

Implements lazy anonymous mmap (defers PTEs, zeroing, and host commit to first touch) and adds an EL1 fast path for common anonymous RW mmaps to cut latency and avoid contention.

• New Features

  • Lazy anon memory: record-only on sys_mmap; zero and commit on first write; PROT_NONE stays a pure reservation.
  • EL1 mmap fast path: per‑vCPU VA arenas and SPSC rings; no vCPU exits or locks; adaptive refill with max guard and automatic fallback to the host path.
  • Page faults: per‑2MiB dirty map skips zeroing clean blocks; batched TLBI per block; repeat faults avoid re-zeroing.
  • Host access: lock‑order‑safe pre‑faulting for futex words, signal frames, I/O buffers, and SysV SHM.
  • API/infra: mmap_lock_acquire/release/cond_wait drain EL1 rings before region checks; new stats counters and tests (including microbench) for lazy faults and fast path behavior; docs describe fast path controls.

• Migration

  • EL1 fast path is on by default for mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, ...). Set ELFUSE_MMAP_FASTPATH=0 to disable. Verbose tracing, the syscall histogram, GDB, and Rosetta force the host path.
  • Fork IPC protocol bumped (ELFM). Update both parent and child when mixing versions.

^{Written for commit a95bd60. Summary will update on new commits.}

[cubic-dev-ai]

8 issues found across 28 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/syscall/signal.c">

<violation number="1" location="src/syscall/signal.c:1768">
P1: Signals on a valid minimum-sized lazy altstack still materialize pages under `sig_lock`, reintroducing the mmap-lock inversion this pre-fault is meant to prevent. Pre-fault the available altstack tail (at least the frame size), rather than requiring the extra 512-byte advisory window to fit.</violation>
</file>

<file name="tests/test-mmap-fastpath.c">

<violation number="1" location="tests/test-mmap-fastpath.c:68">
P2: The SIGSEGV handler installed in `test_fidelity()` is never restored. The `static sigjmp_buf` still points into `test_fidelity()`'s stack frame after the function returns. If a later test (or a fast-path bug) triggers SIGSEGV, the handler would longjmp into dead stack — producing a confusing crash instead of a clean test failure. Save the old `struct sigaction` from `sigaction(SIGSEGV, ...)` and restore it on the way out of `test_fidelity()`.</violation>
</file>

<file name="src/runtime/futex.c">

<violation number="1" location="src/runtime/futex.c:1501">
P2: Unsupported or otherwise invalid futex commands can materialize a lazy mapping before returning their validation error. Validate/dispatch the command before faulting its operand so `-ENOSYS`/`-EINVAL` calls do not commit or alter untouched guest memory.</violation>
</file>

<file name="src/syscall/mem.c">

<violation number="1" location="src/syscall/mem.c:2356">
P1: MAP_FIXED inside this vCPU's unconsumed arena tail can be handed out again by the EL1 bump allocator, overlapping the fixed mapping. Revoke the current tail for every nonzero explicit address, not only non-fixed hints.</violation>
</file>

<file name="tests/bench-mmap-lazy.c">

<violation number="1" location="tests/bench-mmap-lazy.c:87">
P3: When argv[1] is non-numeric (or an empty string), atoll silently returns 0, cap becomes 0, and no sizes are benchmarked without any error or warning. Validate the argument and print a usage message on parse failure.</violation>
</file>

<file name="src/syscall/sysvipc.c">

<violation number="1" location="src/syscall/sysvipc.c:261">
P1: `shmat` can self-deadlock under page-table/materialization allocation failure: failed fault-in is ignored, then `guest_write` retries lazy fault-in and reacquires already-held `mmap_lock`. Handle fault-in failure before copying and return an error/clean up instead of entering `guest_write`.</violation>

<violation number="2" location="src/syscall/sysvipc.c:324">
P1: `shmdt` can hang while detaching when materialization fails, because `guest_read` falls back to lazy fault-in after this ignored failure and recursively takes `mmap_lock`. Check fault-in/copy failures and return a syscall error rather than continuing under the lock.</violation>
</file>

<file name="src/core/guest.c">

<violation number="1" location="src/core/guest.c:1583">
P2: Signal syscalls using an untouched lazy `act`, `oldact`, `set`, or `oldset` buffer now acquire `mmap_lock` while holding `sig_lock`. This reverses the documented lock order and can deadlock against an mmap-then-signal path; pre-fault those syscall buffers before `sig_lock`, or use a no-fault resolve while it is held.</violation>
</file>

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Signals on a valid minimum-sized lazy altstack still materialize pages under sig_lock, reintroducing the mmap-lock inversion this pre-fault is meant to prevent. Pre-fault the available altstack tail (at least the frame size), rather than requiring the extra 512-byte advisory window to fit.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/syscall/signal.c, line 1768:

<comment>Signals on a valid minimum-sized lazy altstack still materialize pages under `sig_lock`, reintroducing the mmap-lock inversion this pre-fault is meant to prevent. Pre-fault the available altstack tail (at least the frame size), rather than requiring the extra 512-byte advisory window to fit.</comment>

<file context>
@@ -1748,8 +1748,31 @@ static int deliver_signal_locked(hv_vcpu_t vcpu,
+        guest_lazy_faultin(g, sp - need, need);
+    thread_entry_t *thr = current_thread;
+    if (thr && thr->altstack_sp != 0 &&
+        !(thr->altstack_flags & LINUX_SS_DISABLE) && thr->altstack_size > need)
+        guest_lazy_faultin(g, thr->altstack_sp + thr->altstack_size - need,
+                           need);
</file context>

Suggested change

	!(thr->altstack_flags & LINUX_SS_DISABLE) && thr->altstack_size > need)
	guest_lazy_faultin(g, thr->altstack_sp + thr->altstack_size - need,
	need);
	!(thr->altstack_flags & LINUX_SS_DISABLE) &&
	thr->altstack_size >= sizeof(linux_rt_sigframe_t)) {
	uint64_t alt_need = thr->altstack_size < need ? thr->altstack_size : need;
	guest_lazy_faultin(g, thr->altstack_sp + thr->altstack_size - alt_need,
	alt_need);
	}

[cubic-dev-ai]

P1: shmdt can hang while detaching when materialization fails, because guest_read falls back to lazy fault-in after this ignored failure and recursively takes mmap_lock. Check fault-in/copy failures and return a syscall error rather than continuing under the lock.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/syscall/sysvipc.c, line 324:

<comment>`shmdt` can hang while detaching when materialization fails, because `guest_read` falls back to lazy fault-in after this ignored failure and recursively takes `mmap_lock`. Check fault-in/copy failures and return a syscall error rather than continuing under the lock.</comment>

<file context>
@@ -312,7 +317,11 @@ int64_t sys_shmdt(guest_t *g, uint64_t shmaddr_gva)
+         * self-deadlock hazard as the shmat copy-in: pages the guest never
+         * touched may still be unmaterialized.
+         */
+        guest_lazy_faultin_locked(g, entry.guest_gva, entry.size);
         guest_read(g, entry.guest_gva, entry.host_addr, entry.size);
     }
</file context>

[cubic-dev-ai]

P1: shmat can self-deadlock under page-table/materialization allocation failure: failed fault-in is ignored, then guest_write retries lazy fault-in and reacquires already-held mmap_lock. Handle fault-in failure before copying and return an error/clean up instead of entering guest_write.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/syscall/sysvipc.c, line 261:

<comment>`shmat` can self-deadlock under page-table/materialization allocation failure: failed fault-in is ignored, then `guest_write` retries lazy fault-in and reacquires already-held `mmap_lock`. Handle fault-in failure before copying and return an error/clean up instead of entering `guest_write`.</comment>

<file context>
@@ -253,7 +253,12 @@ int64_t sys_shmat(guest_t *g, int shmid, uint64_t shmaddr_gva, int shmflg)
+     * guest_write would self-deadlock on it; materialize the fresh anonymous
+     * mapping through the locked variant first.
+     */
+    guest_lazy_faultin_locked(g, (uint64_t) gva, seg_size);
     if (guest_write(g, (uint64_t) gva, host_addr, seg_size) < 0) {
         shmdt(host_addr);
</file context>

[cubic-dev-ai]

P2: The SIGSEGV handler installed in test_fidelity() is never restored. The static sigjmp_buf still points into test_fidelity()'s stack frame after the function returns. If a later test (or a fast-path bug) triggers SIGSEGV, the handler would longjmp into dead stack — producing a confusing crash instead of a clean test failure. Save the old struct sigaction from sigaction(SIGSEGV, ...) and restore it on the way out of test_fidelity().

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/test-mmap-fastpath.c, line 68:

<comment>The SIGSEGV handler installed in `test_fidelity()` is never restored. The `static sigjmp_buf` still points into `test_fidelity()`'s stack frame after the function returns. If a later test (or a fast-path bug) triggers SIGSEGV, the handler would longjmp into dead stack — producing a confusing crash instead of a clean test failure. Save the old `struct sigaction` from `sigaction(SIGSEGV, ...)` and restore it on the way out of `test_fidelity()`.</comment>

<file context>
@@ -0,0 +1,278 @@
+    TEST("unconsumed arena is absent and faults");
+    struct sigaction sa = {.sa_handler = segv_handler};
+    sigemptyset(&sa.sa_mask);
+    if (sigaction(SIGSEGV, &sa, NULL) != 0) {
+        FAIL("sigaction");
+        return;
</file context>

[cubic-dev-ai]

P2: Unsupported or otherwise invalid futex commands can materialize a lazy mapping before returning their validation error. Validate/dispatch the command before faulting its operand so -ENOSYS/-EINVAL calls do not commit or alter untouched guest memory.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/runtime/futex.c, line 1501:

<comment>Unsupported or otherwise invalid futex commands can materialize a lazy mapping before returning their validation error. Validate/dispatch the command before faulting its operand so `-ENOSYS`/`-EINVAL` calls do not commit or alter untouched guest memory.</comment>

<file context>
@@ -1493,6 +1493,16 @@ int64_t sys_futex(guest_t *g,
+     * materializing there would invert the lock order. A futex word in a
+     * mapping the guest never touched reads as zero, matching Linux.
+     */
+    guest_lazy_faultin(g, uaddr, sizeof(uint32_t));
+    if (cmd == FUTEX_REQUEUE || cmd == FUTEX_CMP_REQUEUE ||
+        cmd == FUTEX_WAKE_OP)
</file context>

[cubic-dev-ai]

P2: Signal syscalls using an untouched lazy act, oldact, set, or oldset buffer now acquire mmap_lock while holding sig_lock. This reverses the documented lock order and can deadlock against an mmap-then-signal path; pre-fault those syscall buffers before sig_lock, or use a no-fault resolve while it is held.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/core/guest.c, line 1583:

<comment>Signal syscalls using an untouched lazy `act`, `oldact`, `set`, or `oldset` buffer now acquire `mmap_lock` while holding `sig_lock`. This reverses the documented lock order and can deadlock against an mmap-then-signal path; pre-fault those syscall buffers before `sig_lock`, or use a no-fault resolve while it is held.</comment>

<file context>
@@ -1491,22 +1495,134 @@ static void *gva_resolve_perm(const guest_t *g,
+     */
+    uint64_t want = (avail_limit == UINT64_MAX) ? 1 : avail_limit;
+    if (!ptr) {
+        if (gva_lazy_faultin(g, gva, want, required_perms) < 0)
+            return NULL;
+        return gva_resolve_perm_walk(g, gva, avail, required_perms,
</file context>

[cubic-dev-ai]

P3: When argv[1] is non-numeric (or an empty string), atoll silently returns 0, cap becomes 0, and no sizes are benchmarked without any error or warning. Validate the argument and print a usage message on parse failure.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/bench-mmap-lazy.c, line 87:

<comment>When argv[1] is non-numeric (or an empty string), atoll silently returns 0, cap becomes 0, and no sizes are benchmarked without any error or warning. Validate the argument and print a usage message on parse failure.</comment>

<file context>
@@ -0,0 +1,111 @@
+     */
+    uint64_t cap = ~0ull;
+    if (argc > 1)
+        cap = (uint64_t) atoll(argv[1]) << 30;
+
+    setvbuf(stdout, NULL, _IONBF, 0);
</file context>