Raise host NOFILE capacity for the guest FD table

[Xalestar]

Problem

elfuse exposes a guest FD table with 1024 entries, but the host process may start with a much smaller soft RLIMIT_NOFILE. On a host soft limit of 256, the stress test can open only about 248 guest descriptors before the host runs out of descriptors, even though the guest table still has capacity.

Host capacity and guest-visible NOFILE state also need to remain separate: exposing the internal reserve reports an unreachable limit, while forwarding guest setrlimit to the host can remove capacity needed by fork IPC.

Change

• Require host soft RLIMIT_NOFILE capacity for FD_TABLE_SIZE plus a 256-descriptor host reserve.
• Raise only the host soft limit and preserve its hard limit.
• Fail at startup with a diagnostic when the hard limit cannot provide the required capacity.
• Keep a guest-side 1024/1024 NOFILE limit for getrlimit, setrlimit, prlimit64, proc reporting, and FD allocation.
• Keep guest NOFILE updates from changing the internal host limit.
• Carry guest NOFILE state through fork IPC and restore inherited FDs before applying a lowered soft limit.
• Recheck internal host capacity in fork helpers before receiving the parent FD table.

Design decisions

• The 256-descriptor reserve budgets about 128 descriptors for the bounded blocking-I/O cost (MAX_THREADS = 64, with up to two host FD references per thread), plus headroom for runtime pipes, fork IPC, debugger sockets, sysroot/FUSE plumbing, and smaller concurrent costs.
• ppoll and pselect can retain a number of references that scales with their input FD set. That structural issue cannot be covered by a fixed reserve and should be handled separately.
• If the host hard limit is below the required 1280 descriptors, fail at startup rather than silently reducing the guest-visible limit.

Verification

• Focused test-stress with host soft limit 256: pass
• Focused test-stress with host soft limit 4096: pass
• Host soft/hard limit 256: expected startup failure reports required capacity 1280
• Guest NOFILE reports 1024/1024 without exposing the 256-FD host reserve
• Fork after lowering guest soft and hard NOFILE to 3 preserves an inherited pipe
• make check: 69 tests passed; BusyBox 82 passed, 0 failed, 2 skipped
• make test-matrix-elfuse-aarch64: 235 passed, 0 failed, 4 skipped
• make test-matrix-qemu-aarch64: 215 passed, 0 failed, 24 skipped
• make lint: pass with existing project warnings
• clang-format dry run for all changed C and header files: pass
• git diff --check: pass

Related work

PR #148 addresses the same issue by raising the host limit in fdtable initialization and adding a host-side low-limit regression wrapper. This PR also separates guest-visible NOFILE state from elfuse internal host capacity and preserves that state across fork IPC.

Close #68

[Max042004]

Add "Close #68" at the end of PR and commit message.

Because PR #148 also address same issue. So please also mention this PR.

[Xalestar]

Done. The PR body now mentions PR #148 and ends with Close #68. The latest commit message was also amended to include both references (e189210).

[Max042004]

Is a 64-descriptor host reserve acceptable, or would another policy better match the project?

HOST_FD_RESERVE is fixed at 64, sized (per its comment) for
"runtime pipes, fork IPC, debugger sockets, and sysroot/FUSE plumbing." That
covers process-wide, mostly-singleton overhead, but misses a per-thread cost
that scales with concurrency.

io_block_wait() routes every blocking read/write/
send/recv through host_fd_ref_open(), which
dup()s the guest fd's host fd for the duration of the wait. That dup is held
by every guest thread currently parked in a blocking syscall, and elfuse
supports up to MAX_THREADS = 64 concurrent guest threads. A
thread-pool-style server with all 64 threads blocked in read() at once holds
64 extra host fds that HOST_FD_RESERVE doesn't budget for.

It's not just read/write: sendfile/copy_file_range
(copy_fd_range(), src/syscall/io.c:2297) hold two refs per thread for
the whole transfer (in + out), so the worst case for this term alone is closer
to 2 * MAX_THREADS = 128.

(ppoll/pselect can hold far more refs per thread — up to one per polled
fd — but that's an unbounded, structural issue no fixed constant can absorb;
worth its own follow-up issue rather than folding into this reserve.)

Suggested number

HOST_FD_RESERVE = 256. Breakdown:

• ~128 for the worst-case blocking-I/O term above (64 threads × 2 refs)
• ~16 for the existing singleton overhead (fork IPC socketpair, poll waker
  pipe, asyncio kqueue, log fd, debugger socket, sysroot/FUSE)
• remaining margin for smaller concurrent costs (pty keepalive dups, fork-in-
  flight socketpairs)

When the host hard limit is below the required capacity, should elfuse fail at startup as proposed here, or derive a smaller guest-visible limit?

I prefer fail at startup. MacOS shell fd hard limit is much higher. So if the user hit the hard limit, it mostly because they configured some settings that lower hard limit in the past. It better fail at startup so they can debug.

[Xalestar]

Thanks for the detailed breakdown. Addressed in 50a140a:

• Increased HOST_FD_RESERVE from 64 to 256.
• Documented the bounded 2 * MAX_THREADS = 128 blocking-I/O term and the remaining runtime headroom in the source comment.
• Kept the startup failure when the host hard limit is below the required capacity; the required capacity is now 1280 (1024 guest FDs + 256 reserve).
• Kept ppoll/pselect reference accounting out of this fixed reserve, since it scales with the input FD set and needs separate handling.

I also updated the PR description to record these decisions.

Verification after the change:

• make check: 69/69 passed; BusyBox 82 passed, 0 failed, 2 skipped
• Host soft limit 256 with a sufficient hard limit: test-stress passed (6/6)
• Host soft/hard limit 256: startup failed as expected and reported required capacity 1280
• clang-format --dry-run --Werror src/main.c: passed
• git diff --check: passed

[Max042004]

Please squash to one commit

[Xalestar]

Done. I squashed the three commits into a single commit, d298609, and updated the PR branch with --force-with-lease.

The resulting tree is identical to the pre-squash head (37a8d0e), so this is a history-only change. The commit message retains the problem statement, the 256-descriptor reserve rationale, the guest/host NOFILE separation, the fork IPC changes, the PR #148 reference, and Close #68.