Match macOS du allocation accounting

[nshcr]

Note

AI development disclosure

Both the code changes in this PR and this description were produced by Codex under my direction. I defined the scope and guided the investigation; Codex performed the data collection, cross-version comparisons, source and history tracing, implementation, test development and execution, validation, and final drafting.

I manually reviewed the relevant primary sources and found no serious issues. If this approach is incorrect or does not fit dua's direction, please feel free to modify or reject it, or let me know how it should be improved!

Background

Hi! I found this issue unexpectedly while using dua on macOS 26. When scanning an OrbStack directory mounted through NFS, dua's default allocated-size total did not match the equivalent result from /usr/bin/du.

At first, this looked like a single NFS regression. Testing several dua releases showed that the discrepancy had actually changed direction:

• v2.41.1 and earlier overcounted the tree;
• v2.42.0 and v2.42.1 undercounted it.

Further checks ruled out hard-link deduplication, traversal concurrency, and cancellation between subtrees. The discrepancy came from the allocated-size metadata returned by three different macOS query contracts.

This PR assumes that matching the system du behavior may be desirable for dua, at least for default allocated-size accounting on macOS. It therefore follows the metadata contract used by Apple FTS - the traversal implementation behind macOS /usr/bin/du - rather than adding an OrbStack/NFS-specific workaround.

Reproduction

The comparison uses equivalent default allocated-size modes:

dua aggregate -f bytes "$NFS_ROOT"
BLOCKSIZE=512 /usr/bin/du -s "$NFS_ROOT"

dua reports bytes directly. With BLOCKSIZE=512, macOS du reports 512-byte blocks:

/usr/bin/du: 12,442,673 × 512 = 6,370,648,576 bytes

The main comparison was run sequentially against the same mounted tree:

Implementation	Reported bytes	Delta vs /usr/bin/du
v2.41.0	6,628,753,408	+258,104,832 (+4.051%)
v2.41.1	6,628,753,408	+258,104,832 (+4.051%)
v2.42.0	6,357,613,568	-13,035,008 (-0.205%)
v2.42.1	6,357,612,032	-13,036,544 (-0.205%)
Initial fixed build	6,370,648,576	0
/usr/bin/du	6,370,648,576	Reference

These runs traversed 140,829 entries and reported the same 13 inaccessible paths. A later v2.39.1 check reproduced the v2.41.x byte total, although that older release propagated traversal errors differently. This also shows that the v2.40 walker was not the origin of the overcount.

OrbStack is a live filesystem, so absolute totals describe individual snapshots. The relevant evidence is the stable version-to-version difference under matching entry and error counts, followed by repeated exact agreement between the fix and du on later snapshots.

Root cause

Before v2.42.0, dua ultimately accounted allocated size using per-entry Unix metadata and st_blocks × 512. That is a conventional Unix definition, but it does not guarantee the same filesystem query contract as Apple FTS. On this NFS mount, the per-path metadata query returned a total 258,104,832 bytes larger than /usr/bin/du.

PR #367 introduced a native macOS walker using getattrlistbulk. Its original implementation:

• requested a reduced common/file/directory attribute set;
• called getattrlistbulk(..., options = 0);
• trusted records without first proving that the returned attributes were sufficient to synthesize a complete stat;
• used directory allocation from bulk records, whereas Apple FTS always performs a path lookup for directories.

These differences were not exposed by the existing local-filesystem tests, but on this NFS mount they changed the discrepancy from an overcount to an undercount of 13,035,008 bytes.

PR #369 then reused the same bulk metadata for aggregation roots. This was not the primary cause, but it extended the affected path and reduced the total by another 1,536 bytes on this snapshot.

Implementation

This PR makes dua's macOS bulk walker follow the relevant Apple FTS contract:

• Request the complete common and file attribute sets needed to synthesize stat.
• Leave directory attributes unset, because FTS does not synthesize directory stat fields from bulk records.
• Use FSOPT_PACK_INVAL_ATTRS for full metadata enumeration.
• Decode packed-invalid fields in request order while separately retaining the returned-attribute bitmap.
• Accept bulk metadata only for FTS-recognized vnode types when the required attributes were actually returned.
• Fall back to no-follow path metadata for directories, firmlinks, unknown vnode types, and incomplete records.
• Round allocation bytes to 512-byte accounting units, matching how FTS constructs st_blocks.
• Preserve the existing list-only permission retry and ordinary directory-reading fallback.
• Keep the bulk fast path for complete non-directory records.

The branch is based on #371. Its optional APFS data-fork allocation, clone ID, extended flags, and FSOPT_ATTR_CMN_EXTENDED behavior are retained as extensions of the same packed layout. Missing APFS extension attributes disable clone metadata without invalidating an otherwise complete FTS-compatible record.

The implementation does not inspect the filesystem name and contains no NFS-specific branch.

Validation

The following checks passed on the current branch:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-targets --all-features
git diff --check

All 126 automated tests passed. The macOS coverage includes bulk metadata, APFS clones and data forks, resource forks, symlinks, hard links, permissions, buffer refills, fallback behavior, FTS/APFS request shapes, vnode validation, returned-attribute completeness, and allocation rounding.

Real OrbStack NFS validation confirmed that:

• the default total matched /usr/bin/du exactly;
• the ubuntu and docker subtrees matched independently, ruling out cancellation between positive and negative errors;
• --count-hard-links matched /usr/bin/du -l exactly;
• 1, 4, and 8 threads produced identical totals, entry counts, and errors;
• --apparent-size remained unchanged from v2.42.1;
• the same 13 permission errors were preserved.

After rebasing onto #371, the current head was checked again on a later filesystem snapshot:

Default accounting:
dua                 6,370,499,072 bytes
/usr/bin/du         6,370,499,072 bytes

Counting hard links:
dua                 7,586,685,440 bytes
/usr/bin/du -l      7,586,685,440 bytes

The absolute totals changed with the live filesystem, but equality with the corresponding du mode remained exact. The rebased head also produced the same result with one thread and the default eight threads.

No automated real-NFS fixture is included. Unit tests protect the query, parsing, validation, fallback, and rounding rules; the OrbStack runs provide end-to-end evidence for the original issue.

Performance

A controlled benchmark performed before the rebase onto #371 compared v2.42.1 with the initial fix. Both binaries used Rust 1.95.0 and the same release/all-features configuration. After one warm-up per binary, each workload ran 12 times in alternating order:

Workload	v2.42.1 mean ± sd	Fix mean ± sd	Median, before → after
OrbStack NFS, ~140k entries and 13 errors	0.387 ± 0.014 s	0.403 ± 0.013 s	0.380 → 0.400 s
APFS fixture with 100,101 directories	1.790 ± 0.138 s	1.725 ± 0.162 s	1.820 → 1.755 s

The NFS workload was approximately 17 ms slower on average. The APFS difference was smaller than the observed run-to-run variation. These two workloads did not show a material regression, but they do not cover every filesystem, tree shape, or cache state.

This benchmark predates the rebase onto #371. The rebased head has passed correctness and APFS clone tests, but it has not been re-benchmarked against the current main.

Compatibility and limitations

• The code changes are limited to macOS; Linux and Windows paths are unchanged.
• The accounting contract applies to all macOS filesystems using the bulk path, not only NFS.
• Complete non-directory records retain the bulk fast path.
• Directories now use path metadata, matching Apple FTS; this may have different costs on directory-heavy workloads.
• No CLI, public API, or serialization changes are introduced relative to the current base.
• This PR does not address NFS server availability, retries, timeouts, or mount recovery.
• It does not change --apparent-size or attempt byte-for-byte parity with macOS du -A.
• The real-world validation covers OrbStack NFS and should not be treated as certification for every NFS server or mount configuration.

References

• PR #35: introduce real/apparent size accounting and the v2.41.1 Unix allocated-size call site
• PR #367: introduce the native macOS bulk walker, including its attribute request and getattrlistbulk call
• PR #369: reuse bulk metadata for aggregation roots
• PR #371: add optional APFS clone accounting
• Apple FTS: attribute request and bulk enumeration
• Apple FTS: vnode validation, fallback behavior, and st_blocks construction
• Apple du: FTS traversal and st_blocks aggregation
• Apple getattrlist(2) documentation