fix(reader): resolve sparse Utf8/Binary rows lazily, and stop dropping the fill

[dfa1]

Closes #340.

Started as the allocation fix #340 describes and grew one commit, because the eager merge was also hiding a correctness bug.

The bug

SparseEncodingDecoder.decodeVarBin was handed only fill.is_valid() — a boolean — never the fill scalar itself. Unpatched rows were written into the merged offsets table as zero-length ranges, so every unpatched row of a sparse Utf8/Binary column read back as the empty string instead of the fill. A vortex.sparse utf8 column with a non-null string fill decoded wrong; only its patched rows were ever right.

The Rust SparseArray resolves an unpatched row to the fill value whatever the values encoding is, exactly as this decoder's own primitive path already did. The existing test used fill "x" and asserted only a patched row, so it never saw it.

The change

Two commits:

1. e692269f — a patch-free range stops allocating an (n + 1) all-zeros offsets table to say "every row is the fill"; VarBinConstantArray (added in ConstantEncodingDecoder.decodeString should be lazy, not eager #329) says it in O(1). This is SparseEncodingDecoder allocates an (n+1) offsets table for a zero-patch VarBin column #340 as filed.
2. b72028b7 — VarBinSparseArray, the VarBin member of the LazySparseXxxArray family, resolving row → patch (or the fill) on access over the same SparseArrays.findPatch/walkPatches helpers the primitive variants use. The whole merge goes away: the totalBytes pre-pass, the byte copies, the offsets table, the readVarBinOffset/readUnsignedIdx helpers, and the boundary try/catch that existed only to cover them. Fixing the dropped fill falls out of it — the carrier holds the fill bytes.

Root cause is the same missing-carrier gap #329 closed for vortex.constant and 28cc4f7 for vortex.runend: every primitive type had a lazy sparse carrier, VarBin had none, so decode had nothing lazy to return.

bytesSegment()/segmentIfPresent() follow the "no single contiguous buffer" convention of VarBinChunkedArray, VarBinRunEndArray and VarBinConstantArray, so generic consumers still flatten via VarBinArray.toOffsetMode. That drops a second materialization the old path forced: it called toOffsetMode on the values child, which allocates whenever the pool is view-, dict-, or chunk-backed.

Adversarial tests

Three move from decode time to read time, which is worth reviewing:

test	before	after
non-monotonic value offsets	VortexException at decode	VortexException on the row that uses the bad pair
value offsets past the payload	VortexException at decode	VortexException on the row that uses the bad pair
patch count beyond the offsets child	VortexException at decode	no longer raises

The third is the one to look at. VarBinEncodingDecoder broadcast-fills a short offsets child (i % cap), so the read wraps to a valid pair and stays inside the payload. The old exception was incidental — it came from overrunning a merge buffer that had been sized from the same broken offsets, not from detecting anything. The test now asserts what ADR 0003 actually requires there: no raw JDK exception, no read outside the value buffer. Whether VarBinEncodingDecoder should reject a truncated (as opposed to single-element/constant) offsets child at all is a separate question, not touched here.

Tests

• VarBinSparseArrayTest — new, covering accessors, offset rebasing, the fill copy contract, forEachByteLength vs. per-row search agreement, unsorted indices, toOffsetMode, limited.
• SparseEncodingDecoderTest — the dropped-fill regression (utf8 and binary fills, since they arrive as string_value vs. bytes_value), forEachByteLength, the zero-patch carrier, plus the three adversarial updates.
• VarBinArrayTest — the zero-length constant corner SparseEncodingDecoder allocates an (n+1) offsets table for a zero-patch VarBin column #340 flagged as possibly uncovered; it was.

./mvnw verify green across all 17 modules, integration included (RustWritesJavaReadsIntegrationTest, NullSparseRunEndInteropIntegrationTest).

🤖 Generated with Claude Code

[dfa1]

Review pass applied in 2df5f36. Three of the findings were real:

Unvalidated offset — ProtoPatchesMetadata.offset was the one field never range-checked. Every lazy sparse carrier maps row i to absolute i + offset and the walkers iterate [offset, offset + n), so an offset near Long.MAX_VALUE wrapped that end bound negative: forEachByteLength visited no rows while the per-row binary search still resolved each one — the two accessors disagreeing on the same array. Now guarded next to the patch-count check, which covers the whole LazySparseXxxArray family since they share SparseArrays.walkPatches.

fillBytes conflated "null fill" with "mistyped fill" — an int64_value fill on a Utf8 column is non-null (fillValid == true) but has no bytes, so it fell through to new byte[0] and rendered every unpatched row as a valid "". That is the same shape of wrong answer this PR set out to remove. Now a VortexException, matching ConstantEncodingDecoder on the same mismatch.

Missing ground-truth cover — added. A JNI-written (Rust reference) mostly-null Utf8 column decodes through VarBinSparseArray, asserted by carrier class and not only by values, since the eager and lazy paths agree on every value here. I probed the pinned compressor for the non-null fill half and could not reach it: mostly-one-value → vortex.dict, mostly-empty-string → vortex.fsst over vortex.runend. That half stays unit-covered, and the blocker is recorded in the test javadoc rather than left silent.

Also swept the offset axis in the walk-vs-search agreement test (offsets 0–3, including one that puts a patch behind the window), listed VarBinSparseArray/VarBinRunEndArray in VarBinArray's implementations javadoc, corrected the compatibility table's sparse dtype column (decode covers Primitive/Bool/Utf8/Binary; encode Primitive only), and made varBinPatchCountBeyondValueOffsets say plainly in its doc that it is a deliberate weakening from "raises" to "wrong but safe", with the real detection belonging in VarBinEncodingDecoder.

Not taken: extracting the fillBytes/ConstantEncodingDecoder.decodeString duplication into a shared helper, and the per-row new String(fill, UTF_8) allocation on fill-heavy scans — both are pre-existing and worth their own change rather than widening this one.

./mvnw verify green across all 17 modules.