Enhancing ctable with a new utf8() string type - #677
Merged
Conversation
New first-class column type for high-cardinality/free-text strings: blosc2.utf8(nullable=..., null_value=...) stores each column as two companion NDArrays (int64 row offsets plus a uint8 UTF-8 byte blob), so a row costs exactly its encoded byte length. Bulk reads materialize as numpy StringDType arrays (requires NumPy >= 2.0; a clear error is raised otherwise). This first slice covers storage and roundtrip: - Utf8Spec / utf8() in schema.py, kind "utf8" in the schema compiler. - Utf8Array adapter (new utf8_array.py) with append/extend/flush, pending-aware reads, cluster-based sparse gathers, and O(n - i) in-place cell rewrite (offsets shift on length change). - Storage dispatch in all four TableStorage backends; the byte blob lives at the column key plus a ".utf8" suffix, unreachable from any user column name. delete/rename move both leaves. - Sentinel-based nulls consistent with other scalar columns (policy string_value by default), including is_null/null_count/fillna. - Persistence roundtrips: .b2z/.b2d save/open/load, mmap open, to_cframe/ctable_from_cframe, TreeStore bundling, copy/take/compact. - Comparisons, where(), groupby keys, sorting, and Arrow export raise clear NotImplementedError/TypeError messages for now. vlstring()/string() behavior is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…port CTable.to_arrow()/iter_arrow_batches() export utf8 columns as pa.large_string() (always large — no int32-offset special-casing), building the Arrow null mask from the column's sentinel. CTable.from_arrow()/from_parquet() now map incoming Arrow string/ large_string columns to blosc2.utf8() by default (previously vlstring). This is the sentinel-vs-mask checkpoint from the phase-3 plan: nulls map sentinel<->validity like every other scalar dtype, consistent with int/float/bool/timestamp/fixed-string columns, and column_null_values overrides now work for utf8 (rejected before, since vlstring nulls are native None). binary/large_binary columns are unaffected and still import as vlbytes. No lossiness observed against the tables exercised here, so P5's mask-based null design stays parked. Updated interop tests and the parquet_to_blosc2 CLI's progress-report labels/docstrings, which independently predicted "vlstring" for reporting purposes, to reflect the new utf8 default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…cates
Column.__eq__/__ne__/__lt__/__le__/__gt__/__ge__ now special-case utf8
columns (mirroring the existing dictionary-column special-case),
comparing against a str scalar or another utf8 Column chunk by chunk
in NumPy (StringDType supports ==, !=, and ordering natively) and
returning a physical-length boolean predicate intersected with the
column's live-row mask -- usable directly in t[t.name == "x"] or
t.where(...).
A null value on either side never satisfies any comparison (the same
SQL WHERE rule other nullable columns already follow), via a new
Column._utf8_null_pred() OR'd across operands. Arithmetic and bitwise
operators on utf8 columns still raise NotImplementedError; only
comparisons are implemented here.
blosc2.startswith/endswith already worked against a utf8 Column
operand unmodified -- pinned with a test, no code change needed.
String-expression predicates (t.where("name == 'x'")) still raise,
since numexpr/miniexpr cannot evaluate StringDType.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ap recorded) CTableGroupBy.__init__ no longer rejects utf8 key columns (vlstring/ vlbytes/struct/object/list keys are still rejected). Two small supporting fixes make the existing generic np.unique-based factorization path correct for utf8 keys: - _read_key_chunk pads a chunk read past a utf8 column's logical length with "" (mirroring the P3.a iter_chunks fix) since Utf8Array is sized to the logical row count, not the physical valid_rows capacity. - _factorize_keys casts StringDType key arrays to object dtype before packing into the structured array used for multi-key np.unique, since NumPy structured dtypes reject StringDType fields outright. Everything else needed no changes: StringDType's .kind is "T", so it already falls through the existing fixed-width-string fast path into plain np.unique(); null-sentinel handling, key-column spec propagation to the result table, Python-scalar coercion, sort ordering, and every Cython fast path (which all bail on dtype=None) all worked unmodified. Benchmarked against dictionary-key groupby (bench/ctable/ bench_groupby_keys.py, 1e7 rows, 5-city low-cardinality keys): utf8 is ~17x slower than dict for a single key and ~50x for two keys, well past the 3x target. Profiling isolated the cause to Utf8Array's per-row Python decode loop (a P3.a artifact shared by every bulk utf8 read, not specific to groupby) -- the per-row loop overhead dominates, not the decode call itself, so no small fix closes the gap. A real fix needs a new vectorized byte-hash factorization that never decodes to str for the N-row hot path; that's sized like its own follow-up item, so per the benchmark-gate rule this lands the correct, slower fallback and records the gap instead of merging an unproven fast path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sort_by() now accepts utf8 sort keys. The rejection in
_normalise_sort_keys turned out to be unnecessary rather than merely
lifted: a utf8 column's dtype already resolves to StringDType (not
None), so the existing dtype-is-None rejection branch already skipped
it, and np.issubdtype(StringDType(), np.complexfloating) returns False
cleanly. Two real fixes were needed:
- _build_lex_keys's descending-sort rank-inversion check widened from
"USO" to "USOT" (StringDType's .kind is "T") -- strings can't be
negated with unary minus for a descending lexsort key. Ascending
sort and the null-indicator key needed no changes: np.lexsort/
np.argsort and == already work natively on StringDType arrays.
- _sort_by_inplace and _sorted_copy_from_positions gained a utf8
branch that rebuilds the column via Utf8Array.extend() instead of
bulk slice-assignment, mirroring the existing list-column branch,
since Utf8Array has no bulk __setitem__.
Found and left alone: sorting a table containing a vlstring/vlbytes/
struct/object column already raised before this change (even when
sorting by an unrelated key), because the generic sort-copy fallback
assumes bulk slice-assignment that _ScalarVarLenArray doesn't support.
This predates the utf8 work and is out of scope for a utf8-only phase.
Manually verified the plan's three Goal-section lines run correctly
end-to-end: t[t.name == "Paris"], t.group_by("name").sum("x"), and
t.sort_by("name"). Full test suite (7477 tests) passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ct, fused null masks, buffer-level Arrow export The critical fix: sort_by(inplace=True) and compact() on a file-backed table rebuilt utf8 columns as fresh in-memory Utf8Arrays and only rebound the column reference, so the rewritten rows never reached the store — after close/reopen the column was corrupted and misaligned with its on-disk-sorted siblings. Utf8Array.set_all() now bulk-rewrites through the existing backing offsets/data arrays; compact() also gathers live rows with one clustered fancy-index read instead of two chunk reads per row. Regression tests cover both paths across .b2z/.b2d reopens. Also: - utf8 comparisons compute the null-sentinel mask inside the same chunk pass as the predicate (was: up to four full column scans per nullable filter); comparison dunders pass numpy ufuncs directly instead of string tags. ~2x on a 1M-row nullable filter (419 -> 203 ms). - Arrow export of dense root tables builds pa.LargeStringArray straight from the offsets/bytes buffers (Utf8Array.arrow_slice) with the null mask matched on raw bytes — no per-row decode, no tolist(), no re-encode (3030 -> 1730 ms on 1M rows). Views and tables with deleted rows use a materializing fallback that reuses Column.null_value, _null_mask_for, and _pa_type_from_spec instead of inlining them. - Utf8Array.__setitem__ shifts the raw tail bytes and adds a scalar delta to the tail offsets instead of decoding and re-encoding every following row (21 ms to overwrite row 100 of a 1M-row column). - Utf8Spec storage dispatch imports Utf8Spec once at ctable_storage module top (was: six local imports). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the np.unique-on-StringDType fallback (recorded at 16.9x/49.9x vs dictionary keys) with a factorization that never decodes N rows: - Utf8Array.factorize_span()/Utf8Factorizer group rows by raw byte length, gather each length group column-wise into a (k, L) byte matrix (one reused index vector — ~2x faster than a 2-D fancy-index), hash rows with the same collision-checked mixer as _factorize_fixed_width_str, and decode only the D distinct values. The factorizer keeps a cross-chunk vocabulary (sorted hashes plus representative bytes per length), so rows carrying already-seen values are searchsorted-matched and byte-verified instead of re-sorted. - utf8 key chunks flow through the groupby pipeline as _Utf8KeyChunk (dense int64 string-rank codes + sorted uniques): null masks, live-row masking, and dedup run on integers; single-key dedup is an O(n) bincount. Multi-key packing combines all-integral keys into one composite int64 (Horner over zero-based fields) — np.unique over a structured dtype does field-wise void comparisons and dominated the two-key time; non-integral co-keys keep the structured fallback. - The generic groupby loop batches at ~1 Mi rows instead of the raw validity chunk shape: in-memory tables grown by resize keep their tiny initial 64-row chunks, so the loop ran 156k batches of bookkeeping at 1e7 rows. This helps every generic-path key type. Benchmark (bench_groupby_keys.py, 1e7 rows, 5-city keys, Apple silicon): utf8 key sum: 3304 -> 558 ms, now 2.83x slower than dict keys (gate: <= 3x); two keys int+utf8: 11825 -> 878 ms, now 3.69x slower than dict keys (was 49.9x). utf8 keys are now faster than fixed-width string keys (535 ms). The byte-exact factorization also fixes a correctness gap inherited from np.unique: NumPy merges StringDType values differing only after an embedded NUL, so such keys used to collapse into one group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Benchmarks the three plain-string representations on the real Chicago-taxi company column (1e7 rows, low cardinality) and synthetic high-cardinality free text (2e6 rows, 0-129 chars, multi-byte): ingest, storage footprint, full read, equality filter, groupby key, sort, and Arrow export. Unsupported operations are reported, not silently skipped. Measured numbers and their reading are recorded in the phase-3 plan: utf8 matches vlstring's storage while making filters/groupby/sort work at all, and beats fixed-width string on footprint (7-13x smaller uncompressed), groupby keys, and Arrow export; fixed-width keeps winning raw reads/filters/sorts via the vectorized fast paths. The plan also records the natural follow-up: routing comparisons through Utf8Factorizer codes to close the filter gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…skip utf8 tests utf8 columns require numpy.dtypes.StringDType (NumPy >= 2.0), but the Arrow/Parquet import default had started routing every scalar string column to blosc2.utf8(), which raised on older NumPy — a functional regression for the minimum supported version, and test_utf8.py failed at collection there (module-level StringDType() call). - New utf8_array.have_string_dtype() helper. - from_arrow/from_parquet: on NumPy < 2.0, scalar string columns rejoin the varlen-scalar group and import as vlstring with native-None nulls — the exact pre-utf8 behavior (no auto null sentinel is derived, and a column_null_values override is rejected with the standard varlen message, now reading "vlbytes/vlstring"). - tests/ctable/test_utf8.py skips at module level on NumPy < 2.0; the Arrow/Parquet/CLI tests that assert utf8-specific behavior now assert the vlstring fallback on old NumPy instead of being skipped, keeping interop coverage on the minimum-NumPy CI job. Verified by simulating NumPy-without-StringDType in-process (delattr before import): the four affected test files pass there (204 passed, utf8 module skipped), and the full tests/ctable suite stays green on NumPy 2.x. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route ==, !=, <, <=, >, >= against a str scalar through a pure-NumPy byte-level comparison on Utf8Array's raw offsets/bytes, skipping the per-row decode-to-StringDType loop entirely for filters. Column-vs-Column comparisons keep the existing materializing path. Measured on the 1e7-row taxi company column: s == value drops from ~1900ms to 162ms, ordering ops land at ~367ms (both within plan gates). Also refactors arrow_slice's inline sentinel-null matcher to share the new equal_mask_span helper instead of duplicating the algorithm.
Add a Cython kernel (utf8_ext.pyx, pack_utf8_span) that uses NumPy's StringDType C API (NpyString_pack) to bulk-fill a StringDType array directly from offsets+bytes, replacing the per-row decode loop in Utf8Array._read_persisted_span. Falls back to the old Python loop when the compiled extension is unavailable (lazy _pack_utf8_kernel() import, mirroring the existing groupby_ext pattern); tests cover both paths via a monkeypatch fixture. Two more fixes were needed to actually realize the gain in the common read path: Column's identity-slice fast path excluded utf8 alongside other varlen-scalar kinds even though Utf8Array slices itself efficiently, and Utf8Array._get_many always paid a full StringDType scatter-copy (sort + fancy-index assignment) even for a plain contiguous range, which alone ate most of the kernel's benefit. Measured on the 1e7-row taxi company column: full column read drops from 2472.6ms to 165.3ms (gate was <=500ms), no regressions elsewhere.
Utf8Spec was missing from the create_index rejection list, so indexing a utf8 column failed deep inside with a misleading "indexes are only supported on NDArray" TypeError. Now it raises the same style of clear NotImplementedError that vlstring/vlbytes/struct/object columns get, pointing users to fixed-width string() when an index is needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New "Choosing a string column type" section in the CTable reference: decision path plus a comparison table covering storage layout, per-row cost, filters, groupby/sort, indexing, Arrow interop, and NumPy requirements. utf8 limitations (no string-expression filters, no create_index yet) are footnoted; utf8 added to the Text & binary listing and excluded from the unclassified members page. - Cross-link string()/utf8()/vlstring() docstrings to that section and state each spec's tradeoffs, including utf8's current limitations. - New examples/ctable/utf8_strings.py showing the full utf8 surface: StringDType bulk reads, operator filters, startswith, sentinel nulls, groupby keys, sorting, and Arrow round-trip. - Refresh stale docs that predate the utf8 import default: the parquet guide and arrow_interop.py said Arrow/Parquet strings import as vlstring/fixed-width; they now import as utf8 on NumPy >= 2.0. - Use utf8() for the text column in the getting-started CTable schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… pxds The NpyString C API has been in the NumPy headers since 2.0, but its Cython declarations only appear in the numpy/__init__.pxd of newer NumPy versions. The Pyodide cross-build pins an older NumPy (2.1's pxd has none of them), so cythonizing utf8_ext.pyx failed on wasm32 with "'npy_string_allocator' is not a type identifier". Declare the four symbols locally instead of via `cimport numpy`; they resolve through the API table populated by cnp.import_array(), so no behavior changes. Verified by reproducing the failure with numpy 2.1.3 + Cython 3.2.8, then cythonizing, compiling, and running the kernel in that same env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines
+86
to
+100
| cdef Py_ssize_t i | ||
| cdef int64_t start, length | ||
| cdef int ret = 0 | ||
| try: | ||
| for i in range(n): | ||
| start = rel_ptr[i] | ||
| length = rel_ptr[i + 1] - start | ||
| ret = NpyString_pack( | ||
| allocator, | ||
| <npy_packed_static_string*>(out_data + i * itemsize), | ||
| <const char*>(data_ptr + start), | ||
| <size_t>length, | ||
| ) | ||
| if ret == -1: | ||
| break |
Comment on lines
+621
to
+632
| Successive :meth:`codes_for_span` calls share a vocabulary: each row is | ||
| hashed from its raw bytes and matched against the known values of its | ||
| byte length (``np.searchsorted`` on sorted hashes plus a vectorized | ||
| byte-equality verify), so only rows carrying *new* values pay a sort | ||
| (via :func:`_factorize_byte_rows`). Codes are global across spans, in | ||
| first-appearance order; :meth:`uniques` returns the decoded vocabulary in | ||
| that same order. No row is ever decoded — only the distinct values. | ||
|
|
||
| A hash collision between a new value and a known one is not resolved | ||
| (the new value gets a fresh code); the resulting duplicate vocabulary | ||
| entry is benign because callers merge groups by decoded value, and with | ||
| 64-bit hashes the case is vanishingly rare anyway. |
…atch (I1) extend() now bulk-appends plain-str chunks instead of a per-row Python loop, and _rewrite_from() joins+encodes an all-ASCII batch in one call instead of encoding each row individually. Raising _FLUSH_ROWS (4096 -> 65536) amortizes each flush's NDArray resize/write over far more rows. Combined, taxi-company ingest drops from 3622ms to 870ms, now faster than both string() and vlstring() on the same workload. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rrow steps to_arrow() alone takes 40-90s per column kind on the full taxi workload, dwarfing the ~1s ingest step it's meant to isolate. --ingest-only skips straight to the numbers that matter when iterating on ingest performance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds encode_utf8_span to utf8_ext.pyx: encodes a list of str to concatenated UTF-8 bytes + per-row lengths in a GIL-held C loop using each string's cached UTF-8 representation (PyUnicode_AsUTF8AndSize), avoiding both the per-row Python .encode() calls and the intermediate bytes-list/join allocation that the I1 pure-Python fast path still pays. _rewrite_from tries the kernel first, falling back to I1's join+encode path when the compiled extension is unavailable. Taxi-company ingest: 870.5ms (post-I1) -> 598.6ms, now within ~22% of string() and clearly ahead of vlstring(). Verified full test suite green both with the kernel active and with utf8_ext entirely absent from the environment (falls all the way back to pure Python). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Consistent with how these numbers are now reported elsewhere: 6.05x total (3622ms -> 598.6ms), 1.22x slower than string(), 2.16x faster than vlstring(). Proportions that aren't speedups (time-share breakdown, a data-loss rate) are left as percentages. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- pack_utf8_span (utf8_ext.pyx): validate rel is well-formed (rel[0]==0, non-decreasing, within data's bounds) before the unchecked C loop, instead of trusting the caller silently -- a malformed rel previously risked an out-of-bounds read/crash. - Utf8Factorizer docstring: stop claiming a hash collision's duplicate vocabulary entry is "benign because callers merge groups by decoded value" -- no downstream consumer does that merge, so a collision would actually split a group in two. Still an accepted, vanishingly-rare risk with 64-bit hashes, just not a resolved one. - test_utf8.py: two groupby tests assigned float64 values into the int64 `x` column via implicit cast; use integer inputs explicitly so the test doesn't rely on that coercion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A new schema spec blosc2.utf8() stores strings Arrow-style as two companion NDArrays — int64 row offsets plus a UTF-8 byte blob — so each row costs exactly its encoded length (vs. fixed-width string()'s 4 bytes/char × max-length on every row), and bulk reads materialize as NumPy StringDType arrays. Unlike vlstring() (msgpack cells), utf8 columns are fully operational: