Add pco-inspired lightweight numeric schemes - #9544
Conversation
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Merging this PR will regress 1 benchmark
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -3.8% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.016x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.982x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
File Size Changes (17 files changed, -45.4% overall, 2↑ 15↓)
Totals:
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.975x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 2↑ 2↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -46.5% overall, 0↑ 3↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -44.6% overall, 0↑ 201↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 3↑ 6↓)
duckdb / parquet / ns (0.999x ➖, 3↑ 5↓)
File Size Changes (49 files changed, -45.6% overall, 11↑ 38↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.981x ➖, 1↑ 0↓)
File Size Changes (17 files changed, -45.4% overall, 2↑ 15↓)
Totals:
|
|
please don't add them to the core edition yet |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.987x ➖, 6↑ 3↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -49.5% overall, 0↑ 3↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.048x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.144x ➖, 2↑ 2↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.002x ➖, 2↑ 2↓)
File Size Changes (201 files changed, -41.6% overall, 0↑ 201↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.945x ➖, 3↑ 0↓)
datafusion / parquet / ns (1.034x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.061x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.018x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
random-access / parquet / ns (1.003x ➖, 0↑ 0↓)
random-access / lance / ns (0.993x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.942x ➖, 5↑ 0↓)
vortex / vortex-file-compressed / bytes (0.954x ➖, 4↑ 0↓)
vortex / vortex-file-compressed / ratio (0.946x ➖, 9↑ 0↓)
vortex / parquet / ns (0.999x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
|
Signed-off-by: Will Manning <will@willmanning.io>
DCO Remediation Commit for Will Manning <will@willmanning.io> I, Will Manning <will@willmanning.io>, hereby add my Signed-off-by to this commit: 3e070a6 I, Will Manning <will@willmanning.io>, hereby add my Signed-off-by to this commit: 5786fd7 I, Will Manning <will@willmanning.io>, hereby add my Signed-off-by to this commit: a20651a I, Will Manning <will@willmanning.io>, hereby add my Signed-off-by to this commit: 0f79c92 Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Summary
Add native
BlockResidual,OrderedFloat, andFloatQuantarrays for numeric data that Pco compresses well but ALP and ALP-RD do not. These arrays retain fast scalar access and use composable Vortex children. The selectors use cheap sample analysis and calibrated access-cost factors. The new arrays belong to thepreview:2026.08.0edition, which writers enable through theunstable_encodingsfeature.Changes
Results
The final benchmark compared 17 datasets with five timed iterations per operation. The corpus included GloVe
f32and OpenAI-on-C4f64embeddings.