FastLanes delta padding could inflate max_minus_min span, skipping near-monotone columns

[rapour]

What happened?

When an integer column length is not a multiple of FL_CHUNK_SIZE (1024), delta_compress pads the trailing partial chunk to 1024 elements with T::default() (zero).

If zero-padding places a real value and a zero in the same lane's delta chain, the step 0 - V wraps to ~2^T, inflating the delta span that DeltaScheme::expected_compression_ratio measures via max_minus_min().

Every n % 1024 != 0 gets a padded chunk, but span inflation only occurs when the remainder is also not aligned to the element bit width T:

n % 1024 != 0 -> trailing chunk is padded -> (n % 1024) % T != 0 -> a lane sees a real value to zero step

For u32 (T = 32, LANES = 32), the second condition is n % 32 != 0.

Impact

Near-monotone integer columns with unaligned lengths can be incorrectly rejected by DeltaScheme and BtrBlocks falls back to vortex.sequence (or another scheme) even when delta + bitpacking would win.

Steps to reproduce

You can run this snippet with vortex and vortex-btrblocks as dependencies:

use vortex::VortexSessionDefault;
use vortex::array::{IntoArray, VortexSessionExecute, arrays::PrimitiveArray};
use vortex::array::validity::Validity;
use vortex::buffer::Buffer;
use vortex::compressor::BtrBlocksCompressor;
use vortex::encodings::fastlanes::{FL_CHUNK_SIZE, delta_compress};
use vortex::session::VortexSession;
use vortex_btrblocks::ArrayAndStats;
use vortex_btrblocks::GenerateStatsOptions;

// Same constants as vortex_btrblocks::schemes::integer::delta
const DELTA_PENALTY: f64 = 0.95;
const DELTA_MIN_RATIO: f64 = 1.25;

fn main() -> vortex::error::VortexResult<()> {
    let session = VortexSession::default();
    let mut ctx = session.create_execution_ctx();

    println!("=== DeltaScheme estimator path (max_minus_min) ===\n");

    println!("n     pad  n%32  span          ratio   verdict");
    for n in [1024, 1025, 1056, 2048, 2049] {
        let values: Buffer<u32> = (0..n as u32).map(|i| 1000 + i).collect();
        let array = PrimitiveArray::new(values, Validity::NonNullable);
        let full_width = array.ptype().bit_width() as f64;
        let (_bases, deltas) = delta_compress(&array, &mut ctx)?;
        let stats = ArrayAndStats::new(deltas.into_array(), GenerateStatsOptions::default());
        let span = stats.integer_stats(&mut ctx).erased().max_minus_min();
        let (ratio, verdict) = match span.checked_ilog2() {
            Some(l) => {
                let bits = (l + 1) as f64;
                let ratio = full_width / bits * DELTA_PENALTY;
                let verdict = if ratio <= DELTA_MIN_RATIO { "skip" } else { "select" };
                (ratio, verdict)
            }
            None => (0.0, "skip"),
        };
        println!(
            "{n:<5} {:>4} {:>4}  {span:<12}  {ratio:.2}   {verdict}",
            n % FL_CHUNK_SIZE,
            n % 32,
        );
    }

    println!("=== BtrBlocks encoding tree (default compressor) ===\n");

    for n in [1025usize, 1056] {
        let values: Buffer<u32> = (0..n as u32).map(|i| 1000 + i).collect();
        let array = PrimitiveArray::new(values, Validity::NonNullable).into_array();

        let compressed = BtrBlocksCompressor::default().compress(&array, &mut ctx)?;
        let tree = compressed.display_tree_encodings_only().to_string();
        let uses_delta = tree.contains("fastlanes.delta");

        println!("n={n}: uses fastlanes.delta = {uses_delta}");
        println!("{tree}\n");
    }

    Ok(())
}

=== DeltaScheme estimator path (max_minus_min) ===

n     pad  n%32  span          ratio   verdict
1024     0    0  1             30.40   select
1025     1    1  4294965272    0.95   skip
1056    32    0  1             30.40   select
2048     0    0  1             30.40   select
2049     1    1  4294964248    0.95   skip
=== BtrBlocks encoding tree (default compressor) ===

n=1025: uses fastlanes.delta = false
root: vortex.sequence(u32, len=1025)


n=1056: uses fastlanes.delta = false
root: vortex.sequence(u32, len=1056)

Environment

• vortex = { version = "0.85", features = ["unstable_encodings"] }
• vortex-btrblocks = { version = "0.85", features = ["unstable_encodings"] }

Additional context

No response