feat: add `ListLayout` by mhk197 · Pull Request #8071 · vortex-data/vortex

mhk197 · 2026-05-22T19:18:07Z

Adds ListLayout, a structured layout for DType::List columns that stores elements, offsets, and (when nullable) validity as independent child layouts, each with its own configurable strategy.

ListLayoutStrategy canonicalizes input to ListViewArray, rebuilds it via list_from_list_view, and writes each child concurrently.

ListReader fetches offsets and validity concurrently. Then uses offsets[0]..offsets[-1] to derive a bounded elements range, rebase the offsets to start at zero, fetch only that slice of elements, and assemble the ListArray. This means partial-range reads only fetch the elements they actually need rather than the whole elements buffer.

joseph-isaacs · 2026-05-28T14:20:17Z

+    session: VortexSession,
+    elements: LayoutReaderRef,
+    offsets: LayoutReaderRef,
+    validity: Option<LayoutReaderRef>,


Are we sure this needs a layout?

You can probably can by with this being a segment

joseph-isaacs · 2026-05-28T14:20:48Z

+            offsets,
+            validity,
+            ..
+        } = list_from_list_view(array.execute::<ListViewArray>(&mut exec_ctx)?)?.into_data_parts();


Why to execute to a ListArray?

joseph-isaacs · 2026-05-28T14:22:17Z

+/// Strategy for writing list-typed arrays.
+///
+/// Single-chunk only. The strategy:
+///  1. Canonicalizes the input chunk into a [`ListViewArray`].
+///  2. Calls [`list_from_list_view`] to rebuild it into zero-copy-to-list form
+///     (sorted, gapless, non-overlapping offsets) and produce a [`ListArray`].
+///  3. Writes the `elements`, `offsets`, and (when nullable) `validity` columns into
+///     separately configurable downstream strategies, producing a single [`ListLayout`].
+///


How does the sizing of the splits works, what is it based on?

We just talked through this. This is the layout that really highlights why row-batches suck... and really why engines should be pushing down unnest operations into the source. But... they don't.

For now, the best we can do is register the validity and offsets splits, so we just delegate to the child.

connortsui20 · 2026-05-29T09:09:34Z

Is there a tracking issue that explains why we do not want to store sizes as a (potentially optional) layout child?

It would be nice to see some sort of design doc (or if that already exists, please attach it in the PR description).

github-actions · 2026-05-29T09:16:44Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`fa581c4`		1	Explore Profiling Data

Previous Runs (3)

Status	Commit	Attempt	Link
🟢 Done	`e94f42c`	1	Explore Profiling Data
🟢 Done	`4256223`	2	Explore Profiling Data
🟢 Done	`4256223`	1	Explore Profiling Data

Powered by Polar Signals Cloud

connortsui20

Mostly nits, didn't get a chance to read through writer.rs though

connortsui20 · 2026-05-29T09:06:32Z

+    pub fn new(offsets_ptype: PType) -> Self {
+        let mut metadata = Self::default();
+        metadata.set_offsets_ptype(offsets_ptype);
+        metadata
+    }


This seems like a strange constructor? Why not just do Self { offsets_ptype }?

conversion from enum to i32 in prost needs this apparently, see DictLayout

connortsui20 · 2026-05-29T09:13:41Z

+            // Bound the elements read using offsets[0] and offsets[-1]
+            let elements_range = calculate_elements_range(&offsets, &session)?;
+
+            // Rebase the offsets so they start at zero


Nit: prose comments should end in periods (ask an llm to do this for you!).

connortsui20 · 2026-05-29T09:16:13Z

+            let array = if !mask.all_true() {
+                array.filter(mask)?
+            } else {
+                array
+            };


@joseph-isaacs I feel like we should have this optimization in the .filter method for arrays instead of constructing the filter array and then immediately optimizing it away?

Basically for the "built-in" array methods we might as well do the optimizations immediately.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

…hange Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 linked an issue May 27, 2026 that may be closed by this pull request

Epic: improve List and ListView #8051

Open

1 task

mhk197 removed a link to an issue May 27, 2026

Epic: improve List and ListView #8051

Open

1 task

mhk197 force-pushed the mk/add-list-layout branch from 2dafcdb to b267f99 Compare May 28, 2026 13:53

mhk197 added the changelog/feature A new feature label May 28, 2026

mhk197 linked an issue May 28, 2026 that may be closed by this pull request

Epic: improve List and ListView #8051

Open

1 task

mhk197 removed a link to an issue May 28, 2026

Epic: improve List and ListView #8051

Open

1 task

mhk197 marked this pull request as ready for review May 28, 2026 14:09

mhk197 requested review from connortsui20 and joseph-isaacs May 28, 2026 14:09

connortsui20 requested review from gatesn and robert3005 and removed request for connortsui20 May 28, 2026 14:14

joseph-isaacs reviewed May 28, 2026

View reviewed changes

Comment thread vortex-layout/src/layouts/list/reader.rs

joseph-isaacs reviewed May 28, 2026

View reviewed changes

joseph-isaacs added the action/benchmark Trigger full benchmarks to run on this PR label May 29, 2026

github-actions Bot removed the action/benchmark Trigger full benchmarks to run on this PR label May 29, 2026

connortsui20 reviewed May 29, 2026

View reviewed changes

github-actions Bot added the stale This PR is stale and will be auto-closed soon label Jun 13, 2026

mhk197 requested a review from a team June 17, 2026 00:26

mhk197 marked this pull request as draft June 17, 2026 00:27

mhk197 force-pushed the mk/add-list-layout branch 2 times, most recently from fcd1059 to e94f42c Compare June 17, 2026 04:24

mhk197 added the action/benchmark Trigger full benchmarks to run on this PR label Jun 18, 2026

github-actions Bot removed the action/benchmark Trigger full benchmarks to run on this PR label Jun 18, 2026

mhk197 added action/benchmark-sql Trigger SQL benchmarks to run on this PR and removed stale This PR is stale and will be auto-closed soon labels Jun 18, 2026

mhk197 added 29 commits June 26, 2026 14:45

fix writer

ac981aa

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix

3926e49

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

improve projection eval

d1b0803

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

projection evaluation

4fc585a

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

tests

18d5b06

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

tests

2d876a8

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix test

e020493

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

skip pruning eval

874a261

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

few more tests

76a540d

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

lint fix

f250c66

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix test

f74cab2

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

quick fix

8604883

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

add anylist matcher

1342c1d

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

read validity with all-true mask, not caller's mask

27eb870

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

narrow elements io for sparse mask instead of defering filtering

729334c

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

shortcut on whole-chunk unmasked reads

aa356f8

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

add required fallback to ListLayoutStrategy for non-list input

262a5be

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fmt

850510f

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

cleanup

8b34aa9

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix rebase conflicts

3f71c73

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

use ListLayoutStrategy as default leaf under unstable_encodings

cd031a7

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

recurse into nested lists

29f16cf

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix: update ListLayout::build to use LayoutBuildContext after trait c…

2133967

…hange Signed-off-by: Matt Katz <mhkatz97@gmail.com>

comments

020bcb4

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix

ab45a9e

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

clean up writer

24fa450

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Improve list reader

3fc6eb6

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Fix list docs

39059cc

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Add list filter evaluation

af54f25

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 force-pushed the mk/add-list-layout branch from fa581c4 to af54f25 Compare June 29, 2026 15:31

Uh oh!

Conversation

mhk197 commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

connortsui20 commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

connortsui20 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mhk197 commented May 22, 2026 •

edited

Loading

connortsui20 commented May 29, 2026 •

edited

Loading

github-actions Bot commented May 29, 2026 •

edited

Loading