feat: implement postgres style table partitioning by adsharma · Pull Request #776 · LadybugDB/ladybug

adsharma · 2026-08-04T02:03:54Z

Idea: store partitioned tables in subgraphs which could live on object storage and/or accessed via a columnar network protocol.

-- Hash partitioning on an eligible column.
CREATE NODE TABLE Orders (
    id     INT64  PRIMARY KEY,
    region STRING,
    amount INT64
) PARTITION BY HASH (region) PARTITIONS 4;

-- Range partitioning (bounds are derived; see design below).
CREATE NODE TABLE Events (
    id    INT64 PRIMARY KEY,
    ts    TIMESTAMP,
    value DOUBLE
) PARTITION BY RANGE (ts) PARTITIONS 5;

Each partition lives in a separate subgraph which can be dropped independently.

… storage Add declarative RANGE/HASH partitioning to CREATE NODE TABLE: CREATE NODE TABLE t (...) PARTITION BY (HASH|RANGE) (col) PARTITIONS n; Each partition is backed by its own node-table subgraph (<t>_p<i>); the logical parent owns the schema but no physical storage. Reads on the parent (MATCH) transparently union over every partition via the existing multi-table node scan. Writes to the parent (COPY/CREATE/MERGE) raise an actionable error until routing lands; the partition subgraphs are directly writable. - Parser: PARTITION BY clause + new keywords (regenerated grammar) - Catalog: parent + partition subgraph entries, persistence (storage v44), DROP cascade - Storage: parents skipped in create/checkpoint/serialize/rollback; partitions managed as ordinary node tables - Query: expand parent label to partition subgraphs for scanning - Validation: partition column must be an eligible existing column - Tests: ddl/partitioned.test (5 cases) pass - Docs: docs/partitioning.md with architecture and roadmap (write routing, pruning, ADBC remote partitions)

…r node table CREATE, COPY and MERGE into a partitioned parent now evaluate each row's partition key and route it into the matching partition subgraph (hash(value) % numPartitions, used for HASH and, until declarative range bounds land, RANGE partitions). Single-row inserts route at runtime in NodeInsertExecutor; batched COPY routes consecutive same-partition runs in NodeBatchInsert. Each child is an ordinary NodeTable, so primary-key uniqueness and WAL/MVCC apply per partition. Every node table is now backed by a subgraph (a GraphCatalogEntry in the catalog's `graphs` set), so SHOW_GRAPHS lists node tables and their partition subgraphs. Subgraphs are created and dropped with their table (creation skips WAL logging since the table's create record implies them), follow ALTER TABLE ... RENAME, and DROP GRAPH refuses a node-table subgraph. Adds NodePartitionWriteInfo routing metadata carried from the binder through NodeBatchInsertInfo, updates docs/partitioning.md, and extends the partitioned e2e test with write-routing and show_graphs coverage.

…oncerns Hardens the partitioned-tables feature against the scenarios raised in design review: (a) Partition/subgraph coupling can no longer be broken by accident: - DROP GRAPH <name> now refuses any node-table subgraph with an actionable message (previously it failed with a confusing 'does not exist' even though SHOW_GRAPHS lists it). - DROP TABLE <partition> is refused; dropping one partition would leave the parent holding a dangling childTableID, breaking every parent scan. - ALTER TABLE ... RENAME on the parent renames its <parent>_p<i> partitions (and their subgraphs) along with it. Child renames are not separately WAL-logged: replaying the parent's rename record re-runs the child-rename step, so WAL replay and rollback both stay consistent. (b) Other tables coupled to partitions are refused, never silently detached: - CREATE REL TABLE FROM <partitioned parent> TO x now expands to one FROM-TO pair per partition, so rels attach to real storage instead of the storage-less parent (which silently could never hold a rel). - Dropping a partitioned parent is refused while any rel still references any partition: the existing per-table reference checks now run for every table the cascade would remove, before anything is dropped. - Rel-pattern writes against the parent get an error pointing at the concrete-partition workaround. (c) Updating a partition column is refused at bind time (SET and MERGE ... ON MATCH SET, via parent or partition). Rows cannot move between partitions; leaving a row in a partition that no longer matches its key would break pruning and direct-partition scans. The error gives the delete + re-insert workaround, matching the primary-key restriction. (d) Complexity: storage iteration skips storage-less parents through one shared erasePartitionedParents helper; ALTER on partitioned parents is limited to RENAME/COMMENT with a clean binder error (ADD PROPERTY used to die with 'unordered_map::at'), and partitions cannot be altered directly. docs/partitioning.md gains a Design notes section answering each review concern plus updated limitations/roadmap. Tests: five new e2e cases in ddl/partitioned.test (drop protection, alter restrictions, partition-key update refusal, rel-table expansion and drop blocking).

…air cost CREATE REL TABLE R (FROM Orders TO Orders) on a partitioned parent expands to the full (p_i, p_j) cross product, so edges within a partition and across partitions are both expressible and visible through parent-pattern reads; DETACH DELETE removes cross-partition edges in both directions. Adds an e2e case pinning this down and a design-notes paragraph, including the caveat that a self-rel materializes n^2 rel tables (keep partition counts modest on self-rel tables until pair creation can be lazy).

adsharma force-pushed the partioned_tables branch 3 times, most recently from 9b71128 to c5d4bae Compare August 4, 2026 02:07

adsharma force-pushed the partioned_tables branch 2 times, most recently from 815f03c to f771009 Compare August 18, 2026 19:12

adsharma force-pushed the partioned_tables branch from f771009 to a408ead Compare August 18, 2026 20:45

adsharma added 3 commits August 18, 2026 14:28

adsharma merged commit fcadf85 into main Aug 21, 2026
4 checks passed

adsharma deleted the partioned_tables branch August 21, 2026 20:29

Navigation Menu

Uh oh!

feat: implement postgres style table partitioning - #776

feat: implement postgres style table partitioning#776
adsharma merged 4 commits into
mainfrom
partioned_tables

adsharma commented Aug 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

adsharma commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

adsharma commented Aug 4, 2026 •

edited

Loading