Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Code coverage (gcov/lcov) -- advisory measurement tier.
#
# SQLite's ethic is 100% MC/DC branch coverage. We are nowhere near that, but
# Tier B3 of .agents/test-suite-maturity-plan.md says: MEASURE branch coverage,
# Tier B3 of the project's test-suite maturity plan says: MEASURE branch coverage,
# then aim new DST/PBT/unit tests at the uncovered branches, and ratchet the
# number upward. This workflow does the measuring.
#
Expand Down
22 changes: 11 additions & 11 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ rate the benchmark quantifies.

## Direction, measured baseline, and sequencing

A July 2026 benchmark on a 2-socket `i4i.metal` (128 vCPU, 2 NUMA nodes, 1 TB
A July 2026 benchmark on a 2-socket bare-metal server (128 vCPU, 2 NUMA nodes, 1 TB
RAM, local NVMe RAID0) compared libdb 5.3.30 to WiredTiger as embedded engines
(YCSB-style driver against both C APIs; data on NVMe, not tmpfs; cache sized at
0.5/0.75/1.0/1.5x the working set). It quantified where we stand:
Expand Down Expand Up @@ -94,7 +94,7 @@ shards)
applied once the per-node path is fast.

Every step is gated by the TCL correctness suite, TSan/ASan clean, and a re-run
of the `i4i.metal` libdb-vs-WT benchmark (measured, not asserted — #17). The
of the bare-metal libdb-vs-WT benchmark (measured, not asserted — #17). The
numbered items below are the work units; this section is the order and the
rationale.

Expand Down Expand Up @@ -158,8 +158,8 @@ Add a log-structured merge-tree access method alongside B-tree/Hash/Queue/Recno
for write-heavy workloads. Design synthesized in
[`docs/design/lsm.md`](docs/design/lsm.md) from three implementations plus a
paper: HanoiDB Towers-of-Hanoi levels + SuRF/Bloom filters and **structure-level
adaptation** (SingleIndex - Hybrid - MultiLevel) from `gburd/aether`'s
`src/lsm`, combined with **segment-level adaptive compaction** (per-segment
adaptation** (SingleIndex - Hybrid - MultiLevel) from a structure-adaptive
HanoiDB-style LSM prototype, combined with **segment-level adaptive compaction** (per-segment
leveled - tiered selection via a finite-state controller with cooldowns) from
*"Amethyst: Adaptive Compaction for LSM Trees via Segment-Level Policy
Selection"* (Shankar & Rose). These are two **orthogonal adaptation axes**
Expand Down Expand Up @@ -192,7 +192,7 @@ uncoordinated manner** — concurrent tries (**Ctrie**), **hash array mapped tri
to reduce coordination on the hash directory and per-bucket latching under high
core counts. This review feeds two consumers: the in-place HASH access method,
and the **in-memory key directory of the `LSM-HASH`/Bitcask config (#14)** —
`gburd/libxtc`'s `rexis` Bitcask uses a plain hash directory today, which a
a Bitcask store uses a plain hash directory today, which a
Ctrie/HAMT would make concurrently updatable without a global directory latch.
See [`docs/design/lsm.md`](docs/design/lsm.md) for how the HASH directory and
the log-structured core relate.
Expand All @@ -202,25 +202,25 @@ the log-structured core relate.
### 14. JE-style index-in-WAL with a log cleaner (new config option)
Offer a **log-structured storage model** as a per-database configuration option
for **both B-tree and Hash**, modeled on Berkeley DB **Java Edition (JE)** and
Oracle NoSQL DB (cf. `gburd/noxu`): the index and data live *in the WAL/log*
Oracle NoSQL DB: the index and data live *in the WAL/log*
itself, with a background **cleaner** reclaiming obsolete log segments instead
of updating pages in place. This trades in-place writes for sequential log
writes (great on flash and for write amplification). A **`LSM-HASH`** variant
along these lines resembles Riak's **Bitcask** (append-only log + in-memory key
directory; cf. `gburd/libxtc` `rexis`) and is a natural fit for write-heavy,
directory) and is a natural fit for write-heavy,
point-lookup workloads. Per [`docs/design/lsm.md`](docs/design/lsm.md), this
shares **one log-structured core** with the adaptive LSM (#9): a JE/`noxu`
shares **one log-structured core** with the adaptive LSM (#9): a JE-style
**cleaner** and an LSM **compactor** are the same mechanism viewed from
different access methods (B-tree-in-log uses the cleaner; Hash uses Bitcask
merge). Reuse `noxu`'s VLSN/cleaner learnings, and align the cleaner's
merge). Reuse the index-in-WAL VLSN/cleaner learnings, and align the cleaner's
reclaim-scheduling with the adaptive controller (#9) so log GC and compaction
share one workload-driven policy.

### 15. Scalable replication / HA: quorum systems + Fast Paxos
Rework replication toward flexible, analyzable consensus. Use **quorum systems**
via `gburd/rs-quoracle` (construct and analyze read/write quorum systems to tune
via a quorum-systems toolkit (construct and analyze read/write quorum systems to tune
the latency/fault-tolerance/throughput trade-off) together with **Fast Paxos**
for low-latency commit, following the approach taken in `gburd/noxu`. Goal:
for low-latency commit, following the JE-style index-in-WAL replication approach. Goal:
replication that scales out reads, survives node loss with tunable quorums, and
avoids the leader bottleneck of classic single-master log shipping.

Expand Down
19 changes: 10 additions & 9 deletions docs/design/buffer-swip-aio.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,12 +301,12 @@ and the `futex`/atomic self-time collapse.
- Interaction of trickle/group-commit (#3 in the roadmap) — the async writer and
the WAL group-commit should share the log-flush coordination.

## 11. Reference implementations (sqlxtc, noxu) and refinements
## 11. Reference implementations and refinements

Two sibling projects implement LeanStore/Umbra-style cooling buffer managers;
studying them validates this design and sharpens three points.
Two prototype buffer managers implement LeanStore/Umbra-style cooling; studying
them validates this design and sharpens three points.

**sqlxtc `bufmgr.c`** (libxtc `examples/06_sqlxtc`): frame states
**A cooling clock-sweep buffer manager**: frame states
`FREE/HOT/COOL/LOADED/WRITING`; the eviction state lives in the parent **swip**
and transitions are owned by whoever wins a CAS (loser retries) — exactly the
tagged-swip model here. Its `evict_one` clock sweep is the key:
Expand All @@ -326,11 +326,12 @@ tagged-swip model here. Its `evict_one` clock sweep is the key:
reclaims a clean victim and leaves dirty `COOL` pages for the trickler; it
flushes a dirty page inline only as a last-resort progress guarantee.

**noxu `noxu-evictor`** (Berkeley-DB-JE lineage): a per-operation **`CacheMode`**
(`Default/Unchanged/EvictLn/EvictBin/KeepHot/MakeEvictable`) drives **two
independent tracking sets** — `primary` and `scan_resistant` — and the evictor
drains *scan → primary → dirty* with per-phase quotas. Pluggable LRU/Clock/ARC/
CAR/LIRS; it notes LRU pollutes on scans while ARC/CAR/LIRS resist inherently.
**A CacheMode-driven evictor** (Berkeley DB Java Edition lineage): a
per-operation **`CacheMode`** (`Default/Unchanged/EvictLn/EvictBin/KeepHot/
MakeEvictable`) drives **two independent tracking sets** — `primary` and
`scan_resistant` — and the evictor drains *scan → primary → dirty* with
per-phase quotas. Pluggable LRU/Clock/ARC/CAR/LIRS; it notes LRU pollutes on
scans while ARC/CAR/LIRS resist inherently.

### What this changes here

Expand Down
92 changes: 47 additions & 45 deletions docs/design/lsm.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,46 @@
# LSM design study: aether, libxtc, noxu, and the Amethyst paper
# LSM design study: adaptive compaction, Bitcask, index-in-WAL, and Amethyst

Status: design note (informs ROADMAP items #9 LSM, #13 HASH, #14 index-in-WAL).

This note compares three existing log-structured implementations against the
This note compares three log-structured storage models against the
adaptive-compaction ideas in *Amethyst* and proposes a synthesized LSM design
for libdb.

## Sources studied
## Models studied

### aether (`~/oss/aether`, Rust) — adaptive LSM
### Adaptive LSM (HanoiDB-style, structure-adaptive)
- **Merge structure:** HanoiDB "Towers of Hanoi" — level *k* holds up to
`2^(k-1)` runs; overflow merges a level's runs and promotes one run up.
Bounded write amplification `O(log n)`.
- **Generic backing index:** the per-run/level index is generic (B+tree,
skiplist, RAX).
- **Read filters:** Bloom + **SuRF** (succinct range filter, `src/lsm/surf/`),
with adaptive filter selection by workload.
- **Structure-level adaptation:** `AdaptiveLsm` switches the *whole structure*
between three modes — `SingleIndex` → `Hybrid` (memtable + 1 index) →
`MultiLevel` (full LSM) — driven by rolling-window metrics
(`writes/reads/flushes_per_minute`, `compaction_backlog`) with **hysteresis**
(60 s cooldown) to prevent flapping. It "spawns" complexity under write
pressure and "collapses" back when idle.
- **Merge scheduling:** `MergeStrategy` = Fast / Predictable / HanoiIncremental
— controls the per-level *work budget* and run selection (latency vs smooth
throughput). This is **work scheduling, not layout policy**.
- **Read filters:** Bloom + **SuRF** (succinct range filter), with adaptive
filter selection by workload.
- **Structure-level adaptation:** switch the *whole structure* between three
modes — `SingleIndex` → `Hybrid` (memtable + 1 index) → `MultiLevel` (full
LSM) — driven by rolling-window metrics (writes/reads/flushes per minute,
compaction backlog) with **hysteresis** (a cooldown) to prevent flapping. It
"spawns" complexity under write pressure and "collapses" back when idle.
- **Merge scheduling:** a merge-strategy knob (Fast / Predictable /
HanoiIncremental) controls the per-level *work budget* and run selection
(latency vs smooth throughput). This is **work scheduling, not layout
policy**.
- **Gap:** no leveled-vs-tiered choice; the layout is fixed (Hanoi).

### libxtc (`~/oss/libxtc`, C) — Bitcask (`examples/05_rexis`)
### Bitcask (log-structured hash)
- Append-only data file + **in-memory hash index** (`key -> offset,len,ts`);
put = append + index update, delete = tombstone append, get = index + one
`pread`. Index rebuilt by scanning the log on recovery. Background **merge**
`pread`. Index rebuilt by scanning the log on recovery. A background **merge**
reclaims dead space from superseded entries.
- This is exactly a **log-structured hash** — the `LSM-HASH ≈ Bitcask` model.

### noxu (`~/oss/noxu`, Rust) — JE-style index-in-WAL + cleaner
- A **B+tree whose nodes live in the WAL** (`noxu-log`); `noxu-cleaner` is a
log-GC that reclaims obsolete log segments (the JE / Oracle NoSQL model);
`noxu-evictor` is cache management; `noxu-recovery` does checkpoint/replay.
- `noxu-rep` + **VLSN** (versioned LSN) provide replication; the consensus uses
quorum systems + Fast Paxos (ROADMAP #15).
### JE-style index-in-WAL + cleaner
- A **B+tree whose nodes live in the WAL**; a **cleaner** is a log-GC that
reclaims obsolete log segments (the Berkeley DB Java Edition / Oracle NoSQL
model); a separate evictor handles cache management; recovery does
checkpoint/replay.
- Replication in this model rides on a **VLSN** (versioned LSN); consensus can
use quorum systems + Fast Paxos (ROADMAP #15).
- This is the **index-in-WAL + cleaner** durability model — the cleaner plays
the same role a compactor does in an LSM.

Expand All @@ -50,37 +51,38 @@ for libdb.
so the tree converges to the better policy as the workload shifts between
read- and write-heavy phases (≈2.2× lower runtime than static tiered while
matching leveled read amplification).
- This is **per-segment layout policy adaptation** — orthogonal to aether.
- This is **per-segment layout policy adaptation** — orthogonal to the
structure-adaptive axis above.

## The key insight: two orthogonal adaptation axes, one mechanism

aether and Amethyst adapt *different things* with the *same machinery*
(rolling counters + cooldown/hysteresis):
The structure-adaptive LSM and Amethyst adapt *different things* with the
*same machinery* (rolling counters + cooldown/hysteresis):

| Axis | What it decides | Source | Signal |
|------|-----------------|--------|--------|
| **Structure** (vertical) | SingleIndex vs Hybrid vs MultiLevel — *how much LSM* | aether | env-wide write/read/flush rate |
| **Structure** (vertical) | SingleIndex vs Hybrid vs MultiLevel — *how much LSM* | structure-adaptive LSM | env-wide write/read/flush rate |
| **Policy** (horizontal) | leveled vs tiered *per segment/level* — *how to compact* | Amethyst | per-segment read/write counters |

Neither implementation has both. Combining them yields an LSM that (a) only
pays LSM cost when the workload warrants it, and (b) within the LSM, compacts
each segment with the locally-best policy.
Neither model has both. Combining them yields an LSM that (a) only pays LSM
cost when the workload warrants it, and (b) within the LSM, compacts each
segment with the locally-best policy.

## Proposed libdb LSM design (ROADMAP #9)

1. **Base structure:** HanoiDB Towers-of-Hanoi levels (from aether) for bounded
`O(log n)` write amplification, with HanoiIncremental merge scheduling for
smooth foreground latency.
1. **Base structure:** HanoiDB Towers-of-Hanoi levels for bounded `O(log n)`
write amplification, with incremental merge scheduling for smooth
foreground latency.
2. **Unified adaptive controller** with one metrics/cooldown core driving two
axes:
- *Structure axis* (aether): SingleIndex ⇄ Hybrid ⇄ MultiLevel by env-wide
write pressure, with hysteresis.
- *Structure axis*: SingleIndex ⇄ Hybrid ⇄ MultiLevel by env-wide write
pressure, with hysteresis.
- *Policy axis* (Amethyst): each segment/level tagged leveled or tiered from
per-segment read/write counters, rewritten with cooldowns.
Both reuse the same rolling-window + cooldown primitive (one implementation,
two decision sites) — and the same primitive already used by the SSI marker
GC, keeping the engine's "adaptive" machinery uniform.
3. **Read acceleration:** Bloom + SuRF per run, adaptive selection (aether).
3. **Read acceleration:** Bloom + SuRF per run, adaptive selection.
4. **Generic over the access method's page/index** so the LSM composes with
libdb's existing B-tree/Hash rather than replacing them.

Expand All @@ -89,12 +91,12 @@ each segment with the locally-best policy.
The "index-in-WAL with a cleaner" option is the same idea as LSM compaction,
specialized by access method:

- **HASH → `LSM-HASH` = Bitcask** (libxtc): append-only log + in-memory hash
directory; the LSM "compaction" is Bitcask's dead-space merge. Natural for
write-heavy point-lookup workloads.
- **B-tree → JE-style index-in-WAL + cleaner** (noxu): B+tree nodes in the log,
`cleaner` reclaims obsolete segments. The LSM merger and the JE cleaner are
the same mechanism viewed from two access methods.
- **HASH → `LSM-HASH` = Bitcask**: append-only log + in-memory hash directory;
the LSM "compaction" is Bitcask's dead-space merge. Natural for write-heavy
point-lookup workloads.
- **B-tree → JE-style index-in-WAL + cleaner**: B+tree nodes in the log, a
cleaner reclaims obsolete segments. The LSM merger and the JE cleaner are the
same mechanism viewed from two access methods.

So #9 (adaptive LSM) and #14 (index-in-WAL) share one log-structured core; the
HASH review (#13) should evaluate the Bitcask directory against Ctrie/HAMT for
Expand All @@ -104,6 +106,6 @@ the in-memory index.
- Prototype the unified adaptive controller (structure + segment-policy axes)
as a standalone module, validated with the Amethyst-style phase-shifting
workload before wiring it under an access method.
- Port aether's HanoiDB level math + SuRF as the reference run/level layout.
- Reuse noxu's cleaner/VLSN learnings for the index-in-WAL config and the
replication path (#15).
- Port the HanoiDB level math + SuRF as the reference run/level layout.
- Reuse the index-in-WAL cleaner / VLSN learnings for the index-in-WAL config
and the replication path (#15).
2 changes: 1 addition & 1 deletion docs/design/research/os-aio.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# OS / mpool I/O survey — basis for an internal async-I/O abstraction

Read-only survey of `/Users/gregburd/oss/libdb` @ `master`. No source was
Read-only survey of the libdb tree @ `master`. No source was
modified. All line numbers are from the working tree at survey time
(2026-06-17). Paths are repo-relative.

Expand Down
4 changes: 2 additions & 2 deletions lab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ Example:

Prototype of the two-axis adaptive controller from
[`docs/design/lsm.md`](../../docs/design/lsm.md): one rolling-window + cooldown
core driving both the **structure axis** (SINGLE ⇄ HYBRID ⇄ MULTILEVEL, aether-
style) and the **per-segment policy axis** (LEVELED ⇄ TIERED, Amethyst-style).
core driving both the **structure axis** (SINGLE ⇄ HYBRID ⇄ MULTILEVEL,
structure-adaptive) and the **per-segment policy axis** (LEVELED ⇄ TIERED, Amethyst-style).

Time is a caller-supplied "tick" so the logic is deterministic and testable.

Expand Down
2 changes: 1 addition & 1 deletion lab/lsm/adaptive.h
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
* (see docs/design/lsm.md):
*
* - STRUCTURE axis: how much LSM to run for the whole store
* SINGLE <-> HYBRID <-> MULTILEVEL (aether-style)
* SINGLE <-> HYBRID <-> MULTILEVEL (structure-adaptive)
* decided by env-wide write/read/flush rates with hysteresis.
*
* - POLICY axis: how to compact each segment
Expand Down
4 changes: 2 additions & 2 deletions test/coverage/COVERAGE-TEST-TRIAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ coverage/-O0 resource artifact vs. real engine bug.

## Method

- Worktree: `/tmp/libdb-wt-test-triage`, branch `agent/test-triage` off master.
- A clean checkout of `master` (no `--coverage`).
- Build (NO `--coverage`):
`../dist/configure --enable-debug --enable-diagnostic --enable-test
--with-tcl=/nix/store/mzx2cj89c7phpv7cb6r9hydadnndp53m-tcl-8.6.16/lib && make -j4`.
--with-tcl=/path/to/tcl && make -j4`.
Note: BDB `--enable-debug` compiles at `-g -O2` (adds `-g`, not `-O0`);
`--enable-diagnostic` sets `DIAGNOSTIC 1` so `DB_ASSERT` is live.
- Each test run via the harness: `source test.tcl; run_method <method> testNNN 0 1`,
Expand Down
2 changes: 1 addition & 1 deletion test/coverage/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Code coverage (Tier B3)

SQLite's testing ethic is **100% MC/DC branch coverage**. libdb is nowhere near
that yet — but per Tier B3 of `.agents/test-suite-maturity-plan.md` the point is
that yet — but per Tier B3 of the test-suite maturity plan the point is
to **measure** branch coverage now, aim new DST/PBT/unit tests at the uncovered
branches, and ratchet the number upward over time. This directory holds the
measurement machinery.
Expand Down
2 changes: 1 addition & 1 deletion test/coverage/baseline.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# libdb coverage baseline (Tier B3, .agents/test-suite-maturity-plan.md).
# libdb coverage baseline (Tier B3 of the test-suite maturity plan).
# src/-only, gcov/lcov, representative test subset (see test/coverage/README.md).
# The Coverage workflow warns (advisory) if branch coverage drops >0.5% below this.
# Update after landing new tests that raise coverage (ratchet upward, never down).
Expand Down
2 changes: 1 addition & 1 deletion test/coverage/rank_coverage.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
Prints the least-covered files first (lowest line %, then largest), so future
DST/PBT/unit test work can aim at the biggest uncovered surfaces. Files under
50 lines are skipped as noise. This is the actionable output of Tier B3
(coverage in CI) in .agents/test-suite-maturity-plan.md.
(coverage in CI) in the test-suite maturity plan.

Usage: rank_coverage.py <coverage.info> [min_lines]
"""
Expand Down
2 changes: 1 addition & 1 deletion test/coverage/run_coverage.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# measure branch coverage, then aim new tests at the gaps).
#
# See test/coverage/README.md for how to read the report and find the
# least-covered files. Tier B3 of .agents/test-suite-maturity-plan.md.
# least-covered files. Tier B3 of the test-suite maturity plan.
#
# Usage:
# test/coverage/run_coverage.sh # subset build + report
Expand Down
2 changes: 1 addition & 1 deletion test/faultinject/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ leaves no corruption. libdb routes essentially all allocation through one
seam — `__os_malloc` / `__os_calloc` / `__os_realloc` / `__os_umalloc` /
`__os_urealloc` in `src/os/os_alloc.c` — so a single injection point covers
the library. This is the dynamic complement to the Coccinelle `malloc-leak` /
`mutex-unbalanced` static rules (Tier B1 of `.agents/test-suite-maturity-plan.md`).
`mutex-unbalanced` static rules (Tier B1 of the test-suite maturity plan).

## What's here

Expand Down
Loading
Loading