diff --git a/.gitignore b/.gitignore index 1096fc56e..3510649dd 100644 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,5 @@ lab/bench/ssi_abort_bench # Generated documentation (docs-src/build.py output) docs-build/ docs-src/**/__pycache__/ +rfc/0001/test_adaptive +test_adaptive diff --git a/lab/README.md b/lab/README.md index 50e4de161..0e59d0cff 100644 --- a/lab/README.md +++ b/lab/README.md @@ -56,26 +56,5 @@ Example: ./tproc_h -i -S 1 -m && ./tproc_h -t 8 -w 4 -m -s 30 # MVCC analytic vs writers ``` -## `lab/lsm` — unified adaptive LSM controller - -Prototype of the two-axis adaptive controller from -[`docs/design/lsm.md`](../../docs/design/lsm.md): one rolling-window + cooldown -core driving both the **structure axis** (SINGLE ⇄ HYBRID ⇄ MULTILEVEL, -structure-adaptive) and the **per-segment policy axis** (LEVELED ⇄ TIERED, Amethyst-style). - -Time is a caller-supplied "tick" so the logic is deterministic and testable. - -```sh -cd lab/lsm && make check -``` - -`test_adaptive.c` drives an Amethyst-style phase-shifting workload and asserts: -- a write-heavy phase spawns the structure to `MULTILEVEL`; an idle phase - collapses it back to `SINGLE`, with no oscillation; -- a sustained load reaches `MULTILEVEL` and stays (cooldown anti-flap); -- write-hot segments converge to `TIERED`, read-hot to `LEVELED`, and balanced - segments don't churn. - -Next: replace the tick with a millisecond clock, feed real op counters from the -access-method layer, and have the policy axis emit segment-rewrite decisions to -the compactor / log cleaner (ROADMAP #9, #14). +The adaptive-LSM controller prototype that used to live here has graduated to +[`rfc/0001/`](../rfc/0001/) alongside RFC 0001 (adaptive LSM access method). diff --git a/rfc/0000-template.md b/rfc/0000-template.md new file mode 100644 index 000000000..096ae331e --- /dev/null +++ b/rfc/0000-template.md @@ -0,0 +1,55 @@ +# RFC NNNN: + +- **Status:** Draft +- **Type:** Prospective | Normative +- **Author:** <name> +- **Date:** YYYY-MM-DD +- **Tracking:** <phase / issue / PR, optional> +- **Prototype:** `rfc/NNNN/` (optional) + +--- + +## Summary + +One paragraph: what this proposes, in plain terms. + +## Motivation + +Why now? What problem does it solve, for whom? What breaks or stays slow +without it? Cite evidence where possible (benchmarks in `test/bench`, profiles, +bug reports). + +## North-star check + +Confirm the proposal does not break any of: embedded/no-server operation, ACID, +crash recovery, any access method (B-tree/Hash/Queue/Recno/Heap), multi-process +correctness, on-disk/log/region/ABI format stability. If it *does* touch a +format/ABI, describe the versioned, backward-compatible migration here — this is +the hard review gate. + +## Design + +The proposal itself. Data structures, algorithms, the seams it touches in the +engine, the config/flags it adds. Enough that someone else could implement it. + +## Alternatives considered + +What else could solve this, and why this over those. + +## Risks & open questions + +Correctness risk, performance risk, maintenance cost, what's still unknown. + +## Prototype / evidence + +If `rfc/NNNN/` has a spike, describe what it validated and the results. + +--- + +## Decision + +*(Filled by the reviewer when the RFC is decided.)* + +- **Decision:** Accepted | Rejected — YYYY-MM-DD +- **Rationale:** why. +- **Conditions / follow-ups:** any. diff --git a/docs/design/lsm.md b/rfc/0001-adaptive-lsm.md similarity index 92% rename from docs/design/lsm.md rename to rfc/0001-adaptive-lsm.md index 393ade20e..e3c9abe6d 100644 --- a/docs/design/lsm.md +++ b/rfc/0001-adaptive-lsm.md @@ -1,8 +1,16 @@ -# LSM design study: adaptive compaction, Bitcask, index-in-WAL, and Amethyst +# RFC 0001: Adaptive LSM access method -Status: design note (informs ROADMAP items #9 LSM, #13 HASH, #14 index-in-WAL). +- **Status:** Draft +- **Type:** Prospective +- **Author:** libdb maintainers +- **Date:** 2026-07-31 +- **Prototype:** [`rfc/0001/`](0001/) (adaptive-controller spike) -This note compares three log-structured storage models against the +--- + +## Summary + +This RFC compares three log-structured storage models against the adaptive-compaction ideas in *Amethyst* and proposes a synthesized LSM design for libdb. @@ -40,7 +48,7 @@ for libdb. model); a separate evictor handles cache management; recovery does checkpoint/replay. - Replication in this model rides on a **VLSN** (versioned LSN); consensus can - use quorum systems + Fast Paxos (ROADMAP #15). + use quorum systems + Fast Paxos (see the scalable-replication RFC). - This is the **index-in-WAL + cleaner** durability model — the cleaner plays the same role a compactor does in an LSM. @@ -68,7 +76,7 @@ Neither model has both. Combining them yields an LSM that (a) only pays LSM cost when the workload warrants it, and (b) within the LSM, compacts each segment with the locally-best policy. -## Proposed libdb LSM design (ROADMAP #9) +## Proposed libdb LSM design 1. **Base structure:** HanoiDB Towers-of-Hanoi levels for bounded `O(log n)` write amplification, with incremental merge scheduling for smooth @@ -86,7 +94,7 @@ segment with the locally-best policy. 4. **Generic over the access method's page/index** so the LSM composes with libdb's existing B-tree/Hash rather than replacing them. -## Mapping to durability configs (ROADMAP #14) +## Mapping to durability configs The "index-in-WAL with a cleaner" option is the same idea as LSM compaction, specialized by access method: diff --git a/lab/lsm/Makefile b/rfc/0001/Makefile similarity index 100% rename from lab/lsm/Makefile rename to rfc/0001/Makefile diff --git a/lab/lsm/adaptive.c b/rfc/0001/adaptive.c similarity index 100% rename from lab/lsm/adaptive.c rename to rfc/0001/adaptive.c diff --git a/lab/lsm/adaptive.h b/rfc/0001/adaptive.h similarity index 100% rename from lab/lsm/adaptive.h rename to rfc/0001/adaptive.h diff --git a/lab/lsm/test_adaptive.c b/rfc/0001/test_adaptive.c similarity index 100% rename from lab/lsm/test_adaptive.c rename to rfc/0001/test_adaptive.c diff --git a/docs/design/buffer-swip-aio.md b/rfc/0002-buffer-swip-aio.md similarity index 98% rename from docs/design/buffer-swip-aio.md rename to rfc/0002-buffer-swip-aio.md index 8e3077373..5c09dcc80 100644 --- a/docs/design/buffer-swip-aio.md +++ b/rfc/0002-buffer-swip-aio.md @@ -1,8 +1,18 @@ -# Scalable buffer access: tagged swip, optimistic descent, and async I/O +# RFC 0002: Scalable buffer access — tagged swip, optimistic descent, async I/O -Status: design / plan of record. Companion to -[`scaling-findings.md`](scaling-findings.md). Code anchors verified against -`master` and recorded in `research/mpool-btree.md` and `research/os-aio.md`. +- **Status:** Draft +- **Type:** Prospective +- **Author:** libdb maintainers +- **Date:** 2026-07-31 +- **Prototype:** surveys in [`rfc/0002/`](0002/) + +--- + +## Summary + +Code anchors verified against `master` and recorded in +[`0002/mpool-btree-survey.md`](0002/mpool-btree-survey.md) and +[`0002/os-aio-survey.md`](0002/os-aio-survey.md). ## 1. Problem diff --git a/docs/design/research/mpool-btree.md b/rfc/0002/mpool-btree-survey.md similarity index 100% rename from docs/design/research/mpool-btree.md rename to rfc/0002/mpool-btree-survey.md diff --git a/docs/design/research/os-aio.md b/rfc/0002/os-aio-survey.md similarity index 100% rename from docs/design/research/os-aio.md rename to rfc/0002/os-aio-survey.md diff --git a/rfc/INDEX.md b/rfc/INDEX.md new file mode 100644 index 000000000..20461dbd7 --- /dev/null +++ b/rfc/INDEX.md @@ -0,0 +1,11 @@ +# RFC index + +The register of libdb design proposals. See [`README.md`](README.md) for the +process. Status: Draft · Accepted · Rejected · Superseded · Implemented. + +| # | Title | Status | Type | +|---|-------|--------|------| +| [0001](0001-adaptive-lsm.md) | Adaptive LSM access method (HanoiDB + segment-policy, Bitcask, index-in-WAL) | Draft | Prospective | +| [0002](0002-buffer-swip-aio.md) | Scalable buffer access: tagged swip, optimistic descent, async I/O | Draft | Prospective | + +<!-- Add a row per RFC. Keep the number 4-digit zero-padded and monotonic. --> diff --git a/rfc/README.md b/rfc/README.md new file mode 100644 index 000000000..811ead953 --- /dev/null +++ b/rfc/README.md @@ -0,0 +1,58 @@ +# libdb RFCs + +Design proposals for libdb. An RFC is how a non-trivial change — a new access +method, an on-disk/format change, a performance subsystem, a durability model — +gets written down, reviewed, and accepted or rejected **before** large +implementation effort, so the reasoning survives and the decision is explicit. + +User-facing documentation lives in [`docs_src/`](../docs_src) (published to +libdb.org). This directory is the opposite: internal design intent and the +record of what we decided and why. + +## Layout + +- `NNNN-short-title.md` — the RFC itself (4-digit zero-padded number). +- `NNNN/` — optional per-RFC working directory: prototypes, spikes, data, + scratch code that validates the idea before it's wired into the engine. +- `0000-template.md` — copy this to start a new RFC. +- `INDEX.md` — the register: every RFC, its status, one line. + +## RFC lifecycle / status + +Each RFC carries a `Status:` in its header. The allowed states and the +transitions: + +| Status | Meaning | +|--------|---------| +| **Draft** | Under active writing/discussion. Not yet decided. | +| **Accepted** | Reviewed and approved; implementation may proceed. The RFC is the spec. | +| **Rejected** | Reviewed and declined. Kept for the record (why we said no). | +| **Superseded** | Replaced by a later RFC (name it). | +| **Implemented** | Accepted **and** shipped; links to the code/PRs. | + +`Type:` is `Prospective` (a future direction) or `Normative` (binds current +behavior/format). + +## Review methodology (explicit accept/reject) + +1. **Open**: copy `0000-template.md` to the next free `NNNN-title.md`, fill + Summary/Motivation/Design/Alternatives/Risks, set `Status: Draft`, add a row + to `INDEX.md`. Optionally start `NNNN/` with a prototype. +2. **Review**: the RFC is judged against libdb's north star — a change is + **rejected outright** if it breaks any of: embedded/no-server operation, ACID + guarantees, crash recovery, any access method, multi-process correctness, or + on-disk/log/region/ABI format stability, *unless* the RFC explicitly argues a + versioned, backward-compatible migration. Beyond that gate, review weighs: + correctness risk, performance evidence (measured, not asserted — cite the + `test/bench` microbenchmarks or a reproducible harness), maintenance cost, and + scope. +3. **Decide**: the maintainer records the decision **in the RFC** — flip + `Status:` to `Accepted` or `Rejected` with a dated **Decision** section + stating the rationale and any conditions. A `Rejected` RFC is never deleted; + the "no" and its reasons are the value. +4. **Implement**: an `Accepted` RFC drives the work. When shipped, flip to + `Implemented` and link the PRs/commits. Material deviations from an accepted + RFC require an amendment (a dated note) or a superseding RFC. + +Small, obvious, or purely-internal changes do not need an RFC — this is for +decisions worth remembering.