Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,5 @@ lab/bench/ssi_abort_bench
# Generated documentation (docs-src/build.py output)
docs-build/
docs-src/**/__pycache__/
rfc/0001/test_adaptive
test_adaptive
25 changes: 2 additions & 23 deletions lab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,26 +56,5 @@ Example:
./tproc_h -i -S 1 -m && ./tproc_h -t 8 -w 4 -m -s 30 # MVCC analytic vs writers
```

## `lab/lsm` — unified adaptive LSM controller

Prototype of the two-axis adaptive controller from
[`docs/design/lsm.md`](../../docs/design/lsm.md): one rolling-window + cooldown
core driving both the **structure axis** (SINGLE ⇄ HYBRID ⇄ MULTILEVEL,
structure-adaptive) and the **per-segment policy axis** (LEVELED ⇄ TIERED, Amethyst-style).

Time is a caller-supplied "tick" so the logic is deterministic and testable.

```sh
cd lab/lsm && make check
```

`test_adaptive.c` drives an Amethyst-style phase-shifting workload and asserts:
- a write-heavy phase spawns the structure to `MULTILEVEL`; an idle phase
collapses it back to `SINGLE`, with no oscillation;
- a sustained load reaches `MULTILEVEL` and stays (cooldown anti-flap);
- write-hot segments converge to `TIERED`, read-hot to `LEVELED`, and balanced
segments don't churn.

Next: replace the tick with a millisecond clock, feed real op counters from the
access-method layer, and have the policy axis emit segment-rewrite decisions to
the compactor / log cleaner (ROADMAP #9, #14).
The adaptive-LSM controller prototype that used to live here has graduated to
[`rfc/0001/`](../rfc/0001/) alongside RFC 0001 (adaptive LSM access method).
55 changes: 55 additions & 0 deletions rfc/0000-template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# RFC NNNN: <title>

- **Status:** Draft
- **Type:** Prospective | Normative
- **Author:** <name>
- **Date:** YYYY-MM-DD
- **Tracking:** <phase / issue / PR, optional>
- **Prototype:** `rfc/NNNN/` (optional)

---

## Summary

One paragraph: what this proposes, in plain terms.

## Motivation

Why now? What problem does it solve, for whom? What breaks or stays slow
without it? Cite evidence where possible (benchmarks in `test/bench`, profiles,
bug reports).

## North-star check

Confirm the proposal does not break any of: embedded/no-server operation, ACID,
crash recovery, any access method (B-tree/Hash/Queue/Recno/Heap), multi-process
correctness, on-disk/log/region/ABI format stability. If it *does* touch a
format/ABI, describe the versioned, backward-compatible migration here — this is
the hard review gate.

## Design

The proposal itself. Data structures, algorithms, the seams it touches in the
engine, the config/flags it adds. Enough that someone else could implement it.

## Alternatives considered

What else could solve this, and why this over those.

## Risks & open questions

Correctness risk, performance risk, maintenance cost, what's still unknown.

## Prototype / evidence

If `rfc/NNNN/` has a spike, describe what it validated and the results.

---

## Decision

*(Filled by the reviewer when the RFC is decided.)*

- **Decision:** Accepted | Rejected — YYYY-MM-DD
- **Rationale:** why.
- **Conditions / follow-ups:** any.
20 changes: 14 additions & 6 deletions docs/design/lsm.md → rfc/0001-adaptive-lsm.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,16 @@
# LSM design study: adaptive compaction, Bitcask, index-in-WAL, and Amethyst
# RFC 0001: Adaptive LSM access method

Status: design note (informs ROADMAP items #9 LSM, #13 HASH, #14 index-in-WAL).
- **Status:** Draft
- **Type:** Prospective
- **Author:** libdb maintainers
- **Date:** 2026-07-31
- **Prototype:** [`rfc/0001/`](0001/) (adaptive-controller spike)

This note compares three log-structured storage models against the
---

## Summary

This RFC compares three log-structured storage models against the
adaptive-compaction ideas in *Amethyst* and proposes a synthesized LSM design
for libdb.

Expand Down Expand Up @@ -40,7 +48,7 @@ for libdb.
model); a separate evictor handles cache management; recovery does
checkpoint/replay.
- Replication in this model rides on a **VLSN** (versioned LSN); consensus can
use quorum systems + Fast Paxos (ROADMAP #15).
use quorum systems + Fast Paxos (see the scalable-replication RFC).
- This is the **index-in-WAL + cleaner** durability model — the cleaner plays
the same role a compactor does in an LSM.

Expand Down Expand Up @@ -68,7 +76,7 @@ Neither model has both. Combining them yields an LSM that (a) only pays LSM
cost when the workload warrants it, and (b) within the LSM, compacts each
segment with the locally-best policy.

## Proposed libdb LSM design (ROADMAP #9)
## Proposed libdb LSM design

1. **Base structure:** HanoiDB Towers-of-Hanoi levels for bounded `O(log n)`
write amplification, with incremental merge scheduling for smooth
Expand All @@ -86,7 +94,7 @@ segment with the locally-best policy.
4. **Generic over the access method's page/index** so the LSM composes with
libdb's existing B-tree/Hash rather than replacing them.

## Mapping to durability configs (ROADMAP #14)
## Mapping to durability configs

The "index-in-WAL with a cleaner" option is the same idea as LSM compaction,
specialized by access method:
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
18 changes: 14 additions & 4 deletions docs/design/buffer-swip-aio.md → rfc/0002-buffer-swip-aio.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,18 @@
# Scalable buffer access: tagged swip, optimistic descent, and async I/O
# RFC 0002: Scalable buffer accesstagged swip, optimistic descent, async I/O

Status: design / plan of record. Companion to
[`scaling-findings.md`](scaling-findings.md). Code anchors verified against
`master` and recorded in `research/mpool-btree.md` and `research/os-aio.md`.
- **Status:** Draft
- **Type:** Prospective
- **Author:** libdb maintainers
- **Date:** 2026-07-31
- **Prototype:** surveys in [`rfc/0002/`](0002/)

---

## Summary

Code anchors verified against `master` and recorded in
[`0002/mpool-btree-survey.md`](0002/mpool-btree-survey.md) and
[`0002/os-aio-survey.md`](0002/os-aio-survey.md).

## 1. Problem

Expand Down
File renamed without changes.
File renamed without changes.
11 changes: 11 additions & 0 deletions rfc/INDEX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# RFC index

The register of libdb design proposals. See [`README.md`](README.md) for the
process. Status: Draft · Accepted · Rejected · Superseded · Implemented.

| # | Title | Status | Type |
|---|-------|--------|------|
| [0001](0001-adaptive-lsm.md) | Adaptive LSM access method (HanoiDB + segment-policy, Bitcask, index-in-WAL) | Draft | Prospective |
| [0002](0002-buffer-swip-aio.md) | Scalable buffer access: tagged swip, optimistic descent, async I/O | Draft | Prospective |

<!-- Add a row per RFC. Keep the number 4-digit zero-padded and monotonic. -->
58 changes: 58 additions & 0 deletions rfc/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# libdb RFCs

Design proposals for libdb. An RFC is how a non-trivial change — a new access
method, an on-disk/format change, a performance subsystem, a durability model —
gets written down, reviewed, and accepted or rejected **before** large
implementation effort, so the reasoning survives and the decision is explicit.

User-facing documentation lives in [`docs_src/`](../docs_src) (published to
libdb.org). This directory is the opposite: internal design intent and the
record of what we decided and why.

## Layout

- `NNNN-short-title.md` — the RFC itself (4-digit zero-padded number).
- `NNNN/` — optional per-RFC working directory: prototypes, spikes, data,
scratch code that validates the idea before it's wired into the engine.
- `0000-template.md` — copy this to start a new RFC.
- `INDEX.md` — the register: every RFC, its status, one line.

## RFC lifecycle / status

Each RFC carries a `Status:` in its header. The allowed states and the
transitions:

| Status | Meaning |
|--------|---------|
| **Draft** | Under active writing/discussion. Not yet decided. |
| **Accepted** | Reviewed and approved; implementation may proceed. The RFC is the spec. |
| **Rejected** | Reviewed and declined. Kept for the record (why we said no). |
| **Superseded** | Replaced by a later RFC (name it). |
| **Implemented** | Accepted **and** shipped; links to the code/PRs. |

`Type:` is `Prospective` (a future direction) or `Normative` (binds current
behavior/format).

## Review methodology (explicit accept/reject)

1. **Open**: copy `0000-template.md` to the next free `NNNN-title.md`, fill
Summary/Motivation/Design/Alternatives/Risks, set `Status: Draft`, add a row
to `INDEX.md`. Optionally start `NNNN/` with a prototype.
2. **Review**: the RFC is judged against libdb's north star — a change is
**rejected outright** if it breaks any of: embedded/no-server operation, ACID
guarantees, crash recovery, any access method, multi-process correctness, or
on-disk/log/region/ABI format stability, *unless* the RFC explicitly argues a
versioned, backward-compatible migration. Beyond that gate, review weighs:
correctness risk, performance evidence (measured, not asserted — cite the
`test/bench` microbenchmarks or a reproducible harness), maintenance cost, and
scope.
3. **Decide**: the maintainer records the decision **in the RFC** — flip
`Status:` to `Accepted` or `Rejected` with a dated **Decision** section
stating the rationale and any conditions. A `Rejected` RFC is never deleted;
the "no" and its reasons are the value.
4. **Implement**: an `Accepted` RFC drives the work. When shipped, flip to
`Implemented` and link the PRs/commits. Material deviations from an accepted
RFC require an amendment (a dated note) or a superseding RFC.

Small, obvious, or purely-internal changes do not need an RFC — this is for
decisions worth remembering.
Loading