Define streaming indexed recursive WARP architecture

[flyingrobots]

Summary

• Defines the v20 architecture that eliminates production full-state materialization.
• Uses Roaring ordinals as the selection plane over bounded CAS address pages.
• Specifies stream-only attachments and cycle-safe recursive WARP graph traversal.

Issue

Refs #824

Test plan

• npm run lint:md -- --ignore CHANGELOG.md
• npm run lint:md:code
• npm run lint:docs-topology
• Complete static gates and stable unit-test shards passed in pre-push: 506 files; 7,079 tests passed; 2 skipped.

ADR checks

• This PR does not implement ADR 2 without satisfying ADR 3
• If this PR touches persisted op formats, I linked the ADR 3 readiness issue
• If this PR touches wire compatibility, I confirmed canonical-only ops are still rejected on the wire pre-cutover
• If this PR touches schema constants, I confirmed patch and checkpoint namespaces remain distinct

This is a design-only change and does not modify persisted op formats, wire compatibility, or schema constants.

[coderabbitai]

Warning

Review limit reached

@flyingrobots, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 40 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c7105a2b-f0e6-4148-8375-c63b5f5a1a1c

📥 Commits

Reviewing files that changed from the base of the PR and between 303be3c and ff6d20e.

📒 Files selected for processing (1)

• docs/plans/streaming-indexed-recursive-warp.md

📝 Walkthrough

Walkthrough

The PR adds a 1,027-line v20.0.0 architecture proposal defining bounded streaming, retained graph structures, recursive traversal, migration constraints, enforcement tests, implementation slices, acceptance criteria, and validation commands.

Changes

Streaming Indexed Recursive WARP

Layer / File(s)	Summary
Invariants and architectural constraints docs/plans/streaming-indexed-recursive-warp.md	Defines the strengthened no-full-materialization invariant, release posture, current contracts, and bounded-operation classifications.
Retained graph and streaming execution model docs/plans/streaming-indexed-recursive-warp.md	Specifies retained manifests, ordinal and Roaring selection structures, stream-first payload APIs, recursive traversal, receipts, and bounded global operations.
Migration, failure, and behavioral enforcement docs/plans/streaming-indexed-recursive-warp.md	Describes retained-substrate migration, typed obstructions, source ratchets, behavioral proofs, and fixture policies.
Implementation slices and release acceptance docs/plans/streaming-indexed-recursive-warp.md	Lists implementation slices, issue cross-references, acceptance criteria, open design questions, and validation commands.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

• #824 — The proposal defines the architecture, implementation slices, and acceptance criteria described by this issue.
• #817 — The proposal expands retained-root commitments and bounded causal/state construction.
• #822 — The proposal specifies retained-substrate migration without full-state replay-basis creation.
• #819 — The proposal defines typed recursive references, pinned/live semantics, traversal, and retention.
• #814 — The proposal defines removal of production full-state materialization contracts.
• #816 — The proposal includes the Roaring-ordinal selection plane and streamed index construction.

Poem

I’m a rabbit with a streaming plan,
No giant state heap where burdens span.
Roots stay indexed, queues hop light,
Recursive paths remain in sight.
Tests guard each bounded way—
V20 blooms from docs today!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: defining the streaming indexed recursive WARP architecture.
Description check	✅ Passed	The description matches the template with Summary, Issue, Test plan, and ADR checks filled in.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/plans/streaming-indexed-recursive-warp.md`:
- Around line 511-533: Specify a bounded capacity for the explicit async work
queue used by recursive traversal, including admission backpressure when full
and external spill/resume behavior for overflow. Update the traversal budget
sections associated with the queue and the referenced sections around the queue
implementation so pending child work cannot grow with graph fan-out or violate
the stated heap bound.
- Around line 646-652: The migration plan must specify a bounded decoder for
oversized legacy replay values, rather than merely chunking the CAS stream
before synchronous decoding. Update the legacy component streaming steps to use
an incremental parser or external-memory conversion path that never materializes
the full CBOR value, and revise the associated test to exercise the actual
23,995,927-byte oversized shape while enforcing page/shard bounds.
- Around line 600-604: Update the Export design to preserve deterministic scan
order despite bounded-concurrent payload reads: assign sequence numbers and
buffer completed payloads until the next expected sequence is available, or use
a single ordered writer that applies backpressure before opening more payloads.
Ensure the streaming sink receives records strictly in scan order while
retaining the concurrency bound.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 30dfaaab-3446-4140-8ca0-03b7c86b1bda

📥 Commits

Reviewing files that changed from the base of the PR and between 2fc5b7d and 59c97a2.

📒 Files selected for processing (1)

• docs/plans/streaming-indexed-recursive-warp.md

📜 Review details

⏰ Context from checks skipped due to timeout. (8)

• GitHub Check: coverage-threshold
• GitHub Check: test-bun
• GitHub Check: test-node (22)
• GitHub Check: test-deno
• GitHub Check: type-firewall-lint
• GitHub Check: type-firewall-generated-sdk
• GitHub Check: preflight
• GitHub Check: v19 base/head performance

🧰 Additional context used

🪛 LanguageTool

docs/plans/streaming-indexed-recursive-warp.md

[style] ~118-~118: The double modal “required bounded” is nonstandard (only accepted in certain dialects). Consider “to be bounded”.
Context: ...streaming. - #632 through #634 required bounded node, property, neighborhood, and trave...

(NEEDS_FIXED)

---

[style] ~681-~681: The double modal “Required retained” is nonstandard (only accepted in certain dialects). Consider “to be retained”.
Context: ... retained-root-unavailable | Required retained root is absent | Rebuild or migrate | |...

(NEEDS_FIXED)

---

[grammar] ~725-~725: Use a hyphen to join words.
Context: ... to the CBOR decoder. - A multi-gigabyte generated attachment streams through a b...

(QB_NEW_EN_HYPHEN)

🔇 Additional comments (1)

docs/plans/streaming-indexed-recursive-warp.md (1)

504-507: 🎯 Functional Correctness

Specify root-pinned continuation semantics.

When a budget ends, “continue with cursor” must preserve the resolved live roots, visited-basis set, pending queue, and ordering state. Otherwise resumption can read a newer live root or duplicate/skip records. Make the cursor/receipt binding explicit and test resume across live-locator changes.

Also applies to: 566-576, 675-689

[github-actions]

Release Preflight

• package version: 19.0.0
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.0. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.

[github-actions]

Release Preflight

• package version: 19.0.0
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.0. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.

[github-actions]

Release Preflight

• package version: 19.0.0
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.0. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/plans/streaming-indexed-recursive-warp.md`:
- Around line 800-802: Update the recursive traversal invariant in the plan to
state that ready-queue item and encoded-byte usage may reach, but never exceed,
their configured capacities. Replace the “below both configured capacities”
wording while preserving the existing spill and resume guarantees.
- Around line 604-608: Update the continuation cursor design to persist the
complete ready-queue state before issuance: snapshot all admitted ready-queue
entries into the immutable pending-work log, or include the ready-queue root and
exact dequeue/write offsets in cursor validation. Ensure resume reconstructs the
same queue without dropping or replaying work, preserving the
no-duplicate/no-skip guarantee.
- Around line 687-688: Define the schema transition for
MaterializationRoots.replayBasis before describing migration: specify whether
the root becomes optional, is removed, or is invalidated, and update
GitCasMaterializationReplayBasis.loadRoot and manifest construction accordingly
so migration can build retained roots without decoding a full-state asset while
preserving valid handling of existing checkpoints.
- Around line 711-715: Update the migration design around scripts/v18-to-v19 so
the documented parser path matches the v20 migration boundary. Either retarget
the parser to accept authoritative v19 inputs and the Think-shaped proof
fixture, or explicitly document and test the complete v18 → v19 → v20 chain,
including canonical validation and source-coordinate failures.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 636fa166-aab0-493d-ba63-097de0d7734b

📥 Commits

Reviewing files that changed from the base of the PR and between 59c97a2 and 303be3c.

📒 Files selected for processing (1)

• docs/plans/streaming-indexed-recursive-warp.md

📜 Review details

⏰ Context from checks skipped due to timeout. (15)

• GitHub Check: typecheck-test-advisory
• GitHub Check: coverage-threshold
• GitHub Check: type-firewall-semgrep
• GitHub Check: type-firewall-lint
• GitHub Check: type-firewall-quarantine
• GitHub Check: type-firewall-generated-sdk
• GitHub Check: type-firewall-docs
• GitHub Check: type-firewall-types
• GitHub Check: type-firewall-audit-advisory
• GitHub Check: test-deno
• GitHub Check: test-bun
• GitHub Check: test-node (22)
• GitHub Check: type-firewall-surface
• GitHub Check: v19 base/head performance
• GitHub Check: preflight

🔇 Additional comments (1)

docs/plans/streaming-indexed-recursive-warp.md (1)

13-18: LGTM!

Also applies to: 124-125, 515-519, 551-563, 636-645, 701-709, 738-738, 803-806, 900-914, 1021-1027

[github-actions]

Release Preflight

• package version: 19.0.1
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.1. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.

[github-actions]

Release Preflight

• package version: 19.0.1
• prerelease: false
• npm dist-tag on release: latest
• npm pack dry-run: passed
• jsr publish dry-run: passed

If this PR is from a release/* branch and merges to main, Main Push Release Branch Check will run final preflight and create v19.0.1. A maintainer who is a JSR @git-stunts scope member must then dispatch the Release workflow manually.