fix(history): bound external CLI replay and turn-index rebuild#504
fix(history): bound external CLI replay and turn-index rebuild#504ShiboSheng wants to merge 20 commits into
Conversation
Large SQLite-backed sessions previously collected every event row and copied full argument, result, and content JSON before determining whether the row could contribute turn metadata. That made index migration memory grow with the complete transcript. Fold rusqlite rows directly into turn accumulators, classify tool payload requirements before copying large fields, and publish rebuilt summaries transactionally. Unknown tools retain the conservative metadata behavior so the optimization does not change native SDE results. Verification: - `cargo test -p session_persistence turn_index`: PASS - `cargo test -p orgtrack_core`: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
The storage-specific replay drivers require incremental SQLite BLOB access, native file notifications, compressed streaming support, stable hashing, and process-level RSS assertions that were not enabled in the existing workspace dependency set. Enable the required rusqlite BLOB feature and add the notify, flate2, sha2, and libc wiring used by replay indexing, streamed export, and performance tests. Keep the lockfile aligned with the workspace manifests. Verification: - `cargo check --workspace --all-targets`: PASS on the final branch - Cargo lockfile resolution completed with hooks enabled Pre-commit hook ran. Total eslint: 0, total circular: 1
External history sources do not share one storage format: some append JSONL, some update SQLite or WAL rows, Cursor and Warp use specialized records, and Cline rewrites a whole JSON document. Treating them as one full-history loader caused repeated parsing and session-sized memory growth. Add the authoritative 15-source registry, storage-specific JSONL, SQLite, structured, and whole-document drivers, compact replay index tables, stable cursors, bounded windows, payload locators, and source conformance fixtures. Each source keeps its own parsing and tool-pairing semantics while sharing the same public replay contract. Verification: - `cargo test -p orgtrack_core`: 453 passed, 3 ignored - 30 MiB and 300 MiB replay performance fixtures: PASS - Real 335 MiB Codex Issue 272 core replay: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
A compact source index alone did not remove the memory amplification because application commands, managed CLI mirrors, EventStore hydration, metadata, provenance, and export paths could still request or materialize complete histories. Add the Rust replay commands, watcher lifecycle, bounded EventStore application, managed transcript routing, streamed export, collaboration snapshot ingestion, and compact metadata consumers. Keep replay state separate from native SDE persistence and reject mismatched source or session identities. Verification: - `cargo test -p org2 agent_sessions::event_pipeline`: 403 passed - 327 MiB streamed export hash and buffer-bound test: PASS - Native SDE replay-command isolation assertions: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
Even with bounded Rust readers, renderer and background consumers could preserve the old full-transcript behavior by loading all chunks, sending events back to Rust, or building complete arrays for pagination, Fork, Cloud, Raw Transcript, and collaboration flows. Introduce the typed replay RPC and SessionCore transport, route imported and managed CLI sessions through bounded windows and deltas, virtualize transcript payload access, and migrate Fork, Cloud, hover, provenance, turn loading, and session switching. Native SDE adapters continue using their existing persisted-database mode. Verification: - Full Vitest suite on the final branch: 6,197 passed - `pnpm typecheck`: PASS - Replay RPC schema, request guard, pagination, lifecycle, Fork, Cloud, and transcript tests: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
Bring the issue branch up to the develop snapshot ending at build v1.2.1 so the replay implementation is evaluated against the current source parsers, session directory, cloud synchronization, and dispatch behavior. Resolve source-adapter and cloud-consumer conflicts by retaining bounded replay ownership of external history while incorporating the develop-side parsing and lifecycle changes. No protected local artifacts are included. Verification: - Subsequent branch-wide Rust and Vitest gates recorded in the Issue 443 audit: PASS - Final branch tree is checked independently after the message-only rewrite Pre-commit hook ran. Total eslint: 0, total circular: 1
Cline can expose a temporarily invalid whole-document rewrite and Cursor CLI can temporarily present an incompatible lineage. Re-evaluating the same rejected physical snapshot on every poll wasted CPU and repeatedly exercised large parsing paths. Persist a compact rejection watermark keyed by source identity, parser version, size, timestamp, and fingerprint. Continue serving the previous valid generation until the physical source changes, then clear the watermark only after a valid generation is published atomically. Verification: - Rejected Cline and Cursor CLI snapshot regression tests: PASS - `cargo test -p orgtrack_core`: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
The first prewarm implementation returned a bounded SessionEvent array to JavaScript and then sent the same array back to Rust for EventStore application. That preserved an avoidable Rust to JS to Rust copy and allowed stale prewarm completions to outlive a session visit. Read, normalize, persist Shell payloads, and apply the bounded window directly in Rust. Add prewarm episode guards, reject native SDE identities before allocating replay state, and keep foreground watchers independent from read-only prewarming. Verification: - Replay RPC schema and Cursor prewarm guard tests: PASS - `cargo test -p org2 agent_sessions::event_pipeline`: PASS - Native SDE external-replay call count remains zero Pre-commit hook ran. Total eslint: 0, total circular: 1
Several secondary paths could still retain session-sized data after the initial adapter work: replay caches lacked a complete byte policy, payload roots could be decoded eagerly, and Shell, CLI, exporter, and provenance consumers could bypass bounded projection. Add byte-aware LRU and TTL cache policy, explicit payload encodings and bounded body projections, canonical Shell artifacts, serialized cache access, source routing contracts, and bounded loaders for CLI, export, and provenance. Preserve exact range hydration for callers that explicitly request the original payload. Verification: - 30 MiB to 300 MiB bounded-growth tests: PASS - 10 MiB Shell subprocess and RSS plateau stress tests: PASS - `cargo test -p agent_core`: 3,090 passed, 2 ignored - `cargo test -p perf_utils`: 68 passed Pre-commit hook ran. Total eslint: 0, total circular: 1
Small unit fixtures did not prove the hard memory, IPC, cursor, reset, lifecycle, and source-registration requirements behind Issue 443, and they did not protect the native SDE and prior memory fixes from regression. Add deterministic large-history performance cases, cache and schema assertions, rendered replay coverage, and independent architecture and frontend audit reports. Cover hard event limits, source parity, unchanged polling, streamed payloads, and the Issue 425, 435, and 446 compatibility boundaries. Verification: - Bounded replay performance and cache tests: PASS - Replay RPC contract tests: PASS - Rendered chat replay scenarios: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
Update the issue branch to the develop snapshot that makes release downloads use bounded retries, avoiding a late integration against stale updater and application scaffolding. The merge required no replay-specific behavior change. The bounded external-history implementation and native SDE boundary remain unchanged. Verification: - Subsequent branch-wide Rust and Vitest gates recorded in the Issue 443 audit: PASS - Final branch tree is checked independently after the message-only rewrite Pre-commit hook ran. Total eslint: 0, total circular: 1
Concurrent opens and secondary readers could update the same compact replay cache at the same time, producing avoidable duplicate work and risking stale state publication across OpenCode, export, history, and provenance consumers. Serialize cache mutation per source session, keep read-only consumers on bounded query paths, and prevent stale renderer retention from reintroducing a released replay window. Preserve generation and revision guards while avoiding duplicate EventStore work. Verification: - `cargo test -p orgtrack_core`: PASS - `cargo test -p org2 agent_sessions::event_pipeline`: PASS - Session-scoped retention and replay schema tests: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
Synthetic fixtures could not reproduce the real cost of opening, switching away from, and reopening the 335 MiB Codex Issue 272 session through the rendered Tauri application. Add an isolated real-session E2E path that discovers the source through the production rescan command, opens the bounded replay, performs ten release cycles, samples the complete application process tree, and compares native memory attribution with vmmap. Verification: - Real Issue 272 rendered E2E: PASS - First-open growth: 62.2 MiB within the 400 MiB limit - Ten-cycle measured-tail and backend step growth: 0 MiB - Native versus vmmap difference: 26.6 MiB within tolerance Pre-commit hook ran. Total eslint: 6, total circular: 1
Bring in the develop change that hydrates the initial chat page through block loaders so Issue 443 is tested against the current chat initialization path rather than an obsolete renderer flow. The merge required no replay-specific conflict resolution. Imported sessions remain bounded-replay consumers and native sessions retain the develop block-loader behavior. Verification: - Subsequent SessionCore and chat rendering tests: PASS - Final branch-wide Vitest and Rust gates recorded in the Issue 443 audit: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
Sampling memory only one second after release confused delayed allocator reclamation with retained replay state and produced a renderer-dominated false failure without proving an actual leak. Keep the 250 MiB acceptance threshold unchanged but sample a bounded 30-second idle window after explicit release. Record the lifecycle rationale and verify that request, watcher, payload, and EventStore ownership remain released while the allocator settles. Verification: - Real Issue 272 idle-release E2E: PASS - Settled growth after the bounded idle window: 0 MiB - No threshold was relaxed to manufacture the pass Pre-commit hook ran. Total eslint: 6, total circular: 0
Integrate the develop runtime incremental-scan work before final acceptance so replay routing, source discovery, session aggregation, and collaboration behavior are evaluated on the same code that will receive Issue 443. Resolve conflicts across Codex indexing, shared session event types, replay commands, source descriptors, SessionCore adapters, and collaboration imports. Preserve develop incremental scanning while retaining the exhaustive 15-source bounded replay contract and native SDE separation. Verification: - Merge-conflict focused Vitest suite: 58 passed - Full Rust and Vitest gates were rerun after conflict resolution Pre-commit hook ran. Total eslint: 6, total circular: 0
Update the issue branch to the develop snapshot that expands usage trends before running the final static, Rust, real SQLite, and rendered Issue 272 acceptance gates. The merge introduced no product conflict in bounded replay. Follow-up test fixtures were updated separately to reflect the current native history-mode and webview observer behavior. Verification: - Relevant post-merge targeted tests: PASS - Full Vitest suite after fixture alignment: PASS - Final real-data gates were rerun on the merged tree Pre-commit hook ran. Total eslint: 6, total circular: 0
The latest develop merge changed native history-mode classification and webview visibility observation, leaving two test fixtures modeled on obsolete behavior. Final reports also needed the current real-data measurements rather than pre-merge evidence. Align the native SDE reconciliation fixture with persisted-database mode, update the webview harness for IntersectionObserver and ResizeObserver, tighten real-session discovery, and record the final architecture and rendered-E2E results. No production behavior is changed by the fixture updates. Verification: - Updated targeted Vitest cases: 7 passed - Full Vitest suite: PASS - Real 3.0 GiB SQLite migration and reopen: PASS - Real 335 MiB Issue 272 rendered E2E: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
Merge the final develop snapshot that persists the selected organization so the Issue 443 branch is based on the latest shared application state before delivery. The merge required no bounded-replay conflict resolution. The final static checks, Rust suites, real SQLite migration, and rendered Issue 272 lifecycle test were evaluated after this synchronization. Verification: - `pnpm typecheck` and full Vitest suite: PASS - Required Rust package suites: PASS - Real SQLite and Issue 272 rendered acceptance: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
The audit reports still contained intermediate counts and pre-final-merge wording, which made it difficult for reviewers to distinguish completed acceptance evidence from external infrastructure that remained unavailable. Record the final Vitest count, real SQLite migration and reopen measurements, Issue 272 first-open and lifecycle memory results, native vmmap comparison, and the explicit cloud credential blocker. Keep baseline circular and workspace clippy findings separate from Issue 443 results. Verification: - Final audit evidence cross-checked against completed command output - `git diff --check`: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
92f2065 to
1f34dae
Compare
|
Reviewed with fable: Verdict: high-quality engineering with four blocking-caliber problems. This is really a re-architecture (~53k production/test lines across 290 files), not a "fix": a new bounded-replay subsystem for 15 external CLI sources, a v10→v12 streamed turn-index rebuild, a cloud-transport rewrite, and a staged collaboration-snapshot ingest. The code is unusually disciplined — parameterized SQL throughout, checked arithmetic, TTL'd/capped registries, an atomic rollback-tested migration, and tests that were strengthened rather than gutted. But five parallel deep-dives (Rust replay, event-pipeline commands, turn-index/CLI, frontend sync, frontend API/UI) plus cross-checks surfaced the following, in priority order. Blocking findings
Major operational risks
Moderate Recommendation |
|
What the PR is actually solving The fix's core idea is small: replace "load everything" with bounded windows (max 10 turns / 200 events / 4 MiB per IPC response), backed by a persistent SQLite index of compact rows plus payload locators — pointers back into the source file so large payloads are range-read in ≤256 KiB slices only when actually viewed. The PR's own numbers: first open of the 335 MB Codex session costs 62 MB once, then repeat opens/polls cost ~0. Why that costs 60k lines ~20k — the new replay subsystem in orgtrack-core. "Bounded" can't be implemented generically because the 15 supported sources store sessions in genuinely different shapes: append-only JSONL logs, Cursor's SQLite KV bubble store, whole-JSON-file formats, Qoder's sidecar. So there are six storage drivers plus the shared index, cache policy, registry, metadata projection, and content-addressed payload artifacts — and each driver must handle incremental sync, torn tails mid-write, UTF-8-safe range reads, and turn-header reconstruction on its own format. The 15-source × per-format-driver matrix is the single biggest multiplier (and, per my earlier pass, carries some avoidable copy-paste between drivers). |
|
Worth requesting on this PR
utf8_boundary_at_or_before/after — identical copies in jsonl_driver.rs:1643 and codex_jsonl.rs:1362
Better as follow-ups
Net effect of the pre-merge items: roughly 1.5–2k lines deleted, three findings fixed structurally rather than patched, and the drift-prone format/validation logic reduced to single sources of truth. If you want, I can draft this as review comments on the PR. |
|
Reuse already happens, ad hoc, in three of six drivers: whole_json_driver imports structured_driver::{compact_chunk, range_from_text, rebuild_turns} — the model citizen The three extractions, biggest first
The payoff and the caveat The caveat: don't push past that. The per-format sync bodies for the SQLite/structured/whole-JSON families do genuinely different work (KV bubble traversal vs. table diffing vs. array diffing), and forcing all six under one universal template would be the wrong abstraction. The high-confidence extractions are the ones above, where the code is already identical or already half-shared. |
|
Worth consolidating
replay request epochs — TTL 5 min, cap 64, LRU (external_replay.rs:4814)
Deliberately not consolidating |
2 participants
Summary
Validation
Known external or develop baselines
Relates to #443