diff --git a/docs/architecture-audit-2026-07-23/Issue443.md b/docs/architecture-audit-2026-07-23/Issue443.md new file mode 100644 index 000000000..91fd18f35 --- /dev/null +++ b/docs/architecture-audit-2026-07-23/Issue443.md @@ -0,0 +1,43 @@ +# Issue #443 architecture audit + +Scope: bounded replay for all built-in imported-history sources, managed CLI mirrors, replay consumers, SQLite turn-index projection, and the native-SDE isolation boundary. + +## Ten-layer review + +| Layer | Evidence checked | Verdict | +| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1. Compilation and contracts | Rust/TS wire schemas, targeted `cargo check`, typecheck, serialized compatibility tests | Pass for the #443 change. Workspace `cargo check`, typecheck, lint, full Vitest, circular analysis, and task-scoped `-D warnings` clippy completed. Strict workspace clippy is blocked only by byte-identical `develop` baseline findings listed below. | +| 2. Old/dead paths | Global caller sweep for `loadFullTranscriptChunks`, `supportsWindowedReplay`, `cursorIdeFullRefresh`, `cli_agent_chunks`, and provider full loaders | Pass; production callers are zero. Removed the remaining Cursor full-refresh API. Legacy provider loaders remain only as parser/test compatibility surfaces. | +| 3. Naming consistency | `ReplayCursor`, generation/revision, source/session identity, descriptor/payload naming | Pass | +| 4. Semantic overloading | Core descriptor encoding versus app `PayloadRefEncoding`; EventStore hydration modes | Pass after making generation resets use `set_external_replay_window`. The two encoding enums intentionally separate persistent-core compatibility from the public event wire. | +| 5. Defaults and fallbacks | Unknown source routing, missing adapter behavior, legacy cached payload descriptors, watcher fallback | Pass. Unknown built-ins fail closed; only old cached descriptors may infer encoding. Watcher failure returns the typed active/visible bounded poll fallback. | +| 6. Domain boundaries | Native SDE versus imported/managed CLI, collaboration snapshot secondary reads | Pass. Native adapters never resolve the primary replay target; Rust validates identities before creating watcher/request state. | +| 7. Registry clarity | Rust authoritative 15-source registry, TS metadata mirror, managed native transcript remap | Pass after centralizing an exhaustive compile-time storage-driver classifier shared by sync and payload reads. | +| 8. Wire and memory bounds | 10 turns, 200 events, 4 MiB IPC, 256 KiB payload ranges/cloud segments, EventStore byte cap | Pass in code, wire tests, deterministic 30/300 MiB performance tests, a 3.0 GiB real SQLite DB migration, and both core and rendered-app runs of the 335 MiB Issue 272 Codex JSONL. | +| 9. Entry-point parity | Open, poll, older/seek, prewarm, metadata/hover, provenance, Fork, Cloud, Raw Transcript, JSON/Markdown export | Pass. Background/read-only consumers use compact indexes or bounded scans and do not acquire foreground watchers. | +| 10. Resolver symmetry | Source/session validation, public-to-native cursor remap, generation/revision pinning, reset/release/late-result guards | Pass. Sync and payload range now share the same exhaustive driver routing. | + +## Findings fixed during the audit + +| Severity | Finding | Resolution | +| -------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| P1 | A replay generation reset called generic `EventStore::set`, marking a bounded suffix as fully hydrated. | Use `set_external_replay_window` and assert `HydrationMode::RoundWindow`. | +| P1 | Driver dispatch used guarded matches plus runtime catch-alls, so a future source could compile without a real adapter. | Added exhaustive `replay_driver_kind` and a storage-family contract test. | +| P2 | Cursor IDE's unused `load_full_refresh_for_session` production API remained after consumer migration. | Removed the type/function; retained the single-bubble parser used by SQLite/KV replay. | +| P1 | Payload meaning was inferred from a dotted field path and array export recursion reused the parent path. | Added explicit encoding/body projection and indexed array paths, with legacy-cache compatibility only. | + +## Acceptance boundary + +- All 15 built-in sources must stay `Incremental` and route through the exhaustive driver classifier. +- A native SDE session must make zero external replay calls and retain its existing EventStore/FSM behavior. +- No renderer API may expose a built-in full-transcript loader. +- Completed evidence on the final tree includes 711 Vitest files / 6,426 tests, 482 passing `orgtrack_core` tests (4 intentionally ignored), 416 event-pipeline tests, 3,088 passing `agent_core` tests plus both loopback-only tests rerun successfully outside the sandbox, 68 `perf_utils` tests, 42 `orgtrack_cli` tests, 13 turn-index tests, the #425 ignored RSS stress, streamed 327.1 MiB export hash verification, bounded 30→300 MiB growth, and first-open/reopen of the real DB. +- The Issue 272 core run parsed 52,477 rows from a 335.2 MiB JSONL into a 200-event/10-turn window, then performed an unchanged poll with zero parsed bytes and reopened in 55.64 ms. Cold open completed in 19.637 seconds with 39.4 MiB peak RSS growth. +- The final-tree real SQLite clone migrated 5,227 events / 199 turns from index v11 to v12 in 595.138 ms, reopened in 44.888 ms, and increased peak RSS by 93.0 MiB. The original `~/.orgii/sessions.db` was never opened writable. +- The final-tree rendered Issue 272 run opened and released the real session ten times with 26 events per window. First-open growth was 0 MiB relative to the sampled 609.9 MiB startup baseline, measured-tail step growth was 0 MiB, backend step growth was 0 MiB, and the 30-second idle settled at 494.1 MiB with 0 MiB growth. The #435 native/vmmap comparison was 609.9 MiB versus 566.7 MiB, a 43.2 MiB difference within the acceptance threshold. +- A separate Computer Use pass opened the same 351,503,887-byte Issue 272 source through the visible sidebar and used `Command+5` to inspect runtime behavior. Reopen performed one bounded `es_merge_events`, twelve idle seconds produced zero API calls / zero likely polling, switching away issued `unsubscribe_session_events`, and A→B→A restored the same newest turn without a late result. +- Deterministic performance stayed within every hard gate: 30 MiB cold-open growth was 9.7 MiB, extending to 300 MiB added 52.5 MiB, and the 327.1 MiB streamed export retained SHA-256 `a8ba9a1712ca6758f85f52101855d530b410f6fa12b716c78b0d7932c1ecfa3d` while adding 16.1 MiB peak RSS. +- New #443 production files stay below the 1,500-line hard limit. The largest is the Codex JSONL driver at 1,370 lines; the Qoder sidecar driver is 1,316 lines and the managed-chunk bridge is 1,301 lines. +- The live dual-instance sharing/presence spec now loads correctly and reports all 12 live scenarios as `SKIP`; execution remains externally blocked because no `E2E_CLOUD_*` test credentials are configured. +- `pnpm check:circular` processed 5,849 files and found no circular dependency. ESLint completed with zero errors and two non-blocking warnings in byte-identical `develop` Cloud Realtime code. +- Strict workspace clippy is still blocked by pre-existing findings in files byte-identical to `upstream/develop`: an item-after-test-module in `bin-gateway-chat-cli`, duplicated test modules in `perf_utils`, the existing usage-dashboard argument list, Key Vault enum naming, several existing Agent Core type/trait/style lints, and two existing ORG2 doc/constant-assertion lints. A scoped `-D warnings` run for all six #443 crates passed after allowing only those verified baseline lint classes on the command line; no source-level blanket allow was added. diff --git a/docs/e2e-testing-2026-07-23/Issue443.md b/docs/e2e-testing-2026-07-23/Issue443.md new file mode 100644 index 000000000..663e72d16 --- /dev/null +++ b/docs/e2e-testing-2026-07-23/Issue443.md @@ -0,0 +1,29 @@ +# Issue #443 rendered E2E evidence + +Scope: the real 335 MiB Codex “Issue 272” session, external replay open/switch/close lifecycle, native memory attribution, and the requested dual-instance sharing/presence smoke. + +## Evidence matrix + +| Surface | Result | Production-path evidence | Result details | +| --------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Real Codex first open | PASS | The rendered Tauri app indexes the isolated external root through the production rescan command, resolves the imported Codex session through the production bounded-replay adapter, and opens the chat session. | 26 events were hydrated; first-open Physical Footprint did not exceed the sampled 609.9 MiB startup baseline, against the 400 MiB growth limit. | +| Ten open/release cycles | PASS | Each cycle opens the same real external session, switches back to New Session, releases the replay owner, and measures the app process tree. | Measured-tail step growth was 0 MiB and backend step growth was 0 MiB. | +| Idle release / #446 compatibility | PASS | After the final explicit release, the app remains idle while native process memory is sampled every five seconds for a bounded 30-second window. | The app settled to 0 MiB over baseline without lowering the 250 MiB threshold. | +| #435 native attribution | PASS | Every process row returned by `get_app_memory_snapshot_v1` is compared with `/usr/bin/vmmap -summary`. | Native total 609.9 MiB versus vmmap 566.7 MiB; 43.2 MiB difference is within `max(10%, 50 MiB)`. | +| Hard event bound | PASS | The open-session result returns only compact identity and event count; it does not serialize the full event store back through WebDriver. | Every cycle returned 26 events, below the 200-event hard cap. | +| `Command+5` manual runtime check | PASS | Computer Use opens the 351,503,887-byte source from the visible sidebar, observes the monitor, switches away, and reopens the same session. | One bounded merge per reopen, zero calls during a 12-second idle window, normal unsubscribe on release, and no A→B→A late-result resurrection. | +| Dual-instance sharing/presence | BLOCKED | `cloud-dual-instance-ui.spec.mjs` loads on the current tree and reaches its live credential gate. | All 12 live scenarios are explicitly skipped because `E2E_CLOUD_SUPABASE_URL`, `E2E_CLOUD_ANON_KEY`, and a service-key or email/password credential are absent. | + +## Failure-driven harness correction + +An earlier post-merge run sampled each release after only one second and failed with a renderer-dominated 330.4 MiB “settled” growth even though the backend and measured-tail step growth stayed bounded. The acceptance test preserves the same 250 MiB limit but adds a bounded 30-second idle sampling window, distinguishing delayed allocator reclamation from retained live replay state without manufacturing a pass. + +The first current-head rerun used a brand-new isolated ORGII home and correctly failed before memory measurement because the test tried to open the imported session before any catalog scan. The harness now establishes the same precondition as the real sidebar/data-source flow by calling the production `external_history_rescan_source` command. It also sets `ORGII_EXTERNAL_HISTORY_HOME` to an isolated root containing only the real Issue 272 JSONL. The open/release assertions still use the production session adapter; no full cache or debug-only success path is seeded. + +## Commands + +- `E2E_ISOLATED_RUN=1 E2E_ORGII_HOME= ORGII_EXTERNAL_HISTORY_HOME= E2E_ISSUE_443_REAL_CODEX_SESSION_ID=codexapp-rollout-2026-07-12T01-13-48-019f522b-c7de-7c50-885f-ab6790d68469 E2E_CHAT_RENDERING_SCENARIOS=issue-443-real-codex pnpm test -- --spec './specs/core/external-replay-ui.spec.mjs'`: PASS on the final tree, one real rendered scenario in 2m 41s including service startup and graceful cleanup. +- Computer Use: launched the current debug binary from a uniquely identified temporary app bundle, opened the real Issue 272 source through the normal sidebar, toggled `Command+5`, exercised expand, idle, switch, reopen, and quit, then verified the frontend, IDE server, and WebDriver ports were all released. +- `E2E_ISOLATED_RUN=1 E2E_ORGII_HOME= pnpm test -- --spec './specs/core/cloud-dual-instance-ui.spec.mjs'`: current-tree infrastructure gate reached; all 12 live scenarios explicitly BLOCKED/SKIPPED for missing cloud credentials. + +The large real JSONL and isolated ORGII homes remain outside Git. diff --git a/docs/frontend-ui-audit-2026-07-23/Issue443.md b/docs/frontend-ui-audit-2026-07-23/Issue443.md new file mode 100644 index 000000000..4eb05bb31 --- /dev/null +++ b/docs/frontend-ui-audit-2026-07-23/Issue443.md @@ -0,0 +1,20 @@ +# Issue #443 frontend UI audit + +The repository-listed `frontend-ui-audit` skill was unavailable at both configured paths, so this report applies its documented scope manually to the eight changed TSX files. + +| Line | Element | Verdict | Reason | Suggested change | +| ----------------------------------------- | ------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | +| `TurnPageList.tsx:59` | Virtualized turn selector | keep with reason | Reuses existing dropdown tokens and icon-button primitive; virtualization removes the session-sized derived item array without changing interaction semantics. | None | +| `BlockOutput.tsx:293` | External payload range controls | keep with reason | Uses the shared Button component, bounded 256 KiB reads, request invalidation, disabled states, and an alert role for failures. | None | +| `SessionRawTranscriptContent.tsx:47` | Per-payload range viewer | keep with reason | Uses design-system Button, semantic `article`/`time`/`role=alert`, existing color and spacing tokens, and never assembles the full payload. | None | +| `SessionRawTranscriptContent.tsx:274` | Virtualized external transcript | keep with reason | `react-virtuoso` keeps rows bounded and preserves logical anchors while older pages prepend; native transcripts retain the existing CodeMirror view. | None | +| `SessionRawTranscriptDialog/index.tsx:54` | Raw transcript dialog actions | keep with reason | Reuses Modal, Button and Message; Copy is disabled above the memory budget and streamed Export All is clearly exposed. Existing responsive modal dimensions are retained. | None | +| `SessionRawTranscriptView/index.tsx:15` | Workstation raw view | keep with reason | Delegates to the same audited bounded transcript component, avoiding a second UI pattern. | None | +| `CloudShareImportDialog.tsx:227` | Cloud import client swap | keep with reason | Behavioral transport substitution only; rendered component structure and styling are unchanged. | None | +| `SessionImportExportModal.tsx:128` | Streamed external export | keep with reason | Existing modal UX remains unchanged; only external sessions switch to the Rust streaming exporter. | None | + +Summary: 0 fix, 8 keep-with-reason, 0 abstract. No new arbitrary Tailwind values, duplicate dialog primitives, or basic accessibility regressions were found. + +The final module-splitting commit was rechecked on 2026-07-24. Its TSX changes are import, prop-type, and source-neutral naming moves only; rendered markup, CSS, ARIA, keyboard behavior, and the verdict counts above are unchanged. + +The final hardening pass added localized Raw Transcript status/error text for all 13 shipped locales without changing rendered structure, design-system components, keyboard behavior, or the audit verdict counts. diff --git a/scripts/dev/webpack-server.js b/scripts/dev/webpack-server.js index 86a403aa9..697662891 100755 --- a/scripts/dev/webpack-server.js +++ b/scripts/dev/webpack-server.js @@ -113,9 +113,12 @@ try { // Add compiler hooks for better feedback let isFirstCompile = true; let compileStartTime = Date.now(); + let compileReady = false; + let completedCompile = false; compiler.hooks.compile.tap("ORGIIDevServer", () => { compileStartTime = Date.now(); + compileReady = false; if (!isFirstCompile) { process.stdout.write("WEBPACK_STATUS:recompiling\n"); } @@ -125,6 +128,8 @@ try { const hasErrors = stats.hasErrors(); const hasWarnings = stats.hasWarnings(); const ms = Date.now() - compileStartTime; + completedCompile = true; + compileReady = !hasErrors; if (hasErrors) { // Print the full error details so they appear in the terminal scroll @@ -155,6 +160,11 @@ try { } }); + compiler.orgiiCompileStatus = () => ({ + completed: completedCompile, + ready: compileReady, + }); + // Suppress progress messages for cleaner output compiler.hooks.infrastructureLog.tap("ORGIIDevServer", (name, type, args) => { // Suppress verbose infrastructure logs @@ -190,6 +200,19 @@ const devServerOptions = { ...config.devServer, open: false, // Don't auto-open browser setupMiddlewares: (middlewares, devServer) => { + middlewares.unshift({ + name: "orgii-webpack-readiness", + middleware: (req, res, next) => { + if (req.url !== "/__orgii_webpack_ready__") { + next(); + return; + } + const status = compiler.orgiiCompileStatus(); + res.statusCode = status.ready ? 200 : 503; + res.setHeader("Content-Type", "application/json; charset=utf-8"); + res.end(JSON.stringify(status)); + }, + }); middlewares.unshift({ name: "orgii-dev-request-diagnostics", middleware: (req, res, next) => { diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 04747b6dc..e55c23505 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -5153,6 +5153,7 @@ dependencies = [ "dirs", "dispatch2", "file_ops", + "flate2", "foreign-types", "futures-util", "git", @@ -5164,6 +5165,7 @@ dependencies = [ "libc", "log", "lsp", + "notify", "objc2", "objc2-app-kit", "objc2-foundation", @@ -5248,6 +5250,7 @@ dependencies = [ "core_types", "dirs", "git2", + "libc", "orgtrack_protocol", "orgtrack_sync", "prost", @@ -5258,6 +5261,7 @@ dependencies = [ "serde_json", "sha2", "similar", + "tempfile", ] [[package]] @@ -7258,6 +7262,7 @@ dependencies = [ "chrono", "core_types", "database", + "libc", "orgtrack_core", "rusqlite", "serde", diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index ef475abb5..0dbf07c82 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -106,7 +106,7 @@ tokio = { version = "1", features = ["full"] } tokio-util = { version = "0.7", default-features = false } petgraph = { version = "0.6.4", default-features = false, features = ["serde-1"] } chrono = { version = "0.4", features = ["serde"] } -rusqlite = { version = "0.32", features = ["bundled"] } +rusqlite = { version = "0.32", features = ["bundled", "blob"] } axum = { version = "0.8", features = ["ws"] } # Modern Objective-C bindings (replaces the unmaintained `objc 0.2` / @@ -202,6 +202,7 @@ tauri-plugin-notification = "2" tauri-plugin-webdriver-automation = { version = "0.1", optional = true } portable-pty = "0.9" tokio = { workspace = true } +notify = { workspace = true } uuid = { version = "1", features = ["v4"] } # Shared cross-crate type definitions (Phase 1 of workspace migration). @@ -445,6 +446,7 @@ dirs = "6.0" # Get user directories (~/.orgii/data) chrono = { workspace = true } # Date/time handling zip = { workspace = true } base64 = { workspace = true } +flate2 = "1.1" # AWS SDK for SSO OIDC archived alongside `crates/cursor-bridge` notes. # `aws-config` + `aws-sdk-ssooidc` (and the transitive `aws-lc-sys` BoringSSL diff --git a/src-tauri/crates/agent-core/src/core/interaction/plan_approval/manager.rs b/src-tauri/crates/agent-core/src/core/interaction/plan_approval/manager.rs index 768f4b8da..58181d688 100644 --- a/src-tauri/crates/agent-core/src/core/interaction/plan_approval/manager.rs +++ b/src-tauri/crates/agent-core/src/core/interaction/plan_approval/manager.rs @@ -11,7 +11,9 @@ use super::events::{build_plan_approval_event, PlanApprovalCardStatus}; use super::gc::{persist_blocking, persist_ready_row}; use super::persistence::PlanApprovalStore; use super::resolution::{resolve_pending, PlanResolution}; -use super::snapshot::{auto_approve_deadline_ms, plan_id_for, revision_id_for, PendingPlanApproval}; +use super::snapshot::{ + auto_approve_deadline_ms, plan_id_for, revision_id_for, PendingPlanApproval, +}; use super::watcher::spawn_auto_approve_watcher; pub struct PlanApprovalManager { diff --git a/src-tauri/crates/agent-core/src/core/session/persistence/crud/mod.rs b/src-tauri/crates/agent-core/src/core/session/persistence/crud/mod.rs index 6330d6395..aeb5b4bba 100644 --- a/src-tauri/crates/agent-core/src/core/session/persistence/crud/mod.rs +++ b/src-tauri/crates/agent-core/src/core/session/persistence/crud/mod.rs @@ -24,7 +24,7 @@ mod workspace; pub use migration::ensure_unified_schema; pub use ops::{ backfill_agent_definition_id, delete_session, finalize_terminal_turn_status, - get_child_sessions, get_parent_session, get_session, list_sessions, + get_child_sessions, get_parent_session, get_session, list_root_sessions, list_sessions, mark_stale_running_sessions_abandoned, reconcile_sessions_with_terminal_turn_markers, register_session_delete_mirror_hook, register_session_mirror_hook, update_account_id, update_agent_exec_mode, update_draft_text, update_model, update_model_and_account, update_name, diff --git a/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops.rs b/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops.rs index 40099026c..04276de6c 100644 --- a/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops.rs +++ b/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops.rs @@ -198,11 +198,32 @@ pub fn get_session(session_id: &str) -> SqliteResult SqliteResult> { + list_sessions_with_parent_scope(filter, false) +} + +/// List only top-level sessions with optional filtering. +/// +/// This is the root-only counterpart used by bounded sidebar pagination. +/// Child sessions remain available through [`list_sessions`] and +/// [`get_child_sessions`], but they cannot consume `LIMIT`/`OFFSET` slots +/// intended for user-visible root rows. +pub fn list_root_sessions(filter: &SessionListFilter) -> SqliteResult> { + list_sessions_with_parent_scope(filter, true) +} + +fn list_sessions_with_parent_scope( + filter: &SessionListFilter, + root_only: bool, +) -> SqliteResult> { let conn = get_connection()?; let mut conditions: Vec = Vec::new(); let mut params_vec: Vec> = Vec::new(); + if root_only { + conditions.push("(s.parent_session_id IS NULL OR s.parent_session_id = '')".to_string()); + } + if let Some(ref type_name) = filter.type_name { conditions.push(format!("s.session_type = ?{}", params_vec.len() + 1)); params_vec.push(Box::new(type_name.clone())); diff --git a/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops_tests.rs b/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops_tests.rs index 8da6c238b..d8bf653e1 100644 --- a/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops_tests.rs +++ b/src-tauri/crates/agent-core/src/core/session/persistence/crud/ops_tests.rs @@ -20,10 +20,10 @@ //! `list_sessions` pair without inventing a parallel test fixture. use super::ops::{ - delete_session, finalize_terminal_turn_status, list_sessions, + delete_session, finalize_terminal_turn_status, list_root_sessions, list_sessions, reconcile_sessions_with_terminal_turn_markers, upsert_session, }; -use super::record::UnifiedSessionRecord; +use super::record::{session_type, UnifiedSessionRecord}; use crate::session::persistence; use crate::session::types::{SessionListFilter, SessionStatus}; use core_types::key_source::KeySource; @@ -89,20 +89,33 @@ fn ensure_test_schema() { } fn seed_session(session_id: &str, status: SessionStatus) { + seed_session_with_parent( + session_id, + status, + session_type::CODING, + None, + "2024-01-01T00:00:00Z", + ); +} + +fn seed_session_with_parent( + session_id: &str, + status: SessionStatus, + record_type: &str, + parent_session_id: Option<&str>, + updated_at: &str, +) { ensure_test_schema(); - // Status is the only column under test; everything else is filler - // matching the NOT NULL constraints in the schema. `session_type` - // is forced to a concrete value (`"sde"`) because the default - // `list_sessions` filter already excludes the `gateway` type. - let now = "2024-01-01T00:00:00Z".to_string(); + let created_at = "2024-01-01T00:00:00Z".to_string(); let record = UnifiedSessionRecord { session_id: session_id.to_string(), name: format!("seed-{session_id}"), status: status.as_str().to_string(), - created_at: now.clone(), - updated_at: now, - session_type: "sde".to_string(), + created_at, + updated_at: updated_at.to_string(), + session_type: record_type.to_string(), key_source: KeySource::OwnKey, + parent_session_id: parent_session_id.map(str::to_string), ..Default::default() }; upsert_session(&record).expect("seed upsert"); @@ -188,6 +201,82 @@ fn list_sessions_default_filter_excludes_archived() { ); } +#[test] +fn root_page_offsets_ignore_dense_org_member_children() { + let _sandbox = test_env::sandbox(); + + for (root_index, second) in [50, 30, 10].into_iter().enumerate() { + let root_id = format!("sdeagent-root-{root_index}"); + seed_session_with_parent( + &root_id, + SessionStatus::Completed, + session_type::CODING, + None, + &format!("2026-07-26T12:00:{second:02}Z"), + ); + if root_index < 2 { + for child_index in 1..=9 { + seed_session_with_parent( + &format!("{root_id}:member:{child_index}"), + SessionStatus::Completed, + session_type::ORG_MEMBER, + Some(&root_id), + &format!("2026-07-26T12:00:{:02}Z", second - child_index), + ); + } + } + } + + let first_page_filter = SessionListFilter { + type_names: Some(vec![ + session_type::CODING.to_string(), + session_type::ORG_MEMBER.to_string(), + ]), + limit: Some(3), + offset: Some(0), + ..Default::default() + }; + let first_page = list_root_sessions(&first_page_filter).expect("load first root-only page"); + assert_eq!( + first_page + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + vec!["sdeagent-root-0", "sdeagent-root-1", "sdeagent-root-2"] + ); + + let second_page_filter = SessionListFilter { + limit: Some(3), + offset: Some(2), + ..first_page_filter + }; + let second_page = list_root_sessions(&second_page_filter).expect("load second root-only page"); + assert_eq!( + second_page + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + vec!["sdeagent-root-2"] + ); + + let unscoped = list_sessions(&SessionListFilter { + type_names: Some(vec![ + session_type::CODING.to_string(), + session_type::ORG_MEMBER.to_string(), + ]), + ..Default::default() + }) + .expect("ordinary listing still includes children"); + assert_eq!( + unscoped + .iter() + .filter(|session| session.parent_session_id.is_some()) + .count(), + 18, + "root pagination must not remove children from the ordinary APIs" + ); +} + #[test] fn terminal_turn_finalize_updates_session_status_and_marker() { let _sandbox = test_env::sandbox(); diff --git a/src-tauri/crates/agent-core/src/core/session/persistence/mod.rs b/src-tauri/crates/agent-core/src/core/session/persistence/mod.rs index a4217524b..56ff47fc0 100644 --- a/src-tauri/crates/agent-core/src/core/session/persistence/mod.rs +++ b/src-tauri/crates/agent-core/src/core/session/persistence/mod.rs @@ -24,7 +24,7 @@ mod messages; pub use crud::{ backfill_agent_definition_id, clear_worktree_metadata, delete_session, finalize_terminal_turn_status, get_child_sessions, get_parent_session, get_session, - list_sessions, load_workspace, mark_stale_running_sessions_abandoned, + list_root_sessions, list_sessions, load_workspace, mark_stale_running_sessions_abandoned, reconcile_sessions_with_terminal_turn_markers, register_session_delete_mirror_hook, register_session_mirror_hook, save_workspace, save_worktree_metadata, session_type, update_account_id, update_agent_exec_mode, update_draft_text, update_model, diff --git a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/external_replay.rs b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/external_replay.rs index 2357c1308..6618bd10e 100644 --- a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/external_replay.rs +++ b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/external_replay.rs @@ -13,60 +13,234 @@ use core_types::session_event::{ EventDisplayStatus, SessionEvent, ShellReplayBookmark, ShellReplayRef, ShellReplayState, ShellReplayStatus, }; +use sha2::{Digest, Sha256}; use super::shell_replay::{ - load_complete_replay_state_if_matches, load_replay_state, resolve_replay_root, - ShellReplayStream, ShellReplayTarget, ShellReplayWriter, SHELL_REPLAY_FORMAT_VERSION, - SHELL_REPLAY_FRAME_MAX_BYTES, SHELL_REPLAY_PREVIEW_BYTES, + complete_terminal_prefix_len, load_complete_replay_state_if_matches, load_replay_state, + resolve_replay_root, ShellReplayStream, ShellReplayTarget, ShellReplayWriter, + SHELL_REPLAY_FORMAT_VERSION, SHELL_REPLAY_FRAME_MAX_BYTES, SHELL_REPLAY_PREVIEW_BYTES, + SHELL_REPLAY_RANGE_MAX_BYTES, }; #[derive(Clone, Copy)] -struct OutputPart<'a> { - stream: ShellReplayStream, - text: &'a str, +pub struct ExternalShellInlineSegment<'a> { + pub stream: ShellReplayStream, + pub text: &'a str, +} + +/// One externally-addressable stdout/stderr payload that belongs in a shell +/// replay. `locator` is deliberately opaque to `agent-core`: JSONL byte spans, +/// SQLite row keys and replay-artifact references remain owned by their source +/// adapters. +#[derive(Debug, Clone)] +pub struct ExternalShellReplaySegment { + pub stream: ShellReplayStream, + pub locator: L, + pub expected_bytes: u64, + /// A bounded source-provided tail used only when range hydration fails + /// before the durable replay can be completed. + pub preview: String, +} + +impl ExternalShellReplaySegment { + pub fn new( + stream: ShellReplayStream, + locator: L, + expected_bytes: u64, + preview: impl Into, + ) -> Self { + Self { + stream, + locator, + expected_bytes, + preview: preview.into(), + } + } } /// Persist completed, replay-less shell events produced by external provider /// parsers. Events that already own a replay are left untouched. pub fn persist_external_shell_replays(events: &mut [SessionEvent]) { for event in events { - if event.ui_canonical != core_types::tool_names::RUN_SHELL - || event.shell_replay.is_some() - || event.display_status == EventDisplayStatus::Running - { - continue; + let base = event.call_id.as_deref().unwrap_or(&event.id); + let call_id = external_shell_inline_identity(event) + .map(|identity| format!("{base}-external-{identity}")) + .unwrap_or_else(|| base.to_string()); + persist_external_shell_replay_inline(event, &call_id); + } +} + +/// Inline counterpart with an explicit durable artifact key. Imported replay +/// callers provide a content-addressed key. Length alone is never accepted as +/// identity, so a same-sized output update cannot reuse a stale `.slog`. +pub fn persist_external_shell_replay_inline(event: &mut SessionEvent, artifact_call_id: &str) { + if event.ui_canonical != core_types::tool_names::RUN_SHELL + || event.shell_replay.is_some() + || event.display_status == EventDisplayStatus::Running + { + return; + } + let parts = external_shell_inline_segments(event); + if parts.is_empty() { + return; + } + let expected_bytes = parts.iter().fold(0u64, |total, part| { + total.saturating_add(part.text.len() as u64) + }); + let replay_root = resolve_replay_root(); + match persist_one( + event, + artifact_call_id, + &replay_root, + &parts, + expected_bytes, + ) { + Ok(state) => event.shell_replay = Some(state), + Err(error) => { + // The source transcript remains authoritative, but EventStore + // still needs a bounded visible result instead of an empty card + // when durable import fails. + event.shell_replay = Some(incomplete_preview_state( + event, + artifact_call_id, + &parts, + format!("外部 CLI 完整输出保存失败：{error}"), + )); } - let parts = output_parts(event); - if parts.is_empty() { - continue; + } +} + +/// Persist one external shell event from storage-specific payload locators. +/// +/// The callback is never asked for more than +/// [`SHELL_REPLAY_RANGE_MAX_BYTES`] and each returned buffer is immediately +/// split into `.slog` frames. No complete stdout/stderr `String` is assembled. +/// A callback may return fewer bytes than requested, but returning an empty +/// buffer before the segment's declared length or returning more than the +/// requested range makes the event explicitly incomplete. +pub fn persist_external_shell_replay_segments( + event: &mut SessionEvent, + segments: &[ExternalShellReplaySegment], + read_range: ReadRange, +) where + ReadRange: FnMut(&L, u64, usize) -> Result, String>, +{ + let call_id = event.call_id.clone().unwrap_or_else(|| event.id.clone()); + persist_external_shell_replay_segments_with_call_id(event, &call_id, segments, read_range); +} + +/// Range-backed counterpart with an explicit generation-scoped artifact key. +pub fn persist_external_shell_replay_segments_with_call_id( + event: &mut SessionEvent, + artifact_call_id: &str, + segments: &[ExternalShellReplaySegment], + mut read_range: ReadRange, +) where + ReadRange: FnMut(&L, u64, usize) -> Result, String>, +{ + if event.ui_canonical != core_types::tool_names::RUN_SHELL + || event.shell_replay.is_some() + || event.display_status == EventDisplayStatus::Running + || segments.is_empty() + { + return; + } + + let replay_root = resolve_replay_root(); + let expected_bytes = segments.iter().try_fold(0u64, |total, segment| { + total.checked_add(segment.expected_bytes) + }); + let result = match expected_bytes { + Some(_) => persist_one_from_segments( + event, + artifact_call_id, + &replay_root, + segments, + &mut read_range, + ), + None => Err("external shell replay byte count overflow".to_string()), + }; + match result { + Ok(state) => event.shell_replay = Some(state), + Err(error) => { + event.shell_replay = Some(incomplete_segment_preview_state( + event, + artifact_call_id, + segments, + format!("外部 CLI Shell 完整输出保存失败：{error}"), + )); } - let expected_bytes = parts.iter().fold(0u64, |total, part| { - total.saturating_add(part.text.len() as u64) - }); - let call_id = event.call_id.clone().unwrap_or_else(|| event.id.clone()); - let replay_root = resolve_replay_root(); - match persist_one(event, &call_id, &replay_root, &parts, expected_bytes) { - Ok(state) => event.shell_replay = Some(state), - Err(error) => { - // The source transcript remains authoritative, but EventStore - // still needs a bounded visible result instead of an empty - // card when durable import fails. - event.shell_replay = Some(incomplete_preview_state( - event, - &call_id, - &parts, - format!("外部 CLI 完整输出保存失败：{error}"), - )); + } +} + +fn persist_one_from_segments( + event: &SessionEvent, + call_id: &str, + replay_root: &Path, + segments: &[ExternalShellReplaySegment], + read_range: &mut ReadRange, +) -> Result +where + ReadRange: FnMut(&L, u64, usize) -> Result, String>, +{ + let command = event + .command + .as_deref() + .or_else(|| event.args.get("command").and_then(|value| value.as_str())) + .unwrap_or("external shell command"); + let cwd = event + .args + .get("cwd") + .and_then(|value| value.as_str()) + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from(".")); + let target = ShellReplayTarget::new(&event.session_id, call_id); + let mut writer = ShellReplayWriter::create_detached(replay_root, target, command, &cwd)?; + + let write_result = (|| { + for segment in segments { + let mut offset = 0u64; + let mut frame_buffer = ExternalFrameBuffer::default(); + while offset < segment.expected_bytes { + let remaining = segment.expected_bytes - offset; + let requested = remaining.min(SHELL_REPLAY_RANGE_MAX_BYTES as u64) as usize; + let bytes = read_range(&segment.locator, offset, requested)?; + if bytes.is_empty() { + return Err(format!( + "{} payload ended at byte {offset}, expected {} bytes", + segment.stream.as_wire_str(), + segment.expected_bytes + )); + } + if bytes.len() > requested { + return Err(format!( + "{} payload range returned {} bytes for a {requested}-byte request", + segment.stream.as_wire_str(), + bytes.len() + )); + } + frame_buffer.push(&mut writer, segment.stream, &bytes)?; + offset = offset.saturating_add(bytes.len() as u64); } + frame_buffer.finish(&mut writer, segment.stream)?; } + Ok(()) + })(); + if let Err(error) = write_result { + writer.mark_incomplete(error.clone()); + return Err(error); } + + writer.finalize_at(ShellReplayStatus::Complete, None, event.created_at.clone())?; + load_replay_state(&event.session_id, call_id)? + .ok_or_else(|| "external shell replay manifest missing after finalize".to_string()) } fn persist_one( event: &SessionEvent, call_id: &str, replay_root: &Path, - parts: &[OutputPart<'_>], + parts: &[ExternalShellInlineSegment<'_>], expected_bytes: u64, ) -> Result { if let Some(state) = load_complete_replay_state_if_matches( @@ -119,7 +293,54 @@ fn append_text_bounded( Ok(()) } -fn output_parts(event: &SessionEvent) -> Vec> { +#[derive(Default)] +struct ExternalFrameBuffer { + pending: Vec, +} + +impl ExternalFrameBuffer { + fn push( + &mut self, + writer: &mut ShellReplayWriter, + stream: ShellReplayStream, + mut bytes: &[u8], + ) -> Result<(), String> { + while !bytes.is_empty() { + let available = SHELL_REPLAY_FRAME_MAX_BYTES.saturating_sub(self.pending.len()); + let take = available.min(bytes.len()); + self.pending.extend_from_slice(&bytes[..take]); + bytes = &bytes[take..]; + if self.pending.len() < SHELL_REPLAY_FRAME_MAX_BYTES { + continue; + } + + // Source adapters range over decoded strings, so a range or frame + // boundary may bisect a UTF-8 scalar/ANSI sequence. Keep only that + // tiny suffix for the next frame; never retain the payload body. + let ready = complete_terminal_prefix_len(&self.pending); + if ready == 0 { + return Err("external shell output has no complete terminal prefix".to_string()); + } + writer.append(stream, &self.pending[..ready])?; + self.pending.drain(..ready); + } + Ok(()) + } + + fn finish( + mut self, + writer: &mut ShellReplayWriter, + stream: ShellReplayStream, + ) -> Result<(), String> { + if !self.pending.is_empty() { + writer.append(stream, &self.pending)?; + self.pending.clear(); + } + Ok(()) + } +} + +pub fn external_shell_inline_segments(event: &SessionEvent) -> Vec> { for path in [ &["interleavedOutput"][..], &["aggregated_output"][..], @@ -127,7 +348,7 @@ fn output_parts(event: &SessionEvent) -> Vec> { ] { if let Some(text) = string_at_path(&event.result, path) { if !text.is_empty() { - return vec![OutputPart { + return vec![ExternalShellInlineSegment { stream: ShellReplayStream::Stdout, text, }]; @@ -150,13 +371,13 @@ fn output_parts(event: &SessionEvent) -> Vec> { if stdout.is_some() || stderr.is_some() { let mut parts = Vec::with_capacity(2); if let Some(text) = stdout.filter(|text| !text.is_empty()) { - parts.push(OutputPart { + parts.push(ExternalShellInlineSegment { stream: ShellReplayStream::Stdout, text, }); } if let Some(text) = stderr.filter(|text| !text.is_empty()) { - parts.push(OutputPart { + parts.push(ExternalShellInlineSegment { stream: ShellReplayStream::Stderr, text, }); @@ -167,7 +388,7 @@ fn output_parts(event: &SessionEvent) -> Vec> { if let Some(ExtractedData::Shell(shell)) = event.extracted.as_ref() { if let Some(text) = shell.stream_output.as_deref().or(shell.output.as_deref()) { if !text.is_empty() { - return vec![OutputPart { + return vec![ExternalShellInlineSegment { stream: ShellReplayStream::Stdout, text, }]; @@ -182,7 +403,7 @@ fn output_parts(event: &SessionEvent) -> Vec> { ] { if let Some(text) = string_at_path(&event.result, path) { if !text.is_empty() { - return vec![OutputPart { + return vec![ExternalShellInlineSegment { stream: ShellReplayStream::Stdout, text, }]; @@ -192,6 +413,32 @@ fn output_parts(event: &SessionEvent) -> Vec> { Vec::new() } +/// SHA-256 over ordered `(stream, byte-length, bytes)` tuples. Stream tags and +/// tuple boundaries are part of the digest, so `stdout=A, stderr=B` cannot +/// alias `stdout=AB` or the reversed stream order. +pub fn external_shell_inline_identity(event: &SessionEvent) -> Option { + let parts = external_shell_inline_segments(event); + if parts.is_empty() { + return None; + } + let mut hash = Sha256::new(); + hash.update(b"orgii-external-shell-inline-v1\0"); + for part in parts { + hash.update([match part.stream { + ShellReplayStream::Stdout => 1, + ShellReplayStream::Stderr => 2, + }]); + hash.update((part.text.len() as u64).to_le_bytes()); + hash.update(part.text.as_bytes()); + } + Some( + hash.finalize() + .iter() + .map(|byte| format!("{byte:02x}")) + .collect(), + ) +} + fn first_string_at_paths<'a>(value: &'a serde_json::Value, paths: &[&[&str]]) -> Option<&'a str> { paths.iter().find_map(|path| string_at_path(value, path)) } @@ -207,7 +454,7 @@ fn string_at_path<'a>(value: &'a serde_json::Value, path: &[&str]) -> Option<&'a fn incomplete_preview_state( event: &SessionEvent, call_id: &str, - parts: &[OutputPart<'_>], + parts: &[ExternalShellInlineSegment<'_>], error: String, ) -> ShellReplayState { let mut preview = String::new(); @@ -238,11 +485,98 @@ fn incomplete_preview_state( } } +fn incomplete_segment_preview_state( + event: &SessionEvent, + call_id: &str, + segments: &[ExternalShellReplaySegment], + error: String, +) -> ShellReplayState { + const MARKER: &str = "[external CLI shell replay incomplete]\n"; + let content_budget = SHELL_REPLAY_PREVIEW_BYTES.saturating_sub(MARKER.len()); + let mut content = String::new(); + for segment in segments { + if segment.stream == ShellReplayStream::Stderr { + append_string_tail_bounded(&mut content, "[stderr] ", content_budget); + } + append_string_tail_bounded(&mut content, &segment.preview, content_budget); + } + let mut preview = String::with_capacity(MARKER.len() + content.len()); + preview.push_str(MARKER); + preview.push_str(&content); + ShellReplayState { + replay_ref: ShellReplayRef { + session_id: event.session_id.clone(), + call_id: call_id.to_string(), + format_version: SHELL_REPLAY_FORMAT_VERSION, + }, + bookmark: ShellReplayBookmark::default(), + terminal_preview: preview, + status: ShellReplayStatus::Incomplete, + error: Some(error), + completed_at: Some(event.created_at.clone()), + } +} + +fn append_string_tail_bounded(target: &mut String, value: &str, max_bytes: usize) { + if max_bytes == 0 { + target.clear(); + return; + } + let mut value_start = value.len().saturating_sub(max_bytes); + while value_start < value.len() && !value.is_char_boundary(value_start) { + value_start += 1; + } + let value = &value[value_start..]; + let overflow = target + .len() + .saturating_add(value.len()) + .saturating_sub(max_bytes); + if overflow > 0 { + let mut remove_through = overflow.min(target.len()); + while remove_through < target.len() && !target.is_char_boundary(remove_through) { + remove_through += 1; + } + target.drain(..remove_through); + } + target.push_str(value); +} + #[cfg(test)] mod tests { use super::*; use core_types::session_event::{ActivityStatus, EventDisplayVariant, EventSource}; + #[derive(Debug, Clone)] + struct GeneratedPayload { + seed: u8, + } + + fn generated_payload_range(locator: &GeneratedPayload, offset: u64, length: usize) -> Vec { + (0..length) + .map(|index| b'a' + ((offset + index as u64 + locator.seed as u64) % 26) as u8) + .collect() + } + + fn hash_generated_segments( + segments: &[ExternalShellReplaySegment], + ) -> blake3::Hash { + let mut hasher = blake3::Hasher::new(); + let mut buffer = Vec::with_capacity(64 * 1024); + for segment in segments { + let mut offset = 0u64; + while offset < segment.expected_bytes { + let length = (segment.expected_bytes - offset).min(64 * 1024) as usize; + buffer.clear(); + buffer.extend((0..length).map(|index| { + b'a' + ((offset + index as u64 + segment.locator.seed as u64) % 26) as u8 + })); + hasher.update(&buffer); + offset += length as u64; + } + } + hasher.finalize() + } + fn external_shell_event(output: String) -> SessionEvent { SessionEvent { id: "external-shell-event".to_string(), @@ -293,4 +627,210 @@ mod tests { assert!(state.terminal_preview.ends_with("TAIL")); assert!(state.terminal_preview.len() <= SHELL_REPLAY_PREVIEW_BYTES); } + + #[test] + #[serial_test::serial] + fn ten_mib_locator_segments_stream_into_one_bounded_replay() { + const SIX_MIB: u64 = 6 * 1024 * 1024; + const FOUR_MIB: u64 = 4 * 1024 * 1024; + const TEN_MIB: u64 = SIX_MIB + FOUR_MIB; + + let _sandbox = test_helpers::test_env::sandbox(); + let conn = database::db::get_connection().unwrap(); + database::init_shell_replay_tables(&conn).unwrap(); + let segments = vec![ + ExternalShellReplaySegment::new( + ShellReplayStream::Stdout, + GeneratedPayload { seed: 3 }, + SIX_MIB, + "stdout tail", + ), + ExternalShellReplaySegment::new( + ShellReplayStream::Stderr, + GeneratedPayload { seed: 19 }, + FOUR_MIB, + "stderr tail", + ), + ]; + let expected_hash = hash_generated_segments(&segments); + let mut max_requested = 0usize; + let mut max_returned = 0usize; + let mut event = external_shell_event(String::new()); + + persist_external_shell_replay_segments( + &mut event, + &segments, + |locator, offset, requested| { + assert!(requested <= SHELL_REPLAY_RANGE_MAX_BYTES); + max_requested = max_requested.max(requested); + let bytes = generated_payload_range(locator, offset, requested); + max_returned = max_returned.max(bytes.len()); + Ok(bytes) + }, + ); + + let state = event.shell_replay.as_ref().expect("streamed replay state"); + assert_eq!(state.status, ShellReplayStatus::Complete); + assert_eq!(state.bookmark.visible_bytes, TEN_MIB); + assert_eq!(max_requested, SHELL_REPLAY_RANGE_MAX_BYTES); + assert!(max_returned <= SHELL_REPLAY_RANGE_MAX_BYTES); + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let mut actual_hash = blake3::Hasher::new(); + let mut offset = 0u64; + while offset < TEN_MIB { + let range = runtime + .block_on(super::super::shell_replay::shell_replay_read_range( + event.session_id.clone(), + event.call_id.clone().unwrap(), + state.bookmark.visible_through_sequence, + state.bookmark.visible_bytes, + offset, + SHELL_REPLAY_RANGE_MAX_BYTES as u64, + )) + .unwrap(); + assert!(range.next_offset_bytes > offset); + for frame in range.frames { + actual_hash.update(frame.text.as_bytes()); + } + offset = range.next_offset_bytes; + if range.eof { + break; + } + } + assert_eq!(offset, TEN_MIB); + assert_eq!(actual_hash.finalize(), expected_hash); + + // Same call id and byte length are not content identity. A provider + // update must read the source again and replace the stale `.slog`. + let changed_segments = vec![ + ExternalShellReplaySegment::new( + ShellReplayStream::Stdout, + GeneratedPayload { seed: 29 }, + SIX_MIB, + "changed stdout tail", + ), + ExternalShellReplaySegment::new( + ShellReplayStream::Stderr, + GeneratedPayload { seed: 47 }, + FOUR_MIB, + "changed stderr tail", + ), + ]; + let changed_expected_hash = hash_generated_segments(&changed_segments); + assert_ne!(changed_expected_hash, expected_hash); + let mut reopened = external_shell_event(String::new()); + let mut changed_source_reads = 0_usize; + persist_external_shell_replay_segments( + &mut reopened, + &changed_segments, + |locator, offset, requested| { + changed_source_reads += 1; + Ok(generated_payload_range(locator, offset, requested)) + }, + ); + let changed_state = reopened + .shell_replay + .as_ref() + .expect("updated replay state"); + assert_eq!(changed_state.status, ShellReplayStatus::Complete); + assert!(changed_source_reads > 1); + + let mut changed_actual_hash = blake3::Hasher::new(); + let mut changed_offset = 0_u64; + while changed_offset < TEN_MIB { + let range = runtime + .block_on(super::super::shell_replay::shell_replay_read_range( + reopened.session_id.clone(), + reopened.call_id.clone().unwrap(), + changed_state.bookmark.visible_through_sequence, + changed_state.bookmark.visible_bytes, + changed_offset, + SHELL_REPLAY_RANGE_MAX_BYTES as u64, + )) + .unwrap(); + assert!(range.next_offset_bytes > changed_offset); + for frame in range.frames { + changed_actual_hash.update(frame.text.as_bytes()); + } + changed_offset = range.next_offset_bytes; + if range.eof { + break; + } + } + assert_eq!(changed_offset, TEN_MIB); + assert_eq!(changed_actual_hash.finalize(), changed_expected_hash); + } + + #[test] + fn inline_identity_includes_content_stream_and_segment_boundaries() { + let mut split = external_shell_event(String::new()); + split.result = serde_json::json!({"stdout":"ab","stderr":"c","exit_code":0}); + let split_identity = external_shell_inline_identity(&split).expect("split identity"); + + let mut moved_boundary = external_shell_event(String::new()); + moved_boundary.result = serde_json::json!({"stdout":"a","stderr":"bc","exit_code":0}); + let moved_identity = + external_shell_inline_identity(&moved_boundary).expect("moved boundary identity"); + + let combined = external_shell_event("abc".to_string()); + let combined_identity = + external_shell_inline_identity(&combined).expect("combined identity"); + assert_ne!(split_identity, moved_identity); + assert_ne!(split_identity, combined_identity); + assert_ne!(moved_identity, combined_identity); + + let mut same_length_update = external_shell_event(String::new()); + same_length_update.result = serde_json::json!({"stdout":"ax","stderr":"c","exit_code":0}); + assert_eq!( + external_shell_inline_segments(&same_length_update) + .iter() + .map(|part| part.text.len()) + .sum::(), + 3 + ); + assert_ne!( + split_identity, + external_shell_inline_identity(&same_length_update).expect("updated identity") + ); + } + + #[test] + #[serial_test::serial] + fn truncated_locator_sets_an_explicit_incomplete_preview() { + let _sandbox = test_helpers::test_env::sandbox(); + let conn = database::db::get_connection().unwrap(); + database::init_shell_replay_tables(&conn).unwrap(); + let segments = vec![ExternalShellReplaySegment::new( + ShellReplayStream::Stdout, + "stdout-locator", + 10, + "source-provided tail", + )]; + let mut event = external_shell_event(String::new()); + + persist_external_shell_replay_segments( + &mut event, + &segments, + |_locator, offset, _requested| { + if offset == 0 { + Ok(b"abc".to_vec()) + } else { + Ok(Vec::new()) + } + }, + ); + + let state = event.shell_replay.expect("incomplete replay state"); + assert_eq!(state.status, ShellReplayStatus::Incomplete); + assert!(state + .terminal_preview + .starts_with("[external CLI shell replay incomplete]")); + assert!(state.terminal_preview.contains("source-provided tail")); + assert!(state.terminal_preview.len() <= SHELL_REPLAY_PREVIEW_BYTES); + assert!(state.error.unwrap().contains("ended at byte 3")); + } } diff --git a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay.rs b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay.rs index 31a37391e..82a21e7cb 100644 --- a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay.rs +++ b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay.rs @@ -11,10 +11,10 @@ mod active; mod cleanup; mod range; mod recovery; -mod text; -mod writer; #[cfg(test)] mod tests; +mod text; +mod writer; pub use active::{ active_state, active_states_for_session, ShellReplayAppend, ShellReplayStream, @@ -31,8 +31,8 @@ pub use range::{ pub use recovery::recover_incomplete_replays; pub use writer::ShellReplayWriter; -pub(super) use range::load_complete_replay_state_if_matches; pub use range::__cmd__shell_replay_read_range; +pub(super) use range::load_complete_replay_state_if_matches; pub(super) use text::complete_terminal_prefix_len; #[cfg(test)] pub(super) use text::complete_utf8_prefix_len; diff --git a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/active.rs b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/active.rs index 1b28d2c97..fdcb26721 100644 --- a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/active.rs +++ b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/active.rs @@ -33,8 +33,9 @@ impl ActiveReplayState { } } -pub(super) static ACTIVE_REPLAYS: LazyLock>>> = - LazyLock::new(|| RwLock::new(HashMap::new())); +pub(super) static ACTIVE_REPLAYS: LazyLock< + RwLock>>, +> = LazyLock::new(|| RwLock::new(HashMap::new())); #[derive(Debug, Clone)] pub struct ShellReplayTarget { diff --git a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/tests.rs b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/tests.rs index 47f8e8806..01022adf7 100644 --- a/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/tests.rs +++ b/src-tauri/crates/agent-core/src/core/tools/impls/coding/exec/shell_replay/tests.rs @@ -205,8 +205,7 @@ fn range_aligns_to_complete_sequence_and_never_splits_emoji() { with_test_home(|home| { let root = home.join("replays"); let target = ShellReplayTarget::new("session-utf8", "call-utf8"); - let mut writer = - ShellReplayWriter::create(&root, target, "emit utf8", home, None).unwrap(); + let mut writer = ShellReplayWriter::create(&root, target, "emit utf8", home, None).unwrap(); let text = format!("{}🙂END", "x".repeat(1_000)); let append = writer .append(ShellReplayStream::Stdout, text.as_bytes()) @@ -457,8 +456,7 @@ fn oversized_frame_is_rejected_and_corrupt_length_is_never_allocated() { let root = home.join("replays"); let target = ShellReplayTarget::new("session-corrupt-length", "call-corrupt-length"); let path = { - let mut writer = - ShellReplayWriter::create(&root, target, "emit", home, None).unwrap(); + let mut writer = ShellReplayWriter::create(&root, target, "emit", home, None).unwrap(); let oversized = vec![b'x'; SHELL_REPLAY_FRAME_MAX_BYTES + 1]; assert!(writer .append(ShellReplayStream::Stdout, &oversized) @@ -674,9 +672,7 @@ fn shell_replay_rss_plateau_after_ten_megabyte_warmup() { } let final_peak = peak_rss_bytes(); let delta = final_peak.saturating_sub(warm_peak); - eprintln!( - "shell replay RSS: warm_peak={warm_peak} final_peak={final_peak} delta={delta}" - ); + eprintln!("shell replay RSS: warm_peak={warm_peak} final_peak={final_peak} delta={delta}"); assert!( delta <= 64 * 1024 * 1024, "RSS grew by {delta} bytes after warmup" diff --git a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/cache.rs b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/cache.rs index 7a47878f0..986a363c3 100644 --- a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/cache.rs +++ b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/cache.rs @@ -9,8 +9,8 @@ use std::path::PathBuf; use std::sync::{Arc, LazyLock}; use std::time::Duration; -use super::SkillsLoader; use super::super::types::SkillInfo; +use super::SkillsLoader; use crate::utils::swr_cache::SwrCache; const SKILL_SCAN_CACHE_TTL: Duration = Duration::from_secs(2); diff --git a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/fs_scan.rs b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/fs_scan.rs index 6305037f9..9fe3429ea 100644 --- a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/fs_scan.rs +++ b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/fs_scan.rs @@ -7,9 +7,9 @@ use std::fs; use std::path::{Path, PathBuf}; -use super::SkillsLoader; use super::super::helpers::{collect_bundled_files, estimate_summary_line_tokens, estimate_tokens}; use super::super::types::{DescriptionQuality, SkillInfo}; +use super::SkillsLoader; const DISCOVERED_SKILL_ROOT_MAX_DEPTH: usize = 4; const DISCOVERED_SKILL_ROOT_MAX_ENTRIES: usize = 500; diff --git a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing.rs b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing.rs index 53e1d4519..182c72bd5 100644 --- a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing.rs +++ b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing.rs @@ -1,8 +1,8 @@ //! Per-turn skill listing rendering: entries, budgeted description //! truncation, and the `always: true` skills manifest. -use super::SkillsLoader; use super::super::types::{SkillInfo, SkillListingEntry}; +use super::SkillsLoader; // Listing budget mirrors the reference harness: the listing exists for // discovery only (full bodies load via the `skill` tool), so verbose diff --git a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing_budget_tests.rs b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing_budget_tests.rs index dd28c4050..d4b7b75df 100644 --- a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing_budget_tests.rs +++ b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/listing_budget_tests.rs @@ -24,8 +24,7 @@ fn listing_lines(rendered: &str) -> Vec<&str> { fn per_entry_description_capped_at_250_chars() { let long_desc = "x".repeat(SKILL_LISTING_MAX_DESC_CHARS + 150); let entries = vec![entry("verbose", &long_desc)]; - let rendered = - SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); + let rendered = SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); let line = listing_lines(&rendered)[0]; assert!( line.contains('\u{2026}'), @@ -49,8 +48,7 @@ fn total_budget_trims_descriptions_evenly() { let entries: Vec = (0..50) .map(|i| entry(&format!("s-{i:03}"), &desc)) .collect(); - let rendered = - SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); + let rendered = SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); let lines = listing_lines(&rendered); assert_eq!(lines.len(), 50, "no entry may be dropped"); let total_chars: usize = @@ -77,8 +75,7 @@ fn names_only_floor_never_drops_entries() { let entries: Vec = (0..400) .map(|i| entry(&format!("s-{i:03}"), &desc)) .collect(); - let rendered = - SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); + let rendered = SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); let lines = listing_lines(&rendered); assert_eq!(lines.len(), 400, "floor keeps every invocable name"); for (i, line) in lines.iter().enumerate() { @@ -93,8 +90,7 @@ fn names_only_floor_never_drops_entries() { #[test] fn under_budget_listing_keeps_full_descriptions() { let entries = vec![entry("alpha", "first"), entry("beta", "second")]; - let rendered = - SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); + let rendered = SkillsLoader::format_skill_listing_entries(&entries).expect("listing populated"); assert!(rendered.contains("- **alpha** (workspace): first [available]")); assert!(rendered.contains("- **beta** (workspace): second [available]")); } diff --git a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/metadata.rs b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/metadata.rs index 56140fe3a..b318f8708 100644 --- a/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/metadata.rs +++ b/src-tauri/crates/agent-core/src/specialization/skills/loader/scanner/metadata.rs @@ -1,7 +1,7 @@ //! SKILL.md YAML-frontmatter parsing and requirement checks. -use super::SkillsLoader; use super::super::types::SkillMetadata; +use super::SkillsLoader; impl SkillsLoader { pub(super) fn skill_metadata_applies_to_agent(&self, meta: &SkillMetadata) -> bool { diff --git a/src-tauri/crates/orgtrack-cli/src/commands.rs b/src-tauri/crates/orgtrack-cli/src/commands.rs index 734503481..657bc93dd 100644 --- a/src-tauri/crates/orgtrack-cli/src/commands.rs +++ b/src-tauri/crates/orgtrack-cli/src/commands.rs @@ -1,22 +1,30 @@ //! Command handlers: one function per subcommand (`sources`, `plugins`, `scan`, //! `list`/`search`, `usage`, `show`) plus their table/markdown/CSV renderers. +use std::io::{self, Write}; + use core_types::activity::ActivityChunk; use rusqlite::Connection; use orgtrack_core::session_usage; +use orgtrack_core::sources::imported_history::{ + replay::{ + self, ImportedHistorySourceId, ReplayCursor, ReplayIndexedChunk, ReplayLimits, + ReplayPayloadDescriptor, ReplayPayloadRange, + }, + router as replay_router, +}; use orgtrack_core::sources::registry; use orgtrack_core::usage_dashboard::{ self, TrendBucket, UsageFilter, UsageRoundQuery, UsageSessionRow, UsageSummary, }; -use crate::output::chunk_body; use crate::output::{ - csv_row, formatter_for, md_cell, parse_sort, preview_of, print_usage_session_row, + chunk_body, csv_row, formatter_for, md_cell, parse_sort, preview_of, print_usage_session_row, print_usage_summary, render_template, row_matches, session_label, to_json, truncate, }; use crate::plugin_exec::{ - apply_chunk_processors, apply_session_processors, load_session_chunks, source_of_session, + apply_chunk_processors, apply_session_processors, load_plugin_session_chunks, source_of_session, }; use crate::plugins::{self, FormatterPlugin, LoaderPlugin, ProcessorPlugin}; use crate::scan::{counts_by_source, read_cached, scan_all}; @@ -842,106 +850,5 @@ fn render_check( Ok(()) } -pub(crate) fn cmd_show( - opts: &Options, - plugins: &[LoaderPlugin], - processors: &[ProcessorPlugin], - formatters: &[FormatterPlugin], -) -> Result<(), String> { - let Some(session_id) = opts.positionals.first().cloned() else { - return Err("show needs a session id, e.g. `orgtrack show claude_code-`".into()); - }; - let target = db_target(opts)?; - if !opts.no_scan { - scan_all(&target.path, opts, plugins); - } - let conn = open_conn(&target.path)?; - let chunks = - load_session_chunks(&conn, &session_id, plugins, opts.timeout())?.ok_or_else(|| { - format!("'{session_id}' is not a known imported session id (nothing to show)") - })?; - // Chunk-stage processors reshape the conversation before rendering. - let source = source_of_session(&session_id, plugins); - let chunks = apply_chunk_processors(&session_id, &source, chunks, processors, opts.timeout()); - - if let Some(formatter) = formatter_for(opts, formatters) { - let context = serde_json::json!({ - "command": "show", - "sessionId": session_id, - "chunks": chunks, - }); - return render_template(formatter, &context); - } - match opts.format()? { - Format::Json => println!("{}", to_json(&chunks)?), - Format::Md => print!("{}", render_show_md(&session_id, &chunks)), - Format::Csv => print!("{}", render_show_csv(&chunks)), - Format::Table => { - println!("Session {session_id} — {} activity chunks\n", chunks.len()); - for chunk in &chunks { - let label = if chunk.function.is_empty() { - chunk.action_type.clone() - } else { - format!("{}:{}", chunk.action_type, chunk.function) - }; - println!("[{}] {}", truncate(&chunk.created_at, 19), label); - if let Some(text) = preview_of(&chunk.args).or_else(|| preview_of(&chunk.result)) { - println!(" {}", truncate(&text, 160)); - } - } - } - } - Ok(()) -} - -/// Portable markdown transcript of a session — the export format. Message -/// bodies render as prose; tool calls render as fenced code so a transcript -/// round-trips into any markdown viewer. -pub(crate) fn render_show_md(session_id: &str, chunks: &[ActivityChunk]) -> String { - let mut out = format!("# Session {session_id}\n\n"); - for chunk in chunks { - let role = chunk_role(chunk); - out.push_str(&format!( - "**{role}** · {}\n\n", - truncate(&chunk.created_at, 19) - )); - let body = chunk_body(&chunk.args).or_else(|| chunk_body(&chunk.result)); - match body { - Some(text) if chunk.action_type == "tool_call" => { - out.push_str(&format!("```\n{}\n```\n\n", text.trim_end())) - } - Some(text) => out.push_str(&format!("{}\n\n", text.trim_end())), - None => out.push_str("_(no content)_\n\n"), - } - } - out -} - -pub(crate) fn render_show_csv(chunks: &[ActivityChunk]) -> String { - let mut out = String::from("created_at,role,action_type,function,preview\n"); - for chunk in chunks { - let preview = preview_of(&chunk.args) - .or_else(|| preview_of(&chunk.result)) - .unwrap_or_default(); - out.push_str(&csv_row(&[ - &chunk.created_at, - &chunk_role(chunk), - &chunk.action_type, - &chunk.function, - &preview, - ])); - } - out -} - -/// Human role label for a chunk: `user`, `assistant`, `assistant (thinking)`, -/// or `tool: `. -pub(crate) fn chunk_role(chunk: &ActivityChunk) -> String { - match chunk.action_type.as_str() { - "raw" if chunk.function.contains("user") => "user".to_string(), - "assistant" => "assistant".to_string(), - "thinking" => "assistant (thinking)".to_string(), - "tool_call" => format!("tool: {}", chunk.function), - other => other.to_string(), - } -} +mod show; +pub(crate) use show::cmd_show; diff --git a/src-tauri/crates/orgtrack-cli/src/commands/show.rs b/src-tauri/crates/orgtrack-cli/src/commands/show.rs new file mode 100644 index 000000000..4a45e4ef5 --- /dev/null +++ b/src-tauri/crates/orgtrack-cli/src/commands/show.rs @@ -0,0 +1,883 @@ +use super::*; + +pub(crate) fn cmd_show( + opts: &Options, + plugins: &[LoaderPlugin], + processors: &[ProcessorPlugin], + formatters: &[FormatterPlugin], +) -> Result<(), String> { + let Some(session_id) = opts.positionals.first().cloned() else { + return Err("show needs a session id, e.g. `orgtrack show claude_code-`".into()); + }; + let target = db_target(opts)?; + if !opts.no_scan { + scan_all(&target.path, opts, plugins); + } + let mut conn = open_conn(&target.path)?; + + // Canonical built-in prefixes always take the compact replay path. A + // third-party plugin cannot shadow one and accidentally re-enable a full + // provider transcript load. + if let Some(source) = replay_router::source_for_session(&session_id) { + if let Some(formatter) = formatter_for(opts, formatters) { + eprintln!( + "orgtrack: custom show formatters receive bounded compact pages; deferred payloads remain previews" + ); + return stream_builtin_show_template( + &mut conn, + &session_id, + source.as_str(), + opts, + processors, + formatter, + ); + } + return stream_builtin_show(&mut conn, &session_id, source, opts, processors); + } + + // Plugin loaders are an explicit third-party compatibility boundary. They + // may return a Vec because their protocol is a single JSON response, but no + // built-in source can reach this branch. + let chunks = load_plugin_session_chunks(&conn, &session_id, plugins, opts.timeout())? + .ok_or_else(|| { + format!("'{session_id}' is not a known imported session id (nothing to show)") + })?; + let source = source_of_session(&session_id, plugins); + let chunks = apply_chunk_processors(&session_id, &source, chunks, processors, opts.timeout()); + + if let Some(formatter) = formatter_for(opts, formatters) { + let context = serde_json::json!({ + "command": "show", + "sessionId": session_id, + "chunks": chunks, + }); + return render_template(formatter, &context); + } + match opts.format()? { + Format::Json => println!("{}", to_json(&chunks)?), + Format::Md => print!("{}", render_show_md(&session_id, &chunks)), + Format::Csv => print!("{}", render_show_csv(&chunks)), + Format::Table => { + println!("Session {session_id} — {} activity chunks\n", chunks.len()); + for chunk in &chunks { + let label = if chunk.function.is_empty() { + chunk.action_type.clone() + } else { + format!("{}:{}", chunk.action_type, chunk.function) + }; + println!("[{}] {}", truncate(&chunk.created_at, 19), label); + if let Some(text) = preview_of(&chunk.args).or_else(|| preview_of(&chunk.result)) { + println!(" {}", truncate(&text, 160)); + } + } + } + } + Ok(()) +} + +const SHOW_PAGE_EVENTS: usize = 64; +const SHOW_PAGE_IPC_BYTES: usize = 1024 * 1024; + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +struct StreamedShowSummary { + chunks: usize, + has_more: bool, +} + +/// Visit compact replay pages without ever dropping payload descriptors or +/// accumulating pages into a session-sized vector. +fn for_each_builtin_indexed_show_page( + conn: &mut Connection, + session_id: &str, + max_chunks: usize, + mut visit: impl FnMut( + &mut Connection, + usize, + &ReplayCursor, + bool, + &[ReplayIndexedChunk], + ) -> Result<(), String>, +) -> Result { + let mut cursor: Option = None; + let mut page_index = 0usize; + let mut consumed = 0usize; + let mut has_more = false; + + while consumed < max_chunks { + let previous_sequence = cursor.as_ref().map_or(-1, |cursor| cursor.through_sequence); + let remaining = max_chunks.saturating_sub(consumed); + let limits = ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: remaining.clamp(1, SHOW_PAGE_EVENTS), + max_ipc_bytes: SHOW_PAGE_IPC_BYTES, + }; + let scan = replay_router::scan_activity_chunks_for_session( + conn, + session_id, + cursor.as_ref(), + limits, + )? + .ok_or_else(|| format!("Unknown built-in imported session id: {session_id}"))?; + if scan.chunks.is_empty() + && scan.has_more + && scan.cursor.through_sequence <= previous_sequence + { + return Err(format!( + "Bounded replay scan made no progress for {session_id} after sequence {}", + scan.cursor.through_sequence + )); + } + + let raw_count = scan.chunks.len(); + let scan_has_more = scan.has_more; + let scan_cursor = scan.cursor; + if raw_count > 0 || (!scan_has_more && page_index == 0) { + visit(conn, page_index, &scan_cursor, scan_has_more, &scan.chunks)?; + } + + consumed = consumed.saturating_add(raw_count); + has_more = scan_has_more; + cursor = Some(scan_cursor); + page_index = page_index.saturating_add(1); + if !has_more { + break; + } + } + + Ok(StreamedShowSummary { + chunks: consumed, + has_more, + }) +} + +/// Third-party processors still speak the historical `Vec` +/// protocol. Keep that compatibility boundary page-bounded and prevent it +/// from becoming a built-in provider loader fallback. +fn for_each_builtin_processed_show_page( + conn: &mut Connection, + session_id: &str, + source: &str, + processors: &[ProcessorPlugin], + timeout: std::time::Duration, + max_chunks: usize, + mut visit: impl FnMut(usize, &ReplayCursor, bool, &[ActivityChunk]) -> Result<(), String>, +) -> Result { + for_each_builtin_indexed_show_page( + conn, + session_id, + max_chunks, + |_conn, page_index, cursor, has_more, indexed| { + let chunks = indexed + .iter() + .map(|indexed| indexed.chunk.clone()) + .collect::>(); + let chunks = apply_chunk_processors(session_id, source, chunks, processors, timeout); + visit(page_index, cursor, has_more, &chunks) + }, + ) +} + +fn stream_builtin_show_template( + conn: &mut Connection, + session_id: &str, + source: &str, + opts: &Options, + processors: &[ProcessorPlugin], + formatter: &FormatterPlugin, +) -> Result<(), String> { + let max_chunks = opts.limit.unwrap_or(usize::MAX); + for_each_builtin_processed_show_page( + conn, + session_id, + source, + processors, + opts.timeout(), + max_chunks, + |page_index, cursor, has_more, chunks| { + let context = serde_json::json!({ + "command": "show", + "sessionId": session_id, + "chunks": chunks, + "page": { + "index": page_index, + "hasMore": has_more, + "generation": &cursor.generation, + "revision": cursor.revision, + "throughSequence": cursor.through_sequence, + }, + }); + render_template(formatter, &context) + }, + )?; + Ok(()) +} + +fn stream_builtin_show( + conn: &mut Connection, + session_id: &str, + source: ImportedHistorySourceId, + opts: &Options, + processors: &[ProcessorPlugin], +) -> Result<(), String> { + let format = opts.format()?; + let stdout = io::stdout(); + let mut out = io::BufWriter::new(stdout.lock()); + match format { + Format::Json => out.write_all(b"[\n").map_err(show_write_error)?, + Format::Md => writeln!(out, "# Session {session_id}\n").map_err(show_write_error)?, + Format::Csv => out + .write_all(b"created_at,role,action_type,function,preview\n") + .map_err(show_write_error)?, + Format::Table => writeln!(out, "Session {session_id} — streaming activity chunks\n") + .map_err(show_write_error)?, + } + + let mut wrote_json_chunk = false; + let max_chunks = opts.limit.unwrap_or(usize::MAX); + let summary = if processors.is_empty() { + for_each_builtin_indexed_show_page( + conn, + session_id, + max_chunks, + |conn, _page_index, cursor, _has_more, chunks| { + for indexed in chunks { + if format == Format::Json && wrote_json_chunk { + out.write_all(b",\n").map_err(show_write_error)?; + } + write_indexed_show_chunk( + conn, + &mut out, + source, + session_id, + &cursor.generation, + format, + indexed, + )?; + wrote_json_chunk |= format == Format::Json; + } + Ok(()) + }, + )? + } else { + eprintln!( + "orgtrack: custom chunk processors receive bounded compact pages; deferred payloads remain previews" + ); + for_each_builtin_processed_show_page( + conn, + session_id, + source.as_str(), + processors, + opts.timeout(), + max_chunks, + |_page_index, _cursor, _has_more, chunks| { + for chunk in chunks { + if format == Format::Json && wrote_json_chunk { + out.write_all(b",\n").map_err(show_write_error)?; + } + write_processed_show_chunk(&mut out, format, chunk)?; + wrote_json_chunk |= format == Format::Json; + } + Ok(()) + }, + )? + }; + + match format { + Format::Json => out.write_all(b"\n]\n").map_err(show_write_error)?, + Format::Table => { + writeln!(out, "\n{} activity chunks shown.", summary.chunks) + .map_err(show_write_error)?; + } + Format::Md | Format::Csv => {} + } + out.flush().map_err(show_write_error)?; + if summary.has_more { + eprintln!( + "orgtrack: output stopped at --limit {} (more activity is available)", + opts.limit.unwrap_or(summary.chunks) + ); + } + Ok(()) +} + +fn write_processed_show_chunk( + writer: &mut impl Write, + format: Format, + chunk: &ActivityChunk, +) -> Result<(), String> { + match format { + Format::Json => { + serde_json::to_writer(writer, chunk).map_err(|err| format!("json encode: {err}")) + } + Format::Md => writer + .write_all(render_show_md_chunk(chunk).as_bytes()) + .map_err(show_write_error), + Format::Csv => writer + .write_all(render_show_csv_chunk(chunk).as_bytes()) + .map_err(show_write_error), + Format::Table => write_show_table_chunk(writer, chunk), + } +} + +#[allow(clippy::too_many_arguments)] +fn write_indexed_show_chunk( + conn: &mut Connection, + writer: &mut impl Write, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + format: Format, + indexed: &ReplayIndexedChunk, +) -> Result<(), String> { + if format == Format::Table { + return write_show_table_chunk(writer, &indexed.chunk); + } + let mut read_range = |field_path: &str, offset: u64, max_bytes: usize| { + replay::read_payload_range( + conn, + source, + session_id, + generation, + &indexed.chunk.chunk_id, + field_path, + offset, + Some(max_bytes), + ) + }; + match format { + Format::Json => write_indexed_chunk_json_with_reader(writer, indexed, &mut read_range), + Format::Md => write_indexed_chunk_md_with_reader(writer, indexed, &mut read_range), + Format::Csv => write_indexed_chunk_csv_with_reader(writer, indexed, &mut read_range), + Format::Table => unreachable!("table returned before opening a payload reader"), + } +} + +fn write_indexed_chunk_json_with_reader( + writer: &mut W, + indexed: &ReplayIndexedChunk, + read_range: &mut R, +) -> Result<(), String> +where + W: Write, + R: FnMut(&str, u64, usize) -> Result, +{ + let chunk = &indexed.chunk; + writer + .write_all(b"{\"chunk_id\":") + .map_err(show_write_error)?; + write_small_json(writer, &chunk.chunk_id)?; + writer + .write_all(b",\"session_id\":") + .map_err(show_write_error)?; + write_small_json(writer, &chunk.session_id)?; + writer + .write_all(b",\"action_type\":") + .map_err(show_write_error)?; + write_small_json(writer, &chunk.action_type)?; + writer + .write_all(b",\"function\":") + .map_err(show_write_error)?; + write_small_json(writer, &chunk.function)?; + writer.write_all(b",\"args\":").map_err(show_write_error)?; + write_replay_json_value_with_reader( + writer, + "args", + &chunk.args, + &indexed.payloads, + read_range, + )?; + writer + .write_all(b",\"result\":") + .map_err(show_write_error)?; + write_replay_json_value_with_reader( + writer, + "result", + &chunk.result, + &indexed.payloads, + read_range, + )?; + writer + .write_all(b",\"created_at\":") + .map_err(show_write_error)?; + write_small_json(writer, &chunk.created_at)?; + if let Some(thread_id) = &chunk.thread_id { + writer + .write_all(b",\"thread_id\":") + .map_err(show_write_error)?; + write_small_json(writer, thread_id)?; + } + if let Some(process_id) = &chunk.process_id { + writer + .write_all(b",\"process_id\":") + .map_err(show_write_error)?; + write_small_json(writer, process_id)?; + } + writer.write_all(b"}").map_err(show_write_error) +} + +fn write_small_json(writer: &mut impl Write, value: &impl serde::Serialize) -> Result<(), String> { + serde_json::to_writer(writer, value).map_err(|err| format!("json encode: {err}")) +} + +fn write_replay_json_value_with_reader( + writer: &mut W, + path: &str, + value: &serde_json::Value, + payloads: &[ReplayPayloadDescriptor], + read_range: &mut R, +) -> Result<(), String> +where + W: Write, + R: FnMut(&str, u64, usize) -> Result, +{ + if let Some(payload) = payloads.iter().find(|payload| payload.field_path == path) { + return match payload.resolved_encoding() { + replay::ReplayPayloadEncoding::JsonValue => { + stream_payload_with_reader(writer, path, false, read_range) + } + replay::ReplayPayloadEncoding::Utf8Text => { + writer.write_all(b"\"").map_err(show_write_error)?; + stream_payload_with_reader(writer, path, true, read_range)?; + writer.write_all(b"\"").map_err(show_write_error) + } + replay::ReplayPayloadEncoding::LegacyPathInferred => { + unreachable!("resolved replay payload encoding cannot remain legacy") + } + }; + } + + match value { + serde_json::Value::Null + | serde_json::Value::Bool(_) + | serde_json::Value::Number(_) + | serde_json::Value::String(_) => write_small_json(writer, value), + serde_json::Value::Array(values) => { + writer.write_all(b"[").map_err(show_write_error)?; + for (index, value) in values.iter().enumerate() { + if index > 0 { + writer.write_all(b",").map_err(show_write_error)?; + } + write_replay_json_value_with_reader( + writer, + &format!("{path}.{index}"), + value, + payloads, + read_range, + )?; + } + writer.write_all(b"]").map_err(show_write_error) + } + serde_json::Value::Object(values) => { + writer.write_all(b"{").map_err(show_write_error)?; + let mut wrote_field = false; + for (key, value) in values { + if replay::is_compact_only_replay_field(key) { + continue; + } + if wrote_field { + writer.write_all(b",").map_err(show_write_error)?; + } + wrote_field = true; + write_small_json(writer, key)?; + writer.write_all(b":").map_err(show_write_error)?; + write_replay_json_value_with_reader( + writer, + &format!("{path}.{key}"), + value, + payloads, + read_range, + )?; + } + writer.write_all(b"}").map_err(show_write_error) + } + } +} + +fn stream_payload_with_reader( + writer: &mut W, + field_path: &str, + escape_json_string: bool, + read_range: &mut R, +) -> Result<(), String> +where + W: Write, + R: FnMut(&str, u64, usize) -> Result, +{ + let mut offset = 0u64; + loop { + let range = read_range(field_path, offset, replay::HARD_MAX_PAYLOAD_RANGE_BYTES)?; + if range.field_path != field_path { + return Err(format!( + "Replay payload {field_path} returned mismatched path {}", + range.field_path + )); + } + if range.offset != offset { + return Err(format!( + "Replay payload {field_path} skipped from {offset} to {}", + range.offset + )); + } + if escape_json_string { + write_json_string_content(writer, &range.text)?; + } else { + writer + .write_all(range.text.as_bytes()) + .map_err(show_write_error)?; + } + if range.next_offset <= offset && !range.eof { + return Err(format!( + "Replay payload {field_path} made no progress at {offset}" + )); + } + offset = range.next_offset; + if range.eof { + if offset != range.total_bytes { + return Err(format!( + "Replay payload {field_path} ended at {offset}, expected {}", + range.total_bytes + )); + } + break; + } + } + Ok(()) +} + +fn write_json_string_content(writer: &mut impl Write, text: &str) -> Result<(), String> { + for ch in text.chars() { + match ch { + '"' => writer.write_all(b"\\\"").map_err(show_write_error)?, + '\\' => writer.write_all(b"\\\\").map_err(show_write_error)?, + '\u{08}' => writer.write_all(b"\\b").map_err(show_write_error)?, + '\u{0c}' => writer.write_all(b"\\f").map_err(show_write_error)?, + '\n' => writer.write_all(b"\\n").map_err(show_write_error)?, + '\r' => writer.write_all(b"\\r").map_err(show_write_error)?, + '\t' => writer.write_all(b"\\t").map_err(show_write_error)?, + control if control <= '\u{1f}' => { + write!(writer, "\\u{:04x}", control as u32).map_err(show_write_error)?; + } + other => { + let mut encoded = [0u8; 4]; + writer + .write_all(other.encode_utf8(&mut encoded).as_bytes()) + .map_err(show_write_error)?; + } + } + } + Ok(()) +} + +enum ReplayBodySelection<'a> { + Compact(String), + Payload(&'a ReplayPayloadDescriptor), + Projection(&'a replay::ReplayPayloadBodyProjection), + CompactProjection(String), +} + +fn select_replay_body(indexed: &ReplayIndexedChunk) -> Option> { + select_replay_body_root(indexed, "args", &indexed.chunk.args) + .or_else(|| select_replay_body_root(indexed, "result", &indexed.chunk.result)) +} + +fn select_replay_body_root<'a>( + indexed: &'a ReplayIndexedChunk, + root: &'static str, + value: &'a serde_json::Value, +) -> Option> { + let payloads = indexed + .payloads + .iter() + .filter(|payload| payload_path_is_under(&payload.field_path, root)) + .collect::>(); + if let Some(payload) = payloads.iter().find(|payload| payload.field_path == root) { + if let Some(projection) = payload.body_projection.as_ref() { + return Some(ReplayBodySelection::Projection(projection)); + } + return chunk_body(value).map(ReplayBodySelection::CompactProjection); + } + + let selected_path = selected_compact_body_path(value, root)?; + if let Some(payload) = payloads + .iter() + .find(|payload| payload.field_path == selected_path) + { + return Some(ReplayBodySelection::Payload(payload)); + } + if selected_path == root && !payloads.is_empty() { + return chunk_body(value).map(ReplayBodySelection::CompactProjection); + } + chunk_body(value).map(ReplayBodySelection::Compact) +} + +fn selected_compact_body_path(value: &serde_json::Value, root: &str) -> Option { + match value { + serde_json::Value::Null => None, + serde_json::Value::String(text) => non_blank_path(text, root), + serde_json::Value::Object(map) if map.is_empty() => None, + serde_json::Value::Array(items) if items.is_empty() => None, + serde_json::Value::Object(map) => { + if let Some(text) = map + .get("message") + .and_then(|message| message.get("content")) + .and_then(|content| content.as_str()) + { + if let Some(path) = non_blank_path(text, &format!("{root}.message.content")) { + return Some(path); + } + } + for key in [ + "content", + "text", + "observation", + "cmd", + "command", + "body", + "summary", + "prompt", + "description", + ] { + if let Some(text) = map.get(key).and_then(|value| value.as_str()) { + if let Some(path) = non_blank_path(text, &format!("{root}.{key}")) { + return Some(path); + } + } + } + Some(root.to_string()) + } + _ => Some(root.to_string()), + } +} + +fn non_blank_path(text: &str, path: &str) -> Option { + (!text.trim().is_empty()).then(|| path.to_string()) +} + +fn payload_path_is_under(field_path: &str, root: &str) -> bool { + field_path == root + || field_path + .strip_prefix(root) + .is_some_and(|suffix| suffix.starts_with('.')) +} + +fn write_selected_replay_body( + writer: &mut W, + selection: ReplayBodySelection<'_>, + read_range: &mut R, +) -> Result<(), String> +where + W: Write, + R: FnMut(&str, u64, usize) -> Result, +{ + match selection { + ReplayBodySelection::Compact(text) => { + writer.write_all(text.as_bytes()).map_err(show_write_error) + } + ReplayBodySelection::Payload(payload) => { + stream_payload_with_reader(writer, &payload.field_path, false, read_range) + } + ReplayBodySelection::Projection(projection) => { + writer + .write_all(projection.text.as_bytes()) + .map_err(show_write_error)?; + if projection.truncated { + writer + .write_all(REPLAY_BODY_TRUNCATED_NOTICE) + .map_err(show_write_error)?; + } + Ok(()) + } + ReplayBodySelection::CompactProjection(text) => { + writer + .write_all(text.as_bytes()) + .map_err(show_write_error)?; + writer + .write_all(REPLAY_BODY_TRUNCATED_NOTICE) + .map_err(show_write_error) + } + } +} + +const REPLAY_BODY_TRUNCATED_NOTICE: &[u8] = + b"\n... [large replay body truncated; use --format json or export for the full payload]"; + +fn write_indexed_chunk_md_with_reader( + writer: &mut W, + indexed: &ReplayIndexedChunk, + read_range: &mut R, +) -> Result<(), String> +where + W: Write, + R: FnMut(&str, u64, usize) -> Result, +{ + let chunk = &indexed.chunk; + writeln!( + writer, + "**{}** · {}\n", + chunk_role(chunk), + truncate(&chunk.created_at, 19) + ) + .map_err(show_write_error)?; + let Some(selection) = select_replay_body(indexed) else { + return writer + .write_all(b"_(no content)_\n\n") + .map_err(show_write_error); + }; + if chunk.action_type == "tool_call" { + writer.write_all(b"```\n").map_err(show_write_error)?; + write_selected_replay_body(writer, selection, read_range)?; + writer.write_all(b"\n```\n\n").map_err(show_write_error) + } else { + write_selected_replay_body(writer, selection, read_range)?; + writer.write_all(b"\n\n").map_err(show_write_error) + } +} + +fn write_indexed_chunk_csv_with_reader( + writer: &mut W, + indexed: &ReplayIndexedChunk, + read_range: &mut R, +) -> Result<(), String> +where + W: Write, + R: FnMut(&str, u64, usize) -> Result, +{ + let chunk = &indexed.chunk; + let role = chunk_role(chunk); + for field in [ + chunk.created_at.as_str(), + role.as_str(), + chunk.action_type.as_str(), + chunk.function.as_str(), + ] { + write_csv_field(writer, field)?; + writer.write_all(b",").map_err(show_write_error)?; + } + writer.write_all(b"\"").map_err(show_write_error)?; + if let Some(selection) = select_replay_body(indexed) { + let mut csv_body = CsvBodyWriter { inner: writer }; + write_selected_replay_body(&mut csv_body, selection, read_range)?; + } + writer.write_all(b"\"\n").map_err(show_write_error) +} + +fn write_csv_field(writer: &mut impl Write, field: &str) -> Result<(), String> { + if field.contains([',', '"', '\n', '\r']) { + writer.write_all(b"\"").map_err(show_write_error)?; + let mut escaped = CsvBodyWriter { inner: writer }; + escaped + .write_all(field.as_bytes()) + .map_err(show_write_error)?; + writer.write_all(b"\"").map_err(show_write_error) + } else { + writer.write_all(field.as_bytes()).map_err(show_write_error) + } +} + +struct CsvBodyWriter<'a, W> { + inner: &'a mut W, +} + +impl Write for CsvBodyWriter<'_, W> { + fn write(&mut self, bytes: &[u8]) -> io::Result { + let mut start = 0usize; + for (index, byte) in bytes.iter().enumerate() { + let replacement = match byte { + b'"' => Some(&b"\"\""[..]), + b'\n' => Some(&b" "[..]), + _ => None, + }; + if let Some(replacement) = replacement { + self.inner.write_all(&bytes[start..index])?; + self.inner.write_all(replacement)?; + start = index + 1; + } + } + self.inner.write_all(&bytes[start..])?; + Ok(bytes.len()) + } + + fn flush(&mut self) -> io::Result<()> { + self.inner.flush() + } +} + +fn write_show_table_chunk(writer: &mut impl Write, chunk: &ActivityChunk) -> Result<(), String> { + let label = if chunk.function.is_empty() { + chunk.action_type.clone() + } else { + format!("{}:{}", chunk.action_type, chunk.function) + }; + writeln!(writer, "[{}] {}", truncate(&chunk.created_at, 19), label) + .map_err(show_write_error)?; + if let Some(text) = preview_of(&chunk.args).or_else(|| preview_of(&chunk.result)) { + writeln!(writer, " {}", truncate(&text, 160)).map_err(show_write_error)?; + } + Ok(()) +} + +fn show_write_error(error: io::Error) -> String { + format!("write show output: {error}") +} + +/// Portable markdown transcript of a session — the export format. Message +/// bodies render as prose; tool calls render as fenced code so a transcript +/// round-trips into any markdown viewer. +pub(crate) fn render_show_md(session_id: &str, chunks: &[ActivityChunk]) -> String { + let mut out = format!("# Session {session_id}\n\n"); + for chunk in chunks { + out.push_str(&render_show_md_chunk(chunk)); + } + out +} + +fn render_show_md_chunk(chunk: &ActivityChunk) -> String { + let role = chunk_role(chunk); + let mut out = format!("**{role}** · {}\n\n", truncate(&chunk.created_at, 19)); + let body = chunk_body(&chunk.args).or_else(|| chunk_body(&chunk.result)); + match body { + Some(text) if chunk.action_type == "tool_call" => { + out.push_str(&format!("```\n{}\n```\n\n", text.trim_end())) + } + Some(text) => out.push_str(&format!("{}\n\n", text.trim_end())), + None => out.push_str("_(no content)_\n\n"), + } + out +} + +pub(crate) fn render_show_csv(chunks: &[ActivityChunk]) -> String { + let mut out = String::from("created_at,role,action_type,function,preview\n"); + for chunk in chunks { + out.push_str(&render_show_csv_chunk(chunk)); + } + out +} + +fn render_show_csv_chunk(chunk: &ActivityChunk) -> String { + let preview = preview_of(&chunk.args) + .or_else(|| preview_of(&chunk.result)) + .unwrap_or_default(); + csv_row(&[ + &chunk.created_at, + &chunk_role(chunk), + &chunk.action_type, + &chunk.function, + &preview, + ]) +} + +/// Human role label for a chunk: `user`, `assistant`, `assistant (thinking)`, +/// or `tool: `. +pub(crate) fn chunk_role(chunk: &ActivityChunk) -> String { + match chunk.action_type.as_str() { + "raw" if chunk.function.contains("user") => "user".to_string(), + "assistant" => "assistant".to_string(), + "thinking" => "assistant (thinking)".to_string(), + "tool_call" => format!("tool: {}", chunk.function), + other => other.to_string(), + } +} + +#[cfg(test)] +#[path = "show_tests.rs"] +mod show_stream_tests; diff --git a/src-tauri/crates/orgtrack-cli/src/commands/show_tests.rs b/src-tauri/crates/orgtrack-cli/src/commands/show_tests.rs new file mode 100644 index 000000000..d40c5a703 --- /dev/null +++ b/src-tauri/crates/orgtrack-cli/src/commands/show_tests.rs @@ -0,0 +1,204 @@ +use super::*; + +fn payload( + field_path: &str, + encoding: replay::ReplayPayloadEncoding, + total_bytes: usize, +) -> ReplayPayloadDescriptor { + ReplayPayloadDescriptor { + field_path: field_path.to_string(), + kind: replay::ReplayPayloadKind::ToolOutput, + encoding, + body_projection: None, + spans: Vec::new(), + total_bytes: total_bytes as u64, + source_ordinal: None, + source_key: None, + } +} + +fn range_for(text: &str, field_path: &str, offset: u64, max_bytes: usize) -> ReplayPayloadRange { + let start = offset as usize; + let end = (start + max_bytes).min(text.len()); + ReplayPayloadRange { + event_id: "event-1".to_string(), + field_path: field_path.to_string(), + offset, + next_offset: end as u64, + eof: end == text.len(), + total_bytes: text.len() as u64, + text: text[start..end].to_string(), + } +} + +#[test] +fn streamed_json_restores_root_and_nested_array_payloads() { + let args_full = r#"{"command":"FULL_ROOT_COMMAND"}"#; + let result_full = "FULL_NESTED_\"quoted\"\\path\nnext"; + let indexed = ReplayIndexedChunk { + sequence: 7, + turn_index: 2, + chunk: ActivityChunk::new("codex-test", "tool_call", "shell") + .with_args(serde_json::json!({ + "command": "ARGS_TAIL_PREVIEW", + "_preview": "compact-only" + })) + .with_result(serde_json::json!({ + "content": [{"text": "ARRAY_TAIL_PREVIEW", "kind": "text"}] + })), + payloads: vec![ + payload( + "args", + replay::ReplayPayloadEncoding::JsonValue, + args_full.len(), + ), + payload( + "result.content.0.text", + replay::ReplayPayloadEncoding::Utf8Text, + result_full.len(), + ), + ], + }; + let mut offsets = Vec::new(); + let mut output = Vec::new(); + + write_indexed_chunk_json_with_reader( + &mut output, + &indexed, + &mut |field_path, offset, max_bytes| { + offsets.push((field_path.to_string(), offset)); + let text = match field_path { + "args" => args_full, + "result.content.0.text" => result_full, + other => panic!("unexpected payload path {other}"), + }; + Ok(range_for(text, field_path, offset, max_bytes)) + }, + ) + .expect("streamed JSON should rebuild both payload shapes"); + + let value: serde_json::Value = serde_json::from_slice(&output).expect("valid JSON"); + assert_eq!(value["args"]["command"], "FULL_ROOT_COMMAND"); + assert!(value["args"].get("_preview").is_none()); + assert_eq!(value["result"]["content"][0]["text"], result_full); + assert_eq!(value["result"]["content"][0]["kind"], "text"); + assert!(!String::from_utf8(output).unwrap().contains("TAIL_PREVIEW")); + assert_eq!( + offsets, + vec![ + ("args".to_string(), 0), + ("result.content.0.text".to_string(), 0) + ] + ); +} + +#[test] +fn streamed_markdown_and_csv_use_canonical_payload_not_preview() { + let canonical = "FULL,\"quoted\"\nsecond-line"; + let indexed = ReplayIndexedChunk { + sequence: 1, + turn_index: 0, + chunk: ActivityChunk::new("claude_code-test", "assistant", "assistant") + .with_result(serde_json::json!({"observation": "TAIL_PREVIEW"})), + payloads: vec![payload( + "result.observation", + replay::ReplayPayloadEncoding::Utf8Text, + canonical.len(), + )], + }; + let mut markdown = Vec::new(); + write_indexed_chunk_md_with_reader( + &mut markdown, + &indexed, + &mut |field_path, offset, max_bytes| { + Ok(range_for(canonical, field_path, offset, max_bytes)) + }, + ) + .expect("markdown payload should stream"); + let markdown = String::from_utf8(markdown).unwrap(); + assert!(markdown.contains(canonical)); + assert!(!markdown.contains("TAIL_PREVIEW")); + + let mut csv = Vec::new(); + write_indexed_chunk_csv_with_reader( + &mut csv, + &indexed, + &mut |field_path, offset, max_bytes| { + Ok(range_for(canonical, field_path, offset, max_bytes)) + }, + ) + .expect("CSV payload should stream"); + let csv = String::from_utf8(csv).unwrap(); + assert!(csv.contains("FULL,")); + assert!(csv.contains("\"\"quoted\"\" second-line")); + assert!(!csv.contains("TAIL_PREVIEW")); +} + +#[test] +fn root_body_projection_keeps_markdown_and_csv_bounded_without_payload_reads() { + let mut descriptor = payload( + "args", + replay::ReplayPayloadEncoding::JsonValue, + 10 * 1024 * 1024, + ); + descriptor.body_projection = Some(replay::ReplayPayloadBodyProjection { + field_path: "args.command".to_string(), + text: "cargo test --workspace".to_string(), + truncated: true, + }); + let indexed = ReplayIndexedChunk { + sequence: 1, + turn_index: 0, + chunk: ActivityChunk::new("codex-test", "tool_call", "shell") + .with_args(serde_json::json!({"command":"COMPACT_PREVIEW"})), + payloads: vec![descriptor], + }; + + let mut markdown = Vec::new(); + write_indexed_chunk_md_with_reader(&mut markdown, &indexed, &mut |_, _, _| { + panic!("bounded body projection must not hydrate the root payload") + }) + .expect("projected markdown"); + let markdown = String::from_utf8(markdown).unwrap(); + assert!(markdown.contains("cargo test --workspace")); + assert!(markdown.contains("large replay body truncated")); + assert!(!markdown.contains("COMPACT_PREVIEW")); + + let mut csv = Vec::new(); + write_indexed_chunk_csv_with_reader(&mut csv, &indexed, &mut |_, _, _| { + panic!("bounded body projection must not hydrate the root payload") + }) + .expect("projected CSV"); + let csv = String::from_utf8(csv).unwrap(); + assert!(csv.contains("cargo test --workspace")); + assert!(csv.contains("large replay body truncated")); + assert!(!csv.contains("COMPACT_PREVIEW")); +} + +#[test] +fn explicit_encoding_not_path_shape_controls_json_reconstruction() { + let canonical = r#"{"text":"FULL_OBJECT","kind":"text"}"#; + let indexed = ReplayIndexedChunk { + sequence: 1, + turn_index: 0, + chunk: ActivityChunk::new("codex-test", "assistant", "assistant") + .with_result(serde_json::json!({"content":["COMPACT_PREVIEW"]})), + payloads: vec![payload( + "result.content.0", + replay::ReplayPayloadEncoding::JsonValue, + canonical.len(), + )], + }; + let mut output = Vec::new(); + write_indexed_chunk_json_with_reader( + &mut output, + &indexed, + &mut |field_path, offset, max_bytes| { + Ok(range_for(canonical, field_path, offset, max_bytes)) + }, + ) + .expect("nested JSON value payload"); + let restored: serde_json::Value = serde_json::from_slice(&output).expect("valid JSON"); + assert_eq!(restored["result"]["content"][0]["text"], "FULL_OBJECT"); + assert_eq!(restored["result"]["content"][0]["kind"], "text"); +} diff --git a/src-tauri/crates/orgtrack-cli/src/content_index.rs b/src-tauri/crates/orgtrack-cli/src/content_index.rs index b6538c9eb..753deeba4 100644 --- a/src-tauri/crates/orgtrack-cli/src/content_index.rs +++ b/src-tauri/crates/orgtrack-cli/src/content_index.rs @@ -19,15 +19,29 @@ use core_types::activity::ActivityChunk; use rusqlite::{params, Connection}; -use crate::plugin_exec::load_session_chunks; +use orgtrack_core::sources::imported_history::{ + replay::{ + self, ImportedHistorySourceId, ReplayIndexedChunk, ReplayLimits, ReplayPayloadDescriptor, + ReplayPayloadRange, + }, + router as replay_router, +}; + +use crate::plugin_exec::load_plugin_session_chunks; use crate::plugins::LoaderPlugin; use crate::scan::target_source_ids; use crate::Options; /// Max characters of transcript text indexed per session. const MAX_BODY_CHARS: usize = 256 * 1024; +/// Compact-index rows read per FTS page. The byte limit is the second bound. +const CONTENT_PAGE_EVENTS: usize = 32; +const CONTENT_PAGE_IPC_BYTES: usize = 512 * 1024; /// Sessions re-indexed per write transaction. const BATCH: usize = 64; +/// One retry absorbs a source replacement or transient plugin/IPC failure. +/// Persistent failures remain uncommitted so the next search retries them. +const SESSION_BODY_ATTEMPTS: usize = 2; /// One full-text hit: a session plus a highlighted snippet around the match. pub(crate) struct ContentHit { @@ -119,9 +133,30 @@ pub(crate) fn update( let mut indexed = 0usize; for batch in stale.chunks(BATCH) { + // Replay synchronization needs the writable connection and may create + // its own transaction. Build bounded bodies before opening the FTS + // transaction; at most BATCH * MAX_BODY_CHARS are retained here. + let mut bodies = Vec::with_capacity(batch.len()); + for (candidate_index, (session_id, _, _, _)) in batch.iter().enumerate() { + match retry_session_body(session_id, || { + session_body(conn, session_id, plugins, timeout) + }) { + Ok(body) => bodies.push((candidate_index, body)), + Err(error) => { + // Keep this session's previous FTS row and stale + // fingerprint. One malformed/deleted source must not make + // every healthy session unsearchable, and the unchanged + // state row makes the next command retry this session. + eprintln!("warning: skipped content index for {session_id}: {error}"); + } + } + } + if bodies.is_empty() { + continue; + } let tx = conn.transaction().map_err(|err| err.to_string())?; - for (session_id, source, name, fingerprint) in batch { - let body = session_body(&tx, session_id, plugins, timeout); + for (candidate_index, body) in bodies { + let (session_id, source, name, fingerprint) = &batch[candidate_index]; tx.execute( "DELETE FROM orgtrack_fts WHERE session_id = ?1", [session_id], @@ -148,6 +183,23 @@ pub(crate) fn update( Ok(indexed) } +fn retry_session_body( + session_id: &str, + mut load: impl FnMut() -> Result, +) -> Result { + let mut last_error = None; + for _ in 0..SESSION_BODY_ATTEMPTS { + match load() { + Ok(body) => return Ok(body), + Err(error) => last_error = Some(error), + } + } + Err(format!( + "content body failed after {SESSION_BODY_ATTEMPTS} attempts for {session_id}: {}", + last_error.unwrap_or_else(|| "unknown content loader failure".to_string()) + )) +} + /// Run the FTS5 query and return ranked hits with highlighted snippets. pub(crate) fn search( conn: &Connection, @@ -184,34 +236,258 @@ pub(crate) fn search( /// Flatten a session's chunks into a bounded plain-text body for indexing. fn session_body( - conn: &Connection, + conn: &mut Connection, session_id: &str, plugins: &[LoaderPlugin], timeout: std::time::Duration, -) -> String { - let chunks = load_session_chunks(conn, session_id, plugins, timeout) - .ok() - .flatten() - .unwrap_or_default(); +) -> Result { + if let Some(source) = replay_router::source_for_session(session_id) { + return replay_session_body(conn, source, session_id); + } + + // The full-vector protocol is confined to third-party loaders. Built-in + // prefixes are handled above and can never fall through to it. + let chunks = + load_plugin_session_chunks(conn, session_id, plugins, timeout)?.unwrap_or_default(); + Ok(body_from_chunks(chunks, MAX_BODY_CHARS)) +} + +fn replay_session_body( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, +) -> Result { + let mut body = String::new(); + let mut remaining_chars = MAX_BODY_CHARS; + let mut cursor = None; + while remaining_chars > 0 { + let previous_sequence = cursor + .as_ref() + .map_or(-1, |cursor: &replay::ReplayCursor| cursor.through_sequence); + let scan = replay_router::scan_activity_chunks_for_session( + conn, + session_id, + cursor.as_ref(), + ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: CONTENT_PAGE_EVENTS, + max_ipc_bytes: CONTENT_PAGE_IPC_BYTES, + }, + )? + .ok_or_else(|| format!("Unknown built-in imported session id: {session_id}"))?; + if scan.chunks.is_empty() + && scan.has_more + && scan.cursor.through_sequence <= previous_sequence + { + return Err(format!( + "Bounded replay content scan made no progress for {session_id} after sequence {}", + scan.cursor.through_sequence + )); + } + let scan_has_more = scan.has_more; + let scan_cursor = scan.cursor; + for indexed in &scan.chunks { + append_indexed_chunk_text( + conn, + source, + session_id, + &scan_cursor.generation, + &mut body, + &mut remaining_chars, + indexed, + )?; + if remaining_chars == 0 { + return Ok(body); + } + } + cursor = Some(scan_cursor); + if !scan_has_more { + break; + } + } + Ok(body) +} + +fn body_from_chunks(chunks: impl IntoIterator, max_chars: usize) -> String { let mut body = String::new(); - for chunk in &chunks { - if body.len() >= MAX_BODY_CHARS { + let mut remaining_chars = max_chars; + for chunk in chunks { + append_chunk_text(&mut body, &mut remaining_chars, &chunk); + if remaining_chars == 0 { break; } - append_chunk_text(&mut body, chunk); } - body.truncate(MAX_BODY_CHARS); body } /// Pull the human-readable text out of a chunk (message content, tool command, /// tool output) into `body`. -fn append_chunk_text(body: &mut String, chunk: &ActivityChunk) { +fn append_chunk_text(body: &mut String, remaining_chars: &mut usize, chunk: &ActivityChunk) { for value in [&chunk.result, &chunk.args] { if let Some(text) = text_of(value) { - body.push_str(&text); - body.push('\n'); + append_text_with_budget(body, remaining_chars, &text); + if *remaining_chars == 0 { + return; + } + } + } +} + +fn append_indexed_chunk_text( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + body: &mut String, + remaining_chars: &mut usize, + indexed: &ReplayIndexedChunk, +) -> Result<(), String> { + append_indexed_chunk_text_with_reader( + body, + remaining_chars, + indexed, + |payload, offset, max_bytes| { + replay::read_payload_range( + conn, + source, + session_id, + generation, + &indexed.chunk.chunk_id, + &payload.field_path, + offset, + Some(max_bytes), + ) + }, + ) +} + +fn append_indexed_chunk_text_with_reader( + body: &mut String, + remaining_chars: &mut usize, + indexed: &ReplayIndexedChunk, + mut read_range: impl FnMut( + &ReplayPayloadDescriptor, + u64, + usize, + ) -> Result, +) -> Result<(), String> { + // Preserve the old transcript root order (result, then args), but remove + // every deferred path from the compact projection before indexing it. + // A preview can be a head or a Shell tail, so it must never be treated as + // an already-consumed prefix of the canonical payload. + for (root, value) in [ + ("result", &indexed.chunk.result), + ("args", &indexed.chunk.args), + ] { + let root_payloads = indexed + .payloads + .iter() + .filter(|payload| payload_belongs_to_root(&payload.field_path, root)) + .collect::>(); + let mut projected = value.clone(); + for payload in &root_payloads { + remove_deferred_path(&mut projected, root, &payload.field_path); + } + if let Some(text) = text_of(&projected) { + append_text_with_budget(body, remaining_chars, &text); + } + for payload in root_payloads { + if *remaining_chars == 0 { + return Ok(()); + } + let mut offset = 0u64; + loop { + let max_bytes = remaining_chars + .saturating_mul(4) + .clamp(1, replay::HARD_MAX_PAYLOAD_RANGE_BYTES); + let range = read_range(payload, offset, max_bytes)?; + if range.offset != offset { + return Err(format!( + "Replay payload {}:{} skipped from {offset} to {}", + indexed.chunk.chunk_id, payload.field_path, range.offset + )); + } + append_text_with_budget(body, remaining_chars, &range.text); + if *remaining_chars == 0 || range.eof { + break; + } + if range.next_offset <= offset { + return Err(format!( + "Replay payload {}:{} made no progress at {offset}", + indexed.chunk.chunk_id, payload.field_path + )); + } + offset = range.next_offset; + } + } + } + Ok(()) +} + +fn payload_belongs_to_root(field_path: &str, root: &str) -> bool { + field_path == root + || field_path + .strip_prefix(root) + .is_some_and(|suffix| suffix.starts_with('.')) +} + +fn remove_deferred_path(value: &mut serde_json::Value, root: &str, field_path: &str) { + let Some(relative) = field_path.strip_prefix(root) else { + return; + }; + if relative.is_empty() { + *value = serde_json::Value::Null; + return; + } + let keys = relative + .trim_start_matches('.') + .split('.') + .collect::>(); + remove_json_path(value, &keys); +} + +fn remove_json_path(value: &mut serde_json::Value, keys: &[&str]) { + let Some((key, rest)) = keys.split_first() else { + return; + }; + match value { + serde_json::Value::Object(object) => { + if rest.is_empty() { + object.remove(*key); + } else if let Some(child) = object.get_mut(*key) { + remove_json_path(child, rest); + } + } + serde_json::Value::Array(items) => { + let Some(index) = key.parse::().ok() else { + return; + }; + let Some(child) = items.get_mut(index) else { + return; + }; + if rest.is_empty() { + *child = serde_json::Value::Null; + } else { + remove_json_path(child, rest); + } } + _ => {} + } +} + +fn append_text_with_budget(body: &mut String, remaining_chars: &mut usize, text: &str) { + if *remaining_chars == 0 { + return; + } + let mut written = 0usize; + for ch in text.chars().take(*remaining_chars) { + body.push(ch); + written += 1; + } + *remaining_chars = (*remaining_chars).saturating_sub(written); + if *remaining_chars > 0 { + body.push('\n'); + *remaining_chars -= 1; } } @@ -267,6 +543,8 @@ fn sanitize_query(query: &str) -> String { #[cfg(test)] mod tests { + use std::cell::Cell; + use super::*; #[test] @@ -289,4 +567,242 @@ mod tests { assert!(text_of(&serde_json::json!({})).is_none()); assert!(text_of(&serde_json::json!("")).is_none()); } + + #[test] + fn virtual_three_hundred_mib_transcript_stops_at_the_character_budget() { + const ONE_MIB: usize = 1024 * 1024; + let generated = Cell::new(0usize); + let chunks = (0..300).map(|_| { + generated.set(generated.get() + 1); + ActivityChunk::new("plugin-session", "assistant", "assistant") + .with_result(serde_json::Value::String("x".repeat(ONE_MIB))) + }); + + let body = body_from_chunks(chunks, MAX_BODY_CHARS); + + assert_eq!(body.chars().count(), MAX_BODY_CHARS); + assert_eq!( + generated.get(), + 1, + "the lazy 300 MiB generator must stop after the first bounded chunk" + ); + } + + #[test] + fn character_budget_never_splits_utf8() { + let mut body = String::new(); + let mut remaining = 3usize; + append_text_with_budget(&mut body, &mut remaining, "é你🙂tail"); + assert_eq!(body, "é你🙂"); + assert_eq!(remaining, 0); + assert!(std::str::from_utf8(body.as_bytes()).is_ok()); + } + + #[test] + fn compact_content_pages_are_stricter_than_replay_hard_limits() { + const { + assert!(CONTENT_PAGE_EVENTS <= replay::HARD_MAX_EVENTS); + assert!(CONTENT_PAGE_IPC_BYTES <= replay::HARD_MAX_IPC_BYTES); + } + } + + #[test] + fn session_body_retry_recovers_transient_failures_and_bounds_persistent_ones() { + let attempts = Cell::new(0usize); + let body = retry_session_body("transient", || { + attempts.set(attempts.get() + 1); + if attempts.get() == 1 { + Err("source changed".to_string()) + } else { + Ok("indexed body".to_string()) + } + }) + .expect("second attempt succeeds"); + assert_eq!(body, "indexed body"); + assert_eq!(attempts.get(), SESSION_BODY_ATTEMPTS); + + attempts.set(0); + let error = retry_session_body("broken", || { + attempts.set(attempts.get() + 1); + Err("still broken".to_string()) + }) + .expect_err("persistent failure stays isolated"); + assert_eq!(attempts.get(), SESSION_BODY_ATTEMPTS); + assert!(error.contains("still broken")); + } + + #[test] + fn deferred_payload_replaces_tail_preview_and_keeps_legacy_root_order() { + let canonical = "FULL_START-full-result-FULL_END"; + let indexed = ReplayIndexedChunk { + sequence: 0, + turn_index: 0, + chunk: ActivityChunk::new("codex-test", "tool_call", "shell") + .with_args(serde_json::json!({"command": "later-command"})) + .with_result(serde_json::json!({"observation": "TAIL_PREVIEW"})), + payloads: vec![ReplayPayloadDescriptor { + field_path: "result.observation".to_string(), + kind: replay::ReplayPayloadKind::ToolOutput, + encoding: replay::ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: Vec::new(), + total_bytes: canonical.len() as u64, + source_ordinal: None, + source_key: None, + }], + }; + let mut body = String::new(); + let mut remaining = canonical.chars().count() + "later-command".chars().count() + 2; + let mut offsets = Vec::new(); + + append_indexed_chunk_text_with_reader( + &mut body, + &mut remaining, + &indexed, + |payload, offset, max_bytes| { + assert_eq!(payload.field_path, "result.observation"); + offsets.push(offset); + let start = offset as usize; + let end = (start + max_bytes).min(canonical.len()); + Ok(ReplayPayloadRange { + event_id: indexed.chunk.chunk_id.clone(), + field_path: payload.field_path.clone(), + offset, + next_offset: end as u64, + eof: end == canonical.len(), + total_bytes: canonical.len() as u64, + text: canonical[start..end].to_string(), + }) + }, + ) + .expect("deferred payload should stream"); + + assert_eq!(offsets.first(), Some(&0), "a tail preview is not a prefix"); + assert_eq!(body.matches(canonical).count(), 1); + assert!(!body.contains("TAIL_PREVIEW")); + assert!( + body.find(canonical).expect("canonical result") + < body.find("later-command").expect("later args") + ); + } + + #[test] + fn deferred_payload_gets_the_body_budget_before_later_roots() { + let canonical = "canonical-result"; + let indexed = ReplayIndexedChunk { + sequence: 0, + turn_index: 0, + chunk: ActivityChunk::new("codex-test", "tool_call", "shell") + .with_args(serde_json::json!({"command": "must-not-starve-result"})) + .with_result(serde_json::json!({"observation": "preview-must-not-count"})), + payloads: vec![ReplayPayloadDescriptor { + field_path: "result.observation".to_string(), + kind: replay::ReplayPayloadKind::ToolOutput, + encoding: replay::ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: Vec::new(), + total_bytes: canonical.len() as u64, + source_ordinal: None, + source_key: None, + }], + }; + let mut body = String::new(); + let mut remaining = canonical.len(); + + append_indexed_chunk_text_with_reader( + &mut body, + &mut remaining, + &indexed, + |_payload, offset, max_bytes| { + let start = offset as usize; + let end = (start + max_bytes).min(canonical.len()); + Ok(ReplayPayloadRange { + event_id: indexed.chunk.chunk_id.clone(), + field_path: "result.observation".to_string(), + offset, + next_offset: end as u64, + eof: end == canonical.len(), + total_bytes: canonical.len() as u64, + text: canonical[start..end].to_string(), + }) + }, + ) + .expect("deferred payload should stream"); + + assert_eq!(body, canonical); + assert_eq!(remaining, 0); + } + + #[test] + fn deferred_path_removal_descends_numeric_array_segments() { + let mut result = serde_json::json!({ + "content": [ + {"text": "ARRAY_TAIL_PREVIEW", "kind": "text"}, + {"text": "keep-me"} + ] + }); + + remove_deferred_path(&mut result, "result", "result.content.0.text"); + + assert_eq!(result["content"][0]["text"], serde_json::Value::Null); + assert_eq!(result["content"][0]["kind"], "text"); + assert_eq!(result["content"][1]["text"], "keep-me"); + + // Bad or out-of-range indices are a no-op, not a panic or an + // accidental deletion of an adjacent array item. + let unchanged = result.clone(); + remove_deferred_path(&mut result, "result", "result.content.99.text"); + remove_deferred_path(&mut result, "result", "result.content.nope.text"); + assert_eq!(result, unchanged); + } + + #[test] + fn nested_array_deferred_payload_indexes_only_the_canonical_body() { + let canonical = "FULL_ARRAY_BODY"; + let indexed = ReplayIndexedChunk { + sequence: 0, + turn_index: 0, + chunk: ActivityChunk::new("codex-test", "assistant", "assistant").with_result( + serde_json::json!({ + "content": [{"text": "ARRAY_TAIL_PREVIEW", "kind": "text"}] + }), + ), + payloads: vec![ReplayPayloadDescriptor { + field_path: "result.content.0.text".to_string(), + kind: replay::ReplayPayloadKind::AssistantContent, + encoding: replay::ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: Vec::new(), + total_bytes: canonical.len() as u64, + source_ordinal: None, + source_key: None, + }], + }; + let mut body = String::new(); + let mut remaining = 128; + + append_indexed_chunk_text_with_reader( + &mut body, + &mut remaining, + &indexed, + |payload, offset, max_bytes| { + assert_eq!(payload.field_path, "result.content.0.text"); + let start = offset as usize; + let end = (start + max_bytes).min(canonical.len()); + Ok(ReplayPayloadRange { + event_id: indexed.chunk.chunk_id.clone(), + field_path: payload.field_path.clone(), + offset, + next_offset: end as u64, + eof: end == canonical.len(), + total_bytes: canonical.len() as u64, + text: canonical[start..end].to_string(), + }) + }, + ) + .expect("nested deferred payload should stream"); + + assert!(body.contains(canonical)); + assert!(!body.contains("ARRAY_TAIL_PREVIEW")); + } } diff --git a/src-tauri/crates/orgtrack-cli/src/main.rs b/src-tauri/crates/orgtrack-cli/src/main.rs index 8bf1fb6f7..d39e14d29 100644 --- a/src-tauri/crates/orgtrack-cli/src/main.rs +++ b/src-tauri/crates/orgtrack-cli/src/main.rs @@ -10,8 +10,9 @@ //! 1. [`registry::scan_source`] — discover + cache one provider's sessions. //! 2. [`session_usage::backfill_session_usage`] — project cached sessions //! into the usage table the analytics layer reads. -//! 3. [`usage_dashboard`] / [`imported_history::load_activity_chunks_for_session`] -//! — analyze and replay. +//! 3. [`usage_dashboard`] / +//! [`orgtrack_core::sources::imported_history::router::scan_activity_chunks_for_session`] +//! — analyze and replay through bounded compact-index pages. //! //! This binary is only argument parsing, orchestration, and formatting. diff --git a/src-tauri/crates/orgtrack-cli/src/plugin_exec.rs b/src-tauri/crates/orgtrack-cli/src/plugin_exec.rs index a6dbe88d1..1ba5e846c 100644 --- a/src-tauri/crates/orgtrack-cli/src/plugin_exec.rs +++ b/src-tauri/crates/orgtrack-cli/src/plugin_exec.rs @@ -240,11 +240,12 @@ pub(crate) fn js_str_vec(value: &serde_json::Value, key: &str) -> Vec { .unwrap_or_default() } -/// Load a session's activity chunks, routing plugin sessions (matched by their -/// `session_prefix`) through the plugin's own loader (generic JSONL, or the -/// exec plugin's `load` verb), and everything else through core's built-in -/// provider router. `Ok(None)` = unknown id. -pub(crate) fn load_session_chunks( +/// Load chunks only across the explicit third-party loader boundary. +/// +/// Built-in session ids must be handled by core's bounded replay router before +/// calling this function. There is deliberately no built-in/full-history +/// fallback here; `Ok(None)` means no plugin owns the prefix. +pub(crate) fn load_plugin_session_chunks( conn: &Connection, session_id: &str, plugins: &[LoaderPlugin], @@ -269,7 +270,7 @@ pub(crate) fn load_session_chunks( } }; } - imported_history::load_activity_chunks_for_session(conn, session_id) + Ok(None) } /// The source id a session belongs to, resolved from a plugin `session_prefix`. @@ -515,6 +516,15 @@ mod tests { assert!(exec_session_to_input(&job, &serde_json::json!({"name": "x"})).is_err()); } + #[test] + fn built_in_prefix_never_falls_back_through_plugin_loader() { + let conn = Connection::open_in_memory().expect("in-memory DB"); + let loaded = + load_plugin_session_chunks(&conn, "codexapp-built-in", &[], Duration::from_secs(1)) + .expect("empty plugin registry must not query built-in history"); + assert!(loaded.is_none()); + } + #[test] fn scanned_row_reconstructs_from_json() { let value = serde_json::json!({ diff --git a/src-tauri/crates/orgtrack-core/Cargo.toml b/src-tauri/crates/orgtrack-core/Cargo.toml index ce0f5e5bb..682b59d0e 100644 --- a/src-tauri/crates/orgtrack-core/Cargo.toml +++ b/src-tauri/crates/orgtrack-core/Cargo.toml @@ -13,7 +13,7 @@ path = "src/lib.rs" [features] default = ["sqlite", "repo-sync", "git"] -sqlite = ["dep:rusqlite"] +sqlite = ["dep:rusqlite", "dep:sha2"] repo-sync = ["dep:sha2"] git = ["dep:git2"] source-cursor = [] @@ -45,4 +45,6 @@ git2 = { version = "0.20", default-features = false, features = [ ], optional = true } [dev-dependencies] +libc = "0.2" prost = "0.14.3" +tempfile = "3.13" diff --git a/src-tauri/crates/orgtrack-core/src/development_artifact.rs b/src-tauri/crates/orgtrack-core/src/development_artifact.rs index 2ed5b3170..40ffc1411 100644 --- a/src-tauri/crates/orgtrack-core/src/development_artifact.rs +++ b/src-tauri/crates/orgtrack-core/src/development_artifact.rs @@ -10,6 +10,13 @@ use std::sync::LazyLock; use core_types::extracted::{ExtractedGitArtifactData, GitArtifactKind}; use regex::Regex; +use serde_json::Value; + +/// Compact replay rows keep pre-extracted Git artifacts here when the source +/// output itself is range-backed. This prevents a SHA near the head of a huge +/// output (or a PR URL in its middle) from disappearing when the renderer +/// preview keeps only a bounded head/tail slice. +pub const REPLAY_GIT_ARTIFACTS_FIELD: &str = "_replayGitArtifacts"; static GIT_COMMAND_CONTEXT_RE: LazyLock = LazyLock::new(|| { Regex::new(r"(?i)(^|[;&|()\s])(git|gh)(\s|$)").expect("valid git command context regex") @@ -58,6 +65,28 @@ pub fn parse_git_artifacts(input: GitArtifactParseInput<'_>) -> Vec Vec { + result + .get(REPLAY_GIT_ARTIFACTS_FIELD) + .cloned() + .and_then(|value| serde_json::from_value(value).ok()) + .unwrap_or_default() +} + /// Parse the raw args/result JSON stored in `sessions.db.events`. /// /// The shape mirrors the live shell extractor: completed tool payloads may diff --git a/src-tauri/crates/orgtrack-core/src/profile/signals.rs b/src-tauri/crates/orgtrack-core/src/profile/signals.rs index 114a62a90..f83983084 100644 --- a/src-tauri/crates/orgtrack-core/src/profile/signals.rs +++ b/src-tauri/crates/orgtrack-core/src/profile/signals.rs @@ -346,56 +346,85 @@ fn active_spans(mut stamps: Vec) -> Vec<(i64, i64)> { out } -/// Extract one session's signals from its chunk stream. -pub fn extract(session_id: &str, source: &str, chunks: &[ActivityChunk]) -> SessionSignals { - let mut s = SessionSignals { - session_id: session_id.to_string(), - source: source.to_string(), - signals_version: SIGNALS_VERSION, - ..Default::default() - }; - - let mut stamps: Vec = Vec::new(); - let mut chains: Vec = Vec::new(); - let mut chain = 0i64; - let (mut interrupts, mut questions, mut redirects) = (0i64, 0i64, 0i64); - let (mut postedit_evidence, mut postedit_moveon, mut postedit_paste) = (0i64, 0i64, 0i64); - let mut pending_postedit = false; - let mut first_brief_seen = false; - let (mut lines_added, mut lines_removed) = (0i64, 0i64); - let mut planned_first = false; - let mut edited: BTreeSet = BTreeSet::new(); - let mut edit_kinds: Vec = Vec::new(); +/// Incremental profile projection for one replay scan. +/// +/// External histories are read through bounded replay pages. Keeping the +/// folding state separate from page storage prevents a large transcript from +/// becoming a session-sized `Vec` just to compute aggregates. +pub struct SessionSignalsAccumulator { + signals: SessionSignals, + stamps: Vec, + chains: Vec, + chain: i64, + interrupts: i64, + questions: i64, + redirects: i64, + postedit_evidence: i64, + postedit_moveon: i64, + postedit_paste: i64, + pending_postedit: bool, + first_brief_seen: bool, + lines_added: i64, + lines_removed: i64, + planned_first: bool, + edit_kinds: Vec, +} - for chunk in chunks { +impl SessionSignalsAccumulator { + pub fn new(session_id: &str, source: &str) -> Self { + Self { + signals: SessionSignals { + session_id: session_id.to_string(), + source: source.to_string(), + signals_version: SIGNALS_VERSION, + ..Default::default() + }, + stamps: Vec::new(), + chains: Vec::new(), + chain: 0, + interrupts: 0, + questions: 0, + redirects: 0, + postedit_evidence: 0, + postedit_moveon: 0, + postedit_paste: 0, + pending_postedit: false, + first_brief_seen: false, + lines_added: 0, + lines_removed: 0, + planned_first: false, + edit_kinds: Vec::new(), + } + } + + pub fn push(&mut self, chunk: &ActivityChunk) { if let Some(ms) = parse_ts_ms(&chunk.created_at) { - stamps.push(ms); + self.stamps.push(ms); } match chunk.action_type.as_str() { - ACTION_TYPE_TASK_FAILED => interrupts += 1, // Codex reports aborts explicitly - ACTION_TYPE_ASSISTANT => s.assistant_turns += 1, + ACTION_TYPE_TASK_FAILED => self.interrupts += 1, // Codex reports aborts explicitly + ACTION_TYPE_ASSISTANT => self.signals.assistant_turns += 1, ACTION_TYPE_TOOL_CALL => { - s.tool_calls += 1; - chain += 1; + self.signals.tool_calls += 1; + self.chain += 1; let kind = classify_tool(&chunk.function); - if kind == ToolKind::Todo && !s.has_edit { - planned_first = true; + if kind == ToolKind::Todo && !self.signals.has_edit { + self.planned_first = true; } if kind == ToolKind::Delegate { - s.delegate_calls += 1; + self.signals.delegate_calls += 1; } if kind == ToolKind::Edit { - s.edit_calls += 1; - s.has_edit = true; - pending_postedit = true; + self.signals.edit_calls += 1; + self.signals.has_edit = true; + self.pending_postedit = true; if let Some(p) = tool_path(&chunk.args) { - edit_kinds.push(classify_path(p)); - edited.insert(p.to_string()); + self.edit_kinds.push(classify_path(p)); } // Only some readers attach a diff; absent means unknown, not zero. let num = |k: &str| chunk.result.get(k).and_then(Value::as_i64).unwrap_or(0); - lines_added += num("linesAdded"); - lines_removed += num("linesRemoved"); + self.lines_added += num("linesAdded"); + self.lines_removed += num("linesRemoved"); } } ACTION_TYPE_RAW if chunk.function == FUNCTION_USER_MESSAGE => { @@ -412,94 +441,114 @@ pub fn extract(session_id: &str, source: &str, chunks: &[ActivityChunk]) -> Sess // The interrupt sentinel arrives as an ordinary user message on // Claude Code; it is not a human turn. if interrupt_re().is_match(body) { - interrupts += 1; - continue; + self.interrupts += 1; + return; } let text = clean_user_text(body); if text.is_empty() { - continue; + return; } - if chain > 0 { - chains.push(chain); - s.max_chain = s.max_chain.max(chain); - chain = 0; + if self.chain > 0 { + self.chains.push(self.chain); + self.signals.max_chain = self.signals.max_chain.max(self.chain); + self.chain = 0; } - s.user_turns += 1; + self.signals.user_turns += 1; let words = text.split_whitespace().count() as i64; - s.prompt_words += words; - s.longest_prompt_words = s.longest_prompt_words.max(words); - if !first_brief_seen { - first_brief_seen = true; + self.signals.prompt_words += words; + self.signals.longest_prompt_words = self.signals.longest_prompt_words.max(words); + if !self.first_brief_seen { + self.first_brief_seen = true; let head: String = text.chars().take(BRIEF_SCAN_CHARS).collect(); - s.brief_items = list_item_re().find_iter(&head).count() as i64; - s.brief_constraints = constraint_re().find_iter(&head).count() as i64; - s.brief_targets = target_re().find_iter(&head).count() as i64; + self.signals.brief_items = list_item_re().find_iter(&head).count() as i64; + self.signals.brief_constraints = + constraint_re().find_iter(&head).count() as i64; + self.signals.brief_targets = target_re().find_iter(&head).count() as i64; } - questions += text.matches('?').count() as i64 + text.matches('？').count() as i64; + self.questions += + text.matches('?').count() as i64 + text.matches('？').count() as i64; if redirect_re().is_match(text) { - redirects += 1; + self.redirects += 1; } - if pending_postedit { - pending_postedit = false; - s.postedit_turns += 1; + if self.pending_postedit { + self.pending_postedit = false; + self.signals.postedit_turns += 1; if text.contains("```") || text.matches('\n').count() >= 3 { - postedit_paste += 1; + self.postedit_paste += 1; } if evidence_re().is_match(text) { - postedit_evidence += 1; + self.postedit_evidence += 1; } else if ack_re().is_match(text) || text.chars().count() < SHORT_REPLY_CHARS { - postedit_moveon += 1; + self.postedit_moveon += 1; } } } _ => {} } } - if chain > 0 { - chains.push(chain); - s.max_chain = s.max_chain.max(chain); - } - let users = s.user_turns.max(1) as f64; - let turns = (s.user_turns + s.assistant_turns).max(1) as f64; - let edits = edit_kinds.len().max(1) as f64; - let postedit = s.postedit_turns.max(1) as f64; + pub fn finish(mut self) -> SessionSignals { + if self.chain > 0 { + self.chains.push(self.chain); + self.signals.max_chain = self.signals.max_chain.max(self.chain); + } + + let users = self.signals.user_turns.max(1) as f64; + let turns = (self.signals.user_turns + self.signals.assistant_turns).max(1) as f64; + let edits = self.edit_kinds.len().max(1) as f64; + let postedit = self.signals.postedit_turns.max(1) as f64; - s.tools_per_user = s.tool_calls as f64 / users; - s.mean_chain = if chains.is_empty() { - 0.0 - } else { - chains.iter().sum::() as f64 / chains.len() as f64 - }; - s.interrupt_rate = interrupts as f64 / users; - s.user_share = s.user_turns as f64 / turns; - s.question_rate = questions as f64 / users; - s.redirect_rate = redirects as f64 / users; - - let share = |k: PathKind| edit_kinds.iter().filter(|x| **x == k).count() as f64 / edits; - s.harness_edit_share = share(PathKind::Harness); - s.infra_edit_share = share(PathKind::Infra); - s.doc_edit_share = share(PathKind::Doc); - s.test_edit_share = share(PathKind::Test); - s.product_edit_share = share(PathKind::Product); - - s.postedit_evidence_rate = postedit_evidence as f64 / postedit; - s.postedit_moveon_rate = postedit_moveon as f64 / postedit; - s.postedit_paste_rate = postedit_paste as f64 / postedit; - - s.lines_added = lines_added; - s.lines_removed = lines_removed; - s.planned_first = planned_first; - - let spans = active_spans(stamps); - s.longest_span_secs = spans - .iter() - .map(|(a, b)| (b - a) as f64 / 1000.0) - .fold(0.0, f64::max); - s.started_at_ms = spans.first().map(|(a, _)| *a).unwrap_or(0); - s.active_secs = spans.iter().map(|(a, b)| (b - a) as f64 / 1000.0).sum(); - s.active_spans = spans; - s + self.signals.tools_per_user = self.signals.tool_calls as f64 / users; + self.signals.mean_chain = if self.chains.is_empty() { + 0.0 + } else { + self.chains.iter().sum::() as f64 / self.chains.len() as f64 + }; + self.signals.interrupt_rate = self.interrupts as f64 / users; + self.signals.user_share = self.signals.user_turns as f64 / turns; + self.signals.question_rate = self.questions as f64 / users; + self.signals.redirect_rate = self.redirects as f64 / users; + + let share = |kind: PathKind| { + self.edit_kinds + .iter() + .filter(|candidate| **candidate == kind) + .count() as f64 + / edits + }; + self.signals.harness_edit_share = share(PathKind::Harness); + self.signals.infra_edit_share = share(PathKind::Infra); + self.signals.doc_edit_share = share(PathKind::Doc); + self.signals.test_edit_share = share(PathKind::Test); + self.signals.product_edit_share = share(PathKind::Product); + + self.signals.postedit_evidence_rate = self.postedit_evidence as f64 / postedit; + self.signals.postedit_moveon_rate = self.postedit_moveon as f64 / postedit; + self.signals.postedit_paste_rate = self.postedit_paste as f64 / postedit; + + self.signals.lines_added = self.lines_added; + self.signals.lines_removed = self.lines_removed; + self.signals.planned_first = self.planned_first; + + let spans = active_spans(self.stamps); + self.signals.longest_span_secs = spans + .iter() + .map(|(a, b)| (b - a) as f64 / 1000.0) + .fold(0.0, f64::max); + self.signals.started_at_ms = spans.first().map(|(a, _)| *a).unwrap_or(0); + self.signals.active_secs = spans.iter().map(|(a, b)| (b - a) as f64 / 1000.0).sum(); + self.signals.active_spans = spans; + self.signals + } +} + +/// Extract one session's signals from an already-materialized chunk slice. +pub fn extract(session_id: &str, source: &str, chunks: &[ActivityChunk]) -> SessionSignals { + let mut accumulator = SessionSignalsAccumulator::new(session_id, source); + for chunk in chunks { + accumulator.push(chunk); + } + accumulator.finish() } /// Share of each session's active time during which at least one *other* @@ -651,6 +700,32 @@ mod tests { assert_eq!(s.product_edit_share, 1.0); } + #[test] + fn incremental_pages_preserve_the_existing_signal_projection() { + let chunks = vec![ + user("add a parser test", "2026-07-01T10:00:00Z"), + tool( + "Read", + json!({ "file_path": "src/parser.rs" }), + "2026-07-01T10:00:05Z", + ), + tool( + "Edit", + json!({ "file_path": "src/parser.rs" }), + "2026-07-01T10:00:10Z", + ), + user("still fails with 404?", "2026-07-01T10:01:00Z"), + ]; + let expected = extract("s", "claude_code", &chunks); + let mut accumulator = SessionSignalsAccumulator::new("s", "claude_code"); + for page in chunks.chunks(2) { + for chunk in page { + accumulator.push(chunk); + } + } + assert_eq!(accumulator.finish(), expected); + } + #[test] fn prompt_words_are_counted_from_human_turns() { let chunks = vec![ diff --git a/src-tauri/crates/orgtrack-core/src/profile/store.rs b/src-tauri/crates/orgtrack-core/src/profile/store.rs index 92be9fd2f..465a0666c 100644 --- a/src-tauri/crates/orgtrack-core/src/profile/store.rs +++ b/src-tauri/crates/orgtrack-core/src/profile/store.rs @@ -17,10 +17,14 @@ use std::collections::HashSet; +use core_types::activity::ActivityChunk; use rusqlite::{params, Connection, OptionalExtension}; -use super::signals::{self, SessionSignals, SIGNALS_VERSION}; -use crate::sources::imported_history::load_activity_chunks_for_session; +use super::signals::{SessionSignals, SessionSignalsAccumulator, SIGNALS_VERSION}; +use crate::sources::imported_history::{ + replay::{ReplayCursor, ReplayLimits}, + router::scan_activity_chunks_for_session, +}; /// Table owned by this module. pub const TABLE: &str = "orgtrack_core_session_signals"; @@ -130,18 +134,59 @@ pub fn upsert(conn: &Connection, s: &SessionSignals) -> Result<(), String> { /// Parse a transcript and cache its signals. This is the expensive path; every /// caller should prefer [`load_signals`] and let the backfill do the work. pub fn recompute_session_signals( - conn: &Connection, + conn: &mut Connection, session_id: &str, ) -> Result, String> { - let Some(chunks) = load_activity_chunks_for_session(conn, session_id)? else { - return Ok(None); - }; let source = source_of(conn, session_id).unwrap_or_else(|| "unknown".to_string()); - let s = signals::extract(session_id, &source, &chunks); + let mut accumulator = SessionSignalsAccumulator::new(session_id, &source); + let mut saw_chunk = false; + visit_session_chunks(conn, session_id, |chunk| { + saw_chunk = true; + accumulator.push(chunk); + })?; + if !saw_chunk { + return Ok(None); + } + let s = accumulator.finish(); upsert(conn, &s)?; Ok(Some(s)) } +/// Visit one imported transcript without retaining a session-sized chunk +/// vector. The replay cursor pins one generation/revision for the whole scan; +/// every page remains subject to the normal event and IPC byte limits. +fn visit_session_chunks( + conn: &mut Connection, + session_id: &str, + mut visit: impl FnMut(&ActivityChunk), +) -> Result<(), String> { + let limits = ReplayLimits::default(); + let mut cursor: Option = None; + + loop { + let Some(scan) = + scan_activity_chunks_for_session(conn, session_id, cursor.as_ref(), limits)? + else { + return Ok(()); + }; + for indexed in &scan.chunks { + visit(&indexed.chunk); + } + if !scan.has_more { + return Ok(()); + } + let previous_sequence = cursor + .as_ref() + .map_or(-1, |previous| previous.through_sequence); + if scan.chunks.is_empty() || scan.cursor.through_sequence <= previous_sequence { + return Err(format!( + "Builder Profile replay scan made no progress for {session_id} after sequence {previous_sequence}" + )); + } + cursor = Some(scan.cursor); + } +} + fn source_of(conn: &Connection, session_id: &str) -> Option { for sql in [ "SELECT source FROM orgtrack_core_sessions WHERE session_id = ?1", @@ -365,7 +410,7 @@ fn mark_unreadable(conn: &Connection, session_id: &str) -> Result<(), String> { /// Returns the number of sessions *processed* — extracted or tombstoned — so a /// batch of unreadable transcripts still reports progress and the caller keeps /// draining the backlog behind them. -pub fn backfill_session_signals(conn: &Connection, limit: usize) -> Result { +pub fn backfill_session_signals(conn: &mut Connection, limit: usize) -> Result { if limit == 0 || !table_exists(conn, "imported_history_session_cache")? { return Ok(0); } @@ -583,8 +628,11 @@ mod tests { #[test] fn backfill_is_a_no_op_without_the_imported_cache() { - let conn = db(); - assert_eq!(backfill_session_signals(&conn, 10).expect("backfill"), 0); + let mut conn = db(); + assert_eq!( + backfill_session_signals(&mut conn, 10).expect("backfill"), + 0 + ); } #[test] @@ -604,7 +652,7 @@ mod tests { #[test] fn unreadable_sessions_do_not_stall_the_backfill() { - let conn = db(); + let mut conn = db(); // A minimal imported cache with sessions that have no transcript at // all: every one of them is unreadable by construction. conn.execute_batch( @@ -618,12 +666,12 @@ mod tests { ) .expect("seed imported cache"); - let first = backfill_session_signals(&conn, 10).expect("first pass"); + let first = backfill_session_signals(&mut conn, 10).expect("first pass"); assert_eq!( first, 3, "unreadable sessions must still count as processed, or the drain loop stops with a stranded backlog" ); - let second = backfill_session_signals(&conn, 10).expect("second pass"); + let second = backfill_session_signals(&mut conn, 10).expect("second pass"); assert_eq!( second, 0, "tombstoned sessions must leave the candidate set" @@ -632,15 +680,41 @@ mod tests { } } +#[cfg(test)] +struct RealHistoryCopy { + // Field order matters: close SQLite before TempDir removes the files. + conn: Connection, + _dir: tempfile::TempDir, +} + +#[cfg(test)] +fn open_real_history_copy(path: &str) -> Result { + let dir = tempfile::tempdir().map_err(|err| format!("create history snapshot: {err}"))?; + let copy_path = dir.path().join("sessions.db"); + std::fs::copy(path, ©_path) + .map_err(|err| format!("copy history database into test snapshot: {err}"))?; + for suffix in ["-wal", "-shm"] { + let source = std::path::PathBuf::from(format!("{path}{suffix}")); + if source.exists() { + std::fs::copy(&source, dir.path().join(format!("sessions.db{suffix}"))) + .map_err(|err| format!("copy history database{suffix}: {err}"))?; + } + } + let conn = + Connection::open(©_path).map_err(|err| format!("open history snapshot: {err}"))?; + Ok(RealHistoryCopy { conn, _dir: dir }) +} + /// Read-only smoke test against the real local database. /// /// Ignored by default: it depends on machine-local history, and it must never -/// write to a store the running app owns. Run with +/// write to a store the running app owns. The test copies the database (and +/// any live WAL) into a temporary directory before replay indexing. Run with /// `cargo test -p orgtrack_core real_local_history -- --ignored --nocapture`. #[cfg(test)] mod real_data { use super::*; - use crate::profile; + use crate::profile::{self, signals}; #[test] #[ignore = "requires local session history"] @@ -652,13 +726,10 @@ mod real_data { eprintln!("no local history at {path}; skipping"); return; } - let conn = Connection::open_with_flags( - &path, - rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX, - ) - .expect("open read-only"); + let mut snapshot = open_real_history_copy(&path).expect("open history snapshot"); - let mut statement = conn + let mut statement = snapshot + .conn .prepare( "SELECT session_id, source FROM imported_history_session_cache ORDER BY created_at_ms DESC LIMIT 400", @@ -674,10 +745,16 @@ mod real_data { let mut all = Vec::new(); for (id, source) in &ids { - if let Ok(Some(chunks)) = load_activity_chunks_for_session(&conn, id) { - if chunks.len() >= 3 { - all.push(signals::extract(id, source, &chunks)); - } + let mut accumulator = SessionSignalsAccumulator::new(id, source); + let mut chunk_count = 0usize; + if visit_session_chunks(&mut snapshot.conn, id, |chunk| { + chunk_count += 1; + accumulator.push(chunk); + }) + .is_ok() + && chunk_count >= 3 + { + all.push(accumulator.finish()); } } eprintln!("extracted signals for {} sessions", all.len()); @@ -748,12 +825,9 @@ mod taxonomy { if !std::path::Path::new(&path).exists() { return; } - let conn = Connection::open_with_flags( - &path, - rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX, - ) - .expect("open"); - let mut st = conn + let mut snapshot = open_real_history_copy(&path).expect("open history snapshot"); + let mut st = snapshot + .conn .prepare( "SELECT session_id FROM imported_history_session_cache ORDER BY created_at_ms DESC LIMIT 300", @@ -768,12 +842,12 @@ mod taxonomy { let mut counts: HashMap<(String, String), i64> = HashMap::new(); for id in &ids { - if let Ok(Some(chunks)) = load_activity_chunks_for_session(&conn, id) { - for c in chunks.iter().filter(|c| c.action_type == "tool_call") { + let _ = visit_session_chunks(&mut snapshot.conn, id, |c| { + if c.action_type == "tool_call" { let kind = format!("{:?}", classify_tool(&c.function)); *counts.entry((kind, c.function.clone())).or_default() += 1; } - } + }); } let mut rows: Vec<_> = counts.into_iter().collect(); rows.sort_by_key(|(_, n)| -*n); @@ -804,7 +878,7 @@ mod taxonomy { #[cfg(test)] mod cost { use super::*; - use crate::profile; + use crate::profile::{self, signals}; use std::time::Instant; #[test] diff --git a/src-tauri/crates/orgtrack-core/src/projectors/turn_metadata.rs b/src-tauri/crates/orgtrack-core/src/projectors/turn_metadata.rs index 5bde4fde9..b8d39635d 100644 --- a/src-tauri/crates/orgtrack-core/src/projectors/turn_metadata.rs +++ b/src-tauri/crates/orgtrack-core/src/projectors/turn_metadata.rs @@ -18,7 +18,7 @@ use serde::{Deserialize, Serialize}; use serde_json::Value; use crate::canonical::{ResourceAction, ResourceInteractionOutcome}; -use crate::development_artifact::parse_git_artifacts_from_tool_payload; +use crate::development_artifact::{parse_git_artifacts_from_tool_payload, replay_git_artifacts}; use crate::resource_interaction::{ action_for_tool_name, file_interactions_from_tool, interaction_outcome_from_tool_result, }; @@ -27,6 +27,153 @@ const STATUS_CREATED: &str = "created"; const STATUS_DELETED: &str = "deleted"; const STATUS_MODIFIED: &str = "modified"; +/// The payload and projection work required to derive compact turn metadata +/// from one normalized event. +/// +/// This type deliberately describes projection capabilities rather than a +/// denylist of event names. Callers such as the SQLite turn-index rebuild use +/// it before calling `Row::get` so large JSON columns that cannot contribute +/// metadata never become Rust `String`s in the first place. Unknown tools use +/// the conservative all-enabled shape so adding a provider tool cannot +/// silently drop metadata. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MetadataProjectionRequirements { + load_args_json: bool, + load_result_json: bool, + project_resource_interactions: bool, + project_modified_files: bool, + project_git_artifacts: bool, +} + +impl MetadataProjectionRequirements { + const NONE: Self = Self { + load_args_json: false, + load_result_json: false, + project_resource_interactions: false, + project_modified_files: false, + project_git_artifacts: false, + }; + + const CONSERVATIVE: Self = Self { + load_args_json: true, + load_result_json: true, + project_resource_interactions: true, + project_modified_files: true, + project_git_artifacts: true, + }; + + pub const fn needs_args_json(self) -> bool { + self.load_args_json + } + + pub const fn needs_result_json(self) -> bool { + self.load_result_json + } + + pub const fn projects_resource_interactions(self) -> bool { + self.project_resource_interactions + } + + pub const fn projects_modified_files(self) -> bool { + self.project_modified_files + } + + pub const fn projects_git_artifacts(self) -> bool { + self.project_git_artifacts + } + + pub const fn is_empty(self) -> bool { + !self.load_args_json + && !self.load_result_json + && !self.project_resource_interactions + && !self.project_modified_files + && !self.project_git_artifacts + } +} + +/// Classify the minimum work needed to project one normalized event. +/// +/// Exact known-no-metadata names are intentionally kept narrow. Broad +/// substring matching here would make a future provider tool disappear from +/// historical metadata merely because its display name happened to contain +/// words such as `message` or `node`. +pub fn metadata_projection_requirements( + function_name: Option<&str>, +) -> MetadataProjectionRequirements { + let Some(function_name) = function_name else { + return MetadataProjectionRequirements::NONE; + }; + let normalized = function_name.trim().to_ascii_lowercase(); + + if matches!( + normalized.as_str(), + "user" + | "user_message" + | "assistant" + | "assistant_message" + | "agent_message" + | "thinking" + | "think" + | "llm_thinking" + | "llm_thinking_delta" + | "thinking_delta" + | "reasoning" + | "internal_monologue" + | "reflection" + | "node_repl" + | "node-repl" + | "node repl" + ) { + return MetadataProjectionRequirements::NONE; + } + + // Search result bodies can be enormous. The turn projection's existing + // search policy only needs the compact input side (for example `path`) + // and deliberately does not harvest paths from result matches. + if matches!(normalized.as_str(), "grep" | "ripgrep" | "code_search") { + return MetadataProjectionRequirements { + load_args_json: true, + load_result_json: false, + project_resource_interactions: true, + project_modified_files: false, + project_git_artifacts: false, + }; + } + + if let Some(action) = action_for_tool_name(function_name) { + return MetadataProjectionRequirements { + load_args_json: true, + // Results carry failure status, provider-returned paths, patch + // segments, and exact line statistics. + load_result_json: true, + project_resource_interactions: true, + project_modified_files: is_modifying_action(action), + project_git_artifacts: false, + }; + } + + if matches!( + normalized.as_str(), + "bash" + | "shell" + | "run_command" + | "run_command_line" + | "terminal" + | "terminal_command" + | "execute_command" + ) { + return MetadataProjectionRequirements { + load_args_json: true, + load_result_json: true, + project_resource_interactions: false, + project_modified_files: false, + project_git_artifacts: true, + }; + } + + MetadataProjectionRequirements::CONSERVATIVE +} + /// One file the round wrote to, with summed line stats. Serialized as the /// camelCase shape the frontend `FileChangeInfo` expects. #[derive(Debug, Clone, Serialize, Deserialize, Default, PartialEq, Eq)] @@ -106,27 +253,112 @@ impl TurnMetadataAccumulator { let Some(function_name) = function_name else { return; }; - let args = serde_json::from_str::(args_json).unwrap_or(Value::Null); - let result = serde_json::from_str::(result_json).unwrap_or(Value::Null); - let outcome = interaction_outcome_from_tool_result(&result); - - for interaction in file_interactions_from_tool(function_name, &args, Some(&result)) { - self.merge_resource_interaction( - interaction.file_path, - interaction.action, - outcome, - occurred_at, - ); + let requirements = metadata_projection_requirements(Some(function_name)); + if requirements.is_empty() { + return; } - if outcome != ResourceInteractionOutcome::Failed { - for change in extract_event_files(function_name, &args, &result) { - self.merge_modified_file(change); + let args = requirements + .needs_args_json() + .then(|| serde_json::from_str::(args_json).unwrap_or(Value::Null)); + let result = requirements + .needs_result_json() + .then(|| serde_json::from_str::(result_json).unwrap_or(Value::Null)); + self.add_parsed_event_at( + function_name, + args.as_ref().unwrap_or(&Value::Null), + result.as_ref().unwrap_or(&Value::Null), + occurred_at, + requirements, + ); + + if requirements.projects_git_artifacts() { + for artifact in parse_git_artifacts_from_tool_payload(args_json, result_json) { + self.merge_artifact(artifact); + } + if let Some(result) = result.as_ref() { + for artifact in replay_git_artifacts(result) { + self.merge_artifact(artifact); + } + } + } + } + + /// Fold an event whose normalized payloads are already parsed. + /// + /// Imported-history adapters use this path to avoid serializing a + /// `serde_json::Value` to text only for the projector to parse it again. + pub fn add_event_values_at( + &mut self, + function_name: Option<&str>, + args: &Value, + result: &Value, + occurred_at: &str, + ) { + let Some(function_name) = function_name else { + return; + }; + let requirements = metadata_projection_requirements(Some(function_name)); + if requirements.is_empty() { + return; + } + + self.add_parsed_event_at( + function_name, + if requirements.needs_args_json() { + args + } else { + &Value::Null + }, + if requirements.needs_result_json() { + result + } else { + &Value::Null + }, + occurred_at, + requirements, + ); + + if requirements.projects_git_artifacts() { + // Git artifact parsing intentionally keeps one canonical parser + // for persisted JSON and imported Values. Serialization happens + // only for tool classes that can actually produce Git metadata. + let args_json = serde_json::to_string(args).unwrap_or_else(|_| "null".to_string()); + let result_json = serde_json::to_string(result).unwrap_or_else(|_| "null".to_string()); + for artifact in parse_git_artifacts_from_tool_payload(&args_json, &result_json) { + self.merge_artifact(artifact); + } + for artifact in replay_git_artifacts(result) { + self.merge_artifact(artifact); } } + } - for artifact in parse_git_artifacts_from_tool_payload(args_json, result_json) { - self.merge_artifact(artifact); + fn add_parsed_event_at( + &mut self, + function_name: &str, + args: &Value, + result: &Value, + occurred_at: &str, + requirements: MetadataProjectionRequirements, + ) { + let outcome = interaction_outcome_from_tool_result(result); + + if requirements.projects_resource_interactions() { + for interaction in file_interactions_from_tool(function_name, args, Some(result)) { + self.merge_resource_interaction( + interaction.file_path, + interaction.action, + outcome, + occurred_at, + ); + } + } + + if requirements.projects_modified_files() && outcome != ResourceInteractionOutcome::Failed { + for change in extract_event_files(function_name, args, result) { + self.merge_modified_file(change); + } } } @@ -321,13 +553,10 @@ pub fn project_activity_chunks(chunks: &[ActivityChunk]) -> Vec String { .into_iter() .find_map(|field| chunk.args.get(field).and_then(Value::as_str)) .or_else(|| chunk.args.as_str()) + // Imported providers normalize user bubbles through + // `user_message_chunk`, whose canonical text lives in + // `result.message.content`. Keep metadata previews aligned with the + // rendered replay instead of silently returning an empty string. + .or_else(|| { + chunk + .result + .pointer("/message/content") + .and_then(Value::as_str) + }) + .or_else(|| chunk.result.get("content").and_then(Value::as_str)) .unwrap_or_default() .to_string() } @@ -637,291 +877,4 @@ fn merge_missing_artifact_fields( } #[cfg(test)] -mod tests { - use super::*; - - #[test] - fn ignores_read_only_and_unknown_tools() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event(Some("read_file"), r#"{"file_path":"a.rs"}"#, "{}"); - acc.add_event(None, "{}", "{}"); - assert!(acc.files().is_empty()); - } - - #[test] - fn edit_file_extracts_path_and_line_stats() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("edit_file"), - r#"{"file_path":"src/foo.rs"}"#, - r#"{"success":{"linesAdded":3,"linesRemoved":1}}"#, - ); - let files = acc.files(); - assert_eq!(files.len(), 1); - assert_eq!(files[0].path, "src/foo.rs"); - assert_eq!(files[0].file_name, "foo.rs"); - assert_eq!(files[0].status, "modified"); - assert_eq!(files[0].additions, 3); - assert_eq!(files[0].deletions, 1); - } - - #[test] - fn create_and_delete_status_mapping() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event(Some("create_file"), r#"{"file_path":"new.ts"}"#, "{}"); - acc.add_event(Some("delete_file"), r#"{"file_path":"old.ts"}"#, "{}"); - let files = acc.files(); - assert_eq!(files[0].status, "created"); - assert_eq!(files[1].status, "deleted"); - } - - #[test] - fn file_name_supports_provider_paths_from_both_platforms() { - assert_eq!(file_name_for("src/lib.rs"), "lib.rs"); - assert_eq!(file_name_for(r"C:\repo\src\lib.rs"), "lib.rs"); - } - - #[test] - fn create_file_falls_back_to_content_line_count() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("create_file"), - r#"{"file_path":"note.md","content":"one\ntwo\nthree"}"#, - "{}", - ); - let files = acc.files(); - assert_eq!(files.len(), 1); - assert_eq!(files[0].status, "created"); - assert_eq!(files[0].additions, 3); - assert_eq!(files[0].deletions, 0); - } - - #[test] - fn duplicate_path_merges_and_sums() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("edit_file"), - r#"{"file_path":"a.rs"}"#, - r#"{"linesAdded":2,"linesRemoved":0}"#, - ); - acc.add_event( - Some("edit_file"), - r#"{"file_path":"a.rs"}"#, - r#"{"linesAdded":5,"linesRemoved":3}"#, - ); - let files = acc.files(); - assert_eq!(files.len(), 1); - assert_eq!(files[0].additions, 7); - assert_eq!(files[0].deletions, 3); - } - - #[test] - fn error_result_is_skipped() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("edit_file"), - r#"{"file_path":"a.rs"}"#, - r#"{"content":"Error: permission denied"}"#, - ); - assert!(acc.files().is_empty()); - } - - #[test] - fn apply_patch_uses_segments() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("apply_patch"), - r#"{"patch_text":"*** Update File: a.rs\n"}"#, - r#"{"segments":[ - {"filePath":"a.rs","linesAdded":4,"linesRemoved":1}, - {"filePath":"b.rs","isDeleted":true} - ]}"#, - ); - let files = acc.files(); - assert_eq!(files.len(), 2); - assert_eq!(files[0].path, "a.rs"); - assert_eq!(files[0].additions, 4); - assert_eq!(files[1].path, "b.rs"); - assert_eq!(files[1].status, "deleted"); - } - - #[test] - fn apply_patch_falls_back_to_patch_text_with_line_stats() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("apply_patch"), - r#"{"patch_text":"*** Add File: x.rs\n+one\n+two\n*** Update File: y.rs\n-old\n+new\n context\n"}"#, - "{}", - ); - let files = acc.files(); - let paths: Vec<&str> = files.iter().map(|f| f.path.as_str()).collect(); - assert_eq!(paths, vec!["x.rs", "y.rs"]); - assert_eq!(files[0].additions, 2); - assert_eq!(files[0].deletions, 0); - assert_eq!(files[1].additions, 1); - assert_eq!(files[1].deletions, 1); - } - - #[test] - fn apply_patch_prefers_patch_text_stats_over_file_paths() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("apply_patch"), - r#"{"patch_text":"*** Update File: a.rs\n-old\n+new\n+extra\n"}"#, - r#"{"filePaths":["a.rs"]}"#, - ); - let files = acc.files(); - assert_eq!(files.len(), 1); - assert_eq!(files[0].path, "a.rs"); - assert_eq!(files[0].additions, 2); - assert_eq!(files[0].deletions, 1); - } - - #[test] - fn normalized_codex_edit_file_keeps_apply_patch_line_stats() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("edit_file_by_replace"), - r#"{"action":"apply_patch","patch_text":"*** Update File: src/app.ts\n-old\n+new\n+extra\n"}"#, - r#"{"filePaths":["src/app.ts"]}"#, - ); - - let files = acc.files(); - assert_eq!(files.len(), 1); - assert_eq!(files[0].path, "src/app.ts"); - assert_eq!(files[0].additions, 2); - assert_eq!(files[0].deletions, 1); - } - - #[test] - fn malformed_json_is_tolerated() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event(Some("edit_file"), "{not json", "{also not json"); - assert!(acc.files().is_empty()); - } - - #[test] - fn folds_read_metadata_and_drops_searches_across_provider_tool_names() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event_at( - Some("Read"), - r#"{"file_path":"src/lib.rs"}"#, - "{}", - "2026-07-15T00:00:01Z", - ); - acc.add_event_at( - Some("Grep"), - r#"{"path":"src"}"#, - r#"{"matches":[{"file":"src/lib.rs"},{"path":"src/main.rs"}]}"#, - "2026-07-15T00:00:02Z", - ); - - // search-rows: only the read survives — the Grep contributes neither its - // queried path nor the paths named in its matches. - let interactions = acc.resource_interactions(); - assert_eq!(interactions.len(), 1); - assert!(interactions.iter().any(|item| { - item.path == "src/lib.rs" - && item.action == ResourceAction::Read - && item.outcome == ResourceInteractionOutcome::Succeeded - })); - assert!(!interactions - .iter() - .any(|item| item.action == ResourceAction::Search)); - } - - #[test] - fn records_failed_observation_but_does_not_claim_a_modification() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event_at( - Some("replace"), - r#"{"file_path":"src/lib.rs","new_string":"replacement"}"#, - r#"{"content":"Error: permission denied"}"#, - "2026-07-15T00:00:03Z", - ); - - assert!(acc.modified_files().is_empty()); - assert_eq!(acc.resource_interactions().len(), 1); - assert_eq!( - acc.resource_interactions()[0].outcome, - ResourceInteractionOutcome::Failed - ); - } - - #[test] - fn shell_artifacts_are_projected_without_host_tool_constants() { - let mut acc = TurnMetadataAccumulator::new(); - acc.add_event( - Some("Bash"), - r#"{"command":"git commit -m metadata"}"#, - r#"{"success":{"command":"git commit -m metadata","stdout":"[feature abc1234] metadata","exitCode":0}}"#, - ); - - assert_eq!(acc.git_artifacts().len(), 1); - assert_eq!(acc.git_artifacts()[0].sha.as_deref(), Some("abc1234")); - } - - #[test] - fn imported_activity_projection_uses_user_messages_not_execution_threads() { - let mut first_user = ActivityChunk::new("session-1", "raw", "user_message"); - first_user.chunk_id = "user-1".to_string(); - first_user.created_at = "2026-07-15T00:00:00Z".to_string(); - first_user.args = serde_json::json!({"content": "inspect the code"}); - let mut read = ActivityChunk::new("session-1", "tool_call", "Read"); - read.chunk_id = "read-1".to_string(); - read.thread_id = Some("subagent-9".to_string()); - read.created_at = "2026-07-15T00:00:01Z".to_string(); - read.args = serde_json::json!({"file_path": "src/lib.rs"}); - let mut second_user = ActivityChunk::new("session-1", "raw", "user_message"); - second_user.chunk_id = "user-2".to_string(); - second_user.created_at = "2026-07-15T00:01:00Z".to_string(); - second_user.args = serde_json::json!({"content": "now edit it"}); - let mut edit = ActivityChunk::new("session-1", "tool_call", "replace"); - edit.chunk_id = "edit-1".to_string(); - edit.thread_id = Some("subagent-10".to_string()); - edit.created_at = "2026-07-15T00:01:01Z".to_string(); - edit.args = serde_json::json!({ - "file_path": "src/lib.rs", - "old_string": "old", - "new_string": "new" - }); - - let rounds = project_activity_chunks(&[first_user, read, second_user, edit]); - - assert_eq!(rounds.len(), 2); - assert_eq!(rounds[0].turn_id, "user-1"); - assert_eq!(rounds[1].turn_id, "user-2"); - assert_eq!( - rounds[0].resource_interactions[0].action, - ResourceAction::Read - ); - assert_eq!(rounds[1].modified_files[0].path, "src/lib.rs"); - } - - #[test] - fn lifecycle_markers_keep_active_tail_pending_until_completion() { - let mut user = ActivityChunk::new("session-1", "raw", "user_message"); - user.chunk_id = "user-1".to_string(); - user.created_at = "2026-07-15T00:00:00Z".to_string(); - user.args = serde_json::json!({"content": "edit it"}); - let mut start = ActivityChunk::new("session-1", "task_start", "task_start"); - start.created_at = "2026-07-15T00:00:00Z".to_string(); - let mut edit = ActivityChunk::new("session-1", "tool_call", "edit_file"); - edit.created_at = "2026-07-15T00:00:01Z".to_string(); - edit.args = serde_json::json!({"file_path": "src/lib.rs", "content": "new"}); - - let active = project_activity_chunks(&[user.clone(), start.clone(), edit.clone()]); - assert_eq!(active[0].status, "pending"); - assert_eq!(active[0].ended_at, None); - - let mut complete = ActivityChunk::new("session-1", "task_completed", "task_completed"); - complete.created_at = "2026-07-15T00:00:02Z".to_string(); - let completed = project_activity_chunks(&[user, start, edit, complete]); - assert_eq!(completed[0].status, "completed"); - assert_eq!( - completed[0].ended_at.as_deref(), - Some("2026-07-15T00:00:02Z") - ); - assert_eq!(completed[0].event_count, 2); - } -} +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/projectors/turn_metadata/tests.rs b/src-tauri/crates/orgtrack-core/src/projectors/turn_metadata/tests.rs new file mode 100644 index 000000000..b557a0695 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/projectors/turn_metadata/tests.rs @@ -0,0 +1,418 @@ +use super::*; + +fn add_event_with_legacy_projection( + acc: &mut TurnMetadataAccumulator, + function_name: Option<&str>, + args_json: &str, + result_json: &str, + occurred_at: &str, +) { + let Some(function_name) = function_name else { + return; + }; + let args = serde_json::from_str::(args_json).unwrap_or(Value::Null); + let result = serde_json::from_str::(result_json).unwrap_or(Value::Null); + let outcome = interaction_outcome_from_tool_result(&result); + + for interaction in file_interactions_from_tool(function_name, &args, Some(&result)) { + acc.merge_resource_interaction( + interaction.file_path, + interaction.action, + outcome, + occurred_at, + ); + } + if outcome != ResourceInteractionOutcome::Failed { + for change in extract_event_files(function_name, &args, &result) { + acc.merge_modified_file(change); + } + } + for artifact in parse_git_artifacts_from_tool_payload(args_json, result_json) { + acc.merge_artifact(artifact); + } +} + +#[test] +fn projection_requirements_short_circuit_only_known_safe_tool_classes() { + for name in [ + "user_message", + "assistant", + "assistant_message", + "thinking", + "reasoning", + "node_repl", + ] { + assert!( + metadata_projection_requirements(Some(name)).is_empty(), + "{name} should not load metadata payloads" + ); + } + + let grep = metadata_projection_requirements(Some("Grep")); + assert!(grep.needs_args_json()); + assert!(!grep.needs_result_json()); + assert!(grep.projects_resource_interactions()); + assert!(!grep.projects_modified_files()); + assert!(!grep.projects_git_artifacts()); + + let read = metadata_projection_requirements(Some("Read")); + assert!(read.needs_args_json()); + assert!(read.needs_result_json()); + assert!(read.projects_resource_interactions()); + assert!(!read.projects_modified_files()); + assert!(!read.projects_git_artifacts()); + + let edit = metadata_projection_requirements(Some("edit_file")); + assert!(edit.projects_resource_interactions()); + assert!(edit.projects_modified_files()); + assert!(!edit.projects_git_artifacts()); + + let bash = metadata_projection_requirements(Some("Bash")); + assert!(!bash.projects_resource_interactions()); + assert!(!bash.projects_modified_files()); + assert!(bash.projects_git_artifacts()); + + let future_tool = metadata_projection_requirements(Some("future_provider_tool")); + assert!(future_tool.needs_args_json()); + assert!(future_tool.needs_result_json()); + assert!(future_tool.projects_resource_interactions()); + assert!(future_tool.projects_modified_files()); + assert!(future_tool.projects_git_artifacts()); +} + +#[test] +fn typed_projection_matches_legacy_metadata_for_representative_events() { + let events = [ + ( + "Read", + r#"{"file_path":"src/lib.rs"}"#, + "{}", + "2026-07-15T00:00:01Z", + ), + ( + "edit_file", + r#"{"file_path":"src/lib.rs","new_string":"one\ntwo"}"#, + r#"{"success":{"linesAdded":2,"linesRemoved":1}}"#, + "2026-07-15T00:00:02Z", + ), + ( + "Grep", + r#"{"path":"src"}"#, + r#"{"matches":[{"file":"src/main.rs"}]}"#, + "2026-07-15T00:00:03Z", + ), + ( + "Bash", + r#"{"command":"git commit -m metadata"}"#, + r#"{"success":{"command":"git commit -m metadata","stdout":"[feature abc1234] metadata","exitCode":0}}"#, + "2026-07-15T00:00:04Z", + ), + ( + "future_provider_tool", + r#"{"command":"gh pr create"}"#, + r#"{"output":"https://github.com/acme/repo/pull/42"}"#, + "2026-07-15T00:00:05Z", + ), + ]; + let mut legacy = TurnMetadataAccumulator::new(); + let mut typed = TurnMetadataAccumulator::new(); + for (name, args, result, occurred_at) in events { + add_event_with_legacy_projection(&mut legacy, Some(name), args, result, occurred_at); + typed.add_event_at(Some(name), args, result, occurred_at); + } + + assert_eq!(typed.modified_files(), legacy.modified_files()); + assert_eq!( + typed.resource_interactions(), + legacy.resource_interactions() + ); + assert_eq!( + serde_json::to_value(typed.git_artifacts()).unwrap(), + serde_json::to_value(legacy.git_artifacts()).unwrap() + ); +} + +#[test] +fn ignores_read_only_and_unknown_tools() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event(Some("read_file"), r#"{"file_path":"a.rs"}"#, "{}"); + acc.add_event(None, "{}", "{}"); + assert!(acc.files().is_empty()); +} + +#[test] +fn edit_file_extracts_path_and_line_stats() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("edit_file"), + r#"{"file_path":"src/foo.rs"}"#, + r#"{"success":{"linesAdded":3,"linesRemoved":1}}"#, + ); + let files = acc.files(); + assert_eq!(files.len(), 1); + assert_eq!(files[0].path, "src/foo.rs"); + assert_eq!(files[0].file_name, "foo.rs"); + assert_eq!(files[0].status, "modified"); + assert_eq!(files[0].additions, 3); + assert_eq!(files[0].deletions, 1); +} + +#[test] +fn create_and_delete_status_mapping() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event(Some("create_file"), r#"{"file_path":"new.ts"}"#, "{}"); + acc.add_event(Some("delete_file"), r#"{"file_path":"old.ts"}"#, "{}"); + let files = acc.files(); + assert_eq!(files[0].status, "created"); + assert_eq!(files[1].status, "deleted"); +} + +#[test] +fn file_name_supports_provider_paths_from_both_platforms() { + assert_eq!(file_name_for("src/lib.rs"), "lib.rs"); + assert_eq!(file_name_for(r"C:\repo\src\lib.rs"), "lib.rs"); +} + +#[test] +fn create_file_falls_back_to_content_line_count() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("create_file"), + r#"{"file_path":"note.md","content":"one\ntwo\nthree"}"#, + "{}", + ); + let files = acc.files(); + assert_eq!(files.len(), 1); + assert_eq!(files[0].status, "created"); + assert_eq!(files[0].additions, 3); + assert_eq!(files[0].deletions, 0); +} + +#[test] +fn duplicate_path_merges_and_sums() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("edit_file"), + r#"{"file_path":"a.rs"}"#, + r#"{"linesAdded":2,"linesRemoved":0}"#, + ); + acc.add_event( + Some("edit_file"), + r#"{"file_path":"a.rs"}"#, + r#"{"linesAdded":5,"linesRemoved":3}"#, + ); + let files = acc.files(); + assert_eq!(files.len(), 1); + assert_eq!(files[0].additions, 7); + assert_eq!(files[0].deletions, 3); +} + +#[test] +fn error_result_is_skipped() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("edit_file"), + r#"{"file_path":"a.rs"}"#, + r#"{"content":"Error: permission denied"}"#, + ); + assert!(acc.files().is_empty()); +} + +#[test] +fn apply_patch_uses_segments() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("apply_patch"), + r#"{"patch_text":"*** Update File: a.rs\n"}"#, + r#"{"segments":[ + {"filePath":"a.rs","linesAdded":4,"linesRemoved":1}, + {"filePath":"b.rs","isDeleted":true} + ]}"#, + ); + let files = acc.files(); + assert_eq!(files.len(), 2); + assert_eq!(files[0].path, "a.rs"); + assert_eq!(files[0].additions, 4); + assert_eq!(files[1].path, "b.rs"); + assert_eq!(files[1].status, "deleted"); +} + +#[test] +fn apply_patch_falls_back_to_patch_text_with_line_stats() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("apply_patch"), + r#"{"patch_text":"*** Add File: x.rs\n+one\n+two\n*** Update File: y.rs\n-old\n+new\n context\n"}"#, + "{}", + ); + let files = acc.files(); + let paths: Vec<&str> = files.iter().map(|f| f.path.as_str()).collect(); + assert_eq!(paths, vec!["x.rs", "y.rs"]); + assert_eq!(files[0].additions, 2); + assert_eq!(files[0].deletions, 0); + assert_eq!(files[1].additions, 1); + assert_eq!(files[1].deletions, 1); +} + +#[test] +fn apply_patch_prefers_patch_text_stats_over_file_paths() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("apply_patch"), + r#"{"patch_text":"*** Update File: a.rs\n-old\n+new\n+extra\n"}"#, + r#"{"filePaths":["a.rs"]}"#, + ); + let files = acc.files(); + assert_eq!(files.len(), 1); + assert_eq!(files[0].path, "a.rs"); + assert_eq!(files[0].additions, 2); + assert_eq!(files[0].deletions, 1); +} + +#[test] +fn normalized_codex_edit_file_keeps_apply_patch_line_stats() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("edit_file_by_replace"), + r#"{"action":"apply_patch","patch_text":"*** Update File: src/app.ts\n-old\n+new\n+extra\n"}"#, + r#"{"filePaths":["src/app.ts"]}"#, + ); + + let files = acc.files(); + assert_eq!(files.len(), 1); + assert_eq!(files[0].path, "src/app.ts"); + assert_eq!(files[0].additions, 2); + assert_eq!(files[0].deletions, 1); +} + +#[test] +fn malformed_json_is_tolerated() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event(Some("edit_file"), "{not json", "{also not json"); + assert!(acc.files().is_empty()); +} + +#[test] +fn folds_read_metadata_and_drops_searches_across_provider_tool_names() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event_at( + Some("Read"), + r#"{"file_path":"src/lib.rs"}"#, + "{}", + "2026-07-15T00:00:01Z", + ); + acc.add_event_at( + Some("Grep"), + r#"{"path":"src"}"#, + r#"{"matches":[{"file":"src/lib.rs"},{"path":"src/main.rs"}]}"#, + "2026-07-15T00:00:02Z", + ); + + // search-rows: only the read survives — the Grep contributes neither its + // queried path nor the paths named in its matches. + let interactions = acc.resource_interactions(); + assert_eq!(interactions.len(), 1); + assert!(interactions.iter().any(|item| { + item.path == "src/lib.rs" + && item.action == ResourceAction::Read + && item.outcome == ResourceInteractionOutcome::Succeeded + })); + assert!(!interactions + .iter() + .any(|item| item.action == ResourceAction::Search)); +} + +#[test] +fn records_failed_observation_but_does_not_claim_a_modification() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event_at( + Some("replace"), + r#"{"file_path":"src/lib.rs","new_string":"replacement"}"#, + r#"{"content":"Error: permission denied"}"#, + "2026-07-15T00:00:03Z", + ); + + assert!(acc.modified_files().is_empty()); + assert_eq!(acc.resource_interactions().len(), 1); + assert_eq!( + acc.resource_interactions()[0].outcome, + ResourceInteractionOutcome::Failed + ); +} + +#[test] +fn shell_artifacts_are_projected_without_host_tool_constants() { + let mut acc = TurnMetadataAccumulator::new(); + acc.add_event( + Some("Bash"), + r#"{"command":"git commit -m metadata"}"#, + r#"{"success":{"command":"git commit -m metadata","stdout":"[feature abc1234] metadata","exitCode":0}}"#, + ); + + assert_eq!(acc.git_artifacts().len(), 1); + assert_eq!(acc.git_artifacts()[0].sha.as_deref(), Some("abc1234")); +} + +#[test] +fn imported_activity_projection_uses_user_messages_not_execution_threads() { + let mut first_user = ActivityChunk::new("session-1", "raw", "user_message"); + first_user.chunk_id = "user-1".to_string(); + first_user.created_at = "2026-07-15T00:00:00Z".to_string(); + first_user.args = serde_json::json!({"content": "inspect the code"}); + let mut read = ActivityChunk::new("session-1", "tool_call", "Read"); + read.chunk_id = "read-1".to_string(); + read.thread_id = Some("subagent-9".to_string()); + read.created_at = "2026-07-15T00:00:01Z".to_string(); + read.args = serde_json::json!({"file_path": "src/lib.rs"}); + let mut second_user = ActivityChunk::new("session-1", "raw", "user_message"); + second_user.chunk_id = "user-2".to_string(); + second_user.created_at = "2026-07-15T00:01:00Z".to_string(); + second_user.args = serde_json::json!({"content": "now edit it"}); + let mut edit = ActivityChunk::new("session-1", "tool_call", "replace"); + edit.chunk_id = "edit-1".to_string(); + edit.thread_id = Some("subagent-10".to_string()); + edit.created_at = "2026-07-15T00:01:01Z".to_string(); + edit.args = serde_json::json!({ + "file_path": "src/lib.rs", + "old_string": "old", + "new_string": "new" + }); + + let rounds = project_activity_chunks(&[first_user, read, second_user, edit]); + + assert_eq!(rounds.len(), 2); + assert_eq!(rounds[0].turn_id, "user-1"); + assert_eq!(rounds[1].turn_id, "user-2"); + assert_eq!( + rounds[0].resource_interactions[0].action, + ResourceAction::Read + ); + assert_eq!(rounds[1].modified_files[0].path, "src/lib.rs"); +} + +#[test] +fn lifecycle_markers_keep_active_tail_pending_until_completion() { + let mut user = ActivityChunk::new("session-1", "raw", "user_message"); + user.chunk_id = "user-1".to_string(); + user.created_at = "2026-07-15T00:00:00Z".to_string(); + user.args = serde_json::json!({"content": "edit it"}); + let mut start = ActivityChunk::new("session-1", "task_start", "task_start"); + start.created_at = "2026-07-15T00:00:00Z".to_string(); + let mut edit = ActivityChunk::new("session-1", "tool_call", "edit_file"); + edit.created_at = "2026-07-15T00:00:01Z".to_string(); + edit.args = serde_json::json!({"file_path": "src/lib.rs", "content": "new"}); + + let active = project_activity_chunks(&[user.clone(), start.clone(), edit.clone()]); + assert_eq!(active[0].status, "pending"); + assert_eq!(active[0].ended_at, None); + + let mut complete = ActivityChunk::new("session-1", "task_completed", "task_completed"); + complete.created_at = "2026-07-15T00:00:02Z".to_string(); + let completed = project_activity_chunks(&[user, start, edit, complete]); + assert_eq!(completed[0].status, "completed"); + assert_eq!( + completed[0].ended_at.as_deref(), + Some("2026-07-15T00:00:02Z") + ); + assert_eq!(completed[0].event_count, 2); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/anthropic_jsonl.rs b/src-tauri/crates/orgtrack-core/src/sources/anthropic_jsonl.rs index 54f3040da..a73cd93d6 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/anthropic_jsonl.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/anthropic_jsonl.rs @@ -6,7 +6,7 @@ use std::collections::{HashMap, HashSet}; use std::fs; -use std::io::{BufRead, BufReader}; +use std::io::{BufRead, BufReader, Read}; use std::path::{Path, PathBuf}; use core_types::activity::ActivityChunk; @@ -23,6 +23,8 @@ use crate::sources::imported_history::{ ImportedToolCall, }; +const CATALOG_PREFIX_BYTES: u64 = 1024 * 1024; + const MAX_TOOL_OUTPUT_CHARS: usize = 50_000; /// Config for the generic Anthropic/Claude-style JSONL transcript reader. Any @@ -155,6 +157,47 @@ fn sync_cache(config: &AnthropicJsonlSource, conn: &mut Connection) -> Result<() ) } +pub(crate) fn refresh_catalog( + config: &AnthropicJsonlSource, + conn: &mut Connection, +) -> Result<(), String> { + let discovered = discover_records(config)?; + let signatures = discovered + .iter() + .map(ImportedHistoryDiscoveredRecord::signature) + .collect::>(); + let changed = imported_cache::changed_records_from_conn( + conn, + config.source, + &discovered, + ImportedHistoryDiscoveredRecord::signature, + )?; + let mut inputs = Vec::new(); + for record in changed { + if imported_cache::advance_cached_catalog_record_from_conn( + conn, + config.source, + record, + None, + )? { + continue; + } + // Cold discovery streams one JSONL record at a time. Subsequent + // appends only advance the persisted catalog signature above; body + // deltas belong to the replay adapter. + inputs.push(meta_to_cache_input( + config, + parse_session_meta(config, record)?, + )); + } + imported_cache::sync_source_cache_from_conn( + conn, + config.source, + imported_cache::live_ids_from_signatures(&signatures), + inputs, + ) +} + fn discover_records( config: &AnthropicJsonlSource, ) -> Result, String> { @@ -229,7 +272,6 @@ fn parse_session_meta( config: &AnthropicJsonlSource, record: &ImportedHistoryDiscoveredRecord, ) -> Result { - let turns = read_turns(config, &record.source_path)?; let mut created_at_ms = 0; let mut updated_at_ms = 0; let mut repo_path = None; @@ -240,6 +282,9 @@ fn parse_session_meta( let mut cache_read_tokens = 0; let mut cache_write_tokens = 0; let mut first_user_text = None; + let mut impact = ImportedHistoryImpactStats::default(); + let mut touched_files = std::collections::BTreeSet::new(); + let session_id = format!("{}{}", config.session_prefix, record.source_session_id); let file = fs::File::open(&record.source_path).map_err(|err| { format!( @@ -248,7 +293,7 @@ fn parse_session_meta( record.source_path.display() ) })?; - for line in BufReader::new(file).lines() { + for line in BufReader::new(file.take(CATALOG_PREFIX_BYTES)).lines() { let Ok(line) = line else { continue }; let Ok(parsed) = serde_json::from_str::(line.trim()) else { continue; @@ -269,6 +314,14 @@ fn parse_session_meta( model = Some(parsed.model_id.trim().to_string()); } if let Some(message) = parsed.message { + collect_catalog_edit_impact( + config, + &session_id, + &message, + &normalized_timestamp(&parsed.timestamp), + &mut impact, + &mut touched_files, + ); if model.is_none() && !message.model.trim().is_empty() { model = Some(message.model.trim().to_string()); } @@ -285,9 +338,8 @@ fn parse_session_meta( } let fallback_ms = record.source_mtime_ms / 1_000_000; - let session_id = format!("{}{}", config.session_prefix, record.source_session_id); - let impact = - imported_history::impact_from_edit_chunks(&messages_to_chunks(config, &session_id, &turns)); + impact.touched_files = touched_files.into_iter().collect(); + impact.files_changed = impact.touched_files.len() as i64; Ok(SessionMeta { source_session_id: record.source_session_id.clone(), session_id, @@ -319,6 +371,44 @@ fn parse_session_meta( }) } +fn collect_catalog_edit_impact( + config: &AnthropicJsonlSource, + session_id: &str, + message: &JsonlMessage, + created_at: &str, + impact: &mut ImportedHistoryImpactStats, + touched_files: &mut std::collections::BTreeSet, +) { + for block in content_blocks(&message.content) { + if block_type(&block) != "tool_use" { + continue; + } + let raw_name = block.get("name").and_then(Value::as_str).unwrap_or("tool"); + let (canonical_name, args) = + normalize_tool_call(raw_name, block.get("input").cloned().unwrap_or(Value::Null)); + if canonical_name != imported_history::FUNCTION_EDIT_FILE { + continue; + } + let call = ImportedToolCall { + call_id: block + .get("id") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(), + raw_name: raw_name.to_string(), + canonical_name, + args, + created_at: created_at.to_string(), + }; + let chunk = + imported_history::tool_call_chunk(session_id, config.provider_slug, 0, &call, ""); + let one = imported_history::impact_from_edit_chunks(std::slice::from_ref(&chunk)); + impact.lines_added = impact.lines_added.saturating_add(one.lines_added); + impact.lines_removed = impact.lines_removed.saturating_add(one.lines_removed); + touched_files.extend(one.touched_files); + } +} + fn meta_to_cache_input( config: &AnthropicJsonlSource, meta: SessionMeta, diff --git a/src-tauri/crates/orgtrack-core/src/sources/claude_code/history.rs b/src-tauri/crates/orgtrack-core/src/sources/claude_code/history.rs index d854b2767..e72d7e7fb 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/claude_code/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/claude_code/history.rs @@ -4,26 +4,36 @@ //! converts them into ORGII's canonical `ActivityChunk` shape for read-only //! replay. -use std::collections::{BTreeSet, HashMap, HashSet}; +use std::collections::{HashMap, HashSet}; use std::fs; use std::io::{BufRead, BufReader}; use std::path::{Path, PathBuf}; +#[cfg(test)] +use std::collections::BTreeSet; + use core_types::activity::ActivityChunk; use rusqlite::Connection; -use serde::{Deserialize, Serialize}; +use serde::Deserialize; use serde_json::{json, Value}; +#[cfg(test)] +use serde::Serialize; + use crate::sources::imported_history::{ self, cache as imported_cache, managed_mirror, metadata::{ ImportedHistoryCacheInput, ImportedHistoryDiscoveredRecord, ImportedHistoryImpactStats, - RoundUsage, StoredRoundUsage, SOURCE_CLAUDE_CODE, + SOURCE_CLAUDE_CODE, }, - paths as imported_paths, scan_snapshot, + paths as imported_paths, scan_snapshot, ImportedHistoryRecentPath, ImportedHistorySessionPage, + ImportedHistorySessionRow, ImportedToolCall, +}; + +#[cfg(test)] +use crate::sources::imported_history::{ + metadata::{RoundUsage, StoredRoundUsage}, watermark::{ImportedParseWatermark, WatermarkedTranscriptReader}, - ImportedHistoryRecentPath, ImportedHistorySessionPage, ImportedHistorySessionRow, - ImportedToolCall, }; use super::SESSION_PREFIX as CLAUDE_CODE_SESSION_PREFIX; @@ -35,11 +45,13 @@ const CLAUDE_CODE_PROVIDER_SLUG: &str = "claudecode"; // v8: emit per-round usage rows (imported_history_round_usage). // v9: dedup usage by message.id (one API response spans repeated JSONL lines). const CLAUDE_CODE_METADATA_PARSER_VERSION: i64 = 9; +const CLAUDE_CATALOG_PREFIX_BYTES: u64 = 1024 * 1024; pub type ClaudeCodeHistorySessionRow = ImportedHistorySessionRow; pub type ClaudeCodeHistorySessionPage = ImportedHistorySessionPage; pub type ClaudeCodeRecentPath = ImportedHistoryRecentPath; +#[cfg(test)] #[derive(Debug, Clone)] struct ClaudeCodeHistoryMeta { source_session_id: String, @@ -80,9 +92,11 @@ struct ClaudeJsonlLine { #[serde(default)] summary: String, /// `ai-title` records: the auto-generated title shown in the Claude Code app. + #[cfg(test)] #[serde(default)] ai_title: String, /// `custom-title` records: a user-set title that overrides the AI title. + #[cfg(test)] #[serde(default)] custom_title: String, #[serde(default)] @@ -117,16 +131,19 @@ struct ClaudeMessage { /// Assistant API-response id (`msg_…`). One response is written across /// several JSONL lines that each repeat the cumulative `usage`, so tokens /// are counted once per unique id. + #[cfg(test)] #[serde(default)] id: String, #[serde(default)] model: String, #[serde(default)] content: Value, + #[cfg(test)] #[serde(default)] usage: Option, } +#[cfg(test)] #[derive(Debug, Deserialize)] struct ClaudeUsage { #[serde(default)] @@ -167,7 +184,7 @@ pub fn list_claude_code_history_sessions_paginated( limit: usize, offset: usize, ) -> Result { - sync_claude_code_history_cache(conn)?; + refresh_claude_code_catalog(conn)?; imported_cache::query_imported_session_page_from_conn(conn, SOURCE_CLAUDE_CODE, limit, offset) } @@ -175,7 +192,7 @@ pub fn list_claude_code_recent_paths( conn: &mut Connection, limit: usize, ) -> Result, String> { - sync_claude_code_history_cache(conn)?; + refresh_claude_code_catalog(conn)?; imported_cache::query_imported_recent_paths_from_conn(conn, SOURCE_CLAUDE_CODE, limit) } @@ -213,9 +230,12 @@ pub fn stat_claude_code_history_for_session( } } -fn sync_claude_code_history_cache(conn: &mut Connection) -> Result<(), String> { - let previous_snapshots = - scan_snapshot::read_dir_snapshots_from_conn(conn, SOURCE_CLAUDE_CODE); +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + refresh_claude_code_catalog(conn) +} + +fn refresh_claude_code_catalog(conn: &mut Connection) -> Result<(), String> { + let previous_snapshots = scan_snapshot::read_dir_snapshots_from_conn(conn, SOURCE_CLAUDE_CODE); let mut walker = scan_snapshot::SnapshotDirWalker::new(&previous_snapshots, "jsonl", "Claude"); let discovery = discover_claude_code_history_records(&claude_projects_dirs()?, &mut walker)?; let next_snapshots = walker.into_snapshots(); @@ -227,11 +247,8 @@ fn sync_claude_code_history_cache(conn: &mut Connection) -> Result<(), String> { )?; let ClaudeCodeDiscovery { records: mut discovered, - external_titles, + .. } = discovery; - // Managed (GUI-launched) sessions surface through their code_sessions - // row; the imported twin goes unlistable. Folding the verdict into the - // fingerprint re-parses a session whose managed status flips. let managed_ids = managed_mirror::managed_source_session_ids_from_conn( conn, SOURCE_CLAUDE_CODE, @@ -254,32 +271,18 @@ fn sync_claude_code_history_cache(conn: &mut Connection) -> Result<(), String> { |record| record.signature(), )?; let mut inputs = Vec::new(); - let mut rounds = Vec::new(); - let mut reparsed_ids = Vec::new(); for record in changed { - let stored_watermark = imported_history::watermark::read_parse_watermark_from_conn( + let is_managed = managed_ids.contains(&record.source_session_id); + if imported_cache::advance_cached_catalog_record_from_conn( conn, SOURCE_CLAUDE_CODE, - &record.source_session_id, - )?; - let external_title = external_titles - .get(&record.source_session_id) - .cloned() - .unwrap_or_default(); - let parse = - parse_claude_session_meta_with_title(record, stored_watermark.as_ref(), external_title)?; - imported_history::watermark::write_parse_watermark_from_conn( - conn, - SOURCE_CLAUDE_CODE, - &record.source_session_id, - &parse.watermark, - )?; - if let Some(mut meta) = parse.meta { - let is_managed_history_mirror = managed_ids.contains(&meta.source_session_id); - reparsed_ids.push(meta.session_id.clone()); - rounds.append(&mut meta.rounds); - let mut input = session_meta_to_cache_input(meta); - input.listable = input.listable && !is_managed_history_mirror; + record, + Some(!is_managed), + )? { + continue; + } + if let Some(mut input) = parse_claude_catalog_input(record)? { + input.listable = !is_managed; inputs.push(input); } } @@ -289,18 +292,12 @@ fn sync_claude_code_history_cache(conn: &mut Connection) -> Result<(), String> { imported_cache::live_ids_from_signatures(&signatures), inputs, )?; - imported_cache::write_session_rounds_from_conn(conn, &reparsed_ids, &rounds)?; - // Context-window continuations rewrite the conversation into a new - // session file with the same first-user-message uuid; keep only the - // newest sibling of each family listable. - imported_cache::demote_superseded_continuations_from_conn(conn, SOURCE_CLAUDE_CODE)?; - Ok(()) + imported_cache::demote_superseded_continuations_from_conn(conn, SOURCE_CLAUDE_CODE).map(|_| ()) } #[derive(Debug)] struct ClaudeCodeDiscovery { records: Vec, - external_titles: HashMap, } fn discover_claude_code_history_records( @@ -308,7 +305,6 @@ fn discover_claude_code_history_records( walker: &mut scan_snapshot::SnapshotDirWalker<'_>, ) -> Result { let mut records = Vec::new(); - let mut external_titles = HashMap::new(); for projects_dir in projects_dirs { if !projects_dir.is_dir() { continue; @@ -329,12 +325,6 @@ fn discover_claude_code_history_records( }; let (source_mtime_ms, source_size_bytes) = imported_paths::file_metadata_signature(&path, "Claude")?; - if let Some(title) = title_index.get(&file_stem) { - external_titles.insert( - file_stem.clone(), - imported_history::truncate_name(&title.name, 200), - ); - } records.push(ImportedHistoryDiscoveredRecord { source_session_id: file_stem.clone(), source_path: path, @@ -346,10 +336,7 @@ fn discover_claude_code_history_records( }); } } - Ok(ClaudeCodeDiscovery { - records, - external_titles, - }) + Ok(ClaudeCodeDiscovery { records }) } /// `/subagents/workflows/wf_*/journal.jsonl` files are workflow event @@ -466,7 +453,6 @@ fn claude_source_fingerprint( .unwrap_or_default() } -#[cfg(test)] fn claude_session_title_for_record( record: &ImportedHistoryDiscoveredRecord, ) -> Result { @@ -480,7 +466,6 @@ fn claude_session_title_for_record( .unwrap_or_default()) } -#[cfg(test)] fn claude_sessions_dir_for_session_path(session_path: &Path) -> Option { session_path.ancestors().find_map(|ancestor| { if ancestor.file_name().and_then(|name| name.to_str()) == Some("projects") { @@ -494,6 +479,7 @@ fn claude_sessions_dir_for_session_path(session_path: &Path) -> Option /// exactly the per-file state the old single-pass loop kept in locals, so it /// can be frozen into a parse watermark's `state_json` at a complete-line /// boundary and resumed against only the appended suffix. +#[cfg(test)] #[derive(Debug, Clone, Default, Serialize, Deserialize)] struct ClaudeSessionMetaState { created_at_ms: i64, @@ -530,6 +516,7 @@ struct ClaudeSessionMetaState { first_user_uuid: Option, } +#[cfg(test)] impl ClaudeSessionMetaState { fn feed(&mut self, trimmed: &str, record: &ImportedHistoryDiscoveredRecord) { let parsed: ClaudeJsonlLine = match serde_json::from_str(trimmed) { @@ -749,6 +736,7 @@ impl ClaudeSessionMetaState { } } +#[cfg(test)] struct ClaudeSessionMetaParse { meta: Option, watermark: ImportedParseWatermark, @@ -756,6 +744,7 @@ struct ClaudeSessionMetaParse { resumed: bool, } +#[cfg(test)] fn parse_claude_session_meta_with_title( record: &ImportedHistoryDiscoveredRecord, watermark: Option<&ImportedParseWatermark>, @@ -835,6 +824,121 @@ fn parse_claude_session_meta( Ok(parse_claude_session_meta_incremental(record, None)?.meta) } +fn parse_claude_catalog_input( + record: &ImportedHistoryDiscoveredRecord, +) -> Result, String> { + let mut created_at_ms = 0; + let external_title = claude_session_title_for_record(record)?; + let mut summary_title = String::new(); + let mut first_prompt = String::new(); + let mut model = None; + let mut repo_path = None; + let mut branch = None; + let mut parent_source_session_id = None; + let mut first_user_uuid = None; + + for line in imported_history::read_complete_jsonl_prefix_lines( + &record.source_path, + "Claude", + CLAUDE_CATALOG_PREFIX_BYTES, + )? { + let Ok(parsed) = serde_json::from_str::(line.trim()) else { + continue; + }; + if created_at_ms == 0 { + created_at_ms = parsed + .timestamp + .as_deref() + .and_then(imported_history::parse_iso_to_epoch_ms_opt) + .unwrap_or_default(); + } + if repo_path.is_none() && !parsed.cwd.trim().is_empty() { + repo_path = Some(parsed.cwd.clone()); + } + if branch.is_none() && !parsed.git_branch.trim().is_empty() { + branch = Some(parsed.git_branch.clone()); + } + if parent_source_session_id.is_none() && parsed.is_sidechain { + let candidate = parsed.session_id.trim(); + if !candidate.is_empty() && candidate != record.source_session_id { + parent_source_session_id = Some(candidate.to_string()); + } + } + if first_user_uuid.is_none() && parsed.r#type == "user" && !parsed.uuid.trim().is_empty() { + first_user_uuid = Some(parsed.uuid.trim().to_string()); + } + if summary_title.is_empty() && parsed.r#type == "summary" { + let title = parsed.summary.trim(); + if !title.is_empty() { + summary_title = imported_history::truncate_name(title, 200); + } + } + if let Some(message) = parsed.message { + if first_prompt.is_empty() && parsed.r#type == "user" { + if let Some(text) = claude_content_text(&message.content) { + let text = imported_history::strip_orgii_exec_mode_bridge(&text); + if !text.trim().is_empty() { + first_prompt = imported_history::truncate_name(text, 200); + } + } + } + if model.is_none() + && !message.model.trim().is_empty() + && !message.model.starts_with('<') + { + model = Some(message.model); + } + } + } + + let source_time_ms = record.source_mtime_ms.saturating_div(1_000_000); + if created_at_ms == 0 && source_time_ms == 0 { + return Ok(None); + } + let name = if !external_title.is_empty() { + external_title + } else if !summary_title.is_empty() { + summary_title + } else if !first_prompt.is_empty() { + first_prompt + } else { + record.source_record_key.clone() + }; + Ok(Some(ImportedHistoryCacheInput { + source: SOURCE_CLAUDE_CODE, + source_session_id: record.source_session_id.clone(), + session_id: super::canonical_session_id(&record.source_session_id), + source_path: record.source_path.to_string_lossy().to_string(), + source_record_key: record.source_record_key.clone(), + source_mtime_ms: record.source_mtime_ms, + source_size_bytes: record.source_size_bytes, + source_fingerprint: record.source_fingerprint.clone(), + parser_version: CLAUDE_CODE_METADATA_PARSER_VERSION, + name, + created_at_ms: if created_at_ms > 0 { + created_at_ms + } else { + source_time_ms + }, + updated_at_ms: source_time_ms.max(created_at_ms), + model, + input_tokens: 0, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + repo_path, + branch, + impact: ImportedHistoryImpactStats::default(), + listable: true, + source_metadata_json: imported_cache::continuation_group_metadata_json( + first_user_uuid.as_deref(), + ), + parent_session_id: parent_source_session_id + .map(|uuid| format!("{CLAUDE_CODE_SESSION_PREFIX}{uuid}")), + })) +} + +#[cfg(test)] fn session_meta_to_cache_input(meta: ClaudeCodeHistoryMeta) -> ImportedHistoryCacheInput { ImportedHistoryCacheInput { source: SOURCE_CLAUDE_CODE, @@ -871,6 +975,7 @@ fn session_meta_to_cache_input(meta: ClaudeCodeHistoryMeta) -> ImportedHistoryCa /// results containing a unified-diff-style `structuredPatch`. Each hunk's `lines` /// are prefixed with `+` (added), `-` (removed), or ` ` (context), so this yields /// the same counts a `git diff` would — unlike the old_string/new_string heuristic. +#[cfg(test)] fn collect_claude_impact_from_tool_result( result: &Value, impact: &mut ImportedHistoryImpactStats, @@ -901,6 +1006,7 @@ fn collect_claude_impact_from_tool_result( } } +#[cfg(test)] fn collect_claude_impact_from_item( item: &Value, impact: &mut ImportedHistoryImpactStats, @@ -948,6 +1054,7 @@ fn collect_claude_impact_from_item( } } +#[cfg(test)] fn accumulate_claude_edit_input(input: &Value, impact: &mut ImportedHistoryImpactStats) { if let Some(old_string) = input.get("old_string").and_then(Value::as_str) { impact.lines_removed += count_text_lines(old_string); @@ -957,6 +1064,7 @@ fn accumulate_claude_edit_input(input: &Value, impact: &mut ImportedHistoryImpac } } +#[cfg(test)] fn count_text_lines(text: &str) -> i64 { if text.is_empty() { 0 diff --git a/src-tauri/crates/orgtrack-core/src/sources/claude_code/history_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/claude_code/history_tests.rs index 2da847baf..39c4a3dbe 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/claude_code/history_tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/claude_code/history_tests.rs @@ -34,6 +34,98 @@ fn includes_claude_project_dir_candidates() { } } +#[test] +fn catalog_prefix_is_bounded_for_a_thirty_mib_claude_transcript() { + use std::io::Write; + + let path = std::env::temp_dir().join(format!( + "orgii-claude-catalog-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let mut file = std::fs::File::create(&path).expect("create Claude catalog fixture"); + writeln!( + file, + "{}", + serde_json::json!({ + "type":"user", + "uuid":"first-user-id", + "sessionId":"large-claude", + "cwd":"/work/claude", + "gitBranch":"develop", + "timestamp":"2026-07-22T00:00:00Z", + "message":{"role":"user","content":"bounded Claude catalog","model":"claude-sonnet-4"} + }) + ) + .expect("write Claude metadata line"); + let block = vec![b'x'; 64 * 1024]; + for _ in 0..(30 * 1024 / 64) { + file.write_all(&block).expect("extend Claude transcript"); + } + file.flush().expect("flush Claude fixture"); + let size = file.metadata().expect("Claude fixture metadata").len(); + drop(file); + + let record = ImportedHistoryDiscoveredRecord { + source_session_id: "large-claude".to_string(), + source_path: path.clone(), + source_record_key: "large-claude".to_string(), + source_mtime_ms: 1_774_137_600_000_000_000, + source_size_bytes: size as i64, + source_fingerprint: String::new(), + parser_version: CLAUDE_CODE_METADATA_PARSER_VERSION, + }; + let input = parse_claude_catalog_input(&record) + .expect("parse bounded Claude catalog") + .expect("Claude catalog row"); + assert_eq!(input.name, "bounded Claude catalog"); + assert_eq!(input.repo_path.as_deref(), Some("/work/claude")); + assert_eq!(input.model.as_deref(), Some("claude-sonnet-4")); + assert!(CLAUDE_CATALOG_PREFIX_BYTES < size); + + std::fs::remove_file(path).expect("remove Claude fixture"); +} + +#[test] +fn catalog_reads_complete_final_record_without_a_trailing_newline() { + let path = std::env::temp_dir().join(format!( + "orgii-claude-catalog-no-newline-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let record_json = serde_json::json!({ + "type":"user", + "uuid":"first-user-id", + "sessionId":"claude-no-newline", + "cwd":"/work/claude", + "gitBranch":"develop", + "timestamp":"2026-07-22T00:00:00Z", + "message":{ + "role":"user", + "content":"natural EOF Claude catalog", + "model":"claude-sonnet-4" + } + }) + .to_string(); + std::fs::write(&path, &record_json).expect("write no-newline Claude fixture"); + let record = ImportedHistoryDiscoveredRecord { + source_session_id: "claude-no-newline".to_string(), + source_path: path.clone(), + source_record_key: "claude-no-newline".to_string(), + source_mtime_ms: 1_774_137_600_000_000_000, + source_size_bytes: record_json.len() as i64, + source_fingerprint: String::new(), + parser_version: CLAUDE_CODE_METADATA_PARSER_VERSION, + }; + + let input = parse_claude_catalog_input(&record) + .expect("parse no-newline Claude catalog") + .expect("Claude catalog row"); + + assert_eq!(input.name, "natural EOF Claude catalog"); + std::fs::remove_file(path).expect("remove no-newline Claude fixture"); +} + #[test] fn parses_claude_jsonl_into_replay_chunks() { let temp_dir = @@ -717,7 +809,10 @@ fn resumes_claude_meta_parse_from_watermark() { assert_eq!(resumed_meta.input_tokens, 21 + 55); assert_eq!(resumed_meta.input_tokens, scratch_meta.input_tokens); assert_eq!(resumed_meta.output_tokens, scratch_meta.output_tokens); - assert_eq!(resumed_meta.cache_read_tokens, scratch_meta.cache_read_tokens); + assert_eq!( + resumed_meta.cache_read_tokens, + scratch_meta.cache_read_tokens + ); assert_eq!( resumed_meta.cache_write_tokens, scratch_meta.cache_write_tokens @@ -749,10 +844,8 @@ fn resumes_claude_meta_parse_from_watermark() { } fn journal_filter_fixture(tag: &str) -> std::path::PathBuf { - let temp_dir = std::env::temp_dir().join(format!( - "orgii-claude-journal-{tag}-{}", - std::process::id() - )); + let temp_dir = + std::env::temp_dir().join(format!("orgii-claude-journal-{tag}-{}", std::process::id())); std::fs::remove_dir_all(&temp_dir).ok(); let projects_dir = temp_dir.join("projects"); let project = projects_dir.join("-Users-example-proj"); @@ -762,14 +855,14 @@ fn journal_filter_fixture(tag: &str) -> std::path::PathBuf { let line = r#"{"type":"user","sessionId":"s","timestamp":"2026-04-01T07:06:46.543Z","message":{"role":"user","content":"hello"}} "#; std::fs::write(project.join(format!("{session}.jsonl")), line).expect("write session"); - std::fs::write(workflow_dir.join("journal.jsonl"), "{\"type\":\"started\"}\n") - .expect("write journal"); - std::fs::write(workflow_dir.join("agent-a1.jsonl"), line).expect("write workflow agent"); std::fs::write( - project.join(session).join("subagents/agent-a2.jsonl"), - line, + workflow_dir.join("journal.jsonl"), + "{\"type\":\"started\"}\n", ) - .expect("write subagent"); + .expect("write journal"); + std::fs::write(workflow_dir.join("agent-a1.jsonl"), line).expect("write workflow agent"); + std::fs::write(project.join(session).join("subagents/agent-a2.jsonl"), line) + .expect("write subagent"); temp_dir } @@ -790,14 +883,11 @@ fn excludes_workflow_journal_files_from_discovery_and_collect() { assert!(stems.contains(&"agent-a2")); let previous = HashMap::new(); - let mut walker = imported_history::scan_snapshot::SnapshotDirWalker::new( - &previous, "jsonl", "Claude", - ); - let discovery = discover_claude_code_history_records( - std::slice::from_ref(&projects_dir), - &mut walker, - ) - .expect("discover"); + let mut walker = + imported_history::scan_snapshot::SnapshotDirWalker::new(&previous, "jsonl", "Claude"); + let discovery = + discover_claude_code_history_records(std::slice::from_ref(&projects_dir), &mut walker) + .expect("discover"); let ids = discovery .records .iter() @@ -852,14 +942,11 @@ fn prunes_stale_journal_cache_row_after_discovery_filter() { .expect("seed journal watermark"); let previous = HashMap::new(); - let mut walker = imported_history::scan_snapshot::SnapshotDirWalker::new( - &previous, "jsonl", "Claude", - ); - let discovery = discover_claude_code_history_records( - std::slice::from_ref(&projects_dir), - &mut walker, - ) - .expect("discover"); + let mut walker = + imported_history::scan_snapshot::SnapshotDirWalker::new(&previous, "jsonl", "Claude"); + let discovery = + discover_claude_code_history_records(std::slice::from_ref(&projects_dir), &mut walker) + .expect("discover"); let signatures = discovery .records .iter() @@ -909,14 +996,11 @@ fn snapshot_reuse_keeps_fresh_file_signatures() { std::thread::sleep(std::time::Duration::from_millis(5)); let empty = HashMap::new(); - let mut cold_walker = imported_history::scan_snapshot::SnapshotDirWalker::new( - &empty, "jsonl", "Claude", - ); - let cold = discover_claude_code_history_records( - std::slice::from_ref(&projects_dir), - &mut cold_walker, - ) - .expect("cold discover"); + let mut cold_walker = + imported_history::scan_snapshot::SnapshotDirWalker::new(&empty, "jsonl", "Claude"); + let cold = + discover_claude_code_history_records(std::slice::from_ref(&projects_dir), &mut cold_walker) + .expect("cold discover"); assert_eq!(cold.records.len(), 1); let cold_size = cold.records[0].source_size_bytes; assert!(cold_walker.dirs_enumerated >= 2); @@ -928,14 +1012,11 @@ fn snapshot_reuse_keeps_fresh_file_signatures() { .and_then(|mut file| std::io::Write::write_all(&mut file, b"{\"type\":\"assistant\"}\n")) .expect("append to session"); - let mut warm_walker = imported_history::scan_snapshot::SnapshotDirWalker::new( - &snapshots, "jsonl", "Claude", - ); - let warm = discover_claude_code_history_records( - std::slice::from_ref(&projects_dir), - &mut warm_walker, - ) - .expect("warm discover"); + let mut warm_walker = + imported_history::scan_snapshot::SnapshotDirWalker::new(&snapshots, "jsonl", "Claude"); + let warm = + discover_claude_code_history_records(std::slice::from_ref(&projects_dir), &mut warm_walker) + .expect("warm discover"); assert_eq!(warm_walker.dirs_enumerated, 0); assert_eq!(warm_walker.dirs_reused, 2); assert_eq!(warm.records.len(), 1); @@ -952,9 +1033,8 @@ fn bench_real_home_claude_discovery_cold_vs_warm() { let empty = HashMap::new(); let started = std::time::Instant::now(); - let mut cold_walker = imported_history::scan_snapshot::SnapshotDirWalker::new( - &empty, "jsonl", "Claude", - ); + let mut cold_walker = + imported_history::scan_snapshot::SnapshotDirWalker::new(&empty, "jsonl", "Claude"); let cold = discover_claude_code_history_records(&projects_dirs, &mut cold_walker).expect("cold"); let cold_elapsed = started.elapsed(); @@ -962,9 +1042,8 @@ fn bench_real_home_claude_discovery_cold_vs_warm() { let snapshots = cold_walker.into_snapshots(); let started = std::time::Instant::now(); - let mut warm_walker = imported_history::scan_snapshot::SnapshotDirWalker::new( - &snapshots, "jsonl", "Claude", - ); + let mut warm_walker = + imported_history::scan_snapshot::SnapshotDirWalker::new(&snapshots, "jsonl", "Claude"); let warm = discover_claude_code_history_records(&projects_dirs, &mut warm_walker).expect("warm"); let warm_elapsed = started.elapsed(); diff --git a/src-tauri/crates/orgtrack-core/src/sources/cline/history/discovery.rs b/src-tauri/crates/orgtrack-core/src/sources/cline/history/discovery.rs index 3c1107176..5a54acde2 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cline/history/discovery.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cline/history/discovery.rs @@ -17,7 +17,18 @@ pub(super) fn sync_cline_history_cache(conn: &mut Connection) -> Result<(), Stri let mut inputs = Vec::new(); for record in changed { if let Some(meta) = parse_cline_session_meta(record)? { - inputs.push(session_meta_to_cache_input(meta)); + let mut input = session_meta_to_cache_input(meta); + if let Some(cached) = imported_cache::query_cached_session_from_conn( + conn, + SOURCE_CLINE, + &input.source_session_id, + )? { + // Message traversal belongs to whole-document replay. A + // sidecar/DB catalog refresh must never erase the last valid + // compact impact merely because it did not read messages. + input.impact = cached.impact; + } + inputs.push(input); } } imported_cache::sync_source_cache_from_conn( @@ -180,7 +191,6 @@ pub(super) fn parse_cline_session_meta( .and_then(|raw| serde_json::from_str(&raw).ok()) .unwrap_or_default(); - let transcript = read_transcript(messages_path).unwrap_or_default(); let db_metadata = db_meta .and_then(|meta| meta.metadata_json.as_deref()) .and_then(|raw| serde_json::from_str::(raw).ok()) @@ -195,22 +205,13 @@ pub(super) fn parse_cline_session_meta( .map(|meta| meta.started_at.as_str()) .and_then(imported_history::parse_iso_to_epoch_ms_opt) }) - .or_else(|| transcript.messages.iter().find_map(|m| m.ts)) .filter(|ms| *ms > 0) - .unwrap_or(record.source_mtime_ms); + .unwrap_or(record.source_mtime_ms / 1_000_000); - let updated_at_ms = transcript - .messages - .iter() - .rev() - .find_map(|m| m.ts) + let updated_at_ms = db_meta + .and_then(|meta| imported_history::parse_iso_to_epoch_ms_opt(meta.updated_at.as_str())) .filter(|ms| *ms > 0) - .or_else(|| { - db_meta - .map(|meta| meta.updated_at.as_str()) - .and_then(imported_history::parse_iso_to_epoch_ms_opt) - }) - .unwrap_or(record.source_mtime_ms); + .unwrap_or(record.source_mtime_ms / 1_000_000); let title = session_json .metadata @@ -237,7 +238,6 @@ pub(super) fn parse_cline_session_meta( .filter(|value| !value.is_empty()) .map(str::to_string) }) - .or_else(|| first_user_text(&transcript)) .map(|value| imported_history::truncate_name(&value, 200)) .unwrap_or_else(|| record.source_record_key.clone()); @@ -286,8 +286,11 @@ pub(super) fn parse_cline_session_meta( .unwrap_or_default() }); let session_id = format!("{CLINE_SESSION_PREFIX}{}", record.source_session_id); - let impact = - imported_history::impact_from_edit_chunks(&transcript_to_chunks(&session_id, &transcript)); + // The whole-document replay adapter owns message traversal. Catalog + // refresh deliberately avoids deserializing the messages array merely to + // populate a sidebar card; exact impact is projected from the compact + // replay index when that metadata is requested. + let impact = ImportedHistoryImpactStats::default(); let parent_session_id = db_meta .filter(|meta| meta.is_subagent) .and_then(|meta| meta.parent_session_id.as_deref()) diff --git a/src-tauri/crates/orgtrack-core/src/sources/cline/history/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/cline/history/mod.rs index 9032e58ee..feb824c47 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cline/history/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cline/history/mod.rs @@ -179,6 +179,10 @@ pub fn load_cline_history_for_session( load_cline_history_from_path(session_id, &path) } +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + sync_cline_history_cache(conn) +} + #[cfg(test)] #[path = "../history_tests.rs"] mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/cline/history/transcript.rs b/src-tauri/crates/orgtrack-core/src/sources/cline/history/transcript.rs index 61172d754..c85d6a23a 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cline/history/transcript.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cline/history/transcript.rs @@ -304,22 +304,3 @@ pub(super) fn content_blocks(content: &Value) -> Vec<&Value> { pub(super) fn block_type(block: &Value) -> &str { block.get("type").and_then(Value::as_str).unwrap_or("") } - -pub(super) fn first_user_text(transcript: &ClineTranscript) -> Option { - for message in &transcript.messages { - if message.role != "user" { - continue; - } - for block in content_blocks(&message.content) { - if block_type(block) == "text" { - let text = strip_user_input_wrapper( - block.get("text").and_then(Value::as_str).unwrap_or(""), - ); - if !text.is_empty() { - return Some(text.to_string()); - } - } - } - } - None -} diff --git a/src-tauri/crates/orgtrack-core/src/sources/cline/history_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/cline/history_tests.rs index 7bf6394ed..0a756606f 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cline/history_tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cline/history_tests.rs @@ -66,7 +66,58 @@ fn db_candidate_points_at_cline_session_index() { } #[test] -fn imports_db_indexed_subagent_with_its_own_impact() { +fn catalog_metadata_does_not_read_a_thirty_mib_messages_document() { + use std::io::Write; + + let unique = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("clock") + .as_nanos(); + let path = std::env::temp_dir().join(format!("orgii-cline-catalog-{unique}.messages.json")); + let mut file = std::fs::File::create(&path).expect("create Cline fixture"); + let block = vec![b'x'; 64 * 1024]; + for _ in 0..(30 * 1024 / 64) { + file.write_all(&block).expect("extend Cline document"); + } + file.flush().expect("flush Cline fixture"); + let size = file.metadata().expect("Cline fixture metadata").len(); + drop(file); + + let discovered = ClineDiscoveredRecord { + record: ImportedHistoryDiscoveredRecord { + source_session_id: "large-cline".to_string(), + source_path: path.clone(), + source_record_key: "large-cline".to_string(), + source_mtime_ms: 1_774_137_600_000_000_000, + source_size_bytes: size as i64, + source_fingerprint: "db-index-only".to_string(), + parser_version: CLINE_METADATA_PARSER_VERSION, + }, + db_meta: Some(ClineDbSessionMeta { + session_id: "large-cline".to_string(), + updated_at: "2026-07-22T00:00:01Z".to_string(), + prompt: Some("Cline compact catalog".to_string()), + model: Some("claude-sonnet-4".to_string()), + workspace_root: Some("/work/cline".to_string()), + messages_path: path.to_string_lossy().to_string(), + ..ClineDbSessionMeta::default() + }), + }; + let input = session_meta_to_cache_input( + parse_cline_session_meta(&discovered) + .expect("parse compact Cline metadata") + .expect("Cline catalog row"), + ); + assert_eq!(input.name, "Cline compact catalog"); + assert_eq!(input.repo_path.as_deref(), Some("/work/cline")); + assert_eq!(input.model.as_deref(), Some("claude-sonnet-4")); + assert_eq!(input.impact.files_changed, 0); + + std::fs::remove_file(path).expect("remove Cline fixture"); +} + +#[test] +fn imports_db_indexed_subagent_without_hydrating_its_transcript() { let unique = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .expect("clock") @@ -128,9 +179,9 @@ fn imports_db_indexed_subagent_with_its_own_impact() { assert_eq!(input.source_session_id, "root__agent_child"); assert_eq!(input.parent_session_id.as_deref(), Some("clineapp-root")); assert_eq!(input.name, "Child task"); - assert_eq!(input.impact.touched_files, vec!["src/child.rs"]); - assert_eq!(input.impact.lines_added, 2); - assert_eq!(input.impact.lines_removed, 1); + assert!(input.impact.touched_files.is_empty()); + assert_eq!(input.impact.lines_added, 0); + assert_eq!(input.impact.lines_removed, 0); std::fs::remove_dir_all(dir).expect("remove temp dir"); } diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app/desktop_exec.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app/desktop_exec.rs index 2add00cb7..0133ce553 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/codex/app/desktop_exec.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app/desktop_exec.rs @@ -303,7 +303,7 @@ fn parse_string(value: &str) -> Option { None } -pub(super) fn codex_tool_output_text(output: Option<&Value>) -> String { +pub(crate) fn codex_tool_output_text(output: Option<&Value>) -> String { match output { Some(Value::String(text)) => text.clone(), Some(Value::Array(parts)) => parts diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app/impact.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app/impact.rs index e75a9d5d4..bb1b9548c 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/codex/app/impact.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app/impact.rs @@ -1,15 +1,19 @@ //! Patch/impact extraction from Codex events and tool calls. +#[cfg(test)] use std::collections::BTreeSet; +#[cfg(test)] use serde_json::Value; +#[cfg(test)] use crate::sources::imported_history::{self, metadata::ImportedHistoryImpactStats}; /// Tally impact from a `patch_apply_end` event — Codex's authoritative record /// of a successfully applied patch. `changes` maps each touched path to a /// `{ type, unified_diff }` object; the diff's `+`/`-` lines give exact /// add/remove counts regardless of how the edit was requested. +#[cfg(test)] pub(super) fn collect_codex_impact_from_patch_apply_end( payload: &Value, impact: &mut ImportedHistoryImpactStats, @@ -43,6 +47,7 @@ pub(super) fn collect_codex_impact_from_patch_apply_end( } } +#[cfg(test)] pub(super) fn collect_codex_impact_from_payload( payload: &Value, impact: &mut ImportedHistoryImpactStats, @@ -74,6 +79,7 @@ pub(super) fn collect_codex_impact_from_payload( } } +#[cfg(test)] fn patch_from_codex_args(args: &Value) -> Option { args.get("patch") .and_then(Value::as_str) @@ -81,6 +87,7 @@ fn patch_from_codex_args(args: &Value) -> Option { .map(str::to_string) } +#[cfg(test)] fn accumulate_patch_impact( patch: &str, impact: &mut ImportedHistoryImpactStats, diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app/index.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app/index.rs index e6a083ba8..f78d1a909 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/codex/app/index.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app/index.rs @@ -4,47 +4,59 @@ use std::collections::{HashMap, HashSet}; use std::fs; use std::io::{BufRead, BufReader}; use std::path::{Path, PathBuf}; -use std::time::{SystemTime, UNIX_EPOCH}; use core_types::activity::ActivityChunk; -use rusqlite::Connection; +use rusqlite::{Connection, Transaction}; use serde::Deserialize; use serde_json::Value; use crate::sources::codex::{canonical_session_id, SESSION_PREFIX as CODEX_APP_SESSION_PREFIX}; use crate::sources::imported_history::{ self, cache as imported_cache, - metadata::{ImportedHistoryDiscoveredRecord, SOURCE_CODEX_APP}, + metadata::{ImportedHistoryDiscoveredRecord, ImportedHistoryRecordSignature, SOURCE_CODEX_APP}, paths as imported_paths, scan_snapshot, }; use crate::store::{sqlite::SqliteRecordStore, RecordStore}; -use super::meta::{ - parse_codex_session_meta_with_title, resolve_codex_transcript_for_thread_id_near_path, - session_meta_to_cache_input, -}; -use super::transcript::{ - load_codex_app_from_path, load_codex_app_initial_window_from_path, - load_codex_app_turn_from_path, CodexAppInitialWindow, CodexAppTurnWindow, -}; +use super::meta::resolve_codex_transcript_for_thread_id_near_path; +use super::transcript::load_codex_app_from_path; use super::{ CodexAppRecentPath, CodexAppSessionPage, CodexAppSourceMetadata, CODEX_APP_METADATA_PARSER_VERSION, }; -/// Metadata discovery normally parses changed rollouts to derive repo, title, -/// and rounds. Re-reading a very large rollout while Codex is still appending -/// can allocate several times the file size and immediately become stale. -/// Keep its existing cached row (if any) and parse it once after a quiet window. -const MAX_EAGER_ACTIVE_CODEX_METADATA_BYTES: i64 = 32 * 1024 * 1024; -const ACTIVE_CODEX_METADATA_QUIET_NS: i64 = 10 * 60 * 1_000_000_000; - #[derive(Debug, Clone)] struct CodexSessionIndexEntry { thread_name: String, updated_at: Option, } +#[derive(Debug)] +struct DiscoveredCodexCatalogRecord { + record: ImportedHistoryDiscoveredRecord, + /// Codex's own session index is the authoritative source for a root + /// thread's display title. Keep it separate from the discovery + /// fingerprint so catalog repair never has to reverse-parse an opaque + /// signature string. + authoritative_title: Option, +} + +#[derive(Debug)] +struct CachedCodexCatalogTitle { + name: String, + source_record_key: String, + session_id: String, + signature: ImportedHistoryRecordSignature, + verified_title_signature: Option, +} + +#[derive(Debug)] +struct ReplayAppliedTitleOwnership { + applied_name: String, +} + +const CODEX_TITLE_REPAIR_SIGNATURE_FIELD: &str = "_codexTitleRepairSignature"; + #[derive(Debug, Deserialize)] struct CodexSessionIndexLine { #[serde(default)] @@ -60,7 +72,7 @@ pub fn list_codex_app_sessions_paginated( limit: usize, offset: usize, ) -> Result { - sync_codex_app_cache(conn)?; + refresh_codex_app_catalog(conn)?; imported_cache::query_imported_session_page_from_conn(conn, SOURCE_CODEX_APP, limit, offset) } @@ -68,7 +80,7 @@ pub fn list_codex_app_recent_paths( conn: &mut Connection, limit: usize, ) -> Result, String> { - sync_codex_app_cache(conn)?; + refresh_codex_app_catalog(conn)?; imported_cache::query_imported_recent_paths_from_conn(conn, SOURCE_CODEX_APP, limit) } @@ -76,7 +88,7 @@ pub fn list_codex_app_reconciliation_sessions( conn: &mut Connection, limit: usize, ) -> Result, String> { - sync_codex_app_cache(conn)?; + refresh_codex_app_catalog(conn)?; imported_cache::query_recent_cached_sessions_for_source_from_conn(conn, SOURCE_CODEX_APP, limit) } @@ -91,30 +103,6 @@ pub fn load_codex_app_for_session( Ok(chunks) } -pub fn load_codex_app_initial_window_for_session( - conn: &Connection, - session_id: &str, - recent_turn_count: usize, -) -> Result { - let file_stem = codex_file_stem_from_session_id(session_id)?; - let path = resolve_codex_session_path(conn, file_stem)?; - let mut window = load_codex_app_initial_window_from_path(session_id, &path, recent_turn_count)?; - link_codex_subagent_chunks(conn, session_id, &mut window.chunks)?; - Ok(window) -} - -pub fn load_codex_app_turn_for_session( - conn: &Connection, - session_id: &str, - turn_id: &str, -) -> Result { - let file_stem = codex_file_stem_from_session_id(session_id)?; - let path = resolve_codex_session_path(conn, file_stem)?; - let mut window = load_codex_app_turn_from_path(session_id, &path, turn_id)?; - link_codex_subagent_chunks(conn, session_id, &mut window.chunks)?; - Ok(window) -} - #[derive(Debug, Clone)] struct CodexChildSessionLink { session_id: String, @@ -259,10 +247,14 @@ fn best_codex_child_match( .map(|(index, _)| index) } -fn sync_codex_app_cache(conn: &mut Connection) -> Result<(), String> { +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + refresh_codex_app_catalog(conn) +} + +fn refresh_codex_app_catalog(conn: &mut Connection) -> Result<(), String> { let previous_snapshots = scan_snapshot::read_dir_snapshots_from_conn(conn, SOURCE_CODEX_APP); let mut walker = scan_snapshot::SnapshotDirWalker::new(&previous_snapshots, "jsonl", "Codex"); - let discovery = discover_codex_app_records(&codex_sessions_dirs()?, &mut walker)?; + let mut discovered = discover_codex_app_records(&codex_sessions_dirs()?, &mut walker)?; let next_snapshots = walker.into_snapshots(); scan_snapshot::persist_dir_snapshots_if_changed( conn, @@ -270,73 +262,54 @@ fn sync_codex_app_cache(conn: &mut Connection) -> Result<(), String> { &previous_snapshots, &next_snapshots, )?; - let CodexAppDiscovery { - records: mut discovered, - external_titles, - } = discovery; - // Managed (GUI-launched) Codex sessions surface through their - // code_sessions row (`cli_agent_type = 'codex'`); the imported twin goes - // unlistable. Same pattern as the OpenCode/Claude readers. let managed_ids = crate::sources::imported_history::managed_mirror::managed_source_session_ids_from_conn( conn, "codex", SOURCE_CODEX_APP, )?; - for record in &mut discovered { + for discovered_record in &mut discovered { crate::sources::imported_history::managed_mirror::append_managed_fingerprint( - &mut record.source_fingerprint, - // Suffix match: the imported key is the rollout stem while the - // runner binds the bare thread uuid. + &mut discovered_record.record.source_fingerprint, crate::sources::imported_history::managed_mirror::is_managed_source_session_id( &managed_ids, - &record.source_session_id, + &discovered_record.record.source_session_id, ), ); } + repair_codex_catalog_titles(conn, &discovered)?; let signatures = discovered .iter() - .map(ImportedHistoryDiscoveredRecord::signature) + .map(|record| record.record.signature()) .collect::>(); let changed = imported_cache::changed_records_from_conn(conn, SOURCE_CODEX_APP, &discovered, |record| { - record.signature() + record.record.signature() })?; let mut inputs = Vec::new(); - let mut rounds = Vec::new(); - let mut reparsed_ids = Vec::new(); - let now_ns = unix_epoch_now_ns(); - for record in changed { - if should_defer_active_codex_metadata(record, now_ns) { - continue; - } - let stored_watermark = imported_history::watermark::read_parse_watermark_from_conn( - conn, - SOURCE_CODEX_APP, - &record.source_session_id, - )?; - let external_title = external_titles - .get(&record.source_session_id) - .cloned() - .unwrap_or_default(); - let parse = - parse_codex_session_meta_with_title(record, stored_watermark.as_ref(), external_title)?; - imported_history::watermark::write_parse_watermark_from_conn( + for discovered_record in changed { + let record = &discovered_record.record; + let is_managed = + crate::sources::imported_history::managed_mirror::is_managed_source_session_id( + &managed_ids, + &record.source_session_id, + ); + // Growth of an existing rollout only advances the compact catalog + // signature. Its body is indexed incrementally by bounded replay when + // the session is visible/active; sidebar refresh never starts over. + if imported_cache::advance_cached_catalog_record_from_conn( conn, SOURCE_CODEX_APP, - &record.source_session_id, - &parse.watermark, - )?; - if let Some(mut meta) = parse.meta { - let is_managed_history_mirror = - crate::sources::imported_history::managed_mirror::is_managed_source_session_id( - &managed_ids, - &meta.source_session_id, - ); - reparsed_ids.push(meta.session_id.clone()); - rounds.append(&mut meta.rounds); - let mut input = session_meta_to_cache_input(meta); - input.listable = input.listable && !is_managed_history_mirror; + record, + Some(!is_managed), + )? { + continue; + } + if let Some(mut input) = super::meta::parse_codex_catalog_input_with_title( + record, + discovered_record.authoritative_title.as_deref(), + )? { + input.listable = !is_managed; inputs.push(input); } } @@ -345,77 +318,283 @@ fn sync_codex_app_cache(conn: &mut Connection) -> Result<(), String> { SOURCE_CODEX_APP, imported_cache::live_ids_from_signatures(&signatures), inputs, - )?; - imported_cache::write_session_rounds_from_conn(conn, &reparsed_ids, &rounds) + ) +} + +/// Reassert Codex-owned title precedence without hydrating transcripts. +/// +/// Older replay cursors treated every payload `name` as a session title, so +/// an active session could be renamed from its session-index title to +/// `update_plan`, then to `js`, on successive polls. The derivation baseline +/// is not sufficient to undo that damage because an already-polluted title +/// can itself become the next baseline. Discovery has stronger provenance: +/// use the session index unconditionally, and only inspect the bounded JSONL +/// prefix when a legacy replay-owned title has no index entry. +fn repair_codex_catalog_titles( + conn: &mut Connection, + discovered: &[DiscoveredCodexCatalogRecord], +) -> Result<(), String> { + let cached = load_cached_codex_catalog_titles(conn)?; + let replay_applied_names = load_current_replay_applied_names(conn, &cached)?; + let mut repairs = Vec::new(); + let mut verified_replay_titles = Vec::new(); + + for discovered_record in discovered { + let source_session_id = &discovered_record.record.source_session_id; + let Some(current) = cached.get(source_session_id) else { + continue; + }; + let discovered_signature = discovered_record.record.signature(); + let title_repair_signature = codex_title_repair_signature(&discovered_signature); + let desired = + if let Some(authoritative_title) = discovered_record.authoritative_title.as_deref() { + Some(authoritative_title.to_string()) + } else { + let signature_changed = !imported_cache::record_matches_cached_signature( + ¤t.signature, + &discovered_signature, + ); + let placeholder = is_codex_catalog_placeholder(current, source_session_id); + let already_neutral = current.name.trim() == "Untitled"; + let replay_polluted = + replay_applied_names + .get(source_session_id) + .is_some_and(|ownership| { + ownership.applied_name == current.name.trim() + && current.verified_title_signature.as_deref() + != Some(title_repair_signature.as_str()) + }); + if replay_polluted || placeholder && (!already_neutral || signature_changed) { + let parsed = super::meta::parse_codex_catalog_input_with_title( + &discovered_record.record, + None, + )? + .map(|input| input.name); + if replay_polluted { + // The replay derivation can legitimately have selected + // the same first-user title that discovery now verifies. + // Record the physical source signature so an unchanged + // sidebar refresh does not reopen the JSONL prefix on + // every pass. A replace/append/parser upgrade changes the + // signature and intentionally re-enables verification. + verified_replay_titles + .push((source_session_id.clone(), title_repair_signature)); + } + parsed + } else { + None + } + }; + if let Some(desired) = desired.filter(|desired| desired.trim() != current.name.trim()) { + repairs.push((source_session_id.clone(), desired)); + } + } + + if repairs.is_empty() && verified_replay_titles.is_empty() { + return Ok(()); + } + let tx = conn + .transaction() + .map_err(|err| format!("start Codex catalog-title repair: {err}"))?; + for (source_session_id, desired) in repairs { + tx.execute( + "UPDATE imported_history_session_cache + SET name=?2, updated_at=?3 + WHERE source='codex_app' AND source_session_id=?1", + (source_session_id, desired, chrono::Utc::now().to_rfc3339()), + ) + .map_err(|err| format!("restore Codex catalog title: {err}"))?; + } + for (source_session_id, signature) in verified_replay_titles { + mark_codex_replay_title_verified(&tx, &source_session_id, &signature)?; + } + tx.commit() + .map_err(|err| format!("commit Codex catalog-title repair: {err}")) +} + +fn codex_title_repair_signature(signature: &ImportedHistoryRecordSignature) -> String { + serde_json::json!([ + signature.source_path, + signature.source_mtime_ms, + signature.source_size_bytes, + signature.source_fingerprint, + signature.parser_version + ]) + .to_string() } -fn unix_epoch_now_ns() -> i64 { - SystemTime::now() - .duration_since(UNIX_EPOCH) +fn mark_codex_replay_title_verified( + tx: &Transaction<'_>, + source_session_id: &str, + signature: &str, +) -> Result<(), String> { + let source_metadata_json = tx + .query_row( + "SELECT source_metadata_json + FROM imported_history_session_cache + WHERE source='codex_app' AND source_session_id=?1", + [source_session_id], + |row| row.get::<_, String>(0), + ) + .map_err(|err| format!("read Codex replay-title verification state: {err}"))?; + let mut source_metadata = serde_json::from_str::(&source_metadata_json) .ok() - .and_then(|elapsed| i64::try_from(elapsed.as_nanos()).ok()) - .unwrap_or(i64::MAX) + .and_then(|value| value.as_object().cloned()) + .unwrap_or_default(); + source_metadata.insert( + CODEX_TITLE_REPAIR_SIGNATURE_FIELD.to_string(), + Value::String(signature.to_string()), + ); + tx.execute( + "UPDATE imported_history_session_cache + SET source_metadata_json=?2, updated_at=?3 + WHERE source='codex_app' AND source_session_id=?1", + ( + source_session_id, + Value::Object(source_metadata).to_string(), + chrono::Utc::now().to_rfc3339(), + ), + ) + .map(|_| ()) + .map_err(|err| format!("store Codex replay-title verification state: {err}")) } -fn should_defer_active_codex_metadata( - record: &ImportedHistoryDiscoveredRecord, - now_ns: i64, -) -> bool { - record.source_size_bytes > MAX_EAGER_ACTIVE_CODEX_METADATA_BYTES - && now_ns.saturating_sub(record.source_mtime_ms) < ACTIVE_CODEX_METADATA_QUIET_NS +fn load_cached_codex_catalog_titles( + conn: &Connection, +) -> Result, String> { + let mut statement = conn + .prepare( + "SELECT source_session_id,name,source_record_key,session_id, + source_path,source_mtime_ms,source_size_bytes, + source_fingerprint,parser_version,source_metadata_json + FROM imported_history_session_cache + WHERE source='codex_app'", + ) + .map_err(|err| format!("prepare cached Codex title query: {err}"))?; + let rows = statement + .query_map([], |row| { + let source_session_id = row.get::<_, String>(0)?; + let source_metadata_json = row.get::<_, String>(9)?; + let verified_title_signature = serde_json::from_str::(&source_metadata_json) + .ok() + .and_then(|value| { + value + .get(CODEX_TITLE_REPAIR_SIGNATURE_FIELD) + .and_then(Value::as_str) + .map(str::to_string) + }); + Ok(( + source_session_id.clone(), + CachedCodexCatalogTitle { + name: row.get(1)?, + source_record_key: row.get(2)?, + session_id: row.get(3)?, + signature: ImportedHistoryRecordSignature { + source_session_id, + source_path: row.get(4)?, + source_mtime_ms: row.get(5)?, + source_size_bytes: row.get(6)?, + source_fingerprint: row.get(7)?, + parser_version: row.get(8)?, + }, + verified_title_signature, + }, + )) + }) + .map_err(|err| format!("query cached Codex titles: {err}"))?; + rows.collect::>>() + .map_err(|err| format!("read cached Codex titles: {err}")) } -#[derive(Debug)] -struct CodexAppDiscovery { - records: Vec, - external_titles: HashMap, +fn load_current_replay_applied_names( + conn: &Connection, + cached: &HashMap, +) -> Result, String> { + let mut statement = conn + .prepare( + "SELECT source_session_id,applied_json + FROM imported_replay_catalog_derivations + WHERE source='codex_app'", + ) + .map_err(|err| format!("prepare Codex replay-title ownership query: {err}"))?; + let rows = statement + .query_map([], |row| { + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + }) + .map_err(|err| format!("query Codex replay-title ownership: {err}"))?; + let mut applied_names = HashMap::new(); + for row in rows { + let (source_session_id, applied_json) = + row.map_err(|err| format!("read Codex replay-title ownership: {err}"))?; + let Some(current) = cached.get(&source_session_id) else { + continue; + }; + let applied = serde_json::from_str::(&applied_json).ok(); + let applied_name = applied + .as_ref() + .and_then(|value| value.get("name")) + .and_then(Value::as_str) + .map(str::to_string); + if applied_name.as_deref() == Some(current.name.trim()) { + applied_names.insert( + source_session_id, + ReplayAppliedTitleOwnership { + applied_name: current.name.trim().to_string(), + }, + ); + } + } + Ok(applied_names) +} + +fn is_codex_catalog_placeholder(cached: &CachedCodexCatalogTitle, source_session_id: &str) -> bool { + let name = cached.name.trim(); + name.is_empty() + || name == cached.source_record_key + || name == source_session_id + || name == cached.session_id + || matches!(name, "New Agent" | "Untitled") } fn discover_codex_app_records( sessions_dirs: &[PathBuf], walker: &mut scan_snapshot::SnapshotDirWalker<'_>, -) -> Result { - let mut records = Vec::new(); - let mut external_titles = HashMap::new(); +) -> Result, String> { + let mut sessions = Vec::new(); for sessions_dir in sessions_dirs { - if !sessions_dir.is_dir() { - continue; - } - let title_index = load_codex_session_index_for_sessions_dir(sessions_dir)?; - let mut files = Vec::new(); - walker.collect_files(sessions_dir, &mut files)?; - for path in files { - let Some(file_stem) = path - .file_stem() - .and_then(|value| value.to_str()) - .map(ToString::to_string) - else { - continue; - }; - let (source_mtime_ms, source_size_bytes) = - imported_paths::file_metadata_signature(&path, "Codex")?; - if let Some(entry) = codex_title_entry_for_file_stem(&file_stem, &title_index) { - external_titles.insert( - file_stem.clone(), - imported_history::truncate_name(&entry.thread_name, 200), - ); + if sessions_dir.is_dir() { + let title_index = load_codex_session_index_for_sessions_dir(sessions_dir)?; + let mut files = Vec::new(); + walker.collect_files(sessions_dir, &mut files)?; + for path in files { + let Some(file_stem) = path + .file_stem() + .and_then(|value| value.to_str()) + .map(ToString::to_string) + else { + continue; + }; + let (source_mtime_ms, source_size_bytes) = + imported_paths::file_metadata_signature(&path, "Codex")?; + let source_fingerprint = codex_source_fingerprint(&file_stem, &title_index); + let authoritative_title = codex_title_entry_for_file_stem(&file_stem, &title_index) + .map(|entry| imported_history::truncate_name(entry.thread_name.trim(), 200)); + sessions.push(DiscoveredCodexCatalogRecord { + record: ImportedHistoryDiscoveredRecord { + source_session_id: file_stem.clone(), + source_path: path, + source_record_key: file_stem, + source_mtime_ms, + source_size_bytes, + source_fingerprint, + parser_version: CODEX_APP_METADATA_PARSER_VERSION, + }, + authoritative_title, + }); } - let source_fingerprint = codex_source_fingerprint(&file_stem, &title_index); - records.push(ImportedHistoryDiscoveredRecord { - source_session_id: file_stem.clone(), - source_path: path, - source_record_key: file_stem, - source_mtime_ms, - source_size_bytes, - source_fingerprint, - parser_version: CODEX_APP_METADATA_PARSER_VERSION, - }); } } - Ok(CodexAppDiscovery { - records, - external_titles, - }) + Ok(sessions) } pub(super) fn collect_codex_session_files( @@ -709,93 +888,5 @@ pub(crate) fn codex_sessions_dir_candidates(home: &Path) -> Vec { .map(|root| root.join("sessions")) .collect() } - #[cfg(test)] -mod tests { - use super::*; - use serde_json::json; - - fn discovered_record( - source_size_bytes: i64, - source_mtime_ns: i64, - ) -> ImportedHistoryDiscoveredRecord { - ImportedHistoryDiscoveredRecord { - source_session_id: "rollout-test".to_string(), - source_path: PathBuf::from("rollout-test.jsonl"), - source_record_key: "rollout-test".to_string(), - source_mtime_ms: source_mtime_ns, - source_size_bytes, - source_fingerprint: String::new(), - parser_version: CODEX_APP_METADATA_PARSER_VERSION, - } - } - - #[test] - fn giant_active_codex_metadata_is_deferred_until_quiet() { - let now_ns = 1_750_000_000_000_000_000; - let record = discovered_record( - MAX_EAGER_ACTIVE_CODEX_METADATA_BYTES + 1, - now_ns - ACTIVE_CODEX_METADATA_QUIET_NS + 1, - ); - - assert!(should_defer_active_codex_metadata(&record, now_ns)); - assert!(!should_defer_active_codex_metadata( - &discovered_record( - record.source_size_bytes, - now_ns - ACTIVE_CODEX_METADATA_QUIET_NS - ), - now_ns - )); - } - - #[test] - fn bounded_active_codex_metadata_remains_eager() { - let now_ns = 1_750_000_000_000_000_000; - let record = discovered_record(MAX_EAGER_ACTIVE_CODEX_METADATA_BYTES, now_ns); - - assert!(!should_defer_active_codex_metadata(&record, now_ns)); - } - - #[test] - fn links_spawn_chunk_to_matching_codex_child_and_restores_prompt() { - let mut chunks = vec![ - ActivityChunk::new("codexapp-parent", "tool_call", "subagent").with_args(json!({ - "task_name": "audit_todays_commits", - "description": "audit_todays_commits", - "codexAgentThreadId": "019f-audit" - })), - ]; - chunks[0].created_at = "2026-07-23T10:18:52Z".to_string(); - let mut children = vec![ - CodexChildSessionLink { - session_id: "codexapp-wrong-nearby-child".to_string(), - thread_id: Some("019f-wrong".to_string()), - created_at_ms: 1_753_265_932_100, - metadata: CodexAppSourceMetadata { - first_prompt: Some("wrong prompt".to_string()), - agent_path: Some("/root/other_task".to_string()), - agent_nickname: Some("Wrong".to_string()), - }, - }, - CodexChildSessionLink { - session_id: "codexapp-audit-child".to_string(), - thread_id: Some("019f-audit".to_string()), - created_at_ms: 1_753_265_940_000, - metadata: CodexAppSourceMetadata { - first_prompt: Some("audit today's commit history".to_string()), - agent_path: Some("/root/audit_todays_commits".to_string()), - agent_nickname: Some("Peirce".to_string()), - }, - }, - ]; - - link_codex_subagent_chunks_from_children(&mut chunks, &mut children); - - assert_eq!(chunks[0].args["subagentSessionId"], "codexapp-audit-child"); - assert_eq!(chunks[0].args["prompt"], "audit today's commit history"); - assert_eq!(chunks[0].args["subagent_type"], "Peirce"); - assert_eq!(chunks[0].args["action"], "delegate"); - assert_eq!(children.len(), 1); - assert_eq!(children[0].session_id, "codexapp-wrong-nearby-child"); - } -} +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app/index/tests.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app/index/tests.rs new file mode 100644 index 000000000..8bf9e9965 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app/index/tests.rs @@ -0,0 +1,431 @@ +use super::*; +use serde_json::json; +use std::sync::atomic::{AtomicU64, Ordering}; + +static TITLE_REPAIR_FIXTURE_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +fn title_repair_fixture( + current_name: &str, + transcript_lines: &[Value], +) -> (Connection, DiscoveredCodexCatalogRecord, PathBuf) { + let path = std::env::temp_dir().join(format!( + "orgii-codex-title-repair-{}-{}.jsonl", + std::process::id(), + TITLE_REPAIR_FIXTURE_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )); + let body = transcript_lines + .iter() + .map(Value::to_string) + .collect::>() + .join("\n"); + fs::write(&path, format!("{body}\n")).expect("write Codex title fixture"); + let metadata = fs::metadata(&path).expect("Codex title fixture metadata"); + let source_session_id = "rollout-title-fixture".to_string(); + let record = ImportedHistoryDiscoveredRecord { + source_session_id: source_session_id.clone(), + source_path: path.clone(), + source_record_key: source_session_id.clone(), + source_mtime_ms: 1_774_137_600_000_000_000, + source_size_bytes: i64::try_from(metadata.len()).expect("fixture size"), + source_fingerprint: String::new(), + parser_version: CODEX_APP_METADATA_PARSER_VERSION, + }; + + let conn = Connection::open_in_memory().expect("catalog DB"); + SqliteRecordStore::init_tables(&conn).expect("core schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("catalog schema"); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path,source_record_key, + source_mtime_ms,source_size_bytes,source_fingerprint,parser_version, + name,created_at_ms,updated_at_ms,model,input_tokens,output_tokens, + cache_read_tokens,cache_write_tokens,repo_path,branch,files_changed, + lines_added,lines_removed,touched_files_json,listable, + source_metadata_json,parent_session_id,updated_at + ) VALUES( + 'codex_app',?1,'codexapp-title-fixture',?2,?1, + ?3,?4,'',?5,?6,1,1,'',0,0,0,0,'','',0,0,0,'[]',0, + '{\"adapterOwned\":{\"keep\":true},\"unrelated\":\"preserve-me\"}', + 'codexapp-parent','2026-07-22T00:00:00Z' + )", + ( + &source_session_id, + path.to_string_lossy().to_string(), + record.source_mtime_ms, + record.source_size_bytes, + record.parser_version, + current_name, + ), + ) + .expect("insert cached Codex title"); + conn.execute( + "INSERT INTO imported_replay_catalog_derivations( + source,source_session_id,baseline_json,applied_json,updated_at + ) VALUES('codex_app',?1,'{\"name\":\"older-tool\"}',?2, + '2026-07-22T00:00:00Z')", + ( + &source_session_id, + serde_json::json!({"name": current_name}).to_string(), + ), + ) + .expect("insert replay title ownership"); + + ( + conn, + DiscoveredCodexCatalogRecord { + record, + authoritative_title: None, + }, + path, + ) +} + +fn cached_title(conn: &Connection) -> (String, i64, String) { + conn.query_row( + "SELECT name,listable,parent_session_id + FROM imported_history_session_cache + WHERE source='codex_app' + AND source_session_id='rollout-title-fixture'", + [], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .expect("read repaired Codex title") +} + +fn cached_title_verification_signature(conn: &Connection) -> Option { + cached_source_metadata(conn) + .get(CODEX_TITLE_REPAIR_SIGNATURE_FIELD) + .and_then(Value::as_str) + .map(str::to_string) +} + +fn cached_source_metadata(conn: &Connection) -> Value { + conn.query_row( + "SELECT source_metadata_json + FROM imported_history_session_cache + WHERE source='codex_app' + AND source_session_id='rollout-title-fixture'", + [], + |row| row.get::<_, String>(0), + ) + .expect("read Codex source metadata") + .parse::() + .expect("parse Codex source metadata") +} + +fn assert_unrelated_source_metadata_survives(conn: &Connection) { + let metadata = cached_source_metadata(conn); + assert_eq!( + metadata.pointer("/adapterOwned/keep"), + Some(&Value::Bool(true)) + ); + assert_eq!( + metadata.get("unrelated").and_then(Value::as_str), + Some("preserve-me") + ); +} + +fn publish_title_fixture_projection( + conn: &mut Connection, + record: &ImportedHistoryDiscoveredRecord, + model: Option<&str>, +) { + // `title_repair_fixture` seeds the minimal legacy ownership shape used + // by title repair itself. Replay publication expects the modern full + // snapshot shape, so start a fresh derivation exactly as a post-prune + // replay generation would. + conn.execute( + "DELETE FROM imported_replay_catalog_derivations + WHERE source='codex_app' AND source_session_id=?1", + [&record.source_session_id], + ) + .expect("clear legacy title-only derivation"); + let driver_cursor = serde_json::json!({ + "catalog": crate::sources::imported_history::catalog::ReplayCatalogProjection { + model: model.map(str::to_string), + ..Default::default() + } + }) + .to_string(); + let tx = conn.transaction().expect("replay catalog transaction"); + crate::sources::imported_history::catalog::publish_from_replay_tx( + &tx, + crate::sources::imported_history::replay::ImportedHistorySourceId::CodexApp, + &record.source_session_id, + "title-fixture-generation", + 0, + false, + record.source_mtime_ms, + &driver_cursor, + ) + .expect("publish replay catalog projection"); + tx.commit().expect("commit replay catalog projection"); +} + +#[test] +fn authoritative_index_title_repairs_pollution_without_changing_visibility_or_parent() { + let (mut conn, mut discovered, path) = title_repair_fixture("update_plan", &[]); + discovered.authoritative_title = Some("Human session title".to_string()); + + repair_codex_catalog_titles(&mut conn, &[discovered]).expect("repair from Codex session index"); + + assert_eq!( + cached_title(&conn), + ( + "Human session title".to_string(), + 0, + "codexapp-parent".to_string() + ), + "managed/subagent visibility and parent placement must remain adapter-owned" + ); + fs::remove_file(path).expect("remove title fixture"); +} + +#[test] +fn polluted_title_without_an_index_uses_first_real_user_prompt() { + let (mut conn, discovered, path) = title_repair_fixture( + "update_plan", + &[ + json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"response_item", + "payload":{"type":"custom_tool_call","name":"update_plan"} + }), + json!({ + "timestamp":"2026-07-22T00:00:01Z", + "type":"event_msg", + "payload":{"type":"user_message","message":"Investigate bounded replay"} + }), + json!({ + "timestamp":"2026-07-22T00:00:02Z", + "type":"response_item", + "payload":{"type":"function_call","name":"exec"} + }), + ], + ); + + repair_codex_catalog_titles(&mut conn, &[discovered]).expect("repair from first user prompt"); + + assert_eq!(cached_title(&conn).0, "Investigate bounded replay"); + fs::remove_file(path).expect("remove title fixture"); +} + +#[test] +fn verified_replay_title_does_not_reopen_unchanged_jsonl_prefix() { + let (mut conn, discovered, path) = title_repair_fixture( + "Investigate bounded replay", + &[json!({ + "timestamp":"2026-07-22T00:00:01Z", + "type":"event_msg", + "payload":{"type":"user_message","message":"Investigate bounded replay"} + })], + ); + + repair_codex_catalog_titles(&mut conn, std::slice::from_ref(&discovered)) + .expect("verify replay-owned title from first user prompt"); + assert_eq!(cached_title(&conn).0, "Investigate bounded replay"); + let initial_verified_signature = cached_title_verification_signature(&conn) + .expect("verification must live with adapter metadata"); + + // A logical/no-change replay publication rewrites its derivation + // baseline/applied snapshots. Verification must not be stored in that + // disposable lifecycle because compact-index prune deletes it. + publish_title_fixture_projection(&mut conn, &discovered.record, None); + assert_eq!( + cached_title_verification_signature(&conn).as_deref(), + Some(initial_verified_signature.as_str()) + ); + + // Discovery growth advances only physical signature fields. It must + // preserve adapter-owned metadata instead of forcing another prefix + // read on the next unchanged refresh. + let mut advanced_record = discovered.record.clone(); + advanced_record.source_size_bytes += 1; + imported_cache::advance_cached_catalog_record_from_conn( + &conn, + SOURCE_CODEX_APP, + &advanced_record, + None, + ) + .expect("advance Codex discovery signature"); + assert_eq!( + cached_title_verification_signature(&conn).as_deref(), + Some(initial_verified_signature.as_str()) + ); + + let advanced_discovered = DiscoveredCodexCatalogRecord { + record: advanced_record, + authoritative_title: None, + }; + repair_codex_catalog_titles(&mut conn, std::slice::from_ref(&advanced_discovered)) + .expect("changed discovery signature revalidates while JSONL is available"); + let advanced_verified_signature = + cached_title_verification_signature(&conn).expect("advanced verification signature"); + assert_ne!(advanced_verified_signature, initial_verified_signature); + + fs::remove_file(&path).expect("remove title fixture before unchanged refresh"); + repair_codex_catalog_titles(&mut conn, std::slice::from_ref(&advanced_discovered)) + .expect("verified unchanged title must stay on the metadata-only path"); + assert_eq!(cached_title(&conn).0, "Investigate bounded replay"); +} + +#[test] +fn verified_title_survives_replay_baseline_restore_and_prune() { + let (mut conn, discovered, path) = title_repair_fixture( + "Investigate bounded replay", + &[json!({ + "timestamp":"2026-07-22T00:00:01Z", + "type":"event_msg", + "payload":{"type":"user_message","message":"Investigate bounded replay"} + })], + ); + conn.execute( + "UPDATE imported_history_session_cache SET model='adapter-model' + WHERE source='codex_app' AND source_session_id='rollout-title-fixture'", + [], + ) + .expect("seed adapter-owned baseline"); + + repair_codex_catalog_titles(&mut conn, std::slice::from_ref(&discovered)) + .expect("verify replay-owned title"); + let verified_signature = + cached_title_verification_signature(&conn).expect("verification signature"); + assert_unrelated_source_metadata_survives(&conn); + publish_title_fixture_projection(&mut conn, &discovered.record, Some("replay-model")); + assert_unrelated_source_metadata_survives(&conn); + let projected_model: String = conn + .query_row( + "SELECT model FROM imported_history_session_cache + WHERE source='codex_app' + AND source_session_id='rollout-title-fixture'", + [], + |row| row.get(0), + ) + .expect("read replay-projected model"); + assert_eq!(projected_model, "replay-model"); + + let tx = conn.transaction().expect("replay prune transaction"); + crate::sources::imported_history::catalog::clear_replay_projection_tx( + &tx, + SOURCE_CODEX_APP, + &discovered.record.source_session_id, + ) + .expect("prune replay catalog projection"); + tx.commit().expect("commit replay projection prune"); + + let restored_model: String = conn + .query_row( + "SELECT model FROM imported_history_session_cache + WHERE source='codex_app' + AND source_session_id='rollout-title-fixture'", + [], + |row| row.get(0), + ) + .expect("read restored adapter model"); + assert_eq!(restored_model, "adapter-model"); + assert_eq!( + cached_title_verification_signature(&conn).as_deref(), + Some(verified_signature.as_str()), + "pruning replay-owned fields must not discard adapter verification" + ); + assert_unrelated_source_metadata_survives(&conn); + + fs::remove_file(&path).expect("remove title fixture after prune"); + repair_codex_catalog_titles(&mut conn, std::slice::from_ref(&discovered)) + .expect("post-prune refresh must not reopen the unchanged JSONL"); + assert_unrelated_source_metadata_survives(&conn); +} + +#[test] +fn tool_only_codex_session_uses_neutral_untitled_name() { + let (mut conn, discovered, path) = title_repair_fixture( + "js", + &[json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"response_item", + "payload":{"type":"custom_tool_call","name":"js"} + })], + ); + + repair_codex_catalog_titles(&mut conn, &[discovered]).expect("repair tool-only Codex title"); + + assert_eq!(cached_title(&conn).0, "Untitled"); + fs::remove_file(path).expect("remove title fixture"); +} + +#[test] +fn legacy_record_key_placeholder_repairs_without_a_replay_derivation() { + let (mut conn, discovered, path) = title_repair_fixture("rollout-title-fixture", &[]); + conn.execute( + "DELETE FROM imported_replay_catalog_derivations + WHERE source='codex_app'", + [], + ) + .expect("remove replay derivation"); + + repair_codex_catalog_titles(&mut conn, &[discovered]) + .expect("repair legacy record-key placeholder"); + + assert_eq!(cached_title(&conn).0, "Untitled"); + fs::remove_file(path).expect("remove title fixture"); +} + +#[test] +fn unchanged_untitled_session_does_not_reopen_its_transcript() { + let (mut conn, discovered, path) = title_repair_fixture("Untitled", &[]); + conn.execute( + "DELETE FROM imported_replay_catalog_derivations + WHERE source='codex_app'", + [], + ) + .expect("remove replay derivation"); + fs::remove_file(&path).expect("remove title fixture before refresh"); + + repair_codex_catalog_titles(&mut conn, &[discovered]) + .expect("unchanged neutral title must stay on the metadata-only path"); + + assert_eq!(cached_title(&conn).0, "Untitled"); +} + +#[test] +fn links_spawn_chunk_to_matching_codex_child_and_restores_prompt() { + let mut chunks = vec![ + ActivityChunk::new("codexapp-parent", "tool_call", "subagent").with_args(json!({ + "task_name": "audit_todays_commits", + "description": "audit_todays_commits", + "codexAgentThreadId": "019f-audit" + })), + ]; + chunks[0].created_at = "2026-07-23T10:18:52Z".to_string(); + let mut children = vec![ + CodexChildSessionLink { + session_id: "codexapp-wrong-nearby-child".to_string(), + thread_id: Some("019f-wrong".to_string()), + created_at_ms: 1_753_265_932_100, + metadata: CodexAppSourceMetadata { + first_prompt: Some("wrong prompt".to_string()), + agent_path: Some("/root/other_task".to_string()), + agent_nickname: Some("Wrong".to_string()), + }, + }, + CodexChildSessionLink { + session_id: "codexapp-audit-child".to_string(), + thread_id: Some("019f-audit".to_string()), + created_at_ms: 1_753_265_940_000, + metadata: CodexAppSourceMetadata { + first_prompt: Some("audit today's commit history".to_string()), + agent_path: Some("/root/audit_todays_commits".to_string()), + agent_nickname: Some("Peirce".to_string()), + }, + }, + ]; + + link_codex_subagent_chunks_from_children(&mut chunks, &mut children); + + assert_eq!(chunks[0].args["subagentSessionId"], "codexapp-audit-child"); + assert_eq!(chunks[0].args["prompt"], "audit today's commit history"); + assert_eq!(chunks[0].args["subagent_type"], "Peirce"); + assert_eq!(chunks[0].args["action"], "delegate"); + assert_eq!(children.len(), 1); + assert_eq!(children[0].session_id, "codexapp-wrong-nearby-child"); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app/meta.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app/meta.rs index ccac83233..ec7420941 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/codex/app/meta.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app/meta.rs @@ -1,9 +1,14 @@ //! Codex session-meta parsing and parent-thread resolution. +#[cfg(test)] use std::collections::BTreeSet; +#[cfg(test)] +use std::fs; use std::path::{Path, PathBuf}; -use serde::{Deserialize, Serialize}; +use serde::Deserialize; +#[cfg(test)] +use serde::Serialize; use serde_json::Value; use crate::sources::codex::canonical_session_id; @@ -11,20 +16,34 @@ use crate::sources::imported_history::{ self, metadata::{ ImportedHistoryCacheInput, ImportedHistoryDiscoveredRecord, ImportedHistoryImpactStats, - StoredRoundUsage, SOURCE_CODEX_APP, + SOURCE_CODEX_APP, }, +}; +#[cfg(test)] +use crate::sources::imported_history::{ + metadata::StoredRoundUsage, watermark::{ImportedParseWatermark, WatermarkedTranscriptReader}, }; +#[cfg(test)] use super::impact::{collect_codex_impact_from_patch_apply_end, collect_codex_impact_from_payload}; +#[cfg(test)] +use super::index::codex_session_index_title_for_record; use super::index::{ codex_sessions_dir_for_session_path, codex_thread_id_from_file_stem, collect_codex_session_files, }; use super::transcript::user_message_from_payload; -use super::{ - CodexAppSessionMeta, CodexAppSourceMetadata, CodexJsonlLine, CODEX_APP_METADATA_PARSER_VERSION, -}; +#[cfg(test)] +use super::CodexAppSessionMeta; +use super::{CodexAppSourceMetadata, CodexJsonlLine, CODEX_APP_METADATA_PARSER_VERSION}; + +/// Catalog discovery only needs stable card metadata. Capping the prefix is +/// intentional: an appended 300 MiB rollout must not make a sidebar rescan +/// walk 300 MiB again. Exact turns, tokens, and impact come from the persistent +/// bounded replay index. +const CODEX_CATALOG_PREFIX_BYTES: u64 = 1024 * 1024; +const CODEX_UNTITLED_SESSION_NAME: &str = "Untitled"; #[derive(Debug, Clone, PartialEq, Eq)] pub struct CodexTranscriptLocator { @@ -41,6 +60,7 @@ struct CodexTurnContextPayload { model: String, } +#[cfg(test)] /// Resumable accumulator for one rollout's meta scan. Every field is exactly /// the per-file state the old single-pass loop kept in locals, so it can be /// frozen into a parse watermark's `state_json` at a complete-line boundary @@ -81,6 +101,7 @@ struct CodexSessionMetaState { source_metadata: CodexAppSourceMetadata, } +#[cfg(test)] impl CodexSessionMetaState { fn feed(&mut self, trimmed: &str, record: &ImportedHistoryDiscoveredRecord) { let parsed: CodexJsonlLine = match serde_json::from_str(trimmed) { @@ -218,7 +239,7 @@ impl CodexSessionMetaState { let name = if !title.is_empty() { title } else if self.first_prompt.is_empty() { - record.source_record_key.clone() + CODEX_UNTITLED_SESSION_NAME.to_string() } else { imported_history::truncate_name(&self.first_prompt, 200) }; @@ -269,6 +290,7 @@ impl CodexSessionMetaState { } } +#[cfg(test)] pub(crate) struct CodexSessionMetaParse { pub meta: Option, pub watermark: ImportedParseWatermark, @@ -276,10 +298,10 @@ pub(crate) struct CodexSessionMetaParse { pub resumed: bool, } -pub(crate) fn parse_codex_session_meta_with_title( +#[cfg(test)] +pub(crate) fn parse_codex_session_meta_incremental( record: &ImportedHistoryDiscoveredRecord, watermark: Option<&ImportedParseWatermark>, - external_title: String, ) -> Result { let mut reader = WatermarkedTranscriptReader::open( &record.source_path, @@ -331,9 +353,8 @@ pub(crate) fn parse_codex_session_meta_with_title( record.source_size_bytes, state_json, ); - let meta = tail_state - .unwrap_or(state) - .finish(record, external_title); + let external_title = codex_session_index_title_for_record(record)?; + let meta = tail_state.unwrap_or(state).finish(record, external_title); Ok(CodexSessionMetaParse { meta, watermark: next_watermark, @@ -342,51 +363,138 @@ pub(crate) fn parse_codex_session_meta_with_title( } #[cfg(test)] -pub(crate) fn parse_codex_session_meta_incremental( +pub(crate) fn parse_codex_session_meta( record: &ImportedHistoryDiscoveredRecord, - watermark: Option<&ImportedParseWatermark>, -) -> Result { - let external_title = super::index::codex_session_index_title_for_record(record)?; - parse_codex_session_meta_with_title(record, watermark, external_title) +) -> Result, String> { + Ok(parse_codex_session_meta_incremental(record, None)?.meta) } #[cfg(test)] -pub(crate) fn parse_codex_session_meta( +pub(super) fn parse_codex_catalog_input( record: &ImportedHistoryDiscoveredRecord, -) -> Result, String> { - Ok(parse_codex_session_meta_incremental(record, None)?.meta) +) -> Result, String> { + let external_title = codex_session_index_title_for_record(record)?; + parse_codex_catalog_input_with_title(record, Some(&external_title)) } -pub(super) fn session_meta_to_cache_input(meta: CodexAppSessionMeta) -> ImportedHistoryCacheInput { - let source_metadata_json = meta - .parent_session_id +pub(super) fn parse_codex_catalog_input_with_title( + record: &ImportedHistoryDiscoveredRecord, + authoritative_title: Option<&str>, +) -> Result, String> { + let mut created_at_ms = 0; + let mut external_title = authoritative_title + .map(str::trim) + .filter(|title| !title.is_empty()) + .map(ToString::to_string) + .unwrap_or_default(); + let mut first_prompt = String::new(); + let mut model = None; + let mut repo_path = None; + let mut parent_thread_id = None; + let mut source_metadata = CodexAppSourceMetadata::default(); + + for line in imported_history::read_complete_jsonl_prefix_lines( + &record.source_path, + "Codex", + CODEX_CATALOG_PREFIX_BYTES, + )? { + let Ok(parsed) = serde_json::from_str::(line.trim()) else { + continue; + }; + if created_at_ms == 0 { + created_at_ms = parsed + .timestamp + .as_deref() + .and_then(imported_history::parse_iso_to_epoch_ms_opt) + .unwrap_or_default(); + } + if first_prompt.is_empty() { + if let Some(message) = user_message_from_payload(&parsed.payload) { + first_prompt = message; + } + } + if external_title.is_empty() && parsed.line_type == "session_meta" { + if let Some(title) = session_title_from_payload(&parsed.payload) { + external_title = imported_history::truncate_name(&title, 200); + } + } + if parent_thread_id.is_none() && parsed.line_type == "session_meta" { + parent_thread_id = parent_thread_id_from_session_meta_payload( + &parsed.payload, + codex_thread_id_from_file_stem(&record.source_record_key), + ); + } + if parsed.line_type == "session_meta" { + capture_subagent_source_metadata(&parsed.payload, &mut source_metadata); + } + if model.is_none() || repo_path.is_none() { + if let Ok(turn_context) = + serde_json::from_value::(parsed.payload) + { + if model.is_none() && !turn_context.model.trim().is_empty() { + model = Some(turn_context.model); + } + if repo_path.is_none() && !turn_context.cwd.trim().is_empty() { + repo_path = Some(turn_context.cwd); + } + } + } + if !first_prompt.is_empty() + && model.is_some() + && repo_path.is_some() + && parent_thread_id.is_some() + { + break; + } + } + + let source_time_ms = record.source_mtime_ms.saturating_div(1_000_000); + if created_at_ms == 0 && source_time_ms == 0 { + return Ok(None); + } + let name = if !external_title.is_empty() { + external_title + } else if !first_prompt.is_empty() { + imported_history::truncate_name(&first_prompt, 200) + } else { + CODEX_UNTITLED_SESSION_NAME.to_string() + }; + source_metadata.first_prompt = (!first_prompt.trim().is_empty()).then_some(first_prompt); + let parent_session_id = parent_thread_id + .as_deref() + .and_then(|thread_id| codex_parent_session_id_for_record(record, thread_id)); + let source_metadata_json = parent_session_id .as_ref() - .and_then(|_| serde_json::to_string(&meta.source_metadata).ok()); - ImportedHistoryCacheInput { + .and_then(|_| serde_json::to_string(&source_metadata).ok()); + Ok(Some(ImportedHistoryCacheInput { source: SOURCE_CODEX_APP, - source_session_id: meta.source_session_id, - session_id: meta.session_id, - source_path: meta.source_path, - source_record_key: meta.source_record_key, - source_mtime_ms: meta.source_mtime_ms, - source_size_bytes: meta.source_size_bytes, - source_fingerprint: meta.source_fingerprint, + source_session_id: record.source_session_id.clone(), + session_id: canonical_session_id(&record.source_session_id), + source_path: record.source_path.to_string_lossy().to_string(), + source_record_key: record.source_record_key.clone(), + source_mtime_ms: record.source_mtime_ms, + source_size_bytes: record.source_size_bytes, + source_fingerprint: record.source_fingerprint.clone(), parser_version: CODEX_APP_METADATA_PARSER_VERSION, - name: meta.name, - created_at_ms: meta.created_at_ms, - updated_at_ms: meta.updated_at_ms, - model: meta.model, - input_tokens: meta.input_tokens, - output_tokens: meta.output_tokens, - cache_read_tokens: meta.cache_read_tokens, - cache_write_tokens: meta.cache_write_tokens, - repo_path: meta.repo_path, + name, + created_at_ms: if created_at_ms > 0 { + created_at_ms + } else { + source_time_ms + }, + updated_at_ms: source_time_ms.max(created_at_ms), + model, + input_tokens: 0, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + repo_path, branch: None, - impact: meta.impact, + impact: ImportedHistoryImpactStats::default(), listable: true, source_metadata_json, - parent_session_id: meta.parent_session_id, - } + parent_session_id, + })) } fn capture_subagent_source_metadata(payload: &Value, metadata: &mut CodexAppSourceMetadata) { @@ -516,3 +624,142 @@ fn session_title_from_payload(payload: &Value) -> Option { .find(|value| !value.is_empty()) .map(ToString::to_string) } + +#[cfg(test)] +mod catalog_tests { + use std::io::Write; + + use super::*; + + #[test] + fn catalog_prefix_is_bounded_for_a_thirty_mib_rollout() { + let path = std::env::temp_dir().join(format!( + "orgii-codex-catalog-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let mut file = fs::File::create(&path).expect("create Codex catalog fixture"); + writeln!( + file, + "{}", + serde_json::json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"session_meta", + "payload":{"title":"bounded catalog","cwd":"/work/orgii","model":"gpt-5"} + }) + ) + .expect("write metadata line"); + writeln!( + file, + "{}", + serde_json::json!({ + "timestamp":"2026-07-22T00:00:01Z", + "type":"event_msg", + "payload":{"type":"user_message","message":"first prompt"} + }) + ) + .expect("write prompt line"); + let block = vec![b'x'; 64 * 1024]; + for _ in 0..(30 * 1024 / 64) { + file.write_all(&block).expect("extend large rollout"); + } + file.flush().expect("flush fixture"); + let size = file.metadata().expect("fixture metadata").len(); + drop(file); + assert!(size > 30 * 1024 * 1024 - 1024); + + let record = ImportedHistoryDiscoveredRecord { + source_session_id: "catalog-large".to_string(), + source_path: path.clone(), + source_record_key: "catalog-large".to_string(), + source_mtime_ms: 1_774_137_600_000_000_000, + source_size_bytes: size as i64, + source_fingerprint: String::new(), + parser_version: CODEX_APP_METADATA_PARSER_VERSION, + }; + let input = parse_codex_catalog_input(&record) + .expect("parse bounded catalog") + .expect("catalog row"); + assert_eq!(input.name, "bounded catalog"); + assert_eq!(input.repo_path.as_deref(), Some("/work/orgii")); + assert_eq!(input.model.as_deref(), Some("gpt-5")); + assert_eq!(input.input_tokens, 0); + assert!(CODEX_CATALOG_PREFIX_BYTES < size); + + fs::remove_file(path).expect("remove fixture"); + } + + #[test] + fn catalog_reads_complete_final_record_without_a_trailing_newline() { + let path = std::env::temp_dir().join(format!( + "orgii-codex-catalog-no-newline-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let record_json = serde_json::json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"session_meta", + "payload":{"title":"natural EOF Codex catalog","cwd":"/work/orgii"} + }) + .to_string(); + fs::write(&path, &record_json).expect("write no-newline Codex fixture"); + let record = ImportedHistoryDiscoveredRecord { + source_session_id: "catalog-no-newline".to_string(), + source_path: path.clone(), + source_record_key: "catalog-no-newline".to_string(), + source_mtime_ms: 1_774_137_600_000_000_000, + source_size_bytes: record_json.len() as i64, + source_fingerprint: String::new(), + parser_version: CODEX_APP_METADATA_PARSER_VERSION, + }; + + let input = parse_codex_catalog_input(&record) + .expect("parse no-newline Codex catalog") + .expect("Codex catalog row"); + + assert_eq!(input.name, "natural EOF Codex catalog"); + fs::remove_file(path).expect("remove no-newline Codex fixture"); + } + + #[test] + fn catalog_uses_untitled_when_only_tool_and_system_records_exist() { + let path = std::env::temp_dir().join(format!( + "orgii-codex-catalog-tool-only-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let body = [ + serde_json::json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"session_meta", + "payload":{"cwd":"/work/orgii"} + }), + serde_json::json!({ + "timestamp":"2026-07-22T00:00:01Z", + "type":"response_item", + "payload":{"type":"custom_tool_call","name":"update_plan"} + }), + ] + .into_iter() + .map(|value| value.to_string()) + .collect::>() + .join("\n"); + fs::write(&path, format!("{body}\n")).expect("write tool-only Codex fixture"); + let record = ImportedHistoryDiscoveredRecord { + source_session_id: "catalog-tool-only".to_string(), + source_path: path.clone(), + source_record_key: "catalog-tool-only".to_string(), + source_mtime_ms: 1_774_137_600_000_000_000, + source_size_bytes: body.len() as i64 + 1, + source_fingerprint: String::new(), + parser_version: CODEX_APP_METADATA_PARSER_VERSION, + }; + + let input = parse_codex_catalog_input_with_title(&record, None) + .expect("parse tool-only Codex catalog") + .expect("Codex catalog row"); + + assert_eq!(input.name, "Untitled"); + fs::remove_file(path).expect("remove tool-only Codex fixture"); + } +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app/mod.rs index a79f17f69..6e37d36cc 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/codex/app/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app/mod.rs @@ -8,24 +8,25 @@ use serde::{Deserialize, Serialize}; use serde_json::Value; +#[cfg(test)] +use crate::sources::imported_history::metadata::{ImportedHistoryImpactStats, RoundUsage}; use crate::sources::imported_history::{ - metadata::{ImportedHistoryImpactStats, RoundUsage}, ImportedHistoryRecentPath, ImportedHistorySessionPage, ImportedHistorySessionRow, }; -mod desktop_exec; +pub(crate) mod desktop_exec; mod impact; mod index; mod meta; mod normalize; -mod transcript; +pub(crate) mod transcript; // Public API — preserved at `...::sources::codex::app::*`. +pub(crate) use index::refresh_catalog; pub use index::{ codex_thread_id_from_file_stem, list_codex_app_recent_paths, list_codex_app_reconciliation_sessions, list_codex_app_sessions_paginated, - load_codex_app_for_session, load_codex_app_initial_window_for_session, - load_codex_app_turn_for_session, + load_codex_app_for_session, }; pub use meta::{resolve_codex_transcript_for_thread_id_near_path, CodexTranscriptLocator}; pub(crate) use normalize::normalize_codex_tool_calls; @@ -66,15 +67,20 @@ pub type CodexAppSessionPage = ImportedHistorySessionPage; pub type CodexAppRecentPath = ImportedHistoryRecentPath; #[derive(Debug, Deserialize)] -struct CodexJsonlLine { +pub(crate) struct CodexJsonlLine { #[serde(default)] - timestamp: Option, + pub(crate) timestamp: Option, #[serde(default, rename = "type")] - line_type: String, + pub(crate) line_type: String, #[serde(default)] - payload: Value, + pub(crate) payload: Value, } +#[cfg(test)] +#[expect( + dead_code, + reason = "the test-only compatibility parser preserves the complete legacy metadata shape while individual regression cases assert focused subsets" +)] #[derive(Debug, Clone)] pub(crate) struct CodexAppSessionMeta { source_session_id: String, diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app/transcript.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app/transcript.rs index aa3db575d..4b156b928 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/codex/app/transcript.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app/transcript.rs @@ -749,6 +749,10 @@ fn maybe_record_codex_user_offset( } } +#[allow( + clippy::type_complexity, + reason = "Internal collector returns the existing chunks, turn metadata, and offset cache as one atomic result" +)] fn load_codex_app_from_path_with_mode<'a>( session_id: &'a str, path: &Path, @@ -1237,7 +1241,7 @@ fn resolve_write_stdin_call( ); } -fn record_stdin_event(calls: &mut [ImportedToolCall], continuation: &ImportedToolCall) { +pub(crate) fn record_stdin_event(calls: &mut [ImportedToolCall], continuation: &ImportedToolCall) { let chars = continuation .args .get("chars") @@ -1409,15 +1413,15 @@ fn is_codex_script_wrapper_text(text: &str) -> bool { || trimmed.starts_with("Script error") } -fn background_cell_key(cell_id: &str) -> String { +pub(crate) fn background_cell_key(cell_id: &str) -> String { format!("cell:{cell_id}") } -fn background_session_key(session_id: &str) -> String { +pub(crate) fn background_session_key(session_id: &str) -> String { format!("session:{session_id}") } -fn background_cell_id(output: &str) -> Option { +pub(crate) fn background_cell_id(output: &str) -> Option { output.lines().find_map(|line| { line.trim() .strip_prefix("Script running with cell ID ") @@ -1427,7 +1431,7 @@ fn background_cell_id(output: &str) -> Option { }) } -fn wait_cell_id(calls: &[ImportedToolCall]) -> Option<&str> { +pub(crate) fn wait_cell_id(calls: &[ImportedToolCall]) -> Option<&str> { let [call] = calls else { return None; }; @@ -1437,7 +1441,7 @@ fn wait_cell_id(calls: &[ImportedToolCall]) -> Option<&str> { call.args.get("cell_id").and_then(Value::as_str) } -fn codex_tool_call_chunk( +pub(crate) fn codex_tool_call_chunk( session_id: &str, sequence: usize, call: &ImportedToolCall, @@ -1532,7 +1536,7 @@ fn read_line_limit_from_call(call: &ImportedToolCall) -> Option { .and_then(|value| usize::try_from(value).ok()) } -fn pending_tool_calls_from_payload( +pub(crate) fn pending_tool_calls_from_payload( payload: &Value, created_at: &str, ) -> Option<(String, Vec)> { @@ -1600,7 +1604,10 @@ pub(crate) fn pending_custom_tool_calls_from_payload( Some((call_id, calls)) } -fn web_search_call_from_payload(payload: &Value, created_at: &str) -> Option { +pub(crate) fn web_search_call_from_payload( + payload: &Value, + created_at: &str, +) -> Option { let call_id = payload.get("id")?.as_str()?.to_string(); let action = payload.get("action").cloned().unwrap_or_else(|| json!({})); Some(ImportedToolCall { @@ -1647,7 +1654,7 @@ pub(crate) fn user_message_from_payload(payload: &Value) -> Option { Some(stripped.to_string()) } -fn content_text_from_payload(payload: &Value) -> Option { +pub(crate) fn content_text_from_payload(payload: &Value) -> Option { let content = payload.get("content")?; match content { Value::String(text) => Some(text.clone()), @@ -1673,7 +1680,7 @@ fn content_part_text(part: &Value) -> Option { .map(ToString::to_string) } -fn reasoning_text_from_payload(payload: &Value) -> Option { +pub(crate) fn reasoning_text_from_payload(payload: &Value) -> Option { if let Some(text) = payload.get("content").and_then(Value::as_str) { if !text.trim().is_empty() { return Some(text.to_string()); diff --git a/src-tauri/crates/orgtrack-core/src/sources/codex/app_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/codex/app_tests.rs index e991d2688..b9338018f 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/codex/app_tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/codex/app_tests.rs @@ -1815,7 +1815,9 @@ fn maps_codex_subagent_parent_thread_to_parent_session_id() { meta.source_metadata.agent_path.as_deref(), Some("/root/inspect_session_naming") ); - let cache_input = meta::session_meta_to_cache_input(meta); + let cache_input = meta::parse_codex_catalog_input(&record) + .expect("parse compact catalog input") + .expect("catalog input"); let cached_metadata: Value = serde_json::from_str( cache_input .source_metadata_json @@ -2039,7 +2041,10 @@ fn resumes_codex_meta_parse_from_watermark() { let scratch_meta = scratch.meta.expect("scratch meta"); assert_eq!(resumed_meta.input_tokens, scratch_meta.input_tokens); assert_eq!(resumed_meta.output_tokens, scratch_meta.output_tokens); - assert_eq!(resumed_meta.cache_read_tokens, scratch_meta.cache_read_tokens); + assert_eq!( + resumed_meta.cache_read_tokens, + scratch_meta.cache_read_tokens + ); assert_eq!( resumed_meta.cache_write_tokens, scratch_meta.cache_write_tokens diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/mod.rs index 9c7953a3d..d1ae8960d 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/mod.rs @@ -72,6 +72,7 @@ const CURSOR_CLI_PROVIDER_SLUG: &str = "cursorcli"; /// (`ModelType::CursorCli`). const CURSOR_CLI_AGENT_TYPE: &str = "cursor_cli"; const CURSOR_CLI_METADATA_PARSER_VERSION: i64 = 1; +const CURSOR_CLI_CATALOG_BLOB_BYTES: i64 = 1024 * 1024; const STORE_FILENAME: &str = "store.db"; /// The store's placeholder session name; real titles only exist when the user /// renames the agent, so the placeholder yields to the first prompt. @@ -201,6 +202,10 @@ pub fn cursor_cli_history_candidate_paths() -> Vec { imported_paths::dedupe_paths(roots) } +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + sync_cursor_cli_history_cache(conn) +} + fn sync_cursor_cli_history_cache(conn: &mut Connection) -> Result<(), String> { let mut discovered = discover_cursor_cli_history_records()?; // Managed (GUI-launched) sessions surface through their code_sessions @@ -229,6 +234,15 @@ fn sync_cursor_cli_history_cache(conn: &mut Connection) -> Result<(), String> { )?; let mut inputs = Vec::new(); for record in changed { + let is_managed = managed_ids.contains(&record.source_session_id); + if imported_cache::advance_cached_catalog_record_from_conn( + conn, + SOURCE_CURSOR_CLI, + record, + Some(!is_managed), + )? { + continue; + } // A single locked/corrupt per-session store must not hide every other // session, so unreadable stores are skipped rather than failing the // whole source sync; the unchanged signature retries them next scan. @@ -315,9 +329,8 @@ fn session_meta_from_store_conn( let manifest = read_store_manifest(store_conn, &store_meta.latest_root_blob_id)?.unwrap_or_default(); let session_id = super::canonical_session_id(&record.source_session_id); - let chunks = load_history_from_store_conn(store_conn, &session_id)?; - let impact = imported_history::impact_from_edit_chunks(&chunks); - let first_prompt = first_user_prompt_from_chunks(&chunks); + let (first_prompt, impact) = + scan_manifest_catalog(store_conn, &session_id, &manifest, store_meta.created_at)?; // A user-set agent name wins; the store's "New Agent" placeholder yields // to the first prompt, then to the raw session uuid. @@ -386,22 +399,6 @@ fn session_meta_to_cache_input(meta: CursorCliHistoryMeta) -> ImportedHistoryCac } } -fn first_user_prompt_from_chunks(chunks: &[ActivityChunk]) -> Option { - chunks - .iter() - .find(|chunk| chunk.function == imported_history::FUNCTION_USER_MESSAGE) - .and_then(|chunk| { - chunk - .result - .get("message") - .and_then(|message| message.get("content")) - .and_then(Value::as_str) - }) - .map(str::trim) - .filter(|text| !text.is_empty()) - .map(str::to_string) -} - #[cfg(test)] #[path = "../history_tests.rs"] mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/store.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/store.rs index 16311e1e8..3abcdf2a6 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/store.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/store.rs @@ -94,6 +94,21 @@ pub(super) fn read_blob(conn: &Connection, blob_id: &str) -> Result Result>, String> { + conn.query_row( + "SELECT CASE WHEN length(data) <= ?2 THEN data ELSE NULL END + FROM blobs WHERE id = ?1", + rusqlite::params![blob_id, CURSOR_CLI_CATALOG_BLOB_BYTES], + |row| row.get::<_, Option>>(0), + ) + .optional() + .map(|value| value.flatten()) + .map_err(|err| format!("Failed to read compact Cursor CLI blob {blob_id}: {err}")) +} + pub(super) fn read_store_manifest( conn: &Connection, root_blob_id: &str, diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/transcript.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/transcript.rs index 8b4cac080..d7787d9ef 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/transcript.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_cli/history/transcript.rs @@ -130,6 +130,59 @@ pub(super) fn load_history_from_store_conn( Ok(chunks) } +pub(super) fn scan_manifest_catalog( + store_conn: &Connection, + session_id: &str, + manifest: &CursorStoreManifest, + created_at_ms: i64, +) -> Result<(Option, ImportedHistoryImpactStats), String> { + let created_at = imported_history::epoch_ms_to_iso(created_at_ms); + let mut first_prompt = None; + let mut touched_files = std::collections::BTreeSet::new(); + let mut impact = ImportedHistoryImpactStats::default(); + for blob_id in &manifest.message_blob_ids { + let Some(data) = read_catalog_blob(store_conn, blob_id)? else { + continue; + }; + let Ok(message) = serde_json::from_slice::(&data) else { + continue; + }; + match message.get("role").and_then(Value::as_str) { + Some("user") if first_prompt.is_none() => { + first_prompt = clean_user_text(&message_content_text(message.get("content"))); + } + Some("assistant") => { + for item in message_content_items(message.get("content")) { + if item.get("type").and_then(Value::as_str) != Some("tool-call") { + continue; + } + let Some(call) = tool_call_from_item(item, &created_at) else { + continue; + }; + if call.canonical_name != imported_history::FUNCTION_EDIT_FILE { + continue; + } + let chunk = imported_history::tool_call_chunk( + session_id, + CURSOR_CLI_PROVIDER_SLUG, + 0, + &call, + "", + ); + let one = imported_history::impact_from_edit_chunks(&[chunk]); + impact.lines_added = impact.lines_added.saturating_add(one.lines_added); + impact.lines_removed = impact.lines_removed.saturating_add(one.lines_removed); + touched_files.extend(one.touched_files); + } + } + _ => {} + } + } + impact.touched_files = touched_files.into_iter().collect(); + impact.files_changed = impact.touched_files.len() as i64; + Ok((first_prompt, impact)) +} + fn message_content_text(content: Option<&Value>) -> String { match content { Some(Value::String(text)) => text.clone(), diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/mod.rs index e7b083ccd..270c02879 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/mod.rs @@ -18,8 +18,8 @@ use crate::sources::imported_history::{ }; use super::io::{ - cursor_conversation_index_path, cursor_db_path, open_cursor_conversation_index_db, - open_cursor_db, + cursor_db_path_state, open_cursor_conversation_index_db_state, open_cursor_db, + CursorConversationIndexDbState, CursorDbPathState, }; use super::CURSORIDE_SESSION_PREFIX; // Re-exported so submodule code moved out of this file keeps resolving its @@ -32,12 +32,19 @@ mod sync; use sync::delta_sync; #[cfg(test)] -use sync::{build_inputs_from_index, discover_from_index}; +use sync::{ + build_inputs_from_index, cursor_content_probe_count, cursor_content_probed_ids, + delta_sync_from_connections, demote_definitively_missing_cursor_database, discover_from_index, + preferred_cursor_title, reset_cursor_content_probe_count, CursorStorageSnapshot, + CURSOR_STORAGE_MISSING_IDENTITY, INDEX_BLOB_VALIDATION_FIELD, + NO_INDEX_ACTIVITY_SIGNATURE_FIELD, NO_INDEX_DATABASE_IDENTITY_FIELD, + NO_INDEX_VALIDATION_BATCH_SIZE, +}; -// v6: top-level index rows now bring their `subagentComposerIds` into the cache -// as child sessions with `parent_session_id`, allowing the shared sidebar -// parent/child collapse flow to render Cursor subagents. -const CURSOR_IDE_METADATA_PARSER_VERSION: i64 = 6; +// v7: a top-level composer is listable only after a bounded content check +// confirms a real user bubble. This migrates earlier empty composer shells out +// of the sidebar and derives a title for valid untitled sessions. +const CURSOR_IDE_METADATA_PARSER_VERSION: i64 = 7; const COMPOSER_KEY_PREFIX: &str = "composerData:"; const BUBBLE_KEY_PREFIX: &str = "bubbleId:"; const SOURCE_RECORD_KEY_PREFIX: &str = "cursorDiskKV:"; @@ -93,6 +100,8 @@ struct RawComposerData { struct BubbleHeader { #[serde(default)] bubble_id: String, + #[serde(default, rename = "type")] + bubble_type: i64, } #[derive(Debug, Deserialize)] @@ -115,6 +124,31 @@ struct CursorCacheMetadata { status: String, is_agentic: bool, mode: String, + /// Stable main-database identity used by the no-conversation-index + /// compatibility path. WAL activity is intentionally separate: normal + /// Cursor writes must never reparse every visible cached root. + #[serde(default, skip_serializing_if = "Option::is_none")] + no_index_database_identity: Option, + /// Main/WAL activity watermark used only to retry hidden/missing roots in + /// bounded batches. + #[serde(default, skip_serializing_if = "Option::is_none")] + no_index_activity_signature: Option, + /// Validation guard for a conversation-index row whose composer blob is + /// absent from `state.vscdb`. A successful point-read that proves the blob + /// missing or malformed hides the cached card immediately; transient + /// SQLite/open failures do not stamp this field. Physical DB movement + /// changes the signature and permits promotion without scanning others. + #[serde(default, skip_serializing_if = "Option::is_none")] + index_blob_validation: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +struct CursorIndexBlobValidation { + signature: String, + misses: u8, + #[serde(default)] + database_identity: String, } /// One row of Cursor's `conversation-search.db` `conversations` table — the @@ -132,6 +166,14 @@ struct CursorParentBuild { inputs: Vec, live_child_ids: Vec, child_list_authoritative: bool, + composer_availability: CursorComposerAvailability, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum CursorComposerAvailability { + Available, + MissingOrMalformed, + TemporarilyUnavailable, } impl CursorIndexRow { @@ -201,6 +243,10 @@ pub fn get_cursor_sessions( .collect() } +pub(crate) fn refresh_catalog(cache_conn: &mut Connection) -> Result<(), String> { + delta_sync(cache_conn) +} + pub fn list_for_sidebar( cache_conn: &mut Connection, limit: usize, diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/sync.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/sync.rs index 4e61fb290..a456e599c 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/sync.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/sync.rs @@ -1,7 +1,11 @@ //! Delta-sync pipeline: discover changed conversations from Cursor's index and //! materialize the changed few into normalized cache rows. +#[cfg(test)] +use std::cell::{Cell, RefCell}; use std::collections::{HashMap, HashSet}; +use std::fs; +use std::path::Path; use rusqlite::{params, Connection, OptionalExtension}; @@ -12,33 +16,119 @@ use crate::sources::imported_history::{ use super::*; +pub(super) const CURSOR_STORAGE_MISSING_IDENTITY: &str = "state-vscdb:missing"; +pub(super) const NO_INDEX_DATABASE_IDENTITY_FIELD: &str = "noIndexDatabaseIdentity"; +pub(super) const NO_INDEX_ACTIVITY_SIGNATURE_FIELD: &str = "noIndexActivitySignature"; +pub(super) const INDEX_BLOB_VALIDATION_FIELD: &str = "indexBlobValidation"; +pub(super) const NO_INDEX_VALIDATION_BATCH_SIZE: usize = 64; + +#[cfg(test)] +thread_local! { + static CURSOR_CONTENT_PROBE_COUNT: Cell = const { Cell::new(0) }; + static CURSOR_CONTENT_PROBED_IDS: RefCell> = const { RefCell::new(Vec::new()) }; +} + +#[cfg(test)] +pub(super) fn reset_cursor_content_probe_count() { + CURSOR_CONTENT_PROBE_COUNT.set(0); + CURSOR_CONTENT_PROBED_IDS.with(|ids| ids.borrow_mut().clear()); +} + +#[cfg(test)] +pub(super) fn cursor_content_probe_count() -> usize { + CURSOR_CONTENT_PROBE_COUNT.get() +} + +#[cfg(test)] +pub(super) fn cursor_content_probed_ids() -> Vec { + CURSOR_CONTENT_PROBED_IDS.with(|ids| ids.borrow().clone()) +} + +enum CursorComposerLoad { + Present(Box), + Missing, + Malformed, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct CursorStorageSnapshot { + pub(super) database_identity: String, + pub(super) activity_signature: String, +} + /// Refresh the Cursor metadata cache from `conversation-search.db`. /// /// A cheap indexed read yields per-session change signatures (`updated_at` + /// `root_fingerprint`) without parsing any conversation blob, so only /// genuinely-changed sessions are re-read — the same incremental model the /// file-based sources use, and no per-restart scan of the multi-GB `state.vscdb`. -/// If Cursor's conversation index is absent (very old builds), there's simply -/// nothing to sync. +/// +/// Older Cursor builds do not have the optional conversation index. In that +/// case we do not discover by scanning `state.vscdb`; we only point-check +/// already-cached rows that need the current listability migration. pub(super) fn delta_sync(cache_conn: &mut Connection) -> Result<(), String> { - let Some(index_conn) = open_cursor_conversation_index_db() else { + let cursor_path = match cursor_db_path_state() { + CursorDbPathState::Present(path) => path, + CursorDbPathState::Missing => { + return demote_definitively_missing_cursor_database(cache_conn) + } + CursorDbPathState::Unknown => return Ok(()), + }; + let storage_snapshot = match cursor_storage_snapshot(&cursor_path) { + Ok(snapshot) => snapshot, + // Metadata/TCC failures are indeterminate. Do not hide or stamp + // anything on evidence we could not read. + Err(_) => return Ok(()), + }; + let Some(cursor_conn) = open_cursor_db() else { + // An existing path that cannot be opened is indeterminate (TCC, + // SQLITE_BUSY, permissions, or a replacement in progress). Keep the + // last known projection and leave validation markers untouched. return Ok(()); }; - // A missing/foreign `conversations` table degrades to "no sessions" rather - // than failing the whole session list. - let discovered = discover_from_index(&index_conn).unwrap_or_default(); - - // Content lives in `state.vscdb`; open it only to parse the changed few. Its - // path is the session's store path even when we can't open it (cloud rows). - let cursor_conn = open_cursor_db(); - let source_path = cursor_db_path() - .or_else(cursor_conversation_index_path) - .map(|path| path.to_string_lossy().to_string()) - .unwrap_or_default(); + let index_conn = match open_cursor_conversation_index_db_state() { + CursorConversationIndexDbState::Present(connection) => Some(connection), + CursorConversationIndexDbState::Missing => None, + CursorConversationIndexDbState::Unknown => return Ok(()), + }; + let source_path = cursor_path.to_string_lossy().to_string(); + delta_sync_from_connections( + cache_conn, + index_conn.as_ref(), + Some(&cursor_conn), + &source_path, + Some(&storage_snapshot), + ) +} + +pub(super) fn delta_sync_from_connections( + cache_conn: &mut Connection, + index_conn: Option<&Connection>, + cursor_conn: Option<&Connection>, + source_path: &str, + storage_snapshot: Option<&CursorStorageSnapshot>, +) -> Result<(), String> { + // `Missing` is handled by `delta_sync` before this helper. `None` here + // therefore means the source exists but its read-only connection could + // not be established, which is not evidence that any cached row vanished. + if cursor_conn.is_none() { + return Ok(()); + } + let Some(index_conn) = index_conn else { + return repair_cached_listability_without_index(cache_conn, cursor_conn, storage_snapshot); + }; + let discovered = match discover_from_index(index_conn) { + Ok(discovered) => discovered, + // A transiently unreadable/foreign optional index is not authoritative + // evidence that Cursor deleted every session. It is also not the same + // as a definitively absent optional index, so do not silently fall back + // to a different discovery mode or stamp validation success. + Err(_) => return Ok(()), + }; let signatures = discovered .iter() - .map(|row| row.signature(&source_path)) + .map(|row| row.signature(source_path)) .collect::>(); let live_parent_ids = source_cache::live_ids_from_signatures(&signatures); let live_parent_id_set = live_parent_ids.iter().cloned().collect::>(); @@ -47,18 +137,99 @@ pub(super) fn delta_sync(cache_conn: &mut Connection) -> Result<(), String> { cache_conn, SOURCE_CURSOR_IDE, &discovered, - |row| row.signature(&source_path), + |row| row.signature(source_path), )?; let changed_parent_ids = changed .iter() .map(|row| row.id.clone()) .collect::>(); + let cached_blob_states = cached_cursor_index_blob_states(cache_conn)?; let mut authoritative_changed_parent_ids = HashSet::new(); let mut live_ids = live_parent_ids; let mut inputs = Vec::new(); - for row in changed { - let built = build_inputs_from_index(cursor_conn.as_ref(), row, &source_path)?; + for row in &discovered { + let index_signature = index_blob_validation_signature(row, source_path); + let missing_signature = missing_index_blob_validation_signature( + &index_signature, + storage_snapshot.map(|snapshot| snapshot.activity_signature.as_str()), + ); + let cached_blob_state = cached_blob_states.get(&row.id); + let database_identity = storage_snapshot + .map(|snapshot| snapshot.database_identity.as_str()) + .unwrap_or_default(); + if cached_blob_state + .is_some_and(|state| state.validated_missing(&missing_signature, database_identity)) + { + // The index and physical state DB are unchanged since this shell + // was hidden. Do not point-read the same missing blob again. + continue; + } + let changed = changed_parent_ids.contains(&row.id); + if !changed + && cached_blob_state + .is_some_and(|state| state.validated_present(&index_signature, database_identity)) + { + continue; + } + let mut built = build_inputs_from_index(cursor_conn, row, source_path)?; + match built.composer_availability { + CursorComposerAvailability::Available => {} + CursorComposerAvailability::MissingOrMalformed => { + if let Some(state) = cached_blob_state { + stamp_cursor_index_blob_validation( + cache_conn, + &row.id, + &state.source_metadata_json, + &CursorIndexBlobValidation { + signature: missing_signature, + // Reaching this branch means the index, state DB, + // and point-read all succeeded, but the referenced + // composer is definitively absent or malformed. + // Keep the compact terminal marker so an unchanged + // refresh performs zero content reads; physical DB + // activity changes the signature and retries. + misses: 2, + database_identity: storage_snapshot + .map(|snapshot| snapshot.database_identity.clone()) + .unwrap_or_default(), + }, + true, + )?; + } else { + // A newly indexed row whose blob is already absent would + // otherwise be re-probed forever because no cache + // signature exists. Persist one compact, non-listable + // tombstone; an index or physical-activity change makes it + // retryable, while an unchanged refresh performs zero + // content reads. + built.inputs.push(index_missing_tombstone( + row, + source_path, + &CursorIndexBlobValidation { + signature: missing_signature, + misses: 2, + database_identity: database_identity.to_string(), + }, + )); + } + } + CursorComposerAvailability::TemporarilyUnavailable => {} + } + if built.composer_availability == CursorComposerAvailability::Available { + if let Some(root_input) = built.inputs.first_mut() { + attach_cursor_index_blob_validation( + root_input, + &CursorIndexBlobValidation { + signature: index_signature, + misses: 0, + database_identity: storage_snapshot + .map(|snapshot| snapshot.database_identity.clone()) + .unwrap_or_default(), + }, + ); + } + } if built.child_list_authoritative { authoritative_changed_parent_ids.insert(row.id.clone()); } @@ -84,6 +255,551 @@ pub(super) fn delta_sync(cache_conn: &mut Connection) -> Result<(), String> { Ok(()) } +#[derive(Debug)] +struct CachedCursorIndexBlobState { + source_metadata_json: String, + validation: Option, +} + +fn index_blob_validation_signature(row: &CursorIndexRow, source_path: &str) -> String { + let signature = row.signature(source_path); + serde_json::json!([ + signature.source_mtime_ms, + signature.source_size_bytes, + signature.source_fingerprint, + signature.parser_version + ]) + .to_string() +} + +fn missing_index_blob_validation_signature( + index_signature: &str, + storage_signature: Option<&str>, +) -> String { + serde_json::json!([index_signature, storage_signature.unwrap_or_default()]).to_string() +} + +impl CachedCursorIndexBlobState { + fn validated_present(&self, index_signature: &str, database_identity: &str) -> bool { + self.validation.as_ref().is_some_and(|validation| { + validation.misses == 0 + && validation.signature == index_signature + && validation.database_identity == database_identity + }) + } + + fn validated_missing(&self, missing_signature: &str, database_identity: &str) -> bool { + self.validation.as_ref().is_some_and(|validation| { + validation.misses >= 2 + && validation.signature == missing_signature + && validation.database_identity == database_identity + }) + } +} + +fn index_missing_tombstone( + row: &CursorIndexRow, + source_path: &str, + validation: &CursorIndexBlobValidation, +) -> ImportedHistoryCacheInput { + ImportedHistoryCacheInput { + source: SOURCE_CURSOR_IDE, + source_session_id: row.id.clone(), + session_id: canonical_session_id(&row.id), + source_path: source_path.to_string(), + source_record_key: format!("{SOURCE_RECORD_KEY_PREFIX}{COMPOSER_KEY_PREFIX}{}", row.id), + source_mtime_ms: row.updated_at_ms, + source_size_bytes: row.is_archived as i64, + source_fingerprint: row.root_fingerprint.clone(), + parser_version: CURSOR_IDE_METADATA_PARSER_VERSION, + name: preferred_cursor_title(&row.id, "", &row.title, None), + created_at_ms: row.updated_at_ms, + updated_at_ms: row.updated_at_ms, + model: None, + input_tokens: 0, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + repo_path: None, + branch: None, + impact: ImportedHistoryImpactStats::default(), + listable: false, + source_metadata_json: Some(merged_cursor_validation_metadata( + "", + None, + Some(validation), + )), + parent_session_id: None, + } +} + +fn cached_cursor_index_blob_states( + cache_conn: &Connection, +) -> Result, String> { + let mut statement = cache_conn + .prepare( + "SELECT source_session_id,source_metadata_json + FROM imported_history_session_cache + WHERE source=?1 AND COALESCE(parent_session_id, '')=''", + ) + .map_err(|err| format!("Failed to prepare Cursor index-blob state query: {err}"))?; + let rows = statement + .query_map([SOURCE_CURSOR_IDE], |row| { + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + }) + .map_err(|err| format!("Failed to query Cursor index-blob states: {err}"))?; + let mut states = HashMap::new(); + for row in rows { + let (source_session_id, source_metadata_json) = + row.map_err(|err| format!("Failed to read Cursor index-blob state: {err}"))?; + let validation = serde_json::from_str::(&source_metadata_json) + .ok() + .and_then(|value| value.get(INDEX_BLOB_VALIDATION_FIELD).cloned()) + .and_then(|value| serde_json::from_value::(value).ok()); + states.insert( + source_session_id, + CachedCursorIndexBlobState { + source_metadata_json, + validation, + }, + ); + } + Ok(states) +} + +fn attach_cursor_index_blob_validation( + input: &mut ImportedHistoryCacheInput, + validation: &CursorIndexBlobValidation, +) { + input.source_metadata_json = Some(merged_cursor_validation_metadata( + input.source_metadata_json.as_deref().unwrap_or_default(), + None, + Some(validation), + )); +} + +fn stamp_cursor_index_blob_validation( + cache_conn: &Connection, + source_session_id: &str, + source_metadata_json: &str, + validation: &CursorIndexBlobValidation, + hide: bool, +) -> Result<(), String> { + let metadata = merged_cursor_validation_metadata(source_metadata_json, None, Some(validation)); + cache_conn + .execute( + "UPDATE imported_history_session_cache + SET source_metadata_json=?3, + listable=CASE WHEN ?4 != 0 THEN 0 ELSE listable END, + updated_at=?5 + WHERE source=?1 AND source_session_id=?2 + AND COALESCE(parent_session_id, '')=''", + params![ + SOURCE_CURSOR_IDE, + source_session_id, + metadata, + i64::from(hide), + chrono::Utc::now().to_rfc3339(), + ], + ) + .map(|_| ()) + .map_err(|err| format!("Failed to stamp Cursor index-blob validation: {err}")) +} + +#[derive(Debug)] +struct CachedCursorParent { + source_session_id: String, + source_path: String, + source_record_key: String, + source_mtime_ms: i64, + source_size_bytes: i64, + source_fingerprint: String, + name: String, + created_at_ms: i64, + updated_at_ms: i64, + source_metadata_json: String, +} + +/// Split stable replacement identity from normal SQLite activity. +/// +/// The main file's device/inode pair changes when Cursor replaces its state +/// database, while main/WAL mtime+size changes during ordinary writes. SHM is +/// excluded because our own reads can modify it. +fn cursor_storage_snapshot(path: &Path) -> Result { + let metadata = fs::metadata(path) + .map_err(|err| format!("Failed to read Cursor state database metadata: {err}"))?; + let canonical = fs::canonicalize(path).unwrap_or_else(|_| path.to_path_buf()); + #[cfg(unix)] + let database_identity = { + use std::os::unix::fs::MetadataExt; + format!( + "{}:{}:{}", + canonical.display(), + metadata.dev(), + metadata.ino() + ) + }; + #[cfg(not(unix))] + let database_identity = { + // Windows does not expose Unix device/inode metadata through the + // portable API, but file creation time is stable across normal SQLite + // writes and changes when the database file is replaced. + let created_ns = metadata + .created() + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map(|duration| duration.as_nanos()) + .unwrap_or_default(); + format!("{}:{created_ns}", canonical.display()) + }; + let main_mtime_ns = metadata_mtime_ns(&metadata); + let wal_path = std::path::PathBuf::from(format!("{}-wal", path.to_string_lossy())); + let wal_signature = match fs::metadata(wal_path) { + Ok(wal) => format!("wal:{}:{}", wal.len(), metadata_mtime_ns(&wal)), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => "wal:-".to_string(), + Err(err) => { + return Err(format!( + "Failed to read Cursor WAL metadata for activity watermark: {err}" + )) + } + }; + Ok(CursorStorageSnapshot { + database_identity, + activity_signature: format!("main:{}:{main_mtime_ns}|{wal_signature}", metadata.len()), + }) +} + +fn metadata_mtime_ns(metadata: &fs::Metadata) -> i64 { + metadata + .modified() + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map(|duration| duration.as_nanos() as i64) + .unwrap_or_default() +} + +/// Migrate only known cached roots when Cursor has no conversation index. +/// +/// This intentionally does not enumerate `composerData:%`: that table can be +/// multi-GB. Each candidate performs one composer primary-key lookup plus +/// exact lookups for declared headers and one indexed range over that +/// composer's bubbles. A missing/malformed composer is definitive for the +/// current physical DB generation: preserve its cached metadata, hide it, and +/// stamp that generation so unchanged refreshes perform zero repeated probes. +/// Query errors remain indeterminate and are retried one bounded point-read on +/// a later refresh. +fn repair_cached_listability_without_index( + cache_conn: &mut Connection, + cursor_conn: Option<&Connection>, + storage_snapshot: Option<&CursorStorageSnapshot>, +) -> Result<(), String> { + let Some(storage_snapshot) = storage_snapshot else { + return Ok(()); + }; + let Some(cursor_conn) = cursor_conn else { + // The path exists but a read-only connection could not be opened + // (typically a transient lock). Preserve the last known projection and + // do not stamp success, so the next refresh gets one bounded retry. + return Ok(()); + }; + queue_legacy_no_index_validation(cache_conn)?; + demote_no_index_rows_from_replaced_database(cache_conn, storage_snapshot)?; + let cached = cached_cursor_parents_needing_validation(cache_conn, storage_snapshot)?; + for cached in cached { + let raw = match load_composer_raw(cursor_conn, &cached.source_session_id) { + Ok(CursorComposerLoad::Present(raw)) => raw, + Ok(CursorComposerLoad::Missing | CursorComposerLoad::Malformed) => { + demote_cached_cursor_parent(cache_conn, &cached, storage_snapshot)?; + continue; + } + Err(_) => { + rotate_indeterminate_validation_candidate(cache_conn, &cached.source_session_id)?; + continue; + } + }; + let user_bubbles = + match probe_replayable_user_bubbles(cursor_conn, &cached.source_session_id, &raw) { + Ok(value) => value, + Err(_) => { + rotate_indeterminate_validation_candidate( + cache_conn, + &cached.source_session_id, + )?; + continue; + } + }; + let Ok(mut input) = cache_input_from_raw( + cursor_conn, + &cached.source_session_id, + &cached.source_path, + &cached.source_record_key, + cached.source_mtime_ms, + cached.source_size_bytes, + &cached.source_fingerprint, + &raw, + None, + Some(storage_snapshot), + ) else { + rotate_indeterminate_validation_candidate(cache_conn, &cached.source_session_id)?; + continue; + }; + input.created_at_ms = if input.created_at_ms > 0 { + input.created_at_ms + } else { + cached.created_at_ms + }; + input.updated_at_ms = input.updated_at_ms.max(cached.updated_at_ms); + input.name = preferred_cursor_title( + &cached.source_session_id, + &raw.name, + &cached.name, + user_bubbles.first_user_preview.as_deref(), + ); + input.listable = user_bubbles.has_user_bubble; + source_cache::upsert_imported_session_cache_from_conn( + cache_conn, + std::slice::from_ref(&input), + )?; + } + Ok(()) +} + +/// Move an indeterminate point-read to the back of the bounded retry queue. +/// +/// This updates only cache-maintenance bookkeeping, not parser/listability +/// state or a validation-success watermark. Without it, 64 consistently +/// unreadable rows could occupy every batch and starve all later hidden rows. +fn rotate_indeterminate_validation_candidate( + cache_conn: &Connection, + source_session_id: &str, +) -> Result<(), String> { + cache_conn + .execute( + "UPDATE imported_history_session_cache + SET updated_at=?3 + WHERE source=?1 AND source_session_id=?2 + AND COALESCE(parent_session_id, '')=''", + params![ + SOURCE_CURSOR_IDE, + source_session_id, + chrono::Utc::now().to_rfc3339(), + ], + ) + .map(|_| ()) + .map_err(|err| format!("Failed to rotate Cursor validation retry: {err}")) +} + +fn demote_cached_cursor_parent( + cache_conn: &mut Connection, + cached: &CachedCursorParent, + storage_snapshot: &CursorStorageSnapshot, +) -> Result<(), String> { + let metadata = merged_cursor_validation_metadata( + &cached.source_metadata_json, + Some(storage_snapshot), + None, + ); + cache_conn + .execute( + "UPDATE imported_history_session_cache + SET listable=0, parser_version=?3, source_metadata_json=?4, + updated_at=?5 + WHERE source=?1 AND source_session_id=?2 + AND COALESCE(parent_session_id, '')=''", + params![ + SOURCE_CURSOR_IDE, + cached.source_session_id, + CURSOR_IDE_METADATA_PARSER_VERSION, + metadata, + chrono::Utc::now().to_rfc3339(), + ], + ) + .map(|_| ()) + .map_err(|err| format!("Failed to hide invalid Cursor shell: {err}")) +} + +fn merged_cursor_validation_metadata( + existing: &str, + storage_snapshot: Option<&CursorStorageSnapshot>, + index_blob_validation: Option<&CursorIndexBlobValidation>, +) -> String { + let mut object = serde_json::from_str::(existing) + .ok() + .and_then(|value| value.as_object().cloned()) + .unwrap_or_default(); + if let Some(snapshot) = storage_snapshot { + object.insert( + NO_INDEX_DATABASE_IDENTITY_FIELD.to_string(), + serde_json::Value::String(snapshot.database_identity.clone()), + ); + object.insert( + NO_INDEX_ACTIVITY_SIGNATURE_FIELD.to_string(), + serde_json::Value::String(snapshot.activity_signature.clone()), + ); + } + if let Some(validation) = index_blob_validation { + object.insert( + INDEX_BLOB_VALIDATION_FIELD.to_string(), + serde_json::to_value(validation).unwrap_or(serde_json::Value::Null), + ); + } else { + object.remove(INDEX_BLOB_VALIDATION_FIELD); + } + serde_json::Value::Object(object).to_string() +} + +fn cached_cursor_parents_needing_validation( + cache_conn: &Connection, + storage_snapshot: &CursorStorageSnapshot, +) -> Result, String> { + let mut stmt = cache_conn + .prepare( + "SELECT source_session_id,source_path,source_record_key, + source_mtime_ms,source_size_bytes,source_fingerprint, + name,created_at_ms,updated_at_ms,source_metadata_json + FROM imported_history_session_cache + WHERE source=?1 + AND COALESCE(parent_session_id, '')='' + AND ( + parser_version < ?2 + OR ( + listable=0 + AND ( + CASE WHEN json_valid(source_metadata_json) + THEN COALESCE(json_extract( + source_metadata_json, + '$.noIndexDatabaseIdentity' + ), '') + ELSE '' END != ?3 + OR CASE WHEN json_valid(source_metadata_json) + THEN COALESCE(json_extract( + source_metadata_json, + '$.noIndexActivitySignature' + ), '') + ELSE '' END != ?4 + ) + ) + ) + ORDER BY CASE WHEN parser_version < ?2 THEN 0 ELSE 1 END, + updated_at ASC, source_session_id ASC + LIMIT ?5", + ) + .map_err(|err| format!("Failed to prepare cached Cursor validation query: {err}"))?; + let rows = stmt + .query_map( + params![ + SOURCE_CURSOR_IDE, + CURSOR_IDE_METADATA_PARSER_VERSION, + storage_snapshot.database_identity, + storage_snapshot.activity_signature, + NO_INDEX_VALIDATION_BATCH_SIZE as i64, + ], + |row| { + Ok(CachedCursorParent { + source_session_id: row.get(0)?, + source_path: row.get(1)?, + source_record_key: row.get(2)?, + source_mtime_ms: row.get(3)?, + source_size_bytes: row.get(4)?, + source_fingerprint: row.get(5)?, + name: row.get(6)?, + created_at_ms: row.get(7)?, + updated_at_ms: row.get(8)?, + source_metadata_json: row.get(9)?, + }) + }, + ) + .map_err(|err| format!("Failed to query cached Cursor validation rows: {err}"))?; + let mut out = Vec::new(); + for row in rows { + out.push(row.map_err(|err| format!("Failed to read cached Cursor row: {err}"))?); + } + Ok(out) +} + +fn queue_legacy_no_index_validation(cache_conn: &Connection) -> Result<(), String> { + cache_conn + .execute( + "UPDATE imported_history_session_cache + SET parser_version=?2 + WHERE source=?1 + AND COALESCE(parent_session_id, '')='' + AND listable=1 + AND parser_version>=?3 + AND json_valid(source_metadata_json) + AND json_extract(source_metadata_json, + '$.noIndexValidationSignature') IS NOT NULL + AND json_extract(source_metadata_json, + '$.noIndexDatabaseIdentity') IS NULL", + params![ + SOURCE_CURSOR_IDE, + CURSOR_IDE_METADATA_PARSER_VERSION - 1, + CURSOR_IDE_METADATA_PARSER_VERSION, + ], + ) + .map(|_| ()) + .map_err(|err| format!("Failed to queue legacy Cursor validation watermark: {err}")) +} + +fn demote_no_index_rows_from_replaced_database( + cache_conn: &Connection, + storage_snapshot: &CursorStorageSnapshot, +) -> Result<(), String> { + cache_conn + .execute( + "UPDATE imported_history_session_cache + SET listable=0, + source_metadata_json=json_set( + CASE WHEN json_valid(source_metadata_json) + THEN source_metadata_json ELSE '{}' END, + '$.noIndexDatabaseIdentity', ?2, + '$.noIndexActivitySignature', '' + ), + updated_at=?3 + WHERE source=?1 + AND COALESCE(parent_session_id, '')='' + AND CASE WHEN json_valid(source_metadata_json) + THEN COALESCE(json_extract( + source_metadata_json, + '$.noIndexDatabaseIdentity' + ), '') + ELSE '' END NOT IN ('', ?2)", + params![ + SOURCE_CURSOR_IDE, + storage_snapshot.database_identity, + chrono::Utc::now().to_rfc3339(), + ], + ) + .map(|_| ()) + .map_err(|err| format!("Failed to hide Cursor rows after DB replacement: {err}")) +} + +pub(super) fn demote_definitively_missing_cursor_database( + cache_conn: &Connection, +) -> Result<(), String> { + cache_conn + .execute( + "UPDATE imported_history_session_cache + SET listable=0, parser_version=?2, + source_metadata_json=json_set( + CASE WHEN json_valid(source_metadata_json) + THEN source_metadata_json ELSE '{}' END, + '$.noIndexDatabaseIdentity', ?3, + '$.noIndexActivitySignature', '' + ), + updated_at=?4 + WHERE source=?1 AND COALESCE(parent_session_id, '')=''", + params![ + SOURCE_CURSOR_IDE, + CURSOR_IDE_METADATA_PARSER_VERSION, + CURSOR_STORAGE_MISSING_IDENTITY, + chrono::Utc::now().to_rfc3339(), + ], + ) + .map(|_| ()) + .map_err(|err| format!("Failed to hide Cursor rows for missing database: {err}")) +} + fn cached_cursor_child_ids_by_parent( cache_conn: &Connection, ) -> Result>, String> { @@ -142,9 +858,9 @@ pub(super) fn discover_from_index(index_conn: &Connection) -> Result, row: &CursorIndexRow, @@ -152,8 +868,31 @@ pub(super) fn build_inputs_from_index( ) -> Result { let record_key = format!("{SOURCE_RECORD_KEY_PREFIX}{COMPOSER_KEY_PREFIX}{}", row.id); if let Some(cursor_conn) = cursor_conn { - if let Some(raw) = load_composer_raw(cursor_conn, &row.id)? { - let mut input = cache_input_from_raw( + let raw = match load_composer_raw(cursor_conn, &row.id) { + Ok(CursorComposerLoad::Present(raw)) => Some(raw), + Ok(CursorComposerLoad::Missing | CursorComposerLoad::Malformed) => None, + Err(_) => { + return Ok(CursorParentBuild { + inputs: Vec::new(), + live_child_ids: Vec::new(), + child_list_authoritative: false, + composer_availability: CursorComposerAvailability::TemporarilyUnavailable, + }) + } + }; + if let Some(raw) = raw { + let user_bubbles = match probe_replayable_user_bubbles(cursor_conn, &row.id, &raw) { + Ok(value) => value, + Err(_) => { + return Ok(CursorParentBuild { + inputs: Vec::new(), + live_child_ids: Vec::new(), + child_list_authoritative: false, + composer_availability: CursorComposerAvailability::TemporarilyUnavailable, + }) + } + }; + let mut input = match cache_input_from_raw( cursor_conn, &row.id, source_path, @@ -163,7 +902,25 @@ pub(super) fn build_inputs_from_index( &row.root_fingerprint, &raw, None, - )?; + None, + ) { + Ok(input) => input, + Err(_) => { + return Ok(CursorParentBuild { + inputs: Vec::new(), + live_child_ids: Vec::new(), + child_list_authoritative: false, + composer_availability: CursorComposerAvailability::TemporarilyUnavailable, + }) + } + }; + input.name = preferred_cursor_title( + &row.id, + &raw.name, + &row.title, + user_bubbles.first_user_preview.as_deref(), + ); + input.listable = user_bubbles.has_user_bubble; // Sort/display recency comes from the index's authoritative // `updated_at`, not the composer's possibly-stale last-bubble time. if row.updated_at_ms > 0 { @@ -181,8 +938,9 @@ pub(super) fn build_inputs_from_index( let mut inputs = Vec::with_capacity(live_child_ids.len() + 1); inputs.push(input); for child_id in &live_child_ids { - let Some(child_raw) = load_composer_raw(cursor_conn, child_id)? else { - continue; + let child_raw = match load_composer_raw(cursor_conn, child_id)? { + CursorComposerLoad::Present(raw) => raw, + CursorComposerLoad::Missing | CursorComposerLoad::Malformed => continue, }; let child_parent_id = child_raw .subagent_info @@ -206,67 +964,44 @@ pub(super) fn build_inputs_from_index( &format!("parent:{child_parent_id}"), &child_raw, Some(child_parent_id), + None, )?); } return Ok(CursorParentBuild { inputs, live_child_ids, child_list_authoritative: true, + composer_availability: CursorComposerAvailability::Available, }); } + return Ok(CursorParentBuild { + inputs: Vec::new(), + live_child_ids: Vec::new(), + child_list_authoritative: false, + composer_availability: CursorComposerAvailability::MissingOrMalformed, + }); } + // The conversation index can be written before its composer blob. Do not + // create an index-only visible shell, and do not overwrite a previously + // valid cached row during that transient window. The still-live parent id + // retained by the caller prevents pruning; absence from the cache makes a + // newly-discovered row retry on the next refresh. Ok(CursorParentBuild { - inputs: vec![minimal_cache_input_from_index( - row, - source_path, - &record_key, - )], + inputs: Vec::new(), live_child_ids: Vec::new(), child_list_authoritative: false, + composer_availability: CursorComposerAvailability::TemporarilyUnavailable, }) } -/// Minimal cache row from the index alone — used when the composer blob is -/// unavailable. Lists the session with its title and last-updated time; the -/// rich fields fill in if the blob reappears (the signature stays keyed on the -/// index, so a later scan won't spuriously re-import). -fn minimal_cache_input_from_index( - row: &CursorIndexRow, - source_path: &str, - record_key: &str, -) -> ImportedHistoryCacheInput { - ImportedHistoryCacheInput { - source: SOURCE_CURSOR_IDE, - source_session_id: row.id.clone(), - session_id: super::canonical_session_id(&row.id), - source_path: source_path.to_string(), - source_record_key: record_key.to_string(), - source_mtime_ms: row.updated_at_ms, - source_size_bytes: row.is_archived as i64, - source_fingerprint: row.root_fingerprint.clone(), - parser_version: CURSOR_IDE_METADATA_PARSER_VERSION, - name: row.title.clone(), - created_at_ms: row.updated_at_ms, - updated_at_ms: row.updated_at_ms, - model: None, - input_tokens: 0, - output_tokens: 0, - cache_read_tokens: 0, - cache_write_tokens: 0, - repo_path: None, - branch: None, - impact: ImportedHistoryImpactStats::default(), - listable: true, - source_metadata_json: serde_json::to_string(&CursorCacheMetadata::default()).ok(), - parent_session_id: None, +/// Point-lookup + parse a single `composerData:` row (fast; primary key). +fn load_composer_raw(cursor_conn: &Connection, id: &str) -> Result { + #[cfg(test)] + { + CURSOR_CONTENT_PROBE_COUNT.set(CURSOR_CONTENT_PROBE_COUNT.get() + 1); + CURSOR_CONTENT_PROBED_IDS.with(|ids| ids.borrow_mut().push(id.to_string())); } -} -/// Point-lookup + parse a single `composerData:` row (fast; primary key). -fn load_composer_raw( - cursor_conn: &Connection, - id: &str, -) -> Result, String> { let key = format!("{COMPOSER_KEY_PREFIX}{id}"); let value: Option = cursor_conn .query_row( @@ -277,10 +1012,131 @@ fn load_composer_raw( .optional() .map_err(|err| format!("Failed to read Cursor composer {id}: {err}"))?; let Some(value) = value else { - return Ok(None); + return Ok(CursorComposerLoad::Missing); }; - // A malformed blob shouldn't fail the whole sync — treat it as absent. - Ok(serde_json::from_str(&value).ok()) + // Malformed JSON is stable source content for this physical generation, + // unlike a SQLite read error. Callers may hide/stamp it while keeping + // SQLITE_BUSY/LOCKED and other query failures retryable. + match serde_json::from_str(&value) { + Ok(raw) => Ok(CursorComposerLoad::Present(Box::new(raw))), + Err(_) => Ok(CursorComposerLoad::Malformed), + } +} + +/// Confirm real user content separately from deriving an optional title +/// preview. +/// +/// Header point-lookups preserve Cursor's canonical conversation order. A +/// final composer-key-range probe recovers real bubbles omitted from stale +/// `fullConversationHeadersOnly` metadata without scanning `cursorDiskKV`. +fn probe_replayable_user_bubbles( + cursor_conn: &Connection, + composer_id: &str, + raw: &RawComposerData, +) -> Result { + let mut stmt = cursor_conn + .prepare("SELECT value FROM cursorDiskKV WHERE key=?1") + .map_err(|err| format!("Failed to prepare Cursor user-bubble lookup: {err}"))?; + let mut probe = super::helpers::CursorUserBubbleProbe::default(); + for header in &raw.full_conversation_headers_only { + if header.bubble_id.trim().is_empty() { + continue; + } + // Newer Cursor versions put the type in the header. A zero/missing + // type is still checked because older composer blobs only typed the + // bubble value. + if header.bubble_type != 0 && header.bubble_type != 1 { + continue; + } + let key = format!("{BUBBLE_KEY_PREFIX}{composer_id}:{}", header.bubble_id); + let value = stmt + .query_row([key], |row| row.get::<_, Option>(0)) + .optional() + .map_err(|err| format!("Failed to read Cursor user bubble: {err}"))? + .flatten(); + let Some(value) = value else { + continue; + }; + let Ok(bubble) = serde_json::from_str::(&value) else { + continue; + }; + let bubble_type = if bubble.bubble_type != 0 { + bubble.bubble_type + } else { + header.bubble_type + }; + if bubble_type != 1 { + continue; + } + probe.has_user_bubble = true; + let preview = super::helpers::preview_text(&bubble.text); + if !preview.is_empty() { + probe.first_user_preview = Some(preview); + return Ok(probe); + } + } + let indexed = super::helpers::probe_indexed_cursor_user_bubbles(cursor_conn, composer_id)?; + probe.has_user_bubble |= indexed.has_user_bubble; + if probe.first_user_preview.is_none() { + probe.first_user_preview = indexed.first_user_preview; + } + Ok(probe) +} + +pub(super) fn preferred_cursor_title( + composer_id: &str, + composer_title: &str, + fallback_title: &str, + first_user_text: Option<&str>, +) -> String { + if !is_cursor_placeholder_title(composer_title, composer_id) { + return composer_title.trim().to_string(); + } + if !is_cursor_placeholder_title(fallback_title, composer_id) { + return fallback_title.trim().to_string(); + } + first_user_text.unwrap_or_default().to_string() +} + +fn is_cursor_placeholder_title(title: &str, composer_id: &str) -> bool { + let title = title.trim(); + if title.is_empty() { + return true; + } + let normalized = title.to_ascii_lowercase(); + if matches!( + normalized.as_str(), + "new agent" + | "new chat" + | "untitled" + | "untitled cursor session" + | "cursor session" + | "composer" + ) { + return true; + } + if looks_like_uuid(&normalized) { + return true; + } + let composer_id = composer_id.trim().to_ascii_lowercase(); + normalized == composer_id + || normalized == format!("{CURSORIDE_SESSION_PREFIX}{composer_id}").to_ascii_lowercase() + || normalized == format!("{COMPOSER_KEY_PREFIX}{composer_id}").to_ascii_lowercase() + || normalized + == format!("{SOURCE_RECORD_KEY_PREFIX}{COMPOSER_KEY_PREFIX}{composer_id}") + .to_ascii_lowercase() +} + +fn looks_like_uuid(value: &str) -> bool { + let segments = value.split('-').collect::>(); + segments.len() == 5 + && segments + .iter() + .zip([8, 4, 4, 4, 12]) + .all(|(segment, expected_len)| { + segment.len() == expected_len + && segment.bytes().all(|byte| byte.is_ascii_hexdigit()) + }) } /// Normalize a parsed `composerData` blob into a cache row. @@ -295,6 +1151,7 @@ fn cache_input_from_raw( source_fingerprint: &str, raw: &RawComposerData, parent_source_session_id: Option<&str>, + storage_snapshot: Option<&CursorStorageSnapshot>, ) -> Result { let model = raw .model_config @@ -315,6 +1172,11 @@ fn cache_input_from_raw( status: raw.status.clone(), is_agentic: raw.is_agentic, mode: raw.unified_mode.clone(), + no_index_database_identity: storage_snapshot + .map(|snapshot| snapshot.database_identity.clone()), + no_index_activity_signature: storage_snapshot + .map(|snapshot| snapshot.activity_signature.clone()), + index_blob_validation: None, }; let source_metadata_json = serde_json::to_string(&metadata) .map_err(|err| format!("Failed to encode Cursor metadata cache payload: {err}"))?; diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/tests.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/tests.rs index 88e7788b7..e9bc3988e 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/tests.rs @@ -56,6 +56,13 @@ fn cursor_cache_metadata_round_trips() { status: "completed".to_string(), is_agentic: true, mode: "agent".to_string(), + no_index_database_identity: Some("state-db-1".to_string()), + no_index_activity_signature: Some("main:1:2|wal:-".to_string()), + index_blob_validation: Some(CursorIndexBlobValidation { + signature: "index+state".to_string(), + misses: 2, + database_identity: "state-db-1".to_string(), + }), }; let encoded = serde_json::to_string(&metadata).expect("encode"); let decoded: CursorCacheMetadata = serde_json::from_str(&encoded).expect("decode"); @@ -63,6 +70,24 @@ fn cursor_cache_metadata_round_trips() { assert_eq!(decoded.status, "completed"); assert!(decoded.is_agentic); assert_eq!(decoded.mode, "agent"); + assert_eq!( + decoded.no_index_database_identity.as_deref(), + Some("state-db-1") + ); + assert_eq!( + decoded.no_index_activity_signature.as_deref(), + Some("main:1:2|wal:-") + ); + assert_eq!( + decoded.index_blob_validation.as_ref().map(|value| { + ( + value.signature.as_str(), + value.misses, + value.database_identity.as_str(), + ) + }), + Some(("index+state", 2, "state-db-1")) + ); } fn index_db_with_rows() -> Connection { @@ -128,7 +153,7 @@ fn index_signature_tracks_update_archive_and_fingerprint() { } #[test] -fn build_input_from_index_without_composer_uses_index_fields() { +fn build_input_from_index_without_composer_does_not_create_visible_shell() { let row = CursorIndexRow { id: "c9".into(), title: "Just title".into(), @@ -139,15 +164,11 @@ fn build_input_from_index_without_composer_uses_index_fields() { let built = build_inputs_from_index(None, &row, "/store/state.vscdb").expect("build inputs"); assert!(!built.child_list_authoritative); assert!(built.live_child_ids.is_empty()); - assert_eq!(built.inputs.len(), 1); - let input = &built.inputs[0]; - assert_eq!(input.session_id, format!("{CURSORIDE_SESSION_PREFIX}c9")); - assert_eq!(input.name, "Just title"); - assert_eq!(input.created_at_ms, 4242); - assert_eq!(input.updated_at_ms, 4242); - assert_eq!(input.source_mtime_ms, 4242); - assert!(input.listable); - assert!(input.model.is_none()); + assert!(built.inputs.is_empty()); + assert_eq!( + built.composer_availability, + CursorComposerAvailability::TemporarilyUnavailable + ); } #[test] @@ -169,7 +190,8 @@ fn build_input_from_index_with_composer_reads_rich_metadata() { "file:///repo/orgii/src/a.ts": {"isNewlyCreated": false, "contentKey": "k1"}, "file:///repo/orgii/src/b.ts": {"isNewlyCreated": true, "contentKey": ""}, "file:///repo/orgii/src/untouched.ts": {"isNewlyCreated": false, "contentKey": ""} - } + }, + "fullConversationHeadersOnly": [{"bubbleId": "u1", "type": 1}] }) .to_string(); cursor @@ -178,6 +200,16 @@ fn build_input_from_index_with_composer_reads_rich_metadata() { params![composer], ) .expect("insert composer"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('bubbleId:c1:u1', ?1)", + params![serde_json::json!({ + "bubbleId": "u1", "type": 1, "text": "Help me fix this", + "createdAt": "2026-07-20T00:00:00Z" + }) + .to_string()], + ) + .expect("insert user bubble"); let row = CursorIndexRow { id: "c1".into(), @@ -210,6 +242,110 @@ fn build_input_from_index_with_composer_reads_rich_metadata() { assert_eq!(input.updated_at_ms, 3000); assert_eq!(input.source_mtime_ms, 3000); assert_eq!(input.source_fingerprint, "fp"); + assert!(input.listable); +} + +#[test] +fn build_input_recovers_user_bubble_missing_from_composer_headers() { + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('composerData:missing-header', ?1)", + params![serde_json::json!({ + "composerId": "missing-header", + "name": "", + "createdAt": 1000, + "lastUpdatedAt": 2000, + "fullConversationHeadersOnly": [] + }) + .to_string()], + ) + .expect("insert composer"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('bubbleId:missing-header:user-1', ?1)", + params![serde_json::json!({ + "bubbleId": "user-1", + "type": 1, + "text": "Recover me from the indexed bubble range", + "createdAt": "2026-07-20T00:00:00Z" + }) + .to_string()], + ) + .expect("insert unlisted user bubble"); + + let row = CursorIndexRow { + id: "missing-header".into(), + title: String::new(), + updated_at_ms: 2000, + is_archived: false, + root_fingerprint: "fp".into(), + }; + let built = build_inputs_from_index(Some(&cursor), &row, "/store/state.vscdb") + .expect("build missing-header input"); + + assert_eq!(built.inputs.len(), 1); + assert!(built.inputs[0].listable); + assert_eq!( + built.inputs[0].name, + "Recover me from the indexed bubble range" + ); +} + +#[test] +fn empty_user_text_still_marks_an_untitled_cursor_session_listable() { + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('composerData:empty-user-text', ?1)", + params![serde_json::json!({ + "composerId": "empty-user-text", + "name": "", + "createdAt": 1000, + "lastUpdatedAt": 2000, + "fullConversationHeadersOnly": [{"bubbleId": "user-1", "type": 1}] + }) + .to_string()], + ) + .expect("insert composer"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('bubbleId:empty-user-text:user-1', ?1)", + params![serde_json::json!({ + "bubbleId": "user-1", + "type": 1, + "text": " ", + "createdAt": "2026-07-20T00:00:00Z" + }) + .to_string()], + ) + .expect("insert empty user bubble"); + + let row = CursorIndexRow { + id: "empty-user-text".into(), + title: String::new(), + updated_at_ms: 2000, + is_archived: false, + root_fingerprint: "fp".into(), + }; + let built = build_inputs_from_index(Some(&cursor), &row, "/store/state.vscdb") + .expect("build empty-user-text input"); + + assert_eq!(built.inputs.len(), 1); + assert!(built.inputs[0].listable); + assert!(built.inputs[0].name.is_empty()); } #[test] @@ -226,7 +362,8 @@ fn changed_parent_builds_collapsible_subagent_rows() { "name": "Parent", "createdAt": 1000, "lastUpdatedAt": 3000, - "subagentComposerIds": ["child-1", "child-1", "", "parent-1"] + "subagentComposerIds": ["child-1", "child-1", "", "parent-1"], + "fullConversationHeadersOnly": [{"bubbleId": "u1", "type": 1}] }) .to_string(); let child = serde_json::json!({ @@ -254,6 +391,16 @@ fn changed_parent_builds_collapsible_subagent_rows() { params![child], ) .expect("insert child"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('bubbleId:parent-1:u1', ?1)", + params![serde_json::json!({ + "bubbleId": "u1", "type": 1, "text": "Delegate this task", + "createdAt": "2026-07-20T00:00:00Z" + }) + .to_string()], + ) + .expect("insert parent user bubble"); let row = CursorIndexRow { id: "parent-1".into(), @@ -282,3 +429,764 @@ fn changed_parent_builds_collapsible_subagent_rows() { ); assert_eq!(child_input.name, "Explore codebase"); } + +fn cursor_cache_conn() -> Connection { + let conn = Connection::open_in_memory().expect("open cache db"); + crate::store::sqlite::SqliteRecordStore::init_tables(&conn).expect("init core tables"); + crate::store::sqlite::SqliteRecordStore::init_source_cache_tables(&conn) + .expect("init imported cache tables"); + conn +} + +fn legacy_cursor_cache_input(id: &str, name: &str, listable: bool) -> ImportedHistoryCacheInput { + ImportedHistoryCacheInput { + source: SOURCE_CURSOR_IDE, + source_session_id: id.to_string(), + session_id: canonical_session_id(id), + source_path: "/cursor/state.vscdb".to_string(), + source_record_key: format!("{SOURCE_RECORD_KEY_PREFIX}{COMPOSER_KEY_PREFIX}{id}"), + source_mtime_ms: 100, + source_size_bytes: 10, + source_fingerprint: "legacy".to_string(), + parser_version: 2, + name: name.to_string(), + created_at_ms: 100, + updated_at_ms: 200, + model: None, + input_tokens: 0, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + repo_path: None, + branch: None, + impact: crate::sources::imported_history::metadata::ImportedHistoryImpactStats::default(), + listable, + source_metadata_json: None, + parent_session_id: None, + } +} + +fn cached_cursor_projection(conn: &Connection, id: &str) -> (i64, String, i64) { + conn.query_row( + "SELECT listable,name,parser_version + FROM imported_history_session_cache + WHERE source=?1 AND source_session_id=?2", + params![SOURCE_CURSOR_IDE, id], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .expect("cached Cursor projection") +} + +fn cached_cursor_metadata_string(conn: &Connection, id: &str, field: &str) -> Option { + conn.query_row( + "SELECT source_metadata_json + FROM imported_history_session_cache + WHERE source=?1 AND source_session_id=?2", + params![SOURCE_CURSOR_IDE, id], + |row| row.get::<_, String>(0), + ) + .expect("cached Cursor validation metadata") + .parse::() + .ok() + .and_then(|value| { + value + .get(field) + .and_then(serde_json::Value::as_str) + .map(str::to_string) + }) +} + +fn storage_snapshot(database_identity: &str, activity_signature: &str) -> CursorStorageSnapshot { + CursorStorageSnapshot { + database_identity: database_identity.to_string(), + activity_signature: activity_signature.to_string(), + } +} + +fn cached_cursor_index_blob_validation( + conn: &Connection, + id: &str, +) -> Option { + conn.query_row( + "SELECT source_metadata_json + FROM imported_history_session_cache + WHERE source=?1 AND source_session_id=?2", + params![SOURCE_CURSOR_IDE, id], + |row| row.get::<_, String>(0), + ) + .expect("cached Cursor index-blob metadata") + .parse::() + .ok() + .and_then(|value| value.get(INDEX_BLOB_VALIDATION_FIELD).cloned()) + .and_then(|value| serde_json::from_value(value).ok()) +} + +#[test] +fn missing_conversation_index_demotes_parser_v2_empty_shell_without_pruning_it() { + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input("empty-shell", "", true)], + ) + .expect("seed polluted cache"); + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('composerData:empty-shell', ?1)", + params![serde_json::json!({ + "composerId": "empty-shell", + "name": "", + "createdAt": 100, + "lastUpdatedAt": 200, + "fullConversationHeadersOnly": [] + }) + .to_string()], + ) + .expect("insert empty composer"); + + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:100:10|wal:-")), + ) + .expect("repair cache without optional index"); + + let (listable, name, parser_version) = cached_cursor_projection(&cache, "empty-shell"); + assert_eq!(listable, 0); + assert!(name.is_empty()); + assert_eq!(parser_version, CURSOR_IDE_METADATA_PARSER_VERSION); +} + +#[test] +fn missing_conversation_index_promotes_valid_untitled_session_with_user_preview() { + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input("untitled", "", false)], + ) + .expect("seed legacy cache"); + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('composerData:untitled', ?1)", + params![serde_json::json!({ + "composerId": "untitled", + "name": "", + "createdAt": 100, + "lastUpdatedAt": 200, + "fullConversationHeadersOnly": [{"bubbleId": "user-1", "type": 1}] + }) + .to_string()], + ) + .expect("insert composer"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('bubbleId:untitled:user-1', ?1)", + params![serde_json::json!({ + "bubbleId": "user-1", + "type": 1, + "text": " Explain the authentication flow in plain language ", + "createdAt": "2026-07-20T00:00:00Z" + }) + .to_string()], + ) + .expect("insert user bubble"); + + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:100:10|wal:-")), + ) + .expect("repair cache without optional index"); + + let (listable, name, parser_version) = cached_cursor_projection(&cache, "untitled"); + assert_eq!(listable, 1); + assert_eq!(name, "Explain the authentication flow in plain language"); + assert_eq!(parser_version, CURSOR_IDE_METADATA_PARSER_VERSION); +} + +#[test] +fn missing_source_blob_is_hidden_once_then_repromoted_after_physical_change() { + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input( + "temporarily-missing", + "Real history", + true, + )], + ) + .expect("seed valid cache"); + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:100:10|wal:-")), + ) + .expect("hide missing source blob"); + + let (listable, name, parser_version) = cached_cursor_projection(&cache, "temporarily-missing"); + assert_eq!(listable, 0); + assert_eq!(name, "Real history"); + assert_eq!(parser_version, CURSOR_IDE_METADATA_PARSER_VERSION); + assert_eq!( + cached_cursor_metadata_string( + &cache, + "temporarily-missing", + NO_INDEX_ACTIVITY_SIGNATURE_FIELD, + ) + .as_deref(), + Some("main:100:10|wal:-") + ); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:100:10|wal:-")), + ) + .expect("unchanged missing blob refresh"); + assert_eq!( + cursor_content_probe_count(), + 0, + "a stamped missing blob must not be probed again for the same physical generation" + ); + + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ('composerData:temporarily-missing', ?1)", + params![serde_json::json!({ + "composerId": "temporarily-missing", + "name": "", + "createdAt": 100, + "lastUpdatedAt": 300, + "fullConversationHeadersOnly": [{"bubbleId": "user-1", "type": 1}] + }) + .to_string()], + ) + .expect("restore composer blob"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ( + 'bubbleId:temporarily-missing:user-1', ?1 + )", + params![serde_json::json!({ + "bubbleId": "user-1", + "type": 1, + "text": "Recovered Cursor history", + "createdAt": "2026-07-26T04:00:00Z" + }) + .to_string()], + ) + .expect("restore user bubble"); + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:100:10|wal:4096:200")), + ) + .expect("revalidate restored source blob"); + let (listable, name, parser_version) = cached_cursor_projection(&cache, "temporarily-missing"); + assert_eq!(listable, 1); + assert_eq!(name, "Real history"); + assert_eq!(parser_version, CURSOR_IDE_METADATA_PARSER_VERSION); +} + +#[test] +fn malformed_source_blob_is_hidden_and_stamped_without_discarding_cached_metadata() { + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input( + "temporarily-malformed", + "Keep this title", + true, + )], + ) + .expect("seed cached Cursor card"); + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES ( + 'composerData:temporarily-malformed', '{not-json' + )", + [], + ) + .expect("insert malformed composer blob"); + + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:200:20|wal:-")), + ) + .expect("hide malformed source blob"); + + assert_eq!( + cached_cursor_projection(&cache, "temporarily-malformed"), + ( + 0, + "Keep this title".to_string(), + CURSOR_IDE_METADATA_PARSER_VERSION + ) + ); + assert_eq!( + cached_cursor_metadata_string( + &cache, + "temporarily-malformed", + NO_INDEX_ACTIVITY_SIGNATURE_FIELD, + ) + .as_deref(), + Some("main:200:20|wal:-") + ); +} + +#[test] +fn transient_cursor_read_error_preserves_last_known_card_and_does_not_stamp_success() { + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input( + "temporarily-busy", + "Last known good Cursor history", + true, + )], + ) + .expect("seed last known Cursor card"); + // A missing table produces the same adapter-level read error path used for + // SQLITE_BUSY/LOCKED. It is not evidence that the composer was deleted. + let unreadable_cursor = Connection::open_in_memory().expect("open unreadable Cursor DB"); + delta_sync_from_connections( + &mut cache, + None, + Some(&unreadable_cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:300:30|wal:-")), + ) + .expect("conservatively tolerate one bounded read failure"); + + assert_eq!( + cached_cursor_projection(&cache, "temporarily-busy"), + (1, "Last known good Cursor history".to_string(), 2,) + ); + assert!( + cached_cursor_metadata_string( + &cache, + "temporarily-busy", + NO_INDEX_ACTIVITY_SIGNATURE_FIELD, + ) + .is_none(), + "a failed read must remain retryable instead of being stamped as validation" + ); +} + +#[test] +fn definitively_missing_cursor_database_hides_old_ghosts_without_deleting_cache_rows() { + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input( + "cursor-not-installed", + "Old cached Cursor session", + true, + )], + ) + .expect("seed old Cursor ghost"); + + demote_definitively_missing_cursor_database(&cache) + .expect("hide cards whose physical Cursor DB is absent"); + assert_eq!( + cached_cursor_projection(&cache, "cursor-not-installed"), + ( + 0, + "Old cached Cursor session".to_string(), + CURSOR_IDE_METADATA_PARSER_VERSION + ) + ); + assert_eq!( + cached_cursor_metadata_string( + &cache, + "cursor-not-installed", + NO_INDEX_DATABASE_IDENTITY_FIELD, + ) + .as_deref(), + Some(CURSOR_STORAGE_MISSING_IDENTITY) + ); + + reset_cursor_content_probe_count(); + demote_definitively_missing_cursor_database(&cache).expect("unchanged absent database refresh"); + assert_eq!(cursor_content_probe_count(), 0); +} + +#[test] +fn indexed_missing_blob_is_hidden_on_first_successful_validation_until_storage_changes() { + let id = "indexed-missing"; + let mut cache = cursor_cache_conn(); + let mut cached = legacy_cursor_cache_input(id, "Previously valid Cursor history", true); + // Match the index signature exactly. This proves the migration candidate + // does not depend on changed-record discovery: an old visible card without + // a validation stamp must still be checked once. + cached.source_path = "/cursor/state.vscdb".to_string(); + cached.source_mtime_ms = 300; + cached.source_size_bytes = 0; + cached.source_fingerprint = "fp-new".to_string(); + cached.parser_version = CURSOR_IDE_METADATA_PARSER_VERSION; + source_cache::upsert_imported_session_cache_from_conn(&mut cache, &[cached]) + .expect("seed previously valid indexed card"); + let index = Connection::open_in_memory().expect("open Cursor index"); + index + .execute( + "CREATE TABLE conversations ( + id TEXT, title TEXT, updated_at INTEGER, is_archived INTEGER, + root_fingerprint TEXT, source TEXT + )", + [], + ) + .expect("create Cursor conversation index"); + index + .execute( + "INSERT INTO conversations VALUES (?1,'Indexed title',300,0,'fp-new','local')", + [id], + ) + .expect("insert indexed conversation"); + let cursor = Connection::open_in_memory().expect("open Cursor state DB"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + + let physical_one = storage_snapshot("state-db-1", "main:300:30|wal:-"); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&physical_one), + ) + .expect("first definitive missing-blob refresh"); + assert_eq!( + cached_cursor_projection(&cache, id), + ( + 0, + "Previously valid Cursor history".to_string(), + CURSOR_IDE_METADATA_PARSER_VERSION, + ), + "a successful point-read that proves the composer absent must hide the stale card" + ); + assert_eq!( + cached_cursor_index_blob_validation(&cache, id) + .as_ref() + .map(|value| value.misses), + Some(2) + ); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&physical_one), + ) + .expect("unchanged hidden shell refresh"); + assert_eq!( + cursor_content_probe_count(), + 0, + "stable index + physical signatures must not repeatedly probe the missing blob" + ); + + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![ + format!("{COMPOSER_KEY_PREFIX}{id}"), + serde_json::json!({ + "composerId": id, + "name": "", + "createdAt": 100, + "lastUpdatedAt": 400, + "fullConversationHeadersOnly": [{"bubbleId": "u1", "type": 1}] + }) + .to_string() + ], + ) + .expect("restore indexed composer"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![ + format!("{BUBBLE_KEY_PREFIX}{id}:u1"), + serde_json::json!({ + "bubbleId": "u1", + "type": 1, + "text": "Restored indexed Cursor history", + "createdAt": "2026-07-26T04:00:00Z" + }) + .to_string() + ], + ) + .expect("restore indexed user bubble"); + + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:300:30|wal:4096:400")), + ) + .expect("promote restored indexed composer"); + let (listable, name, parser_version) = cached_cursor_projection(&cache, id); + assert_eq!(listable, 1); + assert_eq!(name, "Indexed title"); + assert_eq!(parser_version, CURSOR_IDE_METADATA_PARSER_VERSION); + assert_eq!( + cached_cursor_index_blob_validation(&cache, id) + .as_ref() + .map(|value| value.misses), + Some(0), + "a successful materialization replaces the missing guard with a compact valid stamp" + ); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:300:30|wal:4096:400")), + ) + .expect("unchanged valid indexed refresh"); + assert_eq!( + cursor_content_probe_count(), + 0, + "a successful validation stamp must keep unchanged indexed cards metadata-only" + ); +} + +#[test] +fn indexed_transient_blob_read_error_preserves_last_known_card_without_stamping() { + let id = "indexed-temporarily-unreadable"; + let mut cache = cursor_cache_conn(); + let mut cached = legacy_cursor_cache_input(id, "Last known indexed history", true); + cached.source_path = "/cursor/state.vscdb".to_string(); + cached.source_mtime_ms = 300; + cached.source_size_bytes = 0; + cached.source_fingerprint = "fp-current".to_string(); + cached.parser_version = CURSOR_IDE_METADATA_PARSER_VERSION; + source_cache::upsert_imported_session_cache_from_conn(&mut cache, &[cached]) + .expect("seed last known indexed card"); + + let index = Connection::open_in_memory().expect("open Cursor index"); + index + .execute( + "CREATE TABLE conversations ( + id TEXT, title TEXT, updated_at INTEGER, is_archived INTEGER, + root_fingerprint TEXT, source TEXT + )", + [], + ) + .expect("create Cursor conversation index"); + index + .execute( + "INSERT INTO conversations VALUES (?1,'Indexed title',300,0,'fp-current','local')", + [id], + ) + .expect("insert indexed conversation"); + + // A missing cursorDiskKV table exercises the same adapter-level query + // error branch as SQLITE_BUSY/LOCKED and permission/open failures. It is + // not positive evidence that the indexed composer was deleted. + let unreadable_cursor = Connection::open_in_memory().expect("open unreadable Cursor DB"); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&unreadable_cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:300:30|wal:-")), + ) + .expect("preserve projection across transient point-read failure"); + + assert_eq!( + cached_cursor_projection(&cache, id), + ( + 1, + "Last known indexed history".to_string(), + CURSOR_IDE_METADATA_PARSER_VERSION, + ) + ); + assert!( + cached_cursor_index_blob_validation(&cache, id).is_none(), + "an indeterminate read must remain retryable instead of stamping a definitive miss" + ); +} + +#[test] +fn cursor_placeholder_titles_yield_to_first_real_user_preview() { + let id = "d3880bd5-3420-4d93-8f48-abef565346b2"; + let preview = Some("Explain the replay architecture"); + for placeholder in [ + "", + id, + "11111111-2222-4333-8444-555555555555", + &format!("{CURSORIDE_SESSION_PREFIX}{id}"), + &format!("{COMPOSER_KEY_PREFIX}{id}"), + &format!("{SOURCE_RECORD_KEY_PREFIX}{COMPOSER_KEY_PREFIX}{id}"), + "New Agent", + "New Chat", + "Untitled", + "Untitled Cursor session", + "Cursor session", + "Composer", + ] { + assert_eq!( + preferred_cursor_title(id, placeholder, "", preview), + "Explain the replay architecture", + "placeholder title {placeholder:?} must not leak into the sidebar" + ); + } + assert_eq!( + preferred_cursor_title(id, "Investigate issue 443", "", preview), + "Investigate issue 443" + ); +} + +#[test] +fn unchanged_no_index_shell_skips_repeated_blob_probes_and_rechecks_on_source_change() { + let id = "d3880bd5-3420-4d93-8f48-abef565346b2"; + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input(id, id, true)], + ) + .expect("seed polluted UUID shell"); + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![ + format!("{COMPOSER_KEY_PREFIX}{id}"), + serde_json::json!({ + "composerId": id, + "name": id, + "createdAt": 100, + "lastUpdatedAt": 200, + "fullConversationHeadersOnly": [] + }) + .to_string() + ], + ) + .expect("insert empty composer shell"); + + let generation_one = storage_snapshot("state-db-1", "main:100:10|wal:-"); + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&generation_one), + ) + .expect("validate empty shell"); + let (listable, name, parser_version) = cached_cursor_projection(&cache, id); + assert_eq!(listable, 0, "an empty composer shell must stay hidden"); + assert!( + name.is_empty(), + "a raw composer UUID must not become a title" + ); + assert_eq!(parser_version, CURSOR_IDE_METADATA_PARSER_VERSION); + + reset_cursor_content_probe_count(); + for _ in 0..2 { + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&generation_one), + ) + .expect("unchanged refresh"); + } + assert_eq!( + cursor_content_probe_count(), + 0, + "two unchanged refreshes must not point-read the composer blob again" + ); + + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![ + format!("{BUBBLE_KEY_PREFIX}{id}:user-1"), + serde_json::json!({ + "bubbleId": "user-1", + "type": 1, + "text": "Show the real Cursor conversation", + "createdAt": "2026-07-26T04:00:00Z" + }) + .to_string() + ], + ) + .expect("append real user bubble"); + + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:100:10|wal:4096:200")), + ) + .expect("revalidate after physical source change"); + let (listable, name, parser_version) = cached_cursor_projection(&cache, id); + assert_eq!(listable, 1); + assert_eq!(name, "Show the real Cursor conversation"); + assert_eq!(parser_version, CURSOR_IDE_METADATA_PARSER_VERSION); + assert!( + cursor_content_probe_count() > 0, + "a changed source must recheck the known composer" + ); +} + +#[path = "tests/validation_regressions.rs"] +mod validation_regressions; diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/tests/validation_regressions.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/tests/validation_regressions.rs new file mode 100644 index 000000000..d053aa26a --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/db/tests/validation_regressions.rs @@ -0,0 +1,542 @@ +use super::*; +use std::collections::HashSet; + +#[test] +fn no_index_validation_is_bounded_and_visible_v7_rows_ignore_wal_churn() { + let mut cache = cursor_cache_conn(); + let cursor = Connection::open_in_memory().expect("open cursor db"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + + let mut legacy = Vec::new(); + for ordinal in 0..(NO_INDEX_VALIDATION_BATCH_SIZE + 6) { + let id = format!("bounded-{ordinal:03}"); + legacy.push(legacy_cursor_cache_input(&id, "", false)); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![ + format!("{COMPOSER_KEY_PREFIX}{id}"), + serde_json::json!({ + "composerId": id, + "name": "", + "createdAt": 100, + "lastUpdatedAt": 200, + "fullConversationHeadersOnly": [] + }) + .to_string() + ], + ) + .expect("insert bounded composer"); + } + source_cache::upsert_imported_session_cache_from_conn(&mut cache, &legacy) + .expect("seed bounded migration batch"); + + let initial = storage_snapshot("state-db-1", "main:100:10|wal:-"); + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&initial), + ) + .expect("run first bounded migration batch"); + assert_eq!( + cursor_content_probe_count(), + NO_INDEX_VALIDATION_BATCH_SIZE, + "one refresh may point-read at most one hard-bounded migration batch" + ); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&initial), + ) + .expect("run remaining migration batch"); + assert_eq!(cursor_content_probe_count(), 6); + + // A visible v7 row has already been validated. Ordinary main/WAL activity + // must not make the no-index fallback reparse every visible root. + cache + .execute( + "DELETE FROM imported_history_session_cache WHERE source=?1", + [SOURCE_CURSOR_IDE], + ) + .expect("clear bounded hidden-row fixture"); + let visible_id = "visible-v7"; + let mut visible = legacy_cursor_cache_input(visible_id, "Visible history", true); + visible.parser_version = CURSOR_IDE_METADATA_PARSER_VERSION; + visible.source_metadata_json = Some( + serde_json::json!({ + "noIndexDatabaseIdentity": "state-db-1", + "noIndexActivitySignature": "main:100:10|wal:-" + }) + .to_string(), + ); + source_cache::upsert_imported_session_cache_from_conn(&mut cache, &[visible]) + .expect("seed validated visible row"); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:100:10|wal:4096:200")), + ) + .expect("ignore ordinary WAL churn for visible rows"); + assert_eq!( + cursor_content_probe_count(), + 0, + "WAL activity may retry hidden rows, never the full visible catalog" + ); +} + +#[test] +fn indeterminate_no_index_reads_rotate_so_later_hidden_rows_are_not_starved() { + let mut cache = cursor_cache_conn(); + let mut hidden = Vec::new(); + for ordinal in 0..(NO_INDEX_VALIDATION_BATCH_SIZE + 6) { + let id = format!("retry-fairness-{ordinal:03}"); + let mut input = legacy_cursor_cache_input(&id, "Hidden history", false); + input.parser_version = CURSOR_IDE_METADATA_PARSER_VERSION; + input.source_metadata_json = Some( + serde_json::json!({ + "noIndexDatabaseIdentity": "state-db-1", + "noIndexActivitySignature": "old-activity" + }) + .to_string(), + ); + hidden.push(input); + } + source_cache::upsert_imported_session_cache_from_conn(&mut cache, &hidden) + .expect("seed hidden retry candidates"); + + // No cursorDiskKV table: every point-read is indeterminate and therefore + // must remain retryable without being stamped as missing. + let unreadable_cursor = Connection::open_in_memory().expect("open unreadable Cursor DB"); + let changed = storage_snapshot("state-db-1", "new-activity"); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + None, + Some(&unreadable_cursor), + "/cursor/state.vscdb", + Some(&changed), + ) + .expect("run first retry batch"); + assert_eq!(cursor_content_probe_count(), NO_INDEX_VALIDATION_BATCH_SIZE); + let first_batch = cursor_content_probed_ids(); + assert_eq!( + first_batch.iter().collect::>().len(), + NO_INDEX_VALIDATION_BATCH_SIZE + ); + + let probes_after_first_batch = cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + None, + Some(&unreadable_cursor), + "/cursor/state.vscdb", + Some(&changed), + ) + .expect("run second retry batch"); + assert_eq!( + cursor_content_probe_count() - probes_after_first_batch, + NO_INDEX_VALIDATION_BATCH_SIZE, + "each retry pass must retain the hard batch limit" + ); + let all_probed = cursor_content_probed_ids() + .into_iter() + .collect::>(); + assert_eq!( + all_probed.len(), + NO_INDEX_VALIDATION_BATCH_SIZE + 6, + "oldest-first retry bookkeeping must cover all 70 rows in two bounded passes" + ); + for input in &hidden { + assert!( + all_probed.contains(&input.source_session_id), + "hidden row {} was starved by earlier read errors", + input.source_session_id + ); + assert_eq!( + cached_cursor_projection(&cache, &input.source_session_id).0, + 0, + "an indeterminate read must not promote or otherwise mutate projection state" + ); + assert_eq!( + cached_cursor_metadata_string( + &cache, + &input.source_session_id, + NO_INDEX_ACTIVITY_SIGNATURE_FIELD, + ) + .as_deref(), + Some("old-activity"), + "retry rotation must not stamp validation success" + ); + } +} + +#[test] +fn legacy_v7_no_index_stamp_is_revalidated_instead_of_blessing_current_database() { + let id = "legacy-v7-watermark"; + let mut cache = cursor_cache_conn(); + let mut cached = legacy_cursor_cache_input(id, "Possibly stale card", true); + cached.parser_version = CURSOR_IDE_METADATA_PARSER_VERSION; + cached.source_metadata_json = Some( + serde_json::json!({ + "noIndexValidationSignature": "old-main-and-wal-signature" + }) + .to_string(), + ); + source_cache::upsert_imported_session_cache_from_conn(&mut cache, &[cached]) + .expect("seed legacy v7 validation watermark"); + let cursor = Connection::open_in_memory().expect("open replacement Cursor DB"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + None, + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-new", "main:20:30|wal:-")), + ) + .expect("revalidate legacy v7 watermark"); + assert_eq!(cursor_content_probe_count(), 1); + assert_eq!( + cached_cursor_projection(&cache, id), + ( + 0, + "Possibly stale card".to_string(), + CURSOR_IDE_METADATA_PARSER_VERSION + ), + "a legacy activity-only marker cannot prove the card belongs to the current DB" + ); + assert_eq!( + cached_cursor_metadata_string(&cache, id, NO_INDEX_DATABASE_IDENTITY_FIELD).as_deref(), + Some("state-db-new") + ); +} + +#[test] +fn new_index_row_with_missing_blob_is_memoized_then_recovers_on_activity_change() { + let id = "new-index-missing"; + let mut cache = cursor_cache_conn(); + let index = Connection::open_in_memory().expect("open Cursor index"); + index + .execute( + "CREATE TABLE conversations ( + id TEXT, title TEXT, updated_at INTEGER, is_archived INTEGER, + root_fingerprint TEXT, source TEXT + )", + [], + ) + .expect("create Cursor conversation index"); + index + .execute( + "INSERT INTO conversations VALUES (?1,'New indexed title',300,0,'fp','local')", + [id], + ) + .expect("insert indexed conversation"); + let cursor = Connection::open_in_memory().expect("open Cursor state DB"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + let stable = storage_snapshot("state-db-1", "main:300:30|wal:-"); + + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&stable), + ) + .expect("memoize missing new index row"); + assert_eq!( + cached_cursor_projection(&cache, id), + ( + 0, + "New indexed title".to_string(), + CURSOR_IDE_METADATA_PARSER_VERSION + ) + ); + let validation = + cached_cursor_index_blob_validation(&cache, id).expect("missing-row validation marker"); + assert_eq!(validation.misses, 2); + assert_eq!(validation.database_identity, "state-db-1"); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&stable), + ) + .expect("skip unchanged missing new index row"); + assert_eq!( + cursor_content_probe_count(), + 0, + "a compact tombstone prevents an unchanged index row from being re-probed" + ); + + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![ + format!("{COMPOSER_KEY_PREFIX}{id}"), + serde_json::json!({ + "composerId": id, + "name": "", + "createdAt": 100, + "lastUpdatedAt": 400, + "fullConversationHeadersOnly": [{"bubbleId": "u1", "type": 1}] + }) + .to_string() + ], + ) + .expect("restore composer"); + cursor + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![ + format!("{BUBBLE_KEY_PREFIX}{id}:u1"), + serde_json::json!({ + "bubbleId": "u1", + "type": 1, + "text": "Recovered new index row", + "createdAt": "2026-07-26T04:00:00Z" + }) + .to_string() + ], + ) + .expect("restore user bubble"); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-1", "main:300:30|wal:4096:400")), + ) + .expect("promote new index row after source activity"); + assert_eq!(cached_cursor_projection(&cache, id).0, 1); +} + +#[test] +fn indexed_present_marker_revalidates_on_database_replacement_without_index_change() { + let id = "replacement-indexed"; + let mut cache = cursor_cache_conn(); + let index = Connection::open_in_memory().expect("open Cursor index"); + index + .execute( + "CREATE TABLE conversations ( + id TEXT, title TEXT, updated_at INTEGER, is_archived INTEGER, + root_fingerprint TEXT, source TEXT + )", + [], + ) + .expect("create Cursor conversation index"); + index + .execute( + "INSERT INTO conversations VALUES (?1,'Replacement title',300,0,'fp','local')", + [id], + ) + .expect("insert indexed conversation"); + let original = Connection::open_in_memory().expect("open original state DB"); + original + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create original cursorDiskKV"); + for (key, value) in [ + ( + format!("{COMPOSER_KEY_PREFIX}{id}"), + serde_json::json!({ + "composerId": id, + "name": "", + "createdAt": 100, + "lastUpdatedAt": 300, + "fullConversationHeadersOnly": [{"bubbleId": "u1", "type": 1}] + }) + .to_string(), + ), + ( + format!("{BUBBLE_KEY_PREFIX}{id}:u1"), + serde_json::json!({ + "bubbleId": "u1", + "type": 1, + "text": "Original DB history", + "createdAt": "2026-07-26T04:00:00Z" + }) + .to_string(), + ), + ] { + original + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![key, value], + ) + .expect("seed original state DB"); + } + + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&original), + "/cursor/state.vscdb", + Some(&storage_snapshot("state-db-old", "main:300:30|wal:-")), + ) + .expect("validate original state DB"); + assert_eq!(cached_cursor_projection(&cache, id).0, 1); + + let replacement = Connection::open_in_memory().expect("open replacement state DB"); + replacement + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create replacement cursorDiskKV"); + let replacement_snapshot = storage_snapshot("state-db-new", "main:300:30|wal:-"); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&replacement), + "/cursor/state.vscdb", + Some(&replacement_snapshot), + ) + .expect("revalidate same index against replaced state DB"); + assert_eq!( + cached_cursor_projection(&cache, id).0, + 0, + "a stable index cannot keep a card visible across physical DB replacement" + ); + let validation = + cached_cursor_index_blob_validation(&cache, id).expect("replacement validation marker"); + assert_eq!(validation.misses, 2); + assert_eq!(validation.database_identity, "state-db-new"); + + reset_cursor_content_probe_count(); + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&replacement), + "/cursor/state.vscdb", + Some(&replacement_snapshot), + ) + .expect("skip unchanged replacement miss"); + assert_eq!(cursor_content_probe_count(), 0); + + for (key, value) in [ + ( + format!("{COMPOSER_KEY_PREFIX}{id}"), + serde_json::json!({ + "composerId": id, + "name": "", + "createdAt": 100, + "lastUpdatedAt": 400, + "fullConversationHeadersOnly": [{"bubbleId": "u2", "type": 1}] + }) + .to_string(), + ), + ( + format!("{BUBBLE_KEY_PREFIX}{id}:u2"), + serde_json::json!({ + "bubbleId": "u2", + "type": 1, + "text": "Replacement DB history", + "createdAt": "2026-07-26T05:00:00Z" + }) + .to_string(), + ), + ] { + replacement + .execute( + "INSERT INTO cursorDiskKV VALUES (?1, ?2)", + params![key, value], + ) + .expect("restore replacement state DB"); + } + delta_sync_from_connections( + &mut cache, + Some(&index), + Some(&replacement), + "/cursor/state.vscdb", + Some(&storage_snapshot( + "state-db-new", + "main:300:30|wal:4096:400", + )), + ) + .expect("promote replacement state DB after activity"); + assert_eq!(cached_cursor_projection(&cache, id).0, 1); + let validation = + cached_cursor_index_blob_validation(&cache, id).expect("recovered validation marker"); + assert_eq!(validation.misses, 0); + assert_eq!(validation.database_identity, "state-db-new"); +} + +#[test] +fn index_open_or_query_failure_preserves_cached_projection_without_stamping() { + let id = "index-query-failure"; + let mut cache = cursor_cache_conn(); + source_cache::upsert_imported_session_cache_from_conn( + &mut cache, + &[legacy_cursor_cache_input(id, "Last known card", true)], + ) + .expect("seed last known card"); + let invalid_index = Connection::open_in_memory().expect("open invalid index"); + let cursor = Connection::open_in_memory().expect("open Cursor DB"); + cursor + .execute( + "CREATE TABLE cursorDiskKV (key TEXT PRIMARY KEY, value TEXT)", + [], + ) + .expect("create cursorDiskKV"); + let snapshot = storage_snapshot("state-db-1", "main:300:30|wal:-"); + + delta_sync_from_connections( + &mut cache, + Some(&invalid_index), + Some(&cursor), + "/cursor/state.vscdb", + Some(&snapshot), + ) + .expect("tolerate index query failure"); + assert_eq!( + cached_cursor_projection(&cache, id), + (1, "Last known card".to_string(), 2) + ); + assert!(cached_cursor_metadata_string(&cache, id, NO_INDEX_DATABASE_IDENTITY_FIELD).is_none()); + + delta_sync_from_connections( + &mut cache, + Some(&invalid_index), + None, + "/cursor/state.vscdb", + Some(&snapshot), + ) + .expect("tolerate state DB open failure"); + assert_eq!( + cached_cursor_projection(&cache, id), + (1, "Last known card".to_string(), 2) + ); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/helpers/session.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/helpers/session.rs index 6abf8f391..470f694e2 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/helpers/session.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/helpers/session.rs @@ -6,37 +6,63 @@ use super::*; // Session helper: list filter & cache-row conversion // ============================================================================ -pub(in crate::sources::cursor_ide) fn is_listable_cursor_session( - row: &super::db::CursorSession, - cursor_conn: Option<&Connection>, -) -> Result { - let Some(conn) = cursor_conn else { - return Ok(false); - }; - if row.name.trim().is_empty() { - return Ok(false); - } - // Fast path: single EXISTS query on cursorDiskKV. - // We only need to know whether the session has at least one user bubble - // (bubble_type == 1). Parsing the JSON value is enough — no blob reads, - // no diff, no full order reconstruction. - // load_bubble_order/load_complete_bubble_order fetches all rows AND - // deserialises every bubble value; that was the ~542% CPU hot path. - let prefix = format!("bubbleId:{}:", row.id); - let upper_bound = format!("bubbleId:{};", row.id); - let found: bool = conn - .query_row( - "SELECT EXISTS( - SELECT 1 FROM cursorDiskKV - WHERE key >= ?1 AND key < ?2 - AND json_extract(value, '$.type') = ?3 - LIMIT 1 - )", +#[derive(Debug, Default)] +pub(in crate::sources::cursor_ide) struct CursorUserBubbleProbe { + pub has_user_bubble: bool, + pub first_user_preview: Option, +} + +/// Inspect only one composer's indexed bubble-key range. +/// +/// Cursor occasionally omits a bubble from +/// `fullConversationHeadersOnly`, so header point-lookups alone are not an +/// authoritative existence check. The primary-key range keeps this bounded to +/// the requested session and avoids the old whole-`cursorDiskKV` scan. +pub(in crate::sources::cursor_ide) fn probe_indexed_cursor_user_bubbles( + conn: &Connection, + composer_id: &str, +) -> Result { + let prefix = format!("bubbleId:{composer_id}:"); + let upper_bound = format!("bubbleId:{composer_id};"); + let mut stmt = conn + .prepare( + "SELECT value + FROM cursorDiskKV + WHERE key >= ?1 AND key < ?2 + AND CASE + WHEN json_valid(value) + THEN json_extract(value, '$.type') + END = ?3 + ORDER BY key ASC", + ) + .map_err(|err| format!("Failed to prepare Cursor user-bubble range lookup: {err}"))?; + let rows = stmt + .query_map( rusqlite::params![prefix, upper_bound, CURSOR_BUBBLE_TYPE_USER], - |r| r.get(0), + |row| row.get::<_, Option>(0), ) - .unwrap_or(false); - Ok(found) + .map_err(|err| format!("Failed to query Cursor user-bubble range: {err}"))?; + let mut probe = CursorUserBubbleProbe::default(); + for row in rows { + let Some(value) = + row.map_err(|err| format!("Failed to read Cursor user-bubble range row: {err}"))? + else { + continue; + }; + let Ok(bubble) = serde_json::from_str::(&value) else { + continue; + }; + if bubble.bubble_type != CURSOR_BUBBLE_TYPE_USER { + continue; + } + probe.has_user_bubble = true; + let preview = preview_text(&bubble.text); + if !preview.is_empty() { + probe.first_user_preview = Some(preview); + break; + } + } + Ok(probe) } /// Convert a cache row to the sidebar-ready session shape. diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/history.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/history.rs index 4741ce601..fec451a4e 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/history.rs @@ -1,9 +1,9 @@ //! Cursor IDE chat history reader //! //! Reads `bubbleId:{composerId}:{bubbleId}` blobs from Cursor's `state.vscdb` -//! and converts each bubble into our canonical [`ActivityChunk`] shape, so the -//! existing event pipeline (`processChunksRust` → `eventStoreProxy` → -//! `ChatHistory`) can render Cursor IDE chat history with no UI-layer changes. +//! and converts individual bubbles into our canonical [`ActivityChunk`] shape. +//! Imported sessions reach this parser through the bounded SQLite/KV replay +//! driver; the renderer no longer owns a Cursor-specific full-refresh path. //! //! ## Read-only contract //! @@ -34,12 +34,13 @@ use super::db as cursor_db; use super::helpers::{ bubbles_to_chunks, build_fallback_user_chunk, build_unloaded_turn_placeholder_chunk, cache_row_to_session_row, composer_source_updated_at, enforce_monotonic_created_at, - is_listable_cursor_session, }; use super::io::{ load_bubbles_by_id, load_complete_bubble_order, load_composer_for_order, open_cursor_db, }; -use super::models::{CursorComposerContext, OrderedBubble, RawComposerHeader}; +use super::models::{ + CursorComposerContext, OrderedBubble, RawBubble, RawComposerForOrder, RawComposerHeader, +}; use super::summaries::{ build_cursor_ide_turn_summaries, cursor_ide_summary_source_fingerprint, load_cached_cursor_ide_turn_summaries, upsert_cursor_ide_turn_summaries, @@ -56,7 +57,7 @@ use super::helpers::{ #[cfg(test)] use super::io::load_content_blob; #[cfg(test)] -use super::models::{RawBubble, RawComposerForOrder, RawCursorSubagentInfo, RawToolFormerData}; +use super::models::{RawCursorSubagentInfo, RawToolFormerData}; #[cfg(test)] use serde_json::{json, Value}; @@ -145,13 +146,6 @@ pub struct CursorIdeInitialWindow { pub has_unloaded_middle: bool, } -#[derive(Debug, Clone, Serialize)] -#[serde(rename_all = "camelCase")] -pub struct CursorIdeFullRefresh { - pub chunks: Vec, - pub turns: Vec, -} - #[derive(Debug, Clone, Serialize)] #[serde(rename_all = "camelCase")] pub struct CursorIdeTurnWindow { @@ -176,14 +170,11 @@ pub fn list_cursor_ide_sessions_paginated( limit: usize, offset: usize, ) -> Result { - // Open Cursor's DB once for `is_listable_cursor_session` (bubble-type header - // check only — no blob reads, no diff). `cache_row_to_session_row` no longer - // needs it: hover-only fields are deferred to `cursor_ide_session_detail`. - let cursor_conn = open_cursor_db(); - let (rows, has_more) = - cursor_db::list_for_sidebar_filtered(cache_conn, limit, offset, |row| { - is_listable_cursor_session(row, cursor_conn.as_ref()) - })?; + // Listability is established once by the adapter's bounded delta sync and + // persisted in the compact cache. Re-probing every visible session's + // bubble range here made unchanged sidebar refreshes touch Cursor's large + // database again. + let (rows, has_more) = cursor_db::list_for_sidebar(cache_conn, limit, offset)?; let mut sessions = rows .into_iter() .map(cache_row_to_session_row) @@ -251,72 +242,31 @@ pub fn load_history_for_session(session_id: &str) -> Result, Ok(chunks) } -pub fn load_full_refresh_for_session( - cache_conn: &mut Connection, +/// Normalize a single Cursor KV bubble for bounded replay. +/// +/// The replay driver owns ordering/diffing and calls this only for a new or +/// changed bubble, so this wrapper does not allocate a full bubble list. +pub(crate) fn replay_chunk_from_bubble_json( + conn: &Connection, session_id: &str, -) -> Result { - let composer_id = strip_session_prefix(session_id); - - let cursor_conn = match open_cursor_db() { - Some(conn) => conn, - None => { - return Ok(CursorIdeFullRefresh { - chunks: vec![], - turns: vec![], - }) - } - }; - - let composer = load_composer_for_order(&cursor_conn, composer_id)?; - let composer_context = CursorComposerContext::from_composer(&composer); - let order = load_complete_bubble_order( - &cursor_conn, - composer_id, - &composer.full_conversation_headers_only, - )?; - let source_updated_at = - composer_source_updated_at(&cursor_conn, composer_id, &composer, &order)?; - if order.is_empty() { - return Ok(CursorIdeFullRefresh { - chunks: vec![], - turns: vec![], - }); - } - - let total_bubble_count = order.len(); - let bubbles = load_bubbles_by_id(&cursor_conn, composer_id, &order)?; - let source_fingerprint = cursor_ide_summary_source_fingerprint( - composer.created_at, - composer.last_updated_at, - source_updated_at, - &order, - ); - let turns = match load_cached_cursor_ide_turn_summaries( - cache_conn, - session_id, - source_updated_at, - total_bubble_count, - &source_fingerprint, - )? { - Some(cached_summaries) => cached_summaries, - None => { - let fresh_summaries = build_cursor_ide_turn_summaries(&order, &bubbles); - upsert_cursor_ide_turn_summaries( - cache_conn, - session_id, - composer_id, - source_updated_at, - total_bubble_count, - &source_fingerprint, - &fresh_summaries, - )?; - fresh_summaries - } + bubble_id: &str, + header_type: i64, + raw_json: &str, + composer_json: &str, +) -> Result, String> { + let raw = serde_json::from_str::(raw_json) + .map_err(|err| format!("Failed to parse Cursor replay bubble {bubble_id}: {err}"))?; + let composer = serde_json::from_str::(composer_json) + .map_err(|err| format!("Failed to parse Cursor replay composer: {err}"))?; + let context = CursorComposerContext::from_composer(&composer); + let bubble = OrderedBubble { + bubble_id: bubble_id.to_string(), + bubble_type: header_type, + raw, }; - - let mut chunks = bubbles_to_chunks(&cursor_conn, session_id, &bubbles, &composer_context); - enforce_monotonic_created_at(&mut chunks); - Ok(CursorIdeFullRefresh { chunks, turns }) + Ok(bubbles_to_chunks(conn, session_id, &[bubble], &context) + .into_iter() + .next()) } pub fn load_initial_window_for_session( diff --git a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/io.rs b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/io.rs index 13aae9cbe..814bb6ccd 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/io.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/cursor_ide/io.rs @@ -15,6 +15,21 @@ use super::models::{OrderedBubble, RawBubble, RawComposerForOrder, RawComposerHe const SQLITE_IN_QUERY_CHUNK_SIZE: usize = 500; +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum CursorDbPathState { + Present(PathBuf), + Missing, + /// Home/path metadata could not be resolved (for example macOS TCC + /// denial). Unknown is not evidence that Cursor or its history is absent. + Unknown, +} + +pub(super) enum CursorConversationIndexDbState { + Present(Connection), + Missing, + Unknown, +} + /// Open Cursor's global `state.vscdb` read-only. /// /// Returns `None` if the file does not exist (Cursor not installed / not yet @@ -60,27 +75,45 @@ fn cursor_global_storage_dir() -> Option { } pub(super) fn cursor_db_path() -> Option { - let path = cursor_global_storage_dir()?.join("state.vscdb"); - path.exists().then_some(path) + match cursor_db_path_state() { + CursorDbPathState::Present(path) => Some(path), + CursorDbPathState::Missing | CursorDbPathState::Unknown => None, + } } -/// Cursor's newer conversation index — a small, indexed SQLite listing every -/// conversation (`id`, `title`, `updated_at`, `is_archived`, `root_fingerprint`) -/// next to `state.vscdb`. Lets discovery avoid scanning the multi-GB `state.vscdb`. -/// `None` on older Cursor builds that predate it. -pub(super) fn cursor_conversation_index_path() -> Option { - let path = cursor_global_storage_dir()?.join("conversation-search.db"); - path.exists().then_some(path) +pub(super) fn cursor_db_path_state() -> CursorDbPathState { + let Some(storage_dir) = cursor_global_storage_dir() else { + return CursorDbPathState::Unknown; + }; + classify_cursor_db_path(storage_dir.join("state.vscdb")) } -/// Open the conversation index read-only, or `None` if absent/unreadable. -pub(super) fn open_cursor_conversation_index_db() -> Option { - let path = cursor_conversation_index_path()?; - Connection::open_with_flags( - &path, - OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX, - ) - .ok() +fn classify_cursor_db_path(path: PathBuf) -> CursorDbPathState { + match path.try_exists() { + Ok(true) => CursorDbPathState::Present(path), + Ok(false) => CursorDbPathState::Missing, + Err(_) => CursorDbPathState::Unknown, + } +} + +/// Open Cursor's optional conversation index while preserving the difference +/// between an older install that definitively has no index and an index that +/// exists but is temporarily unreadable. Only the former may use the bounded +/// no-index compatibility path. +pub(super) fn open_cursor_conversation_index_db_state() -> CursorConversationIndexDbState { + let Some(storage_dir) = cursor_global_storage_dir() else { + return CursorConversationIndexDbState::Unknown; + }; + match classify_cursor_db_path(storage_dir.join("conversation-search.db")) { + CursorDbPathState::Present(path) => Connection::open_with_flags( + &path, + OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX, + ) + .map(CursorConversationIndexDbState::Present) + .unwrap_or(CursorConversationIndexDbState::Unknown), + CursorDbPathState::Missing => CursorConversationIndexDbState::Missing, + CursorDbPathState::Unknown => CursorConversationIndexDbState::Unknown, + } } pub(super) fn load_complete_bubble_order( @@ -278,6 +311,9 @@ pub(super) fn load_content_blob(conn: &Connection, content_id: &str) -> Option Connection { let conn = Connection::open_in_memory().expect("open in-memory db"); @@ -350,4 +386,24 @@ mod tests { assert_eq!(ids, vec!["early-user", "early-edit", "header-user"]); } + + #[test] + fn path_state_distinguishes_definitive_absence_from_presence() { + let path = std::env::temp_dir().join(format!( + "orgii-cursor-path-state-{}-{}.vscdb", + std::process::id(), + PATH_STATE_FIXTURE_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )); + assert_eq!( + classify_cursor_db_path(path.clone()), + CursorDbPathState::Missing + ); + + std::fs::write(&path, b"sqlite fixture").expect("create Cursor path-state fixture"); + assert_eq!( + classify_cursor_db_path(path.clone()), + CursorDbPathState::Present(path.clone()) + ); + std::fs::remove_file(path).expect("remove Cursor path-state fixture"); + } } diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache.rs index 8a44cb1e6..c13293c84 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache.rs @@ -14,9 +14,9 @@ use super::metadata::{ RoundUsage, }; use super::{ - effective_limit, recent_paths_from_rows, row_from_input, ImportedHistoryRecentPath, - ImportedHistoryRowInput, ImportedHistorySessionPage, ImportedHistorySessionRow, - ImportedHistorySidebarPage, ImportedHistorySidebarRow, + effective_limit, epoch_ms_to_iso, repo_name_from_path, row_from_input, + ImportedHistoryRecentPath, ImportedHistoryRowInput, ImportedHistorySessionPage, + ImportedHistorySessionRow, ImportedHistorySidebarPage, ImportedHistorySidebarRow, }; #[derive(Debug, Clone)] @@ -387,6 +387,67 @@ pub fn query_imported_sidebar_page_from_conn( limit: usize, offset: usize, ) -> Result { + query_imported_sidebar_scoped_page_from_conn( + conn, + ImportedHistorySidebarPageQuery { + source, + start_ms, + end_ms, + repo_path: None, + missing_repo_path: false, + before_updated_at_ms: None, + before_session_id: None, + limit, + offset, + }, + ) +} + +pub struct ImportedHistorySidebarPageQuery<'a> { + pub source: &'a str, + pub start_ms: Option, + pub end_ms: Option, + pub repo_path: Option<&'a str>, + pub missing_repo_path: bool, + pub before_updated_at_ms: Option, + pub before_session_id: Option<&'a str>, + pub limit: usize, + pub offset: usize, +} + +/// Query a bounded sidebar page within one optional workspace scope. +/// +/// Date and workspace predicates are applied before `LIMIT`/`OFFSET`. This is +/// important for the grouped sidebar: advancing the `/repo-a` cursor must not +/// consume rows that belong to `/repo-b`, and loading Older must not consume +/// Today rows. `missing_repo_path` is distinct from no workspace filter. +pub fn query_imported_sidebar_scoped_page_from_conn( + conn: &Connection, + query: ImportedHistorySidebarPageQuery<'_>, +) -> Result { + let ImportedHistorySidebarPageQuery { + source, + start_ms, + end_ms, + repo_path, + missing_repo_path, + before_updated_at_ms, + before_session_id, + limit, + offset, + } = query; + if repo_path.is_some() && missing_repo_path { + return Err( + "Imported sidebar scope cannot request both repo_path and missing_repo_path" + .to_string(), + ); + } + if before_updated_at_ms.is_some() != before_session_id.is_some() { + return Err( + "Imported sidebar before_updated_at_ms and before_session_id must be provided together" + .to_string(), + ); + } let limit = effective_limit(limit); let mut range_sql = String::new(); let mut values = vec![SqlValue::from(source.to_string())]; @@ -398,10 +459,36 @@ pub fn query_imported_sidebar_page_from_conn( values.push(SqlValue::from(end_ms)); range_sql.push_str(&format!(" AND updated_at_ms < ?{}", values.len())); } + if let Some(repo_path) = repo_path { + values.push(SqlValue::from(repo_path.to_string())); + range_sql.push_str(&format!( + " AND RTRIM(cache.repo_path, '/') = RTRIM(?{}, '/')", + values.len() + )); + } else if missing_repo_path { + range_sql.push_str(" AND TRIM(cache.repo_path) = ''"); + } + if let Some(before_updated_at_ms) = before_updated_at_ms { + values.push(SqlValue::from(before_updated_at_ms)); + let updated_param = values.len(); + values.push(SqlValue::from( + before_session_id.unwrap_or_default().to_string(), + )); + let session_param = values.len(); + range_sql.push_str(&format!( + " AND (cache.updated_at_ms < ?{updated_param} + OR (cache.updated_at_ms = ?{updated_param} + AND cache.session_id < ?{session_param}))" + )); + } let limit_param = values.len() + 1; let offset_param = values.len() + 2; values.push(SqlValue::from(limit.saturating_add(1) as i64)); - values.push(SqlValue::from(offset as i64)); + values.push(SqlValue::from(if before_updated_at_ms.is_some() { + 0 + } else { + offset as i64 + })); let sql = format!( "SELECT session_id, name, created_at_ms, updated_at_ms, cache.repo_path, model, files_changed, lines_added, lines_removed, touched_files_json, @@ -414,8 +501,7 @@ pub fn query_imported_sidebar_page_from_conn( AND cache.listable = 1 AND cache.parent_session_id = '' {range_sql} - ORDER BY cache.updated_at_ms DESC, cache.created_at_ms DESC, - cache.source_session_id ASC + ORDER BY cache.updated_at_ms DESC, cache.session_id DESC LIMIT ?{limit_param} OFFSET ?{offset_param}" ); let mut stmt = conn @@ -440,36 +526,53 @@ pub fn query_imported_sidebar_page_from_conn( .map_err(|err| { rusqlite::Error::FromSqlConversionFailure(14, Type::Text, Box::new(err)) })?; - Ok(ImportedHistorySidebarRow { - session_id: row.get(0)?, - name: row.get(1)?, - created_at: super::epoch_ms_to_iso(row.get(2)?), - updated_at: super::epoch_ms_to_iso(row.get(3)?), - status: None, - is_active: None, - repo_path: non_empty_string(repo_path), - repo_root_path: repo_root_path.and_then(non_empty_string), - repo_remote_urls, - storage_path: non_empty_string(source_path), - model: non_empty_string(model), - total_tokens: input_tokens + output_tokens, - files_changed: row.get(6)?, - lines_added: row.get(7)?, - lines_removed: row.get(8)?, - touched_files, - }) + let session_id: String = row.get(0)?; + let updated_at_ms: i64 = row.get(3)?; + Ok(( + ImportedHistorySidebarRow { + session_id: session_id.clone(), + name: row.get(1)?, + created_at: super::epoch_ms_to_iso(row.get(2)?), + updated_at: super::epoch_ms_to_iso(updated_at_ms), + status: None, + is_active: None, + repo_path: non_empty_string(repo_path), + repo_root_path: repo_root_path.and_then(non_empty_string), + repo_remote_urls, + storage_path: non_empty_string(source_path), + model: non_empty_string(model), + total_tokens: input_tokens + output_tokens, + files_changed: row.get(6)?, + lines_added: row.get(7)?, + lines_removed: row.get(8)?, + touched_files, + }, + super::ImportedHistorySidebarCursor { + updated_at_ms, + session_id, + }, + )) }) .map_err(|err| format!("Failed to query imported sidebar rows for {source}: {err}"))?; - let mut sessions = Vec::new(); + let mut rows_with_cursors = Vec::new(); for row in rows { - sessions.push( + rows_with_cursors.push( row.map_err(|err| format!("Failed to read imported sidebar row for {source}: {err}"))?, ); } - let has_more = sessions.len() > limit; - sessions.truncate(limit); - Ok(ImportedHistorySidebarPage { sessions, has_more }) + let has_more = rows_with_cursors.len() > limit; + rows_with_cursors.truncate(limit); + let next_cursor = rows_with_cursors.last().map(|(_, cursor)| cursor.clone()); + let sessions = rows_with_cursors + .into_iter() + .map(|(session, _)| session) + .collect(); + Ok(ImportedHistorySidebarPage { + sessions, + has_more, + next_cursor, + }) } pub fn query_imported_recent_paths_from_conn( @@ -477,16 +580,34 @@ pub fn query_imported_recent_paths_from_conn( source: &str, limit: usize, ) -> Result, String> { - let rows = query_cached_sessions_from_conn(conn, source, i64::MAX as usize, 0)?; - Ok(recent_paths_from_rows( - &rows - .into_iter() - .map(|session| session.to_row()) - .collect::>(), - ) - .into_iter() - .take(effective_limit(limit)) - .collect()) + // Group and bound inside SQLite. The old implementation decoded every + // cached session (including touched-files JSON) into two Rust vectors and + // only then truncated to `limit`, so a lightweight Spotlight/settings + // query still grew with the number of historical sessions. + let mut statement = conn + .prepare( + "SELECT TRIM(repo_path) AS path, MAX(updated_at_ms), COUNT(*) + FROM imported_history_session_cache + WHERE source=?1 AND listable=1 AND parent_session_id='' + AND TRIM(repo_path)!='' + GROUP BY TRIM(repo_path) + ORDER BY MAX(updated_at_ms) DESC, path ASC + LIMIT ?2", + ) + .map_err(|err| format!("Failed to prepare imported recent-path query: {err}"))?; + let rows = statement + .query_map(params![source, effective_limit(limit) as i64], |row| { + let path = row.get::<_, String>(0)?; + Ok(ImportedHistoryRecentPath { + name: repo_name_from_path(&path), + path, + last_used_at: epoch_ms_to_iso(row.get::<_, i64>(1)?), + session_count: row.get::<_, i64>(2)?.max(0) as usize, + }) + }) + .map_err(|err| format!("Failed to query imported recent paths: {err}"))?; + rows.map(|row| row.map_err(|err| format!("Failed to read imported recent path: {err}"))) + .collect() } pub fn get_cached_source_path_from_conn( @@ -517,7 +638,10 @@ pub fn get_cached_source_path_by_suffix_from_conn( conn.query_row( "SELECT source_path FROM imported_history_session_cache \ WHERE source = ?1 \ - AND (source_session_id = ?2 OR source_session_id LIKE '%-' || ?2) \ + AND ( \ + source_session_id = ?2 \ + OR substr(source_session_id, -(length(?2) + 1)) = ('-' || ?2) \ + ) \ ORDER BY updated_at_ms DESC LIMIT 1", params![source, source_session_id], |row| row.get::<_, String>(0), @@ -836,8 +960,7 @@ fn query_cached_session_by_session_id_impl( let Some(session) = sessions.into_iter().next() else { return Ok(None); }; - if !include_continuation_superseded - && has_newer_continuation_sibling(conn, &source, &session)? + if !include_continuation_superseded && has_newer_continuation_sibling(conn, &source, &session)? { return Ok(None); } @@ -1007,6 +1130,51 @@ where .collect()) } +/// Advance only the discovery signature for an already-cached catalog row. +/// +/// Sidebar rescans must notice that an external transcript grew without +/// re-reading its body merely to rebuild metadata we already persisted. The +/// bounded replay index owns body/token/impact updates; this helper preserves +/// those fields and updates only source identity, freshness, and the optional +/// listability decision derived from cheap source indexes. +pub fn advance_cached_catalog_record_from_conn( + conn: &Connection, + source: &str, + record: &super::metadata::ImportedHistoryDiscoveredRecord, + listable: Option, +) -> Result { + let updated_at_ms = record.source_mtime_ms.saturating_div(1_000_000); + let changed = conn + .execute( + "UPDATE imported_history_session_cache + SET source_path = ?3, + source_record_key = ?4, + source_mtime_ms = ?5, + source_size_bytes = ?6, + source_fingerprint = ?7, + parser_version = ?8, + updated_at_ms = MAX(updated_at_ms, ?9), + listable = CASE WHEN ?10 IS NULL THEN listable ELSE ?10 END, + updated_at = ?11 + WHERE source = ?1 AND source_session_id = ?2", + params![ + source, + record.source_session_id, + record.source_path.to_string_lossy(), + record.source_record_key, + record.source_mtime_ms, + record.source_size_bytes, + record.source_fingerprint, + record.parser_version, + updated_at_ms, + listable.map(i64::from), + Utc::now().to_rfc3339(), + ], + ) + .map_err(|err| format!("Failed to advance imported catalog record: {err}"))?; + Ok(changed > 0) +} + pub fn live_ids_from_signatures(signatures: &[ImportedHistoryRecordSignature]) -> Vec { let mut seen = HashSet::new(); signatures diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache_tests.rs index 90fedd0db..df98cc407 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache_tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/cache_tests.rs @@ -67,6 +67,25 @@ fn cache_query_paginates_newest_first() { assert_eq!(page.sessions[1].session_id, "codex_app-mid"); } +#[test] +fn cached_source_suffix_lookup_treats_percent_and_underscore_literally() { + let mut conn = fixture_conn(); + upsert_imported_session_cache_from_conn( + &mut conn, + &[ + input(SOURCE_CODEX_APP, "rollout-literal%_", 100), + input(SOURCE_CODEX_APP, "rollout-literalXY", 200), + ], + ) + .expect("upsert wildcard fixtures"); + + let source_path = + get_cached_source_path_by_suffix_from_conn(&conn, SOURCE_CODEX_APP, "literal%_") + .expect("literal suffix lookup"); + + assert_eq!(source_path.as_deref(), Some("/tmp/rollout-literal%_.jsonl")); +} + #[test] fn source_stats_batch_counts_roots_children_and_last_activity() { let mut conn = fixture_conn(); @@ -171,6 +190,157 @@ fn sidebar_query_paginates_within_one_date_bucket() { assert_eq!(second.sessions[0].session_id, "codex_app-old"); } +#[test] +fn sidebar_scope_filters_workspace_before_limit_and_offset() { + let mut conn = fixture_conn(); + let mut rows = Vec::new(); + for (id, updated_at, repo_path) in [ + ("a-new", 260, Some("/repo-a/")), + ("b-new", 250, Some("/repo-b")), + ("a-mid", 240, Some("/repo-a")), + ("b-mid", 230, Some("/repo-b")), + ("a-old", 220, Some("/repo-a")), + ("missing", 210, None), + ] { + let mut row = input(SOURCE_CODEX_APP, id, updated_at); + row.repo_path = repo_path.map(str::to_string); + rows.push(row); + } + upsert_imported_session_cache_from_conn(&mut conn, &rows).expect("upsert"); + + let first = query_imported_sidebar_scoped_page_from_conn( + &conn, + ImportedHistorySidebarPageQuery { + source: SOURCE_CODEX_APP, + start_ms: Some(200), + end_ms: Some(300), + repo_path: Some("/repo-a"), + missing_repo_path: false, + before_updated_at_ms: None, + before_session_id: None, + limit: 2, + offset: 0, + }, + ) + .expect("first repo page"); + let second = query_imported_sidebar_scoped_page_from_conn( + &conn, + ImportedHistorySidebarPageQuery { + source: SOURCE_CODEX_APP, + start_ms: Some(200), + end_ms: Some(300), + repo_path: Some("/repo-a"), + missing_repo_path: false, + before_updated_at_ms: None, + before_session_id: None, + limit: 2, + offset: 2, + }, + ) + .expect("second repo page"); + let missing = query_imported_sidebar_scoped_page_from_conn( + &conn, + ImportedHistorySidebarPageQuery { + source: SOURCE_CODEX_APP, + start_ms: Some(200), + end_ms: Some(300), + repo_path: None, + missing_repo_path: true, + before_updated_at_ms: None, + before_session_id: None, + limit: 10, + offset: 0, + }, + ) + .expect("missing repo page"); + + assert!(first.has_more); + assert_eq!( + first + .sessions + .iter() + .map(|row| row.session_id.as_str()) + .collect::>(), + vec!["codex_app-a-new", "codex_app-a-mid"] + ); + assert!(!second.has_more); + assert_eq!(second.sessions[0].session_id, "codex_app-a-old"); + assert_eq!(missing.sessions[0].session_id, "codex_app-missing"); +} + +#[test] +fn imported_sidebar_seek_cursor_is_stable_across_newer_inserts_and_updates() { + let mut conn = fixture_conn(); + upsert_imported_session_cache_from_conn( + &mut conn, + &[ + input(SOURCE_CODEX_APP, "four", 400), + input(SOURCE_CODEX_APP, "three", 300), + input(SOURCE_CODEX_APP, "two", 200), + input(SOURCE_CODEX_APP, "one", 100), + ], + ) + .expect("seed imported seek rows"); + + let first = query_imported_sidebar_scoped_page_from_conn( + &conn, + ImportedHistorySidebarPageQuery { + source: SOURCE_CODEX_APP, + start_ms: None, + end_ms: None, + repo_path: None, + missing_repo_path: false, + before_updated_at_ms: None, + before_session_id: None, + limit: 2, + offset: 0, + }, + ) + .expect("first imported seek page"); + assert_eq!( + first + .sessions + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + ["codex_app-four", "codex_app-three"] + ); + let cursor = first.next_cursor.expect("first-page cursor"); + + upsert_imported_session_cache_from_conn( + &mut conn, + &[ + input(SOURCE_CODEX_APP, "new", 600), + input(SOURCE_CODEX_APP, "four", 500), + ], + ) + .expect("mutate imported head"); + let second = query_imported_sidebar_scoped_page_from_conn( + &conn, + ImportedHistorySidebarPageQuery { + source: SOURCE_CODEX_APP, + start_ms: None, + end_ms: None, + repo_path: None, + missing_repo_path: false, + before_updated_at_ms: Some(cursor.updated_at_ms), + before_session_id: Some(&cursor.session_id), + limit: 2, + // A stable cursor wins over a stale legacy offset. + offset: 99, + }, + ) + .expect("second imported seek page"); + assert_eq!( + second + .sessions + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + ["codex_app-two", "codex_app-one"] + ); +} + #[test] fn cache_pruning_is_source_scoped() { let mut conn = fixture_conn(); @@ -230,6 +400,35 @@ fn cache_recent_paths_are_deduped_and_limited() { assert_eq!(paths[0].session_count, 2); } +#[test] +fn cache_recent_paths_group_in_sql_without_decoding_session_payloads() { + let mut conn = fixture_conn(); + let inputs = (0..500) + .map(|index| { + let mut row = input(SOURCE_CODEX_APP, &format!("session-{index}"), index); + row.repo_path = Some(format!("/tmp/repo-{}", index % 25)); + row + }) + .collect::>(); + upsert_imported_session_cache_from_conn(&mut conn, &inputs).expect("upsert many sessions"); + // The old recent-path path decoded this unrelated JSON column for every + // session before applying its limit. A compact SQL aggregate must neither + // read it nor fail because an old cache row contains malformed metadata. + conn.execute( + "UPDATE imported_history_session_cache SET touched_files_json='not-json'", + [], + ) + .expect("poison unrelated session payload"); + + let paths = query_imported_recent_paths_from_conn(&conn, SOURCE_CODEX_APP, 5) + .expect("bounded SQL recent paths"); + assert_eq!(paths.len(), 5); + assert!(paths.iter().all(|path| path.session_count == 20)); + assert!(paths + .windows(2) + .all(|pair| pair[0].last_used_at >= pair[1].last_used_at)); +} + #[test] fn cache_single_session_lookup_returns_source_metadata() { let mut conn = fixture_conn(); @@ -423,8 +622,7 @@ fn canonical_lookup_skips_continuation_superseded_siblings() { let mut subagent = input(SOURCE_CODEX_APP, "child", 300); subagent.source_metadata_json = group; subagent.parent_session_id = Some("codex_app-gen2".to_string()); - upsert_imported_session_cache_from_conn(&mut conn, &[older, newest, subagent]) - .expect("upsert"); + upsert_imported_session_cache_from_conn(&mut conn, &[older, newest, subagent]).expect("upsert"); // The superseded sibling resolves to None whether or not the election ran. assert!( @@ -462,12 +660,10 @@ fn including_superseded_lookup_resolves_demoted_continuation_siblings() { // The vanished-session sweep's existence check must see the demoted // sibling: it still exists locally and its shared cloud row must survive // a context-window continuation. - let (_, demoted) = query_cached_session_by_session_id_including_superseded_from_conn( - &conn, - "codex_app-gen1", - ) - .expect("query gen1 including superseded") - .expect("demoted sibling resolves"); + let (_, demoted) = + query_cached_session_by_session_id_including_superseded_from_conn(&conn, "codex_app-gen1") + .expect("query gen1 including superseded") + .expect("demoted sibling resolves"); assert_eq!(demoted.source_session_id, "gen1"); assert!( query_cached_session_by_session_id_from_conn(&conn, "codex_app-gen1") @@ -494,8 +690,7 @@ fn canonical_lookup_tolerates_legacy_non_json_metadata_rows() { let mut newest = input(SOURCE_CODEX_APP, "gen2", 200); newest.source_metadata_json = group; let keyless = input(SOURCE_CODEX_APP, "journal", 300); - upsert_imported_session_cache_from_conn(&mut conn, &[older, newest, keyless]) - .expect("upsert"); + upsert_imported_session_cache_from_conn(&mut conn, &[older, newest, keyless]).expect("upsert"); conn.execute( "UPDATE imported_history_session_cache SET source_metadata_json = 'not-json' \ WHERE source = ?1 AND source_session_id = 'journal'", @@ -525,10 +720,9 @@ fn canonical_lookup_tolerates_legacy_non_json_metadata_rows() { .expect("winner lookup succeeds despite corrupt sibling rows") .expect("winner resolves"); assert_eq!(winner.source_session_id, "gen2"); - let (_, keyless_row) = - query_cached_session_by_session_id_from_conn(&conn, "codex_app-journal") - .expect("corrupt-metadata row still resolves by id") - .expect("corrupt-metadata row present"); + let (_, keyless_row) = query_cached_session_by_session_id_from_conn(&conn, "codex_app-journal") + .expect("corrupt-metadata row still resolves by id") + .expect("corrupt-metadata row present"); assert_eq!(keyless_row.source_session_id, "journal"); } @@ -587,8 +781,7 @@ fn source_cache_signature_tracks_upserts_demotions_and_prunes() { .expect("signature after demotion"); assert_ne!(after_second, after_demotion); - prune_missing_records_from_conn(&conn, SOURCE_CODEX_APP, &["gen2".to_string()]) - .expect("prune"); + prune_missing_records_from_conn(&conn, SOURCE_CODEX_APP, &["gen2".to_string()]).expect("prune"); let after_prune = query_source_cache_signature_from_conn(&conn, SOURCE_CODEX_APP) .expect("signature after prune"); assert_ne!(after_demotion, after_prune); diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/catalog.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/catalog.rs new file mode 100644 index 000000000..cd71be4e4 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/catalog.rs @@ -0,0 +1,839 @@ +//! Compact catalog refresh for external-history directory surfaces. +//! +//! Session listing itself reads only `imported_history_session_cache`. Source +//! discovery and metadata refresh are explicit operations routed through this +//! exhaustive registry, so adding a replay source cannot silently put a +//! transcript-hydrating loader back on the sidebar path. + +use rusqlite::{params, Connection, OptionalExtension, Transaction}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use super::replay::ImportedHistorySourceId; +use super::{self as imported_history, cache as imported_cache}; +use crate::projectors::turn_metadata::{metadata_projection_requirements, TurnMetadataAccumulator}; + +/// Small adapter-owned metadata fold persisted inside a replay driver's +/// cursor. It grows by fields, never by transcript length. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(default, rename_all = "camelCase")] +pub(crate) struct ReplayCatalogProjection { + pub title: Option, + pub title_priority: u8, + pub model: Option, + pub repo_path: Option, + pub branch: Option, + pub input_tokens: i64, + pub output_tokens: i64, + pub cache_read_tokens: i64, + pub cache_write_tokens: i64, + pub tokens_observed: bool, + pub parent_session_id: Option, + pub parent_observed: bool, + pub continuation_group_key: Option, + pub continuation_observed: bool, + pub created_at_ms: Option, + pub updated_at_ms: Option, + // Codex reports cumulative counters which can reset after compaction. + pub last_cumulative_input: i64, + pub last_cumulative_output: i64, + pub last_cumulative_cache_read: i64, + pub last_cumulative_cache_write: i64, + pub last_usage_message_id: Option, +} + +/// Snapshot of the mixed-ownership imported catalog row around one replay +/// projection. Discovery-owned signature fields are included as guards: if an +/// adapter refresh changes the row after replay publication, cache eviction +/// must preserve that newer row instead of restoring an obsolete baseline. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +struct ReplayCatalogRowSnapshot { + session_id: String, + source_path: String, + source_record_key: String, + source_mtime_ms: i64, + source_size_bytes: i64, + source_fingerprint: String, + parser_version: i64, + listable: i64, + name: String, + created_at_ms: i64, + updated_at_ms: i64, + model: String, + input_tokens: i64, + output_tokens: i64, + cache_read_tokens: i64, + cache_write_tokens: i64, + repo_path: String, + branch: String, + files_changed: i64, + lines_added: i64, + lines_removed: i64, + touched_files_json: String, + source_metadata_json: String, + parent_session_id: String, + updated_at: String, +} + +impl ReplayCatalogProjection { + pub(crate) fn observe_jsonl( + &mut self, + source: ImportedHistorySourceId, + raw: &Value, + source_session_id: &str, + ) { + self.observe_timestamp( + raw.get("timestamp") + .or_else(|| raw.get("createdAt")) + .or_else(|| raw.get("message_summary_time")), + ); + Self::set_nonempty_path( + &mut self.repo_path, + raw, + &["cwd", "project", "workspaceRoot"], + ); + Self::set_nonempty_path(&mut self.branch, raw, &["gitBranch", "git_branch"]); + + if source == ImportedHistorySourceId::Trae { + if let Some(intent) = nonempty_value(raw.get("intent")) { + self.set_title(intent, 1); + } + } + for (key, priority) in [ + ("summary", 2), + ("aiTitle", 3), + ("ai_title", 3), + ("customTitle", 4), + ("custom_title", 4), + ] { + if let Some(title) = nonempty_value(raw.get(key)) { + self.set_title(title, priority); + } + } + + let message = raw + .get("message") + .filter(|value| value.is_object()) + .or_else(|| { + (raw.get("type").and_then(Value::as_str) == Some("message")).then_some(raw) + }); + if let Some(message) = message { + if let Some(model) = nonempty_value( + message + .get("model") + .or_else(|| message.get("modelId")) + .or_else(|| raw.get("modelId")), + ) { + self.model = Some(model.to_string()); + } + let role = message + .get("role") + .and_then(Value::as_str) + .or_else(|| raw.get("role").and_then(Value::as_str)) + .or_else(|| raw.get("type").and_then(Value::as_str)); + if role == Some("user") && self.title_priority < 1 { + if let Some(text) = first_content_text(message.get("content")) { + self.set_title(&text, 1); + } + } + self.observe_message_usage(message); + } + + if source == ImportedHistorySourceId::ClaudeCode { + if !self.continuation_observed + && raw.get("type").and_then(Value::as_str) == Some("user") + { + self.continuation_observed = true; + self.continuation_group_key = nonempty_value(raw.get("uuid")).map(str::to_string); + } + if raw.get("isSidechain").is_some() { + self.parent_observed = true; + self.parent_session_id = raw + .get("isSidechain") + .and_then(Value::as_bool) + .filter(|value| *value) + .and_then(|_| nonempty_value(raw.get("sessionId"))) + .filter(|parent| *parent != source_session_id) + .map(|parent| format!("{}{parent}", source.descriptor().session_prefix)); + } + } + } + + pub(crate) fn observe_codex( + &mut self, + line_type: &str, + timestamp: Option<&str>, + payload: &Value, + source_session_id: &str, + ) { + self.observe_timestamp(timestamp.map(Value::from).as_ref()); + // `name` is overloaded across Codex payloads: function/tool calls use + // it for values such as `exec`, `update_plan`, and `js`. Only the + // session metadata envelope owns session-title fields. + if line_type == "session_meta" { + if let Some(title) = [ + "title", + "name", + "threadName", + "thread_name", + "conversationTitle", + "conversation_title", + ] + .into_iter() + .find_map(|key| nonempty_value(payload.get(key))) + { + self.set_title(title, 3); + } + } + if let Some(model) = nonempty_value(payload.get("model")) { + self.model = Some(model.to_string()); + } + if let Some(cwd) = nonempty_value(payload.get("cwd")) { + self.repo_path = Some(cwd.to_string()); + } + if self.title_priority < 1 + && payload.get("type").and_then(Value::as_str) == Some("user_message") + { + if let Some(message) = nonempty_value(payload.get("message")) { + let message = imported_history::strip_orgii_exec_mode_bridge(message); + if !message.trim().is_empty() { + self.set_title(message, 1); + } + } + } + if line_type == "session_meta" { + let is_subagent = payload.get("thread_source").and_then(Value::as_str) + == Some("subagent") + || payload.pointer("/source/subagent").is_some(); + if is_subagent { + self.parent_observed = true; + self.parent_session_id = [ + payload.get("parent_thread_id"), + payload.pointer("/source/subagent/thread_spawn/parent_thread_id"), + payload.get("forked_from_id"), + ] + .into_iter() + .flatten() + .find_map(Value::as_str) + .map(str::trim) + .filter(|parent| !parent.is_empty() && *parent != source_session_id) + .map(|parent| { + format!( + "{}{parent}", + ImportedHistorySourceId::CodexApp + .descriptor() + .session_prefix + ) + }); + } + } + if payload.get("type").and_then(Value::as_str) == Some("token_count") { + if let Some(total) = payload + .pointer("/info/total_token_usage") + .or_else(|| payload.get("total_token_usage")) + { + let field = |name| total.get(name).and_then(Value::as_i64).unwrap_or(0); + let current_input = field("input_tokens"); + let current_cache_read = field("cached_input_tokens"); + let current_cache_write = field("cache_write_input_tokens"); + let current_output = field("output_tokens") + field("reasoning_output_tokens"); + self.input_tokens += cumulative_delta(current_input, self.last_cumulative_input); + self.cache_read_tokens += + cumulative_delta(current_cache_read, self.last_cumulative_cache_read); + self.cache_write_tokens += + cumulative_delta(current_cache_write, self.last_cumulative_cache_write); + self.output_tokens += cumulative_delta(current_output, self.last_cumulative_output); + self.last_cumulative_input = current_input; + self.last_cumulative_cache_read = current_cache_read; + self.last_cumulative_cache_write = current_cache_write; + self.last_cumulative_output = current_output; + self.tokens_observed = true; + } + } + } + + fn observe_timestamp(&mut self, value: Option<&Value>) { + let Some(ms) = value.and_then(value_epoch_ms) else { + return; + }; + self.created_at_ms = Some(self.created_at_ms.map_or(ms, |old| old.min(ms))); + self.updated_at_ms = Some(self.updated_at_ms.map_or(ms, |old| old.max(ms))); + } + + fn observe_message_usage(&mut self, message: &Value) { + let Some(usage) = message.get("usage") else { + return; + }; + let message_id = nonempty_value(message.get("id")).map(str::to_string); + if message_id.is_some() && message_id == self.last_usage_message_id { + return; + } + self.last_usage_message_id = message_id; + let field = |name| usage.get(name).and_then(Value::as_i64).unwrap_or(0); + let cache_read = field("cache_read_input_tokens"); + let cache_write = field("cache_creation_input_tokens"); + self.input_tokens += + field("input_tokens") + field("prompt_tokens") + cache_read + cache_write; + self.output_tokens += field("output_tokens") + field("completion_tokens"); + self.cache_read_tokens += cache_read; + self.cache_write_tokens += cache_write; + self.tokens_observed = true; + } + + fn set_title(&mut self, value: &str, priority: u8) { + let value = imported_history::truncate_name(value.trim(), 200); + if !value.is_empty() && priority >= self.title_priority { + self.title = Some(value); + self.title_priority = priority; + } + } + + fn set_nonempty_path(slot: &mut Option, raw: &Value, keys: &[&str]) { + if slot.is_none() { + *slot = keys + .iter() + .find_map(|key| nonempty_value(raw.get(*key))) + .map(str::to_string); + } + } +} + +/// Refresh one source's compact session catalog. +/// +/// Implementations may inspect source-owned indexes or stream lightweight +/// metadata, but must not return or retain a session-sized `ActivityChunk` +/// vector. Transcript bodies remain owned by the bounded replay adapters. +pub fn refresh_source( + conn: &mut Connection, + source: ImportedHistorySourceId, +) -> Result<(), String> { + match source { + ImportedHistorySourceId::ClaudeCode => { + crate::sources::claude_code::history::refresh_catalog(conn) + } + ImportedHistorySourceId::CodexApp => crate::sources::codex::app::refresh_catalog(conn), + ImportedHistorySourceId::CursorIde => crate::sources::cursor_ide::db::refresh_catalog(conn), + ImportedHistorySourceId::CursorCli => { + crate::sources::cursor_cli::history::refresh_catalog(conn) + } + ImportedHistorySourceId::OpenCode => { + crate::sources::opencode::history::refresh_catalog(conn) + } + ImportedHistorySourceId::Windsurf => { + crate::sources::windsurf::history::refresh_catalog(conn) + } + ImportedHistorySourceId::WorkBuddy => crate::sources::workbuddy::refresh_catalog(conn), + ImportedHistorySourceId::Trae => crate::sources::trae::history::refresh_catalog(conn), + ImportedHistorySourceId::Cline => crate::sources::cline::history::refresh_catalog(conn), + ImportedHistorySourceId::Warp => crate::sources::warp::history::refresh_catalog(conn), + ImportedHistorySourceId::ZCode => crate::sources::zcode::history::refresh_catalog(conn), + ImportedHistorySourceId::Qoder => crate::sources::qoder::history::refresh_catalog(conn), + ImportedHistorySourceId::MimoCode => { + crate::sources::mimo_code::history::refresh_catalog(conn) + } + ImportedHistorySourceId::Omp => crate::sources::omp::history::refresh_catalog(conn), + ImportedHistorySourceId::QoderCli => { + crate::sources::qoder_cli::history::refresh_catalog(conn) + } + } +} + +fn read_replay_catalog_row( + tx: &Transaction<'_>, + source: &str, + source_session_id: &str, +) -> Result, String> { + tx.query_row( + "SELECT session_id,source_path,source_record_key,source_mtime_ms, + source_size_bytes,source_fingerprint,parser_version,listable, + name,created_at_ms,updated_at_ms,model,input_tokens,output_tokens, + cache_read_tokens,cache_write_tokens,repo_path,branch,files_changed, + lines_added,lines_removed,touched_files_json,source_metadata_json, + parent_session_id,updated_at + FROM imported_history_session_cache + WHERE source=?1 AND source_session_id=?2", + params![source, source_session_id], + |row| { + Ok(ReplayCatalogRowSnapshot { + session_id: row.get(0)?, + source_path: row.get(1)?, + source_record_key: row.get(2)?, + source_mtime_ms: row.get(3)?, + source_size_bytes: row.get(4)?, + source_fingerprint: row.get(5)?, + parser_version: row.get(6)?, + listable: row.get(7)?, + name: row.get(8)?, + created_at_ms: row.get(9)?, + updated_at_ms: row.get(10)?, + model: row.get(11)?, + input_tokens: row.get(12)?, + output_tokens: row.get(13)?, + cache_read_tokens: row.get(14)?, + cache_write_tokens: row.get(15)?, + repo_path: row.get(16)?, + branch: row.get(17)?, + files_changed: row.get(18)?, + lines_added: row.get(19)?, + lines_removed: row.get(20)?, + touched_files_json: row.get(21)?, + source_metadata_json: row.get(22)?, + parent_session_id: row.get(23)?, + updated_at: row.get(24)?, + }) + }, + ) + .optional() + .map_err(|err| format!("read imported replay catalog row: {err}")) +} + +fn replay_catalog_baseline( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + current: &ReplayCatalogRowSnapshot, +) -> Result { + let stored = tx + .query_row( + "SELECT baseline_json,applied_json + FROM imported_replay_catalog_derivations + WHERE source=?1 AND source_session_id=?2", + params![source.as_str(), source_session_id], + |row| Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)), + ) + .optional() + .map_err(|err| format!("read replay catalog derivation: {err}"))?; + let Some((baseline_json, applied_json)) = stored else { + return Ok(current.clone()); + }; + let baseline = serde_json::from_str::(&baseline_json) + .map_err(|err| format!("decode replay catalog baseline: {err}"))?; + let applied = serde_json::from_str::(&applied_json) + .map_err(|err| format!("decode applied replay catalog snapshot: {err}"))?; + // Equality includes adapter-owned signature fields. Any intervening source + // refresh rebases the baseline before the next replay overlay. + Ok(if &applied == current { + baseline + } else { + current.clone() + }) +} + +fn store_replay_catalog_derivation( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + baseline: &ReplayCatalogRowSnapshot, + applied: &ReplayCatalogRowSnapshot, +) -> Result<(), String> { + tx.execute( + "INSERT INTO imported_replay_catalog_derivations( + source,source_session_id,baseline_json,applied_json,updated_at + ) VALUES(?1,?2,?3,?4,?5) + ON CONFLICT(source,source_session_id) DO UPDATE SET + baseline_json=excluded.baseline_json, + applied_json=excluded.applied_json, + updated_at=excluded.updated_at", + params![ + source.as_str(), + source_session_id, + serde_json::to_string(baseline) + .map_err(|err| format!("encode replay catalog baseline: {err}"))?, + serde_json::to_string(applied) + .map_err(|err| format!("encode applied replay catalog snapshot: {err}"))?, + chrono::Utc::now().to_rfc3339(), + ], + ) + .map(|_| ()) + .map_err(|err| format!("store replay catalog derivation: {err}")) +} + +/// Remove one replay-owned catalog overlay without rolling back a newer +/// adapter refresh. This runs in the same transaction as compact-index prune. +pub(crate) fn clear_replay_projection_tx( + tx: &Transaction<'_>, + source: &str, + source_session_id: &str, +) -> Result<(), String> { + let stored = tx + .query_row( + "SELECT baseline_json,applied_json + FROM imported_replay_catalog_derivations + WHERE source=?1 AND source_session_id=?2", + params![source, source_session_id], + |row| Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)), + ) + .optional() + .map_err(|err| format!("read replay catalog projection for prune: {err}"))?; + let Some((baseline_json, applied_json)) = stored else { + return Ok(()); + }; + let baseline = serde_json::from_str::(&baseline_json) + .map_err(|err| format!("decode replay catalog prune baseline: {err}"))?; + let applied = serde_json::from_str::(&applied_json) + .map_err(|err| format!("decode replay catalog prune snapshot: {err}"))?; + let current = read_replay_catalog_row(tx, source, source_session_id)?; + if current.as_ref() == Some(&applied) { + tx.execute( + "UPDATE imported_history_session_cache SET + name=?3,created_at_ms=?4,updated_at_ms=?5,model=?6, + input_tokens=?7,output_tokens=?8,cache_read_tokens=?9, + cache_write_tokens=?10,repo_path=?11,branch=?12, + files_changed=?13,lines_added=?14,lines_removed=?15, + touched_files_json=?16,source_metadata_json=?17, + parent_session_id=?18,updated_at=?19 + WHERE source=?1 AND source_session_id=?2", + params![ + source, + source_session_id, + baseline.name, + baseline.created_at_ms, + baseline.updated_at_ms, + baseline.model, + baseline.input_tokens, + baseline.output_tokens, + baseline.cache_read_tokens, + baseline.cache_write_tokens, + baseline.repo_path, + baseline.branch, + baseline.files_changed, + baseline.lines_added, + baseline.lines_removed, + baseline.touched_files_json, + baseline.source_metadata_json, + baseline.parent_session_id, + baseline.updated_at, + ], + ) + .map_err(|err| format!("restore adapter catalog baseline: {err}"))?; + } + tx.execute( + "DELETE FROM imported_replay_catalog_derivations + WHERE source=?1 AND source_session_id=?2", + params![source, source_session_id], + ) + .map(|_| ()) + .map_err(|err| format!("delete replay catalog derivation: {err}")) +} + +/// Publish replay-derived card metadata inside the same transaction that +/// publishes the replay generation. A crash therefore exposes either the old +/// replay+catalog pair or the new pair, never a mixed state. +#[allow(clippy::too_many_arguments)] +pub(crate) fn publish_from_replay_tx( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + total_events: u64, + fold_event_metadata: bool, + source_mtime_ns: i64, + driver_cursor_json: &str, +) -> Result<(), String> { + let catalog_before = read_replay_catalog_row(tx, source.as_str(), source_session_id)?; + let catalog_baseline = catalog_before + .as_ref() + .map(|current| replay_catalog_baseline(tx, source, source_session_id, current)) + .transpose()?; + let projection = serde_json::from_str::(driver_cursor_json) + .ok() + .and_then(|cursor| cursor.get("catalog").cloned()) + .and_then(|value| serde_json::from_value::(value).ok()) + .unwrap_or_default(); + let complete_projection = if fold_event_metadata { + let indexed_events = tx + .query_row( + "SELECT COUNT(*) FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("count replay catalog events: {err}"))? + .max(0) as u64; + indexed_events == total_events + } else { + false + }; + let mut impact = imported_history::metadata::ImportedHistoryImpactStats::default(); + let mut first_user_title = None; + + if complete_projection { + let mut statement = tx + .prepare( + "SELECT function_name,created_at,args_preview_json,result_preview_json + FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + ORDER BY sequence ASC", + ) + .map_err(|err| format!("prepare compact replay catalog projection: {err}"))?; + let mut rows = statement + .query(params![source.as_str(), source_session_id, generation]) + .map_err(|err| format!("query compact replay catalog projection: {err}"))?; + let mut metadata = TurnMetadataAccumulator::new(); + while let Some(row) = rows + .next() + .map_err(|err| format!("stream compact replay catalog row: {err}"))? + { + let function_name: String = row.get(0).map_err(|err| err.to_string())?; + let created_at: String = row.get(1).map_err(|err| err.to_string())?; + if function_name == imported_history::FUNCTION_USER_MESSAGE { + if first_user_title.is_none() { + let args: String = row.get(2).map_err(|err| err.to_string())?; + let result: String = row.get(3).map_err(|err| err.to_string())?; + first_user_title = compact_user_title(&args, &result); + } + continue; + } + let requirements = metadata_projection_requirements(Some(&function_name)); + if requirements.is_empty() { + continue; + } + let args = if requirements.needs_args_json() { + row.get::<_, String>(2).map_err(|err| err.to_string())? + } else { + String::new() + }; + let result = if requirements.needs_result_json() { + row.get::<_, String>(3).map_err(|err| err.to_string())? + } else { + String::new() + }; + metadata.add_event_at(Some(&function_name), &args, &result, &created_at); + } + let modified_files = metadata.modified_files(); + impact.touched_files = modified_files + .iter() + .map(|file| file.path.clone()) + .collect(); + impact.files_changed = impact.touched_files.len() as i64; + impact.lines_added = modified_files + .iter() + .map(|file| i64::from(file.additions)) + .sum(); + impact.lines_removed = modified_files + .iter() + .map(|file| i64::from(file.deletions)) + .sum(); + } + + let replay_title = projection.title.as_deref().or(first_user_title.as_deref()); + let (title, title_priority) = if source == ImportedHistorySourceId::CodexApp { + // Codex's session index owns the user-visible title. Replay may only + // fill an adapter placeholder; this also recovers the adapter value + // when an older projection polluted the live row with a tool name. + let title = catalog_baseline + .as_ref() + .and_then(|baseline| { + (!is_catalog_name_placeholder(baseline, source_session_id)) + .then_some(baseline.name.as_str()) + }) + .or(replay_title) + .or_else(|| { + catalog_baseline + .as_ref() + .map(|baseline| baseline.name.trim()) + .filter(|name| !name.is_empty()) + }); + // The selected title has already passed the source-vs-replay + // ownership check above. Use the strong-update branch so an old + // replay overlay is replaced atomically by its recovered baseline. + let title_priority = if title.is_some() { 2 } else { 0 }; + (title, title_priority) + } else { + // Other providers retain their existing catalog-enrichment contract: + // explicit provider metadata can replace discovery names, while the + // first user prompt only fills an empty title. + ( + replay_title, + if projection.title.is_some() { + projection.title_priority + } else if first_user_title.is_some() { + 1 + } else { + 0 + }, + ) + }; + let source_mtime_ms = source_mtime_ns.saturating_div(1_000_000); + let projected_updated_at = projection.updated_at_ms.unwrap_or_default(); + let updated_at_ms = source_mtime_ms.max(projected_updated_at); + let metadata_json = merged_continuation_metadata(tx, source, source_session_id, &projection)?; + + tx.execute( + "UPDATE imported_history_session_cache SET + name = CASE + WHEN COALESCE(?3, '') = '' THEN name + WHEN ?4 >= 2 THEN ?3 + WHEN name = '' OR name = source_record_key OR name = source_session_id + OR name = 'New Agent' THEN ?3 + ELSE name END, + model = CASE WHEN COALESCE(?5, '') != '' THEN ?5 ELSE model END, + repo_path = CASE WHEN COALESCE(?6, '') != '' THEN ?6 ELSE repo_path END, + branch = CASE WHEN COALESCE(?7, '') != '' THEN ?7 ELSE branch END, + input_tokens = CASE WHEN ?8 != 0 THEN ?9 ELSE input_tokens END, + output_tokens = CASE WHEN ?8 != 0 THEN ?10 ELSE output_tokens END, + cache_read_tokens = CASE WHEN ?8 != 0 THEN ?11 ELSE cache_read_tokens END, + cache_write_tokens = CASE WHEN ?8 != 0 THEN ?12 ELSE cache_write_tokens END, + files_changed = CASE WHEN ?13 != 0 THEN ?14 ELSE files_changed END, + lines_added = CASE WHEN ?13 != 0 THEN ?15 ELSE lines_added END, + lines_removed = CASE WHEN ?13 != 0 THEN ?16 ELSE lines_removed END, + touched_files_json = CASE WHEN ?13 != 0 THEN ?17 ELSE touched_files_json END, + parent_session_id = CASE WHEN ?18 != 0 THEN COALESCE(?19, '') ELSE parent_session_id END, + source_metadata_json = CASE WHEN ?20 != 0 THEN ?21 ELSE source_metadata_json END, + created_at_ms = CASE WHEN created_at_ms <= 0 AND ?22 > 0 THEN ?22 ELSE created_at_ms END, + updated_at_ms = MAX(updated_at_ms, ?23), + updated_at = ?24 + WHERE source=?1 AND source_session_id=?2", + params![ + source.as_str(), + source_session_id, + title, + title_priority as i64, + projection.model, + projection.repo_path, + projection.branch, + i64::from(projection.tokens_observed), + projection.input_tokens, + projection.output_tokens, + projection.cache_read_tokens, + projection.cache_write_tokens, + i64::from(complete_projection), + impact.files_changed, + impact.lines_added, + impact.lines_removed, + serde_json::to_string(&impact.touched_files) + .map_err(|err| format!("encode replay catalog touched files: {err}"))?, + i64::from(projection.parent_observed), + projection.parent_session_id, + i64::from(projection.continuation_observed), + metadata_json, + projection.created_at_ms.unwrap_or_default(), + updated_at_ms, + chrono::Utc::now().to_rfc3339(), + ], + ) + .map_err(|err| format!("publish compact replay catalog metadata: {err}"))?; + if let Some(baseline) = catalog_baseline.as_ref() { + let applied = + read_replay_catalog_row(tx, source.as_str(), source_session_id)?.ok_or_else(|| { + "Imported replay catalog row disappeared while publishing projection".to_string() + })?; + store_replay_catalog_derivation(tx, source, source_session_id, baseline, &applied)?; + } + Ok(()) +} + +fn merged_continuation_metadata( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + projection: &ReplayCatalogProjection, +) -> Result { + if !projection.continuation_observed { + return Ok(String::new()); + } + let existing = tx + .query_row( + "SELECT source_metadata_json FROM imported_history_session_cache + WHERE source=?1 AND source_session_id=?2", + params![source.as_str(), source_session_id], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|err| format!("read replay catalog metadata JSON: {err}"))? + .unwrap_or_default(); + let mut object = serde_json::from_str::(&existing) + .ok() + .and_then(|value| value.as_object().cloned()) + .unwrap_or_default(); + if let Some(key) = projection.continuation_group_key.as_deref() { + object.insert( + imported_cache::CONTINUATION_GROUP_KEY_FIELD.to_string(), + Value::String(key.to_string()), + ); + } else { + object.remove(imported_cache::CONTINUATION_GROUP_KEY_FIELD); + } + Ok(Value::Object(object).to_string()) +} + +fn compact_user_title(args_json: &str, result_json: &str) -> Option { + let args = serde_json::from_str::(args_json).unwrap_or(Value::Null); + let result = serde_json::from_str::(result_json).unwrap_or(Value::Null); + let title = [ + args.get("content"), + args.get("message"), + args.get("prompt"), + args.get("text"), + result.pointer("/message/content"), + result.get("content"), + result.get("message"), + ] + .into_iter() + .flatten() + .find_map(Value::as_str) + .map(imported_history::strip_orgii_exec_mode_bridge) + .map(str::trim) + .filter(|title| !title.is_empty()) + .map(|title| imported_history::truncate_name(title, 200)); + title +} + +fn is_catalog_name_placeholder( + snapshot: &ReplayCatalogRowSnapshot, + source_session_id: &str, +) -> bool { + let name = snapshot.name.trim(); + name.is_empty() + || name == snapshot.source_record_key + || name == source_session_id + || name == snapshot.session_id + || matches!(name, "New Agent" | "Untitled") +} + +fn nonempty_value(value: Option<&Value>) -> Option<&str> { + value + .and_then(Value::as_str) + .map(str::trim) + .filter(|v| !v.is_empty()) +} + +fn first_content_text(content: Option<&Value>) -> Option { + match content { + Some(Value::String(text)) => Some(text.clone()), + Some(Value::Array(items)) => items.iter().find_map(|item| { + let kind = item.get("type").and_then(Value::as_str); + if kind.is_some_and(|kind| !matches!(kind, "text" | "input_text")) { + return None; + } + item.get("text") + .or_else(|| item.get("content")) + .and_then(Value::as_str) + .map(str::to_string) + }), + _ => None, + } +} + +fn value_epoch_ms(value: &Value) -> Option { + value + .as_i64() + .or_else(|| value.as_u64().and_then(|value| i64::try_from(value).ok())) + .or_else(|| { + value + .as_str() + .and_then(imported_history::parse_iso_to_epoch_ms_opt) + }) +} + +fn cumulative_delta(current: i64, previous: i64) -> i64 { + if current >= previous { + current - previous + } else { + current + } +} + +#[cfg(test)] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/catalog/tests.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/catalog/tests.rs new file mode 100644 index 000000000..9081f6b48 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/catalog/tests.rs @@ -0,0 +1,539 @@ +use super::*; +use crate::store::sqlite::SqliteRecordStore; + +fn catalog_fixture() -> Connection { + let conn = Connection::open_in_memory().expect("catalog DB"); + SqliteRecordStore::init_tables(&conn).expect("core schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("catalog schema"); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_record_key,name, + model,repo_path,branch,input_tokens,files_changed,lines_added, + lines_removed,touched_files_json,source_metadata_json, + source_mtime_ms,source_size_bytes + ) VALUES('codex_app','fixture','codexapp-fixture','fixture','fixture', + 'old-model','/old','old',7,9,9,9,'[\"old.rs\"]','{\"keep\":true}', + 999,999)", + [], + ) + .expect("cache row"); + for (sequence, function, args, result) in [ + ( + 0_i64, + imported_history::FUNCTION_USER_MESSAGE, + "{}", + r#"{"message":{"content":"bounded replay title"}}"#, + ), + ( + 1_i64, + imported_history::FUNCTION_EDIT_FILE, + r#"{"file_path":"src/new.rs","old_content":"old","new_content":"new\nextra"}"#, + r#"{"status":"completed","linesAdded":2,"linesRemoved":1}"#, + ), + ] { + conn.execute( + "INSERT INTO imported_replay_events( + source,source_session_id,generation,sequence,event_id,turn_index, + action_type,function_name,created_at,args_preview_json, + result_preview_json,content_hash + ) VALUES('codex_app','fixture','g1',?1,?2,0,'raw',?3, + '2026-07-22T00:00:00Z',?4,?5,?2)", + params![ + sequence, + format!("event-{sequence}"), + function, + args, + result + ], + ) + .expect("replay event"); + } + conn +} + +fn current_catalog_snapshot(conn: &mut Connection) -> ReplayCatalogRowSnapshot { + let tx = conn + .transaction() + .expect("read catalog snapshot transaction"); + let snapshot = read_replay_catalog_row(&tx, "codex_app", "fixture") + .expect("read catalog snapshot") + .expect("fixture catalog row"); + tx.commit().expect("finish catalog snapshot transaction"); + snapshot +} + +fn derivation_count(conn: &Connection) -> i64 { + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_catalog_derivations + WHERE source='codex_app' AND source_session_id='fixture'", + [], + |row| row.get(0), + ) + .expect("count catalog derivations") +} + +fn publish_fixture_projection(conn: &mut Connection) { + let cursor = serde_json::json!({ + "catalog": ReplayCatalogProjection { + model: Some("gpt-5".to_string()), + repo_path: Some("/work/orgii".to_string()), + branch: Some("develop".to_string()), + input_tokens: 100, + output_tokens: 20, + tokens_observed: true, + continuation_group_key: Some("first-user".to_string()), + continuation_observed: true, + updated_at_ms: Some(1_774_137_600_000), + ..ReplayCatalogProjection::default() + } + }) + .to_string(); + let tx = conn.transaction().expect("catalog transaction"); + publish_from_replay_tx( + &tx, + ImportedHistorySourceId::CodexApp, + "fixture", + "g1", + 2, + true, + 1_774_137_600_000_000_000, + &cursor, + ) + .expect("publish catalog projection"); + tx.commit().expect("commit catalog projection"); +} + +#[test] +fn catalog_registry_is_exhaustive_for_replay_sources() { + // The exhaustive match in `refresh_source` is the compile-time guard; + // this assertion also documents the external-history contract count. + assert_eq!(ImportedHistorySourceId::ALL.len(), 15); +} + +#[test] +fn codex_catalog_uses_only_session_metadata_or_the_first_user_message_for_titles() { + let mut projection = ReplayCatalogProjection::default(); + projection.observe_codex( + "event_msg", + None, + &serde_json::json!({ + "type": "user_message", + "message": "First genuine request" + }), + "fixture", + ); + for (payload_type, tool_name) in [ + ("function_call", "exec"), + ("function_call", "exec_command"), + ("custom_tool_call", "update_plan"), + ("custom_tool_call", "js"), + ] { + projection.observe_codex( + "response_item", + None, + &serde_json::json!({ + "type": payload_type, + "name": tool_name + }), + "fixture", + ); + } + projection.observe_codex( + "event_msg", + None, + &serde_json::json!({ + "type": "user_message", + "message": "Later request must not replace the first" + }), + "fixture", + ); + + assert_eq!(projection.title.as_deref(), Some("First genuine request")); + assert_eq!(projection.title_priority, 1); + + projection.observe_codex( + "session_meta", + None, + &serde_json::json!({ + "title": "Transcript session title", + "name": "not selected because title is explicit" + }), + "fixture", + ); + projection.observe_codex( + "response_item", + None, + &serde_json::json!({ + "type": "function_call", + "name": "exec" + }), + "fixture", + ); + assert_eq!( + projection.title.as_deref(), + Some("Transcript session title") + ); + assert_eq!(projection.title_priority, 3); +} + +#[test] +fn replay_catalog_preserves_authoritative_source_title() { + let mut conn = catalog_fixture(); + conn.execute( + "UPDATE imported_history_session_cache + SET name='Session index title' + WHERE source='codex_app' AND source_session_id='fixture'", + [], + ) + .expect("set authoritative source title"); + let cursor = serde_json::json!({ + "catalog": ReplayCatalogProjection { + title: Some("Replay-derived title".to_string()), + title_priority: 3, + ..ReplayCatalogProjection::default() + } + }) + .to_string(); + let tx = conn.transaction().expect("catalog transaction"); + publish_from_replay_tx( + &tx, + ImportedHistorySourceId::CodexApp, + "fixture", + "g1", + 2, + true, + 1_774_137_600_000_000_000, + &cursor, + ) + .expect("publish catalog"); + tx.commit().expect("commit catalog"); + + let name: String = conn + .query_row( + "SELECT name FROM imported_history_session_cache + WHERE source='codex_app' AND source_session_id='fixture'", + [], + |row| row.get(0), + ) + .expect("read source title"); + assert_eq!(name, "Session index title"); +} + +#[test] +fn adapter_title_refresh_rebases_a_polluted_replay_derivation() { + let mut conn = catalog_fixture(); + conn.execute( + "UPDATE imported_history_session_cache + SET name='js' + WHERE source='codex_app' AND source_session_id='fixture'", + [], + ) + .expect("set already-polluted baseline"); + let baseline = current_catalog_snapshot(&mut conn); + conn.execute( + "UPDATE imported_history_session_cache SET name='update_plan' + WHERE source='codex_app' AND source_session_id='fixture'", + [], + ) + .expect("simulate old polluted replay title"); + let polluted = current_catalog_snapshot(&mut conn); + let tx = conn.transaction().expect("store old projection ownership"); + store_replay_catalog_derivation( + &tx, + ImportedHistorySourceId::CodexApp, + "fixture", + &baseline, + &polluted, + ) + .expect("store old replay derivation"); + tx.commit().expect("commit old replay derivation"); + + // A Codex catalog rescan now reasserts its source-owned title directly + // instead of restoring the already-polluted replay baseline. + conn.execute( + "UPDATE imported_history_session_cache SET name='Session index title' + WHERE source='codex_app' AND source_session_id='fixture'", + [], + ) + .expect("apply authoritative adapter refresh"); + + let corrected_cursor = serde_json::json!({ + "catalog": ReplayCatalogProjection { + title: Some("First genuine request".to_string()), + title_priority: 1, + ..ReplayCatalogProjection::default() + } + }) + .to_string(); + let tx = conn.transaction().expect("catalog repair transaction"); + publish_from_replay_tx( + &tx, + ImportedHistorySourceId::CodexApp, + "fixture", + "g1", + 2, + true, + 1_774_137_600_000_000_000, + &corrected_cursor, + ) + .expect("publish corrected catalog"); + tx.commit().expect("commit corrected catalog"); + + assert_eq!( + current_catalog_snapshot(&mut conn).name, + "Session index title" + ); + assert_eq!( + derivation_count(&conn), + 1, + "the repaired replay overlay must retain its ownership guard" + ); +} + +#[test] +fn replay_catalog_publish_updates_only_projected_fields() { + let mut conn = catalog_fixture(); + let projection = ReplayCatalogProjection { + model: Some("gpt-5".to_string()), + repo_path: Some("/work/orgii".to_string()), + branch: Some("develop".to_string()), + input_tokens: 100, + output_tokens: 20, + tokens_observed: true, + continuation_group_key: Some("first-user".to_string()), + continuation_observed: true, + updated_at_ms: Some(1_774_137_600_000), + ..ReplayCatalogProjection::default() + }; + let cursor = serde_json::json!({"catalog": projection}).to_string(); + let tx = conn.transaction().expect("catalog transaction"); + publish_from_replay_tx( + &tx, + ImportedHistorySourceId::CodexApp, + "fixture", + "g1", + 2, + true, + 1_774_137_600_000_000_000, + &cursor, + ) + .expect("publish catalog"); + tx.commit().expect("commit catalog"); + + let row = conn + .query_row( + "SELECT name,model,repo_path,branch,input_tokens,output_tokens, + files_changed,lines_added,lines_removed,touched_files_json, + source_metadata_json + FROM imported_history_session_cache + WHERE source='codex_app' AND source_session_id='fixture'", + [], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, String>(2)?, + row.get::<_, String>(3)?, + row.get::<_, i64>(4)?, + row.get::<_, i64>(5)?, + row.get::<_, i64>(6)?, + row.get::<_, i64>(7)?, + row.get::<_, i64>(8)?, + row.get::<_, String>(9)?, + row.get::<_, String>(10)?, + )) + }, + ) + .expect("published row"); + assert_eq!(row.0, "bounded replay title"); + assert_eq!(row.1, "gpt-5"); + assert_eq!(row.2, "/work/orgii"); + assert_eq!(row.3, "develop"); + assert_eq!((row.4, row.5), (100, 20)); + assert_eq!((row.6, row.7, row.8), (1, 2, 1)); + assert_eq!(row.9, r#"["src/new.rs"]"#); + let metadata: Value = serde_json::from_str(&row.10).expect("metadata JSON"); + assert_eq!(metadata.get("keep"), Some(&Value::Bool(true))); + assert_eq!( + metadata + .get(imported_cache::CONTINUATION_GROUP_KEY_FIELD) + .and_then(Value::as_str), + Some("first-user") + ); +} + +#[test] +fn replay_catalog_publish_rolls_back_with_replay_transaction() { + let mut conn = catalog_fixture(); + let cursor = serde_json::json!({ + "catalog": ReplayCatalogProjection { + model: Some("must-rollback".to_string()), + ..ReplayCatalogProjection::default() + } + }) + .to_string(); + { + let tx = conn.transaction().expect("catalog transaction"); + publish_from_replay_tx( + &tx, + ImportedHistorySourceId::CodexApp, + "fixture", + "g1", + 2, + true, + 1, + &cursor, + ) + .expect("publish then rollback"); + // Dropping an uncommitted transaction rolls back both replay and + // catalog publication. + } + let model: String = conn + .query_row( + "SELECT model FROM imported_history_session_cache + WHERE source='codex_app' AND source_session_id='fixture'", + [], + |row| row.get(0), + ) + .expect("rolled-back model"); + assert_eq!(model, "old-model"); + assert_eq!( + derivation_count(&conn), + 0, + "the baseline/applied guard must roll back with its projection" + ); +} + +#[test] +fn replay_catalog_prune_restores_unchanged_adapter_baseline() { + let mut conn = catalog_fixture(); + let baseline = current_catalog_snapshot(&mut conn); + publish_fixture_projection(&mut conn); + let applied = current_catalog_snapshot(&mut conn); + assert_ne!(applied, baseline, "fixture must exercise a real overlay"); + assert_eq!(derivation_count(&conn), 1); + + let tx = conn.transaction().expect("catalog prune transaction"); + clear_replay_projection_tx(&tx, "codex_app", "fixture") + .expect("clear unchanged replay projection"); + tx.commit().expect("commit catalog prune"); + + assert_eq!(current_catalog_snapshot(&mut conn), baseline); + assert_eq!(derivation_count(&conn), 0); +} + +#[test] +fn replay_catalog_prune_preserves_newer_adapter_refresh() { + let mut conn = catalog_fixture(); + publish_fixture_projection(&mut conn); + conn.execute( + "UPDATE imported_history_session_cache SET + source_fingerprint='adapter-new-fingerprint', + source_mtime_ms=123456, + name='adapter-title', + model='adapter-model', + repo_path='/adapter/repo', + source_metadata_json='{\"adapter\":true}', + updated_at='2026-07-23T12:00:00Z' + WHERE source='codex_app' AND source_session_id='fixture'", + [], + ) + .expect("simulate adapter refresh"); + let refreshed = current_catalog_snapshot(&mut conn); + + let tx = conn.transaction().expect("catalog prune transaction"); + clear_replay_projection_tx(&tx, "codex_app", "fixture") + .expect("clear replay projection after adapter refresh"); + tx.commit().expect("commit catalog prune"); + + assert_eq!( + current_catalog_snapshot(&mut conn), + refreshed, + "eviction must not roll a newer adapter row back to the old baseline" + ); + assert_eq!(derivation_count(&conn), 0); +} + +#[test] +fn replay_catalog_prune_rolls_back_atomically() { + let mut conn = catalog_fixture(); + publish_fixture_projection(&mut conn); + let applied = current_catalog_snapshot(&mut conn); + assert_eq!(derivation_count(&conn), 1); + + { + let tx = conn.transaction().expect("catalog prune transaction"); + clear_replay_projection_tx(&tx, "codex_app", "fixture") + .expect("clear replay projection before rollback"); + // Simulate an interrupted prune by dropping the transaction. + } + + assert_eq!(current_catalog_snapshot(&mut conn), applied); + assert_eq!( + derivation_count(&conn), + 1, + "projection and guard must remain visible together after rollback" + ); +} + +#[test] +fn replay_catalog_publish_never_clears_unavailable_fields() { + let mut conn = catalog_fixture(); + let cursor = serde_json::json!({ + "catalog": ReplayCatalogProjection::default() + }) + .to_string(); + let tx = conn.transaction().expect("catalog transaction"); + publish_from_replay_tx( + &tx, + ImportedHistorySourceId::CodexApp, + "fixture", + "g1", + 99, + true, + 123_000_000, + &cursor, + ) + .expect("publish partial catalog"); + tx.commit().expect("commit partial catalog"); + + let row = conn + .query_row( + "SELECT name,model,repo_path,branch,input_tokens,files_changed, + lines_added,lines_removed,touched_files_json, + source_metadata_json,source_mtime_ms,source_size_bytes + FROM imported_history_session_cache + WHERE source='codex_app' AND source_session_id='fixture'", + [], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, String>(2)?, + row.get::<_, String>(3)?, + row.get::<_, i64>(4)?, + row.get::<_, i64>(5)?, + row.get::<_, i64>(6)?, + row.get::<_, i64>(7)?, + row.get::<_, String>(8)?, + row.get::<_, String>(9)?, + row.get::<_, i64>(10)?, + row.get::<_, i64>(11)?, + )) + }, + ) + .expect("preserved catalog row"); + assert_eq!(row.0, "fixture"); + assert_eq!(row.1, "old-model"); + assert_eq!(row.2, "/old"); + assert_eq!(row.3, "old"); + assert_eq!(row.4, 7); + assert_eq!((row.5, row.6, row.7), (9, 9, 9)); + assert_eq!(row.8, r#"["old.rs"]"#); + assert_eq!(row.9, r#"{"keep":true}"#); + assert_eq!( + (row.10, row.11), + (999, 999), + "replay snapshots must not overwrite adapter-owned discovery signatures" + ); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/metadata.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/metadata.rs index dbf250345..42e7f3d3a 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/metadata.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/metadata.rs @@ -25,6 +25,26 @@ pub const SOURCE_QODER: &str = "qoder"; pub const SOURCE_MIMO_CODE: &str = "mimo_code"; pub const SOURCE_OMP: &str = "omp"; pub const SOURCE_QODER_CLI: &str = "qoder_cli"; +/// Transcript-bearing sources that must each have a bounded replay adapter. +/// The replay registry contract test compares against this inventory, so a +/// newly imported source cannot silently inherit a full-history fallback. +pub const IMPORTED_HISTORY_SOURCES: [&str; 15] = [ + SOURCE_CLAUDE_CODE, + SOURCE_CODEX_APP, + SOURCE_CURSOR_IDE, + SOURCE_CURSOR_CLI, + SOURCE_OPENCODE, + SOURCE_WINDSURF, + SOURCE_WORKBUDDY, + SOURCE_TRAE, + SOURCE_CLINE, + SOURCE_WARP, + SOURCE_ZCODE, + SOURCE_QODER, + SOURCE_MIMO_CODE, + SOURCE_OMP, + SOURCE_QODER_CLI, +]; // Hook-only sources: ORGII installs a managed PostToolUse command hook for // these CLIs and records their file-interaction provenance, but does not yet // import their session transcripts. Kept out of `is_imported_history_source` @@ -35,24 +55,7 @@ pub const SOURCE_KIMI: &str = "kimi"; pub const SOURCE_ANTIGRAVITY: &str = "antigravity"; pub fn is_imported_history_source(source: &str) -> bool { - matches!( - source, - SOURCE_CLAUDE_CODE - | SOURCE_CODEX_APP - | SOURCE_CURSOR_IDE - | SOURCE_CURSOR_CLI - | SOURCE_OPENCODE - | SOURCE_WINDSURF - | SOURCE_WORKBUDDY - | SOURCE_TRAE - | SOURCE_CLINE - | SOURCE_WARP - | SOURCE_ZCODE - | SOURCE_QODER - | SOURCE_MIMO_CODE - | SOURCE_OMP - | SOURCE_QODER_CLI - ) + IMPORTED_HISTORY_SOURCES.contains(&source) } #[derive(Debug, Clone)] diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/mod.rs index f702c5793..64a220f7d 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/mod.rs @@ -1,14 +1,19 @@ pub mod cache; +pub mod catalog; pub mod managed_mirror; pub mod managed_roots; pub mod metadata; pub mod paths; +pub mod replay; #[cfg(feature = "git")] pub mod repo_identity; +pub mod router; pub mod scan_snapshot; pub mod watermark; use std::collections::{BTreeSet, HashMap}; +use std::fs; +use std::io::{BufRead, BufReader, Read}; use std::path::Path; use chrono::TimeZone; @@ -38,123 +43,70 @@ pub const FUNCTION_GLOB_FILE_SEARCH: &str = "glob_file_search"; pub const FUNCTION_AWAIT_OUTPUT: &str = "await_output"; pub const DEFAULT_LIST_LIMIT: usize = 200; -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum ImportedHistoryLoader { - ClaudeCode, - Codex, - Cursor, - CursorCli, - OpenCode, - Windsurf, - WorkBuddy, - Trae, - Cline, - Warp, - ZCode, - Qoder, - MimoCode, - Omp, - QoderCli, -} - -fn imported_history_loader(session_id: &str) -> Option { - if session_id.starts_with(super::claude_code::SESSION_PREFIX) { - Some(ImportedHistoryLoader::ClaudeCode) - } else if session_id.starts_with(super::codex::SESSION_PREFIX) { - Some(ImportedHistoryLoader::Codex) - } else if session_id.starts_with(super::cursor_ide::CURSORIDE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Cursor) - } else if session_id.starts_with(super::cursor_cli::SESSION_PREFIX) { - Some(ImportedHistoryLoader::CursorCli) - } else if session_id.starts_with(super::opencode::history::OPENCODE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::OpenCode) - } else if session_id.starts_with(super::windsurf::history::WINDSURF_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Windsurf) - } else if session_id.starts_with(super::workbuddy::WORKBUDDY_SESSION_PREFIX) { - Some(ImportedHistoryLoader::WorkBuddy) - } else if session_id.starts_with(super::trae::history::TRAE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Trae) - } else if session_id.starts_with(super::cline::history::CLINE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Cline) - } else if session_id.starts_with(super::warp::history::WARP_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Warp) - } else if session_id.starts_with(super::zcode::history::ZCODE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::ZCode) - } else if session_id.starts_with(super::qoder::history::QODER_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Qoder) - } else if session_id.starts_with(super::mimo_code::history::MIMO_CODE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::MimoCode) - } else if session_id.starts_with(super::omp::history::OMP_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Omp) - } else if session_id.starts_with(super::qoder_cli::history::QODER_CLI_SESSION_PREFIX) { - Some(ImportedHistoryLoader::QoderCli) - } else { - None - } -} - -/// Load one imported provider session through its existing canonical history -/// reader. `None` means the id is not owned by an imported-history provider; -/// `Some(empty)` is a known provider session whose source currently has no -/// readable chunks. +/// Read only complete UTF-8 JSONL records from a bounded file prefix. /// -/// This is the single provider router for cross-provider projections such as -/// per-round Orgtrack metadata. It deliberately delegates parsing to the -/// established source modules instead of introducing another transcript -/// reader. -pub fn load_activity_chunks_for_session( - conn: &rusqlite::Connection, - session_id: &str, -) -> Result>, String> { - let chunks = match imported_history_loader(session_id) { - Some(ImportedHistoryLoader::ClaudeCode) => { - super::claude_code::history::load_claude_code_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Codex) => { - super::codex::app::load_codex_app_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Cursor) => { - super::cursor_ide::history::load_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::CursorCli) => { - super::cursor_cli::history::load_cursor_cli_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::OpenCode) => { - super::opencode::history::load_opencode_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::Windsurf) => { - super::windsurf::history::load_windsurf_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::WorkBuddy) => { - super::workbuddy::load_workbuddy_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Trae) => { - super::trae::history::load_trae_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Cline) => { - super::cline::history::load_cline_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Warp) => { - super::warp::history::load_warp_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::ZCode) => { - super::zcode::history::load_zcode_history_for_session(session_id)? +/// `BufRead::lines()` attempts to decode the final buffer even when +/// `Read::take()` stopped in the middle of a multi-byte character. Large +/// external transcripts commonly hit that byte boundary, so catalog scans +/// must ignore the partial tail instead of rejecting the whole source. +pub(crate) fn read_complete_jsonl_prefix_lines( + path: &Path, + source_label: &str, + max_bytes: u64, +) -> Result, String> { + let file = fs::File::open(path) + .map_err(|error| format!("Failed to open {source_label} catalog prefix: {error}"))?; + let source_len = file + .metadata() + .map_err(|error| format!("Failed to inspect {source_label} catalog prefix: {error}"))? + .len(); + let prefix_is_capped = source_len > max_bytes; + // Pin this read to the size observed above. If the active provider appends + // concurrently, that new record belongs to the next catalog refresh. + let mut reader = BufReader::new(file.take(source_len.min(max_bytes))); + let mut lines = Vec::new(); + let mut buffer = Vec::new(); + + loop { + buffer.clear(); + let bytes_read = reader + .read_until(b'\n', &mut buffer) + .map_err(|error| format!("Failed to read {source_label} catalog prefix: {error}"))?; + if bytes_read == 0 { + break; } - Some(ImportedHistoryLoader::Qoder) => { - super::qoder::history::load_qoder_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::MimoCode) => { - super::mimo_code::history::load_mimo_code_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Omp) => { - super::omp::history::load_omp_history_for_session(conn, session_id)? + if !buffer.ends_with(b"\n") { + // A byte-capped prefix cannot prove that this is a complete + // record. At natural EOF, however, JSONL producers commonly omit + // the final newline; preserve that record only when both UTF-8 + // and JSON are complete. + if prefix_is_capped { + break; + } + if buffer.ends_with(b"\r") { + buffer.pop(); + } + let Ok(line) = std::str::from_utf8(&buffer) else { + break; + }; + if serde_json::from_str::(line).is_ok() { + lines.push(line.to_string()); + } + break; } - Some(ImportedHistoryLoader::QoderCli) => { - super::qoder_cli::history::load_qoder_cli_history_for_session(conn, session_id)? + buffer.pop(); + if buffer.ends_with(b"\r") { + buffer.pop(); } - None => return Ok(None), - }; - Ok(Some(chunks)) + let line = std::str::from_utf8(&buffer) + .map_err(|error| { + format!("Failed to read {source_label} catalog prefix as UTF-8: {error}") + })? + .to_string(); + lines.push(line); + } + + Ok(lines) } #[derive(Debug, Clone, Serialize)] @@ -229,11 +181,20 @@ pub struct ImportedHistorySidebarRow { pub touched_files: Vec, } +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ImportedHistorySidebarCursor { + pub updated_at_ms: i64, + pub session_id: String, +} + #[derive(Debug, Clone, Serialize)] #[serde(rename_all = "camelCase")] pub struct ImportedHistorySidebarPage { pub sessions: Vec, pub has_more: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub next_cursor: Option, } #[derive(Debug, Clone, Serialize)] @@ -265,7 +226,7 @@ pub struct ImportedHistoryRowInput { pub parent_session_id: Option, } -#[derive(Debug, Clone)] +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] pub struct ImportedToolCall { pub call_id: String, pub raw_name: String, @@ -776,35 +737,95 @@ pub fn truncate_name(name: &str, max_len: usize) -> String { } #[cfg(test)] -mod impact_tests { +mod jsonl_prefix_tests { + use std::io::Write; + use super::*; + fn jsonl_prefix_fixture(label: &str, bytes: &[u8]) -> std::path::PathBuf { + let path = std::env::temp_dir().join(format!( + "orgii-jsonl-prefix-{label}-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, bytes).expect("write JSONL prefix fixture"); + path + } + #[test] - fn routes_every_imported_provider_to_its_existing_history_loader() { - let cases = [ - ("claudecodeapp-id", ImportedHistoryLoader::ClaudeCode), - ("codexapp-id", ImportedHistoryLoader::Codex), - ("cursoride-id", ImportedHistoryLoader::Cursor), - ("cursorcliapp-id", ImportedHistoryLoader::CursorCli), - ("opencodeapp-id", ImportedHistoryLoader::OpenCode), - ("windsurfapp-id", ImportedHistoryLoader::Windsurf), - ("workbuddyapp-id", ImportedHistoryLoader::WorkBuddy), - ("traeapp-id", ImportedHistoryLoader::Trae), - ("clineapp-id", ImportedHistoryLoader::Cline), - ("warpapp-id", ImportedHistoryLoader::Warp), - ("zcodeapp-id", ImportedHistoryLoader::ZCode), - ("qoderapp-id", ImportedHistoryLoader::Qoder), - ("mimocodeapp-id", ImportedHistoryLoader::MimoCode), - ("ompapp-id", ImportedHistoryLoader::Omp), - ("qodercliapp-id", ImportedHistoryLoader::QoderCli), - ]; - - for (session_id, expected) in cases { - assert_eq!(imported_history_loader(session_id), Some(expected)); - } - assert_eq!(imported_history_loader("org2-native-id"), None); + fn bounded_jsonl_prefix_ignores_a_partial_utf8_tail() { + let path = std::env::temp_dir().join(format!( + "orgii-jsonl-prefix-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let first_line = "{\"type\":\"session_meta\"}\n"; + let partial_line_prefix = "{\"message\":\""; + let mut file = fs::File::create(&path).expect("create JSONL prefix fixture"); + file.write_all(first_line.as_bytes()) + .expect("write complete prefix line"); + file.write_all(partial_line_prefix.as_bytes()) + .expect("write partial prefix line"); + file.write_all("中\"}\n".as_bytes()) + .expect("write multibyte prefix tail"); + file.flush().expect("flush JSONL prefix fixture"); + drop(file); + + let max_bytes = (first_line.len() + partial_line_prefix.len() + 1) as u64; + let lines = read_complete_jsonl_prefix_lines(&path, "test", max_bytes) + .expect("read bounded JSONL prefix"); + + assert_eq!(lines, vec!["{\"type\":\"session_meta\"}"]); + fs::remove_file(path).expect("remove JSONL prefix fixture"); } + #[test] + fn natural_eof_accepts_a_complete_json_record_without_newline() { + let record = br#"{"type":"session_meta","payload":{"id":"no-newline"}}"#; + let path = jsonl_prefix_fixture("natural-eof", record); + + let lines = read_complete_jsonl_prefix_lines(&path, "test", 1024) + .expect("read natural-EOF JSONL prefix"); + + assert_eq!( + lines, + vec!["{\"type\":\"session_meta\",\"payload\":{\"id\":\"no-newline\"}}"] + ); + fs::remove_file(path).expect("remove JSONL prefix fixture"); + } + + #[test] + fn natural_eof_ignores_an_incomplete_json_record_without_newline() { + let path = jsonl_prefix_fixture( + "partial-json", + b"{\"type\":\"session_meta\"}\n{\"payload\":{\"id\":\"partial\"", + ); + + let lines = read_complete_jsonl_prefix_lines(&path, "test", 1024) + .expect("read partial natural-EOF JSONL prefix"); + + assert_eq!(lines, vec!["{\"type\":\"session_meta\"}"]); + fs::remove_file(path).expect("remove JSONL prefix fixture"); + } + + #[test] + fn natural_eof_ignores_a_partial_utf8_record_without_newline() { + let mut bytes = b"{\"type\":\"session_meta\"}\n{\"payload\":\"".to_vec(); + bytes.extend_from_slice(&[0xe4, 0xb8]); + let path = jsonl_prefix_fixture("partial-utf8", &bytes); + + let lines = read_complete_jsonl_prefix_lines(&path, "test", 1024) + .expect("read partial UTF-8 natural-EOF JSONL prefix"); + + assert_eq!(lines, vec!["{\"type\":\"session_meta\"}"]); + fs::remove_file(path).expect("remove JSONL prefix fixture"); + } +} + +#[cfg(test)] +mod impact_tests { + use super::*; + #[test] fn impact_collector_counts_normalized_edit_and_patch_paths() { let edit = ActivityChunk::new("session", ACTION_TYPE_TOOL_CALL, FUNCTION_EDIT_FILE) diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/cache_policy.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/cache_policy.rs new file mode 100644 index 000000000..14533bd44 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/cache_policy.rs @@ -0,0 +1,837 @@ +//! Byte-bounded, rebuildable cache eviction for imported replay indexes. +//! +//! The provider transcript is never touched here. Every deletion is scoped to +//! the caller's ORGII-owned SQLite connection, which naturally isolates +//! separate homes/runtimes and lets the next bounded replay rebuild the index. + +use std::collections::BTreeMap; +use std::time::Duration; + +use chrono::{DateTime, Utc}; +use rusqlite::{params, Connection, Transaction, TransactionBehavior}; +use serde::{Deserialize, Serialize}; + +pub const DEFAULT_REPLAY_CACHE_MAX_BYTES: u64 = 512 * 1024 * 1024; +pub const DEFAULT_REPLAY_CACHE_TARGET_BYTES: u64 = 384 * 1024 * 1024; +pub const DEFAULT_REPLAY_CACHE_TTL: Duration = Duration::from_secs(7 * 24 * 60 * 60); +pub const DEFAULT_REPLAY_CACHE_PROTECT_RECENT: Duration = Duration::from_secs(3 * 60); + +const APPROX_ROW_OVERHEAD_BYTES: i64 = 32; +const MAX_EVICTION_SCOPES_PER_RUN: usize = 8; +const MAX_EVICTION_BYTES_PER_RUN: u64 = 128 * 1024 * 1024; + +#[derive(Clone, Copy)] +enum ReplayCacheTableKind { + SourceScoped, + ShellManifest, +} + +struct ReplayCacheTable { + name: &'static str, + kind: ReplayCacheTableKind, +} + +const REPLAY_CACHE_TABLES: &[ReplayCacheTable] = &[ + ReplayCacheTable { + name: "imported_replay_catalog_derivations", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_payload_artifact_refs", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_payload_artifacts", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_changes", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_structured_events", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_structured_rows", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_source_rows", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_events", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_turns", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_rejected_snapshots", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_shell_manifests", + kind: ReplayCacheTableKind::ShellManifest, + }, + ReplayCacheTable { + name: "imported_replay_shell_segments", + kind: ReplayCacheTableKind::SourceScoped, + }, + ReplayCacheTable { + name: "imported_replay_state", + kind: ReplayCacheTableKind::SourceScoped, + }, +]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ReplayCachePolicy { + pub max_bytes: u64, + pub target_bytes: u64, + pub ttl: Duration, + pub protect_recent: Duration, +} + +impl Default for ReplayCachePolicy { + fn default() -> Self { + Self { + max_bytes: DEFAULT_REPLAY_CACHE_MAX_BYTES, + target_bytes: DEFAULT_REPLAY_CACHE_TARGET_BYTES, + ttl: DEFAULT_REPLAY_CACHE_TTL, + protect_recent: DEFAULT_REPLAY_CACHE_PROTECT_RECENT, + } + } +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum ReplayCacheEvictionReason { + Ttl, + ByteBudget, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct ReplayCacheEviction { + pub source_id: String, + pub source_session_id: String, + pub last_accessed_at: String, + pub approx_bytes: u64, + pub reason: ReplayCacheEvictionReason, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct ReplayCachePruneReport { + pub before_bytes: u64, + pub after_bytes: u64, + pub evicted_bytes: u64, + pub evicted_entries: u64, + pub ttl_evictions: u64, + pub budget_evictions: u64, + pub protected_entries: u64, + pub over_budget: bool, + pub evictions: Vec, +} + +#[derive(Debug, Clone)] +struct CacheEntry { + source: String, + source_session_id: String, + last_accessed_at: String, + last_accessed_ms: i64, + approx_bytes: u64, + protected: bool, +} + +#[derive(Debug)] +struct CacheScopeMeasurement { + last_accessed_at: String, + last_accessed_ms: i64, + approx_bytes: u64, +} + +type CacheScopeMeasurements = BTreeMap<(String, String), CacheScopeMeasurement>; + +fn record_scope_access( + scopes: &mut CacheScopeMeasurements, + source: String, + source_session_id: String, + timestamp: String, +) { + let timestamp_ms = DateTime::parse_from_rfc3339(×tamp) + .map(|value| value.timestamp_millis()) + .unwrap_or(i64::MIN); + scopes + .entry((source, source_session_id)) + .and_modify(|current| { + if timestamp_ms > current.last_accessed_ms { + current.last_accessed_at.clone_from(×tamp); + current.last_accessed_ms = timestamp_ms; + } + }) + .or_insert(CacheScopeMeasurement { + last_accessed_at: timestamp, + last_accessed_ms: timestamp_ms, + approx_bytes: 0, + }); +} + +fn try_select_eviction_candidate( + entries: &[CacheEntry], + selected: &mut Vec<(usize, ReplayCacheEvictionReason)>, + selected_flags: &mut [bool], + selected_bytes: &mut u64, + projected_bytes: &mut u64, + index: usize, + reason: ReplayCacheEvictionReason, +) -> bool { + if selected.len() >= MAX_EVICTION_SCOPES_PER_RUN { + return false; + } + let next_bytes = selected_bytes.saturating_add(entries[index].approx_bytes); + // An oversized oldest scope is allowed by itself so repeated maintenance + // can always make progress. Once a scope is selected we preserve LRU order + // instead of skipping a large old scope in favour of newer small ones. + if !selected.is_empty() && next_bytes > MAX_EVICTION_BYTES_PER_RUN { + return false; + } + selected.push((index, reason)); + selected_flags[index] = true; + *selected_bytes = next_bytes; + *projected_bytes = projected_bytes.saturating_sub(entries[index].approx_bytes); + true +} + +fn select_eviction_candidates( + entries: &[CacheEntry], + policy: ReplayCachePolicy, + ttl_ms: i64, + ttl_cutoff: i64, +) -> Vec<(usize, ReplayCacheEvictionReason)> { + let mut selected = Vec::new(); + let mut selected_flags = vec![false; entries.len()]; + let mut selected_bytes = 0_u64; + let mut projected_bytes = entries.iter().fold(0_u64, |total, entry| { + total.saturating_add(entry.approx_bytes) + }); + + for (index, entry) in entries.iter().enumerate() { + let expired = ttl_ms == 0 || entry.last_accessed_ms <= ttl_cutoff; + if !entry.protected + && expired + && !try_select_eviction_candidate( + entries, + &mut selected, + &mut selected_flags, + &mut selected_bytes, + &mut projected_bytes, + index, + ReplayCacheEvictionReason::Ttl, + ) + { + return selected; + } + } + + let target_bytes = policy.target_bytes.min(policy.max_bytes); + if projected_bytes > policy.max_bytes { + for (index, entry) in entries.iter().enumerate() { + if projected_bytes <= target_bytes { + break; + } + if entry.protected || selected_flags[index] { + continue; + } + if !try_select_eviction_candidate( + entries, + &mut selected, + &mut selected_flags, + &mut selected_bytes, + &mut projected_bytes, + index, + ReplayCacheEvictionReason::ByteBudget, + ) { + break; + } + } + } + selected +} + +/// Prune only the rebuildable replay cache stored in `conn`. +/// +/// TTL eviction is applied first, then byte-budget LRU eviction when usage is +/// above `max_bytes`. Recently touched entries are never removed; if they are +/// the only remaining entries, the cache may temporarily stay over budget. +/// One maintenance call handles at most eight scopes and approximately 128 +/// MiB, using one short transaction per scope; an oversized oldest scope is +/// allowed to run alone so later calls can still make progress. +pub fn prune_cache( + conn: &mut Connection, + policy: ReplayCachePolicy, +) -> Result { + prune_cache_at(conn, policy, Utc::now().timestamp_millis()) +} + +fn prune_cache_at( + conn: &mut Connection, + policy: ReplayCachePolicy, + now_ms: i64, +) -> Result { + prune_cache_at_with_hook(conn, policy, now_ms, |_| Ok(())) +} + +fn prune_cache_at_with_hook( + conn: &mut Connection, + policy: ReplayCachePolicy, + now_ms: i64, + before_delete: F, +) -> Result +where + F: FnOnce(&Connection) -> Result<(), String>, +{ + prune_cache_at_with_hooks(conn, policy, now_ms, before_delete, |_| Ok(())) +} + +fn prune_cache_at_with_hooks( + conn: &mut Connection, + policy: ReplayCachePolicy, + now_ms: i64, + before_delete: F, + mut after_scope_transaction: G, +) -> Result +where + F: FnOnce(&Connection) -> Result<(), String>, + G: FnMut(usize) -> Result<(), String>, +{ + let protect_ms = duration_millis(policy.protect_recent); + let ttl_ms = duration_millis(policy.ttl); + let protect_cutoff = now_ms.saturating_sub(protect_ms); + let ttl_cutoff = now_ms.saturating_sub(ttl_ms); + // Expensive byte accounting stays outside the writer critical section. + let mut entries = load_cache_entries(conn)?; + for entry in &mut entries { + entry.protected = protect_ms > 0 && entry.last_accessed_ms >= protect_cutoff; + } + entries.sort_by(|left, right| { + left.last_accessed_ms + .cmp(&right.last_accessed_ms) + .then_with(|| left.source.cmp(&right.source)) + .then_with(|| left.source_session_id.cmp(&right.source_session_id)) + }); + + let before_bytes = entries.iter().fold(0_u64, |total, entry| { + total.saturating_add(entry.approx_bytes) + }); + let selected = select_eviction_candidates(&entries, policy, ttl_ms, ttl_cutoff); + + let protected_entries = entries.iter().filter(|entry| entry.protected).count() as u64; + let had_candidates = !selected.is_empty(); + let mut evictions = Vec::with_capacity(selected.len()); + if had_candidates { + // Tests use this hook to model a real delivery touching a selected + // manifest after the cold measurement but before the writer lock. + before_delete(conn)?; + for (scope_index, (index, reason)) in selected.into_iter().enumerate() { + let entry = &entries[index]; + let tx = conn + .transaction_with_behavior(TransactionBehavior::Immediate) + .map_err(|err| format!("start replay cache prune transaction: {err}"))?; + let eviction = + match load_scope_last_access(&tx, &entry.source, &entry.source_session_id)? { + Some((_, current_access_ms)) => { + let entered_protection = + protect_ms > 0 && current_access_ms >= protect_cutoff; + if current_access_ms > entry.last_accessed_ms || entered_protection { + None + } else { + delete_replay_entry(&tx, &entry.source, &entry.source_session_id)?; + Some(ReplayCacheEviction { + source_id: entry.source.clone(), + source_session_id: entry.source_session_id.clone(), + last_accessed_at: entry.last_accessed_at.clone(), + approx_bytes: entry.approx_bytes, + reason, + }) + } + } + None => None, + }; + tx.commit() + .map_err(|err| format!("commit replay cache prune transaction: {err}"))?; + if let Some(eviction) = eviction { + evictions.push(eviction); + } + // The test hook runs only after the transaction has committed, so + // a peer writer succeeding here proves the RESERVED lock was + // released between source/session scopes. + after_scope_transaction(scope_index + 1)?; + } + } + + // Recount only after a prune attempt, and never while holding the writer + // lock. With no candidates the original snapshot is already the report. + let after_bytes = if had_candidates { + total_cache_bytes(conn)? + } else { + before_bytes + }; + let ttl_evictions = evictions + .iter() + .filter(|eviction| eviction.reason == ReplayCacheEvictionReason::Ttl) + .count() as u64; + let budget_evictions = evictions.len() as u64 - ttl_evictions; + Ok(ReplayCachePruneReport { + before_bytes, + after_bytes, + evicted_bytes: before_bytes.saturating_sub(after_bytes), + evicted_entries: evictions.len() as u64, + ttl_evictions, + budget_evictions, + protected_entries, + over_budget: after_bytes > policy.max_bytes, + evictions, + }) +} + +fn duration_millis(duration: Duration) -> i64 { + duration.as_millis().min(i64::MAX as u128) as i64 +} + +fn load_scope_last_access( + conn: &Connection, + source: &str, + source_session_id: &str, +) -> Result, String> { + // This is the short revalidation path used while holding the writer lock. + // Every branch is keyed by compact metadata; payload BLOBs are not opened. + let mut statement = conn + .prepare( + "WITH access_watermarks(accessed_at) AS ( + SELECT updated_at + FROM imported_replay_state + WHERE source=?1 AND source_session_id=?2 + UNION ALL + SELECT rejected_at + FROM imported_replay_rejected_snapshots + WHERE source=?1 AND source_session_id=?2 + UNION ALL + SELECT updated_at + FROM imported_replay_catalog_derivations + WHERE source=?1 AND source_session_id=?2 + UNION ALL + SELECT manifest.accessed_at + FROM imported_replay_shell_segments AS segment + JOIN imported_replay_shell_manifests AS manifest + ON manifest.session_id=segment.session_id + AND manifest.call_id=segment.call_id + WHERE segment.source=?1 AND segment.source_session_id=?2 + ) + SELECT accessed_at + FROM access_watermarks + ORDER BY julianday(accessed_at) DESC, accessed_at DESC + LIMIT 1", + ) + .map_err(|err| format!("prepare replay cache access revalidation: {err}"))?; + let mut rows = statement + .query(params![source, source_session_id]) + .map_err(|err| format!("query replay cache access revalidation: {err}"))?; + let latest = if let Some(row) = rows + .next() + .map_err(|err| format!("read replay cache access revalidation: {err}"))? + { + let timestamp = row.get::<_, String>(0).map_err(|err| err.to_string())?; + let timestamp_ms = DateTime::parse_from_rfc3339(×tamp) + .map(|value| value.timestamp_millis()) + .unwrap_or(i64::MIN); + Some((timestamp, timestamp_ms)) + } else { + None + }; + Ok(latest) +} + +fn load_cache_entries(conn: &Connection) -> Result, String> { + // A rejected first snapshot can exist before a valid replay state, and a + // catalog derivation is a defensive third base watermark. Normal replay + // rows are transactionally published with state, so scanning large + // event/artifact tables just to rediscover those scopes would turn pruning + // into another full-history hotspot. Shell-only scopes are joined from + // their compact manifest/segment tables below. + let mut scopes = CacheScopeMeasurements::new(); + for (table, timestamp_column) in [ + ("imported_replay_state", "updated_at"), + ("imported_replay_rejected_snapshots", "rejected_at"), + ("imported_replay_catalog_derivations", "updated_at"), + ] { + let mut statement = conn + .prepare(&format!( + "SELECT source,source_session_id,{timestamp_column} FROM {table}" + )) + .map_err(|err| format!("prepare replay cache timestamps from {table}: {err}"))?; + let mut rows = statement + .query([]) + .map_err(|err| format!("query replay cache timestamps from {table}: {err}"))?; + while let Some(row) = rows + .next() + .map_err(|err| format!("read replay cache timestamp from {table}: {err}"))? + { + let source = row.get::<_, String>(0).map_err(|err| err.to_string())?; + let source_session_id = row.get::<_, String>(1).map_err(|err| err.to_string())?; + let timestamp = row.get::<_, String>(2).map_err(|err| err.to_string())?; + record_scope_access(&mut scopes, source, source_session_id, timestamp); + } + } + + // Managed readerless/collaboration delivery can own canonical replay + // payloads and Shell manifests without ever creating replay state. Discover + // those scopes through the compact locator tables and their ORGII access + // watermark. This query never visits the payload BLOB table. + let mut statement = conn + .prepare( + "SELECT segment.source,segment.source_session_id,MAX(manifest.accessed_at) + FROM imported_replay_shell_segments AS segment + JOIN imported_replay_shell_manifests AS manifest + ON manifest.session_id=segment.session_id + AND manifest.call_id=segment.call_id + GROUP BY segment.source,segment.source_session_id", + ) + .map_err(|err| format!("prepare readerless Shell cache timestamps: {err}"))?; + let mut rows = statement + .query([]) + .map_err(|err| format!("query readerless Shell cache timestamps: {err}"))?; + while let Some(row) = rows + .next() + .map_err(|err| format!("read readerless Shell cache timestamp: {err}"))? + { + let source = row.get::<_, String>(0).map_err(|err| err.to_string())?; + let source_session_id = row.get::<_, String>(1).map_err(|err| err.to_string())?; + let timestamp = row.get::<_, String>(2).map_err(|err| err.to_string())?; + record_scope_access(&mut scopes, source, source_session_id, timestamp); + } + drop(rows); + drop(statement); + + aggregate_cache_bytes(conn, &mut scopes)?; + + let mut entries = Vec::with_capacity(scopes.len()); + for ((source, source_session_id), measurement) in scopes { + entries.push(CacheEntry { + source, + source_session_id, + last_accessed_at: measurement.last_accessed_at, + last_accessed_ms: measurement.last_accessed_ms, + approx_bytes: measurement.approx_bytes, + protected: false, + }); + } + Ok(entries) +} + +fn total_cache_bytes(conn: &Connection) -> Result { + load_cache_entries(conn).map(|entries| { + entries.into_iter().fold(0_u64, |total, entry| { + total.saturating_add(entry.approx_bytes) + }) + }) +} + +struct ReplayCacheByteAggregation { + table: &'static str, + query: &'static str, +} + +// Each table is aggregated once for every maintenance snapshot. Adding source +// scopes therefore changes result rows, not the number of SQL statements. +const REPLAY_CACHE_BYTE_AGGREGATIONS: &[ReplayCacheByteAggregation] = &[ + ReplayCacheByteAggregation { + table: "imported_replay_state", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 56 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(source_identity AS BLOB)) + + LENGTH(CAST(driver_cursor_json AS BLOB)) + + LENGTH(CAST(updated_at AS BLOB)) + ),0) + FROM imported_replay_state + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_turns", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 32 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(turn_id AS BLOB)) + + LENGTH(CAST(started_at AS BLOB)) + + COALESCE(LENGTH(CAST(ended_at AS BLOB)),0) + ),0) + FROM imported_replay_turns + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_events", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 56 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(event_id AS BLOB)) + + LENGTH(CAST(action_type AS BLOB)) + + LENGTH(CAST(function_name AS BLOB)) + + LENGTH(CAST(created_at AS BLOB)) + + LENGTH(CAST(args_preview_json AS BLOB)) + + LENGTH(CAST(result_preview_json AS BLOB)) + + COALESCE(LENGTH(CAST(thread_id AS BLOB)),0) + + COALESCE(LENGTH(CAST(process_id AS BLOB)),0) + + LENGTH(CAST(payloads_json AS BLOB)) + + LENGTH(CAST(content_hash AS BLOB)) + ),0) + FROM imported_replay_events + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_source_rows", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 24 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(source_key AS BLOB)) + + LENGTH(CAST(content_hash AS BLOB)) + + COALESCE(LENGTH(CAST(event_id AS BLOB)),0) + ),0) + FROM imported_replay_source_rows + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_structured_rows", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 8 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(source_key AS BLOB)) + + LENGTH(CAST(content_hash AS BLOB)) + ),0) + FROM imported_replay_structured_rows + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_structured_events", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 8 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(source_key AS BLOB)) + + LENGTH(CAST(local_key AS BLOB)) + + LENGTH(CAST(event_id AS BLOB)) + ),0) + FROM imported_replay_structured_events + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_changes", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 16 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(event_id AS BLOB)) + + LENGTH(CAST(change_kind AS BLOB)) + ),0) + FROM imported_replay_changes + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_payload_artifact_refs", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(event_id AS BLOB)) + + LENGTH(CAST(field_path AS BLOB)) + + LENGTH(CAST(content_hash AS BLOB)) + ),0) + FROM imported_replay_payload_artifact_refs + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_payload_artifacts", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(content_hash AS BLOB)) + + LENGTH(payload) + ),0) + FROM imported_replay_payload_artifacts + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_shell_manifests", + // DISTINCT collapses all segment ordinals/generations that point at + // one manifest key before the join. The manifest payload columns are + // therefore counted once per replay scope and no artifact BLOB is + // selected into Rust. + query: "SELECT scope.source,scope.source_session_id,COALESCE(SUM( + ?1 + 24 + + LENGTH(CAST(manifest.session_id AS BLOB)) + + LENGTH(CAST(manifest.logical_call_id AS BLOB)) + + LENGTH(CAST(manifest.call_id AS BLOB)) + + LENGTH(CAST(manifest.identity_hash AS BLOB)) + + LENGTH(CAST(manifest.terminal_preview AS BLOB)) + + COALESCE(LENGTH(CAST(manifest.completed_at AS BLOB)),0) + + LENGTH(CAST(manifest.accessed_at AS BLOB)) + ),0) + FROM ( + SELECT DISTINCT source,source_session_id,session_id,call_id + FROM imported_replay_shell_segments + ) AS scope + JOIN imported_replay_shell_manifests AS manifest + ON manifest.session_id=scope.session_id + AND manifest.call_id=scope.call_id + GROUP BY scope.source,scope.source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_shell_segments", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 48 + + LENGTH(CAST(session_id AS BLOB)) + + LENGTH(CAST(call_id AS BLOB)) + + LENGTH(CAST(stream AS BLOB)) + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(generation AS BLOB)) + + LENGTH(CAST(content_hash AS BLOB)) + ),0) + FROM imported_replay_shell_segments + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_rejected_snapshots", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + 24 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(source_identity AS BLOB)) + + LENGTH(CAST(sample_fingerprint AS BLOB)) + + LENGTH(CAST(rejection_kind AS BLOB)) + + LENGTH(CAST(rejected_at AS BLOB)) + ),0) + FROM imported_replay_rejected_snapshots + GROUP BY source,source_session_id", + }, + ReplayCacheByteAggregation { + table: "imported_replay_catalog_derivations", + query: "SELECT source,source_session_id,COALESCE(SUM( + ?1 + + LENGTH(CAST(source AS BLOB)) + + LENGTH(CAST(source_session_id AS BLOB)) + + LENGTH(CAST(baseline_json AS BLOB)) + + LENGTH(CAST(applied_json AS BLOB)) + + LENGTH(CAST(updated_at AS BLOB)) + ),0) + FROM imported_replay_catalog_derivations + GROUP BY source,source_session_id", + }, +]; + +#[cfg(test)] +std::thread_local! { + static CACHE_BYTE_AGGREGATION_QUERY_COUNT: std::cell::Cell = + const { std::cell::Cell::new(0) }; +} + +fn aggregate_cache_bytes( + conn: &Connection, + scopes: &mut CacheScopeMeasurements, +) -> Result<(), String> { + // TEXT values are cast to BLOB before LENGTH so multibyte UTF-8 is counted + // in bytes. Integer values contribute their stored-cell approximation, + // never their numeric value (notably `indexed_size_bytes`, which describes + // the provider file and must not inflate the ORGII cache measurement). + for aggregation in REPLAY_CACHE_BYTE_AGGREGATIONS { + #[cfg(test)] + CACHE_BYTE_AGGREGATION_QUERY_COUNT.with(|count| count.set(count.get() + 1)); + let mut statement = conn + .prepare(aggregation.query) + .map_err(|err| format!("prepare replay cache {} bytes: {err}", aggregation.table))?; + let mut rows = statement + .query([APPROX_ROW_OVERHEAD_BYTES]) + .map_err(|err| format!("query replay cache {} bytes: {err}", aggregation.table))?; + while let Some(row) = rows + .next() + .map_err(|err| format!("read replay cache {} bytes: {err}", aggregation.table))? + { + let source = row.get::<_, String>(0).map_err(|err| err.to_string())?; + let source_session_id = row.get::<_, String>(1).map_err(|err| err.to_string())?; + let bytes = row.get::<_, i64>(2).map_err(|err| err.to_string())?.max(0) as u64; + // Preserve the prior orphan-row semantics: only state/rejection/ + // catalog watermarks and Shell locator scopes are eligible cache + // entries. A stray payload row cannot invent an entry without a + // reliable last-access timestamp. + if let Some(scope) = scopes.get_mut(&(source, source_session_id)) { + scope.approx_bytes = scope.approx_bytes.saturating_add(bytes); + } + } + } + Ok(()) +} + +fn delete_replay_entry( + tx: &Transaction<'_>, + source: &str, + source_session_id: &str, +) -> Result<(), String> { + // The Shell manifest key is renderer-session/call, while its segments own + // the replay source scope. Remove matching manifests before their segment + // locators disappear; FK cascade handles segments when enabled and the + // source-scoped pass below is the deterministic fallback. Native `.slog` + // rows/files are intentionally outside this replay cache policy. + tx.execute( + "DELETE FROM imported_replay_shell_manifests AS manifest + WHERE EXISTS ( + SELECT 1 FROM imported_replay_shell_segments AS segment + WHERE segment.session_id=manifest.session_id + AND segment.call_id=manifest.call_id + AND segment.source=?1 + AND segment.source_session_id=?2 + )", + params![source, source_session_id], + ) + .map_err(|err| format!("delete replay Shell manifests: {err}"))?; + + // Keep the discovery/path binding so the next bounded open can rebuild + // directly from the provider-owned source. The catalog helper restores the + // pre-replay baseline only when no newer adapter refresh superseded it. + super::super::catalog::clear_replay_projection_tx(tx, source, source_session_id)?; + + for table in REPLAY_CACHE_TABLES + .iter() + .filter(|table| matches!(table.kind, ReplayCacheTableKind::SourceScoped)) + { + tx.execute( + &format!( + "DELETE FROM {} WHERE source=?1 AND source_session_id=?2", + table.name + ), + params![source, source_session_id], + ) + .map_err(|err| format!("delete replay cache row from {}: {err}", table.name))?; + } + Ok(()) +} + +#[cfg(test)] +#[path = "tests/cache_policy.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/conformance_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/conformance_tests.rs new file mode 100644 index 000000000..29e780e8d --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/conformance_tests.rs @@ -0,0 +1,897 @@ +//! Real-shape, redacted fixtures for every imported-history replay adapter. +//! +//! JSONL and whole-document fixtures are copied verbatim to a temporary +//! source. SQLite fixtures describe provider rows and are materialized into +//! the provider's real table layout before the public replay API is called. +//! The assertions therefore exercise source parsing, compact indexing, +//! bounded reads, lazy turn hydration, and stable event identities rather +//! than merely checking that fixture files exist. + +use std::collections::{BTreeMap, BTreeSet}; +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; + +use prost::Message as _; +use prost_reflect::{DescriptorPool, DynamicMessage}; +use rusqlite::{params, Connection}; +use serde::Deserialize; +use serde_json::{json, Value}; + +use super::{ + open_window, poll_delta, read_payload_range, read_turn_window_at_index, scan_window_after, + ImportedHistorySourceId, ReplayLimits, ReplayStorageFamily, HARD_MAX_EVENTS, + HARD_MAX_IPC_BYTES, HARD_MAX_TURNS, +}; +use crate::store::sqlite::SqliteRecordStore; + +const MANIFEST_JSON: &str = include_str!("fixtures/manifest.json"); +const WARP_FILE_DESCRIPTOR_SET: &[u8] = include_bytes!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../proto/warp_multi_agent_v1.descriptor.pb" +)); +const WARP_TASK_PROTO_NAME: &str = "warp.multi_agent.v1.Task"; + +static NEXT_FIXTURE_ROOT: AtomicU64 = AtomicU64::new(0); + +#[derive(Debug, Deserialize)] +struct FixtureManifest { + fixtures: Vec, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct FixtureSpec { + source_id: String, + storage_family: ReplayStorageFamily, + source_session_id: String, + file: String, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct PartFixture { + session_id: String, + parts: Vec, +} + +#[derive(Debug, Deserialize)] +struct PartFixtureRow { + role: String, + data: Value, +} + +#[derive(Debug, Deserialize)] +struct KvFixture { + composer: Value, + bubbles: Vec, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct CursorCliFixture { + agent_id: String, + created_at: i64, + messages: Vec, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct WarpFixture { + conversation_id: String, + tasks: Vec, +} + +struct FixtureRoot(PathBuf); + +impl FixtureRoot { + fn new() -> Self { + let ordinal = NEXT_FIXTURE_ROOT.fetch_add(1, Ordering::Relaxed); + let path = std::env::temp_dir().join(format!( + "orgii-replay-conformance-{}-{ordinal}", + std::process::id() + )); + std::fs::create_dir(&path).expect("create isolated replay fixture root"); + Self(path) + } +} + +impl Drop for FixtureRoot { + fn drop(&mut self) { + if let Err(error) = std::fs::remove_dir_all(&self.0) { + eprintln!( + "failed to clean replay fixture root {}: {error}", + self.0.display() + ); + } + } +} + +fn manifest() -> FixtureManifest { + serde_json::from_str(MANIFEST_JSON).expect("replay fixture manifest") +} + +fn fixture_text(file: &str) -> &'static str { + match file { + "claude_code.jsonl" => include_str!("fixtures/claude_code.jsonl"), + "codex_app.jsonl" => include_str!("fixtures/codex_app.jsonl"), + "cursor_ide.json" => include_str!("fixtures/cursor_ide.json"), + "cursor_cli.json" => include_str!("fixtures/cursor_cli.json"), + "opencode.json" => include_str!("fixtures/opencode.json"), + "windsurf.json" => include_str!("fixtures/windsurf.json"), + "workbuddy.jsonl" => include_str!("fixtures/workbuddy.jsonl"), + "trae.jsonl" => include_str!("fixtures/trae.jsonl"), + "cline.json" => include_str!("fixtures/cline.json"), + "warp.json" => include_str!("fixtures/warp.json"), + "zcode.json" => include_str!("fixtures/zcode.json"), + "qoder.jsonl" => include_str!("fixtures/qoder.jsonl"), + "mimo_code.json" => include_str!("fixtures/mimo_code.json"), + "omp.jsonl" => include_str!("fixtures/omp.jsonl"), + "qoder_cli.jsonl" => include_str!("fixtures/qoder_cli.jsonl"), + unknown => panic!("fixture manifest references an undeclared file: {unknown}"), + } +} + +fn initialize_cache( + source: ImportedHistorySourceId, + source_session_id: &str, + source_path: &Path, +) -> (Connection, String) { + let cache = Connection::open_in_memory().expect("open compact replay cache"); + SqliteRecordStore::init_tables(&cache).expect("initialize replay tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("initialize source cache tables"); + let display_session_id = format!( + "{}{}", + source.descriptor().session_prefix, + source_session_id + ); + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,?2,?3,?4)", + params![ + source.as_str(), + source_session_id, + display_session_id, + source_path.to_string_lossy() + ], + ) + .expect("register replay fixture source"); + (cache, display_session_id) +} + +fn materialize_source(spec: &FixtureSpec, source: ImportedHistorySourceId, root: &Path) -> PathBuf { + let text = fixture_text(&spec.file); + let path = match spec.storage_family { + ReplayStorageFamily::JsonLines | ReplayStorageFamily::WholeJson => { + let path = root.join(&spec.file); + std::fs::write(&path, text).expect("write text replay fixture"); + path + } + ReplayStorageFamily::SqliteWal => materialize_part_db(spec, text, root), + ReplayStorageFamily::SqliteKeyValue => materialize_kv_db(spec, text, root), + ReplayStorageFamily::SqliteManifestBlob => materialize_cursor_cli_db(spec, text, root), + ReplayStorageFamily::SqliteTaskBlob => materialize_warp_db(spec, text, root), + }; + assert_eq!( + source.descriptor().storage_family, + spec.storage_family, + "{} fixture uses the adapter's declared storage family", + source.as_str() + ); + assert!(path.is_file(), "materialized source {}", path.display()); + path +} + +fn materialize_part_db(spec: &FixtureSpec, text: &str, root: &Path) -> PathBuf { + let fixture: PartFixture = serde_json::from_str(text).expect("part fixture JSON"); + assert_eq!(fixture.session_id, spec.source_session_id); + let path = root.join(format!("{}.sqlite", spec.source_id)); + let source = Connection::open(&path).expect("create SQLite/WAL fixture"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE session( + id TEXT PRIMARY KEY,time_created INTEGER,time_updated INTEGER + ); + CREATE TABLE message( + id TEXT PRIMARY KEY,session_id TEXT,time_created INTEGER,data TEXT + ); + CREATE TABLE part( + id TEXT PRIMARY KEY,message_id TEXT,session_id TEXT, + time_created INTEGER,data TEXT + );", + ) + .expect("create SQLite/WAL provider schema"); + source + .execute( + "INSERT INTO session(id,time_created,time_updated) VALUES(?1,1,?2)", + params![spec.source_session_id, fixture.parts.len() as i64 + 1], + ) + .expect("insert provider session"); + for (ordinal, part) in fixture.parts.into_iter().enumerate() { + let message_id = format!("message-{ordinal:04}"); + let part_id = format!("part-{ordinal:04}"); + let timestamp = ordinal as i64 + 1; + source + .execute( + "INSERT INTO message(id,session_id,time_created,data) VALUES(?1,?2,?3,?4)", + params![ + message_id, + spec.source_session_id, + timestamp, + json!({"role":part.role}).to_string() + ], + ) + .expect("insert provider message"); + source + .execute( + "INSERT INTO part(id,message_id,session_id,time_created,data) + VALUES(?1,?2,?3,?4,?5)", + params![ + part_id, + message_id, + spec.source_session_id, + timestamp, + part.data.to_string() + ], + ) + .expect("insert provider part"); + } + source + .execute_batch("PRAGMA wal_checkpoint(TRUNCATE);") + .expect("checkpoint provider fixture"); + drop(source); + path +} + +fn materialize_kv_db(spec: &FixtureSpec, text: &str, root: &Path) -> PathBuf { + let fixture: KvFixture = serde_json::from_str(text).expect("KV fixture JSON"); + assert_eq!( + fixture.composer["composerId"].as_str(), + Some(spec.source_session_id.as_str()) + ); + let path = root.join(format!("{}.sqlite", spec.source_id)); + let source = Connection::open(&path).expect("create SQLite/KV fixture"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE cursorDiskKV(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("create SQLite/KV provider schema"); + source + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2)", + params![ + format!("composerData:{}", spec.source_session_id), + fixture.composer.to_string() + ], + ) + .expect("insert KV composer"); + for bubble in fixture.bubbles { + let bubble_id = bubble["bubbleId"] + .as_str() + .expect("KV bubble has stable id"); + source + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2)", + params![ + format!("bubbleId:{}:{bubble_id}", spec.source_session_id), + bubble.to_string() + ], + ) + .expect("insert KV bubble"); + } + source + .execute_batch("PRAGMA wal_checkpoint(TRUNCATE);") + .expect("checkpoint KV fixture"); + drop(source); + path +} + +fn materialize_cursor_cli_db(spec: &FixtureSpec, text: &str, root: &Path) -> PathBuf { + let fixture: CursorCliFixture = serde_json::from_str(text).expect("Cursor CLI fixture JSON"); + assert_eq!(fixture.agent_id, spec.source_session_id); + let path = root.join(format!("{}.sqlite", spec.source_id)); + let source = Connection::open(&path).expect("create Cursor CLI fixture"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE blobs(id TEXT PRIMARY KEY,data BLOB); + CREATE TABLE meta(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("create Cursor CLI provider schema"); + let mut message_ids = Vec::with_capacity(fixture.messages.len()); + for (ordinal, message) in fixture.messages.into_iter().enumerate() { + let id = hex_encode(&[(ordinal as u8).saturating_add(1); 32]); + source + .execute( + "INSERT INTO blobs(id,data) VALUES(?1,?2)", + params![ + id, + serde_json::to_vec(&message).expect("Cursor message JSON") + ], + ) + .expect("insert Cursor message blob"); + message_ids.push(id); + } + let root_id = hex_encode(&[0xfe; 32]); + source + .execute( + "INSERT INTO blobs(id,data) VALUES(?1,?2)", + params![root_id, cursor_manifest(&message_ids)], + ) + .expect("insert Cursor manifest blob"); + let meta = json!({ + "agentId":fixture.agent_id, + "latestRootBlobId":root_id, + "createdAt":fixture.created_at, + }) + .to_string(); + source + .execute( + "INSERT INTO meta(key,value) VALUES('0',?1)", + [hex_encode(meta.as_bytes())], + ) + .expect("publish Cursor manifest root"); + source + .execute_batch("PRAGMA wal_checkpoint(TRUNCATE);") + .expect("checkpoint Cursor fixture"); + drop(source); + path +} + +fn materialize_warp_db(spec: &FixtureSpec, text: &str, root: &Path) -> PathBuf { + let fixture: WarpFixture = serde_json::from_str(text).expect("Warp fixture JSON"); + assert_eq!(fixture.conversation_id, spec.source_session_id); + let path = root.join(format!("{}.sqlite", spec.source_id)); + let source = Connection::open(&path).expect("create Warp fixture"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE agent_conversations( + id INTEGER PRIMARY KEY,conversation_id TEXT,conversation_data TEXT, + last_modified_at TEXT,summary TEXT + ); + CREATE TABLE agent_tasks( + id INTEGER PRIMARY KEY,conversation_id TEXT,task_id TEXT, + task BLOB,last_modified_at TEXT + );", + ) + .expect("create Warp provider schema"); + source + .execute( + "INSERT INTO agent_conversations( + id,conversation_id,conversation_data,last_modified_at,summary + ) VALUES(1,?1,'{}','2026-07-22 00:00:04','{}')", + [spec.source_session_id.as_str()], + ) + .expect("insert Warp conversation"); + for (ordinal, task) in fixture.tasks.into_iter().enumerate() { + let task_id = task["id"] + .as_str() + .expect("Warp task has stable id") + .to_string(); + source + .execute( + "INSERT INTO agent_tasks( + id,conversation_id,task_id,task,last_modified_at + ) VALUES(?1,?2,?3,?4,?5)", + params![ + ordinal as i64 + 1, + spec.source_session_id, + task_id, + encode_warp_task(task), + format!("2026-07-22 00:00:{:02}", ordinal + 1) + ], + ) + .expect("insert Warp task blob"); + } + source + .execute_batch("PRAGMA wal_checkpoint(TRUNCATE);") + .expect("checkpoint Warp fixture"); + drop(source); + path +} + +fn cursor_manifest(ids: &[String]) -> Vec { + let mut manifest = Vec::with_capacity(ids.len() * 34); + for id in ids { + let decoded = hex_decode(id).expect("Cursor fixture blob id"); + manifest.extend_from_slice(&[0x0a, 32]); + manifest.extend_from_slice(&decoded); + } + manifest +} + +fn encode_warp_task(task: Value) -> Vec { + let pool = DescriptorPool::decode(WARP_FILE_DESCRIPTOR_SET) + .expect("load Warp provider protobuf descriptor"); + let descriptor = pool + .get_message_by_name(WARP_TASK_PROTO_NAME) + .expect("Warp task descriptor"); + let encoded = task.to_string(); + let mut deserializer = serde_json::Deserializer::from_str(&encoded); + DynamicMessage::deserialize(descriptor, &mut deserializer) + .expect("real-shape Warp task JSON") + .encode_to_vec() +} + +fn hex_encode(bytes: &[u8]) -> String { + bytes.iter().map(|byte| format!("{byte:02x}")).collect() +} + +fn hex_decode(text: &str) -> Option> { + if text.len() % 2 != 0 { + return None; + } + text.as_bytes() + .chunks_exact(2) + .map(|pair| { + let pair = std::str::from_utf8(pair).ok()?; + u8::from_str_radix(pair, 16).ok() + }) + .collect() +} + +fn collect_all_event_ids( + cache: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, +) -> Vec { + let mut after_sequence = -1_i64; + let mut event_ids = Vec::new(); + for page in 0..64 { + let scan = scan_window_after( + cache, + source, + session_id, + after_sequence, + ReplayLimits { + max_turns: 1, + max_events: 2, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }, + ) + .unwrap_or_else(|error| panic!("{} scan page {page}: {error}", source.as_str())); + assert!(scan.chunks.len() <= 2, "{} scan event cap", source.as_str()); + assert!( + scan.stats.ipc_bytes <= HARD_MAX_IPC_BYTES as u64, + "{} scan wire cap", + source.as_str() + ); + for indexed in &scan.chunks { + assert!( + indexed.sequence > after_sequence, + "{} scan sequence advances", + source.as_str() + ); + event_ids.push(indexed.chunk.chunk_id.clone()); + } + if !scan.has_more { + return event_ids; + } + assert!( + !scan.chunks.is_empty(), + "{} scan must not stall while more rows exist", + source.as_str() + ); + assert!( + scan.cursor.through_sequence > after_sequence, + "{} scan cursor advances", + source.as_str() + ); + after_sequence = scan.cursor.through_sequence; + } + panic!("{} scan exceeded fixture page budget", source.as_str()); +} + +fn assert_source_conformance(spec: &FixtureSpec, root: &Path) { + let source = ImportedHistorySourceId::parse(&spec.source_id).expect("registered source id"); + let path = materialize_source(spec, source, root); + let (mut cache, session_id) = initialize_cache(source, &spec.source_session_id, &path); + + let hard_bounded = open_window( + &mut cache, + source, + &session_id, + ReplayLimits { + max_turns: usize::MAX, + max_events: usize::MAX, + max_ipc_bytes: usize::MAX, + }, + ) + .unwrap_or_else(|error| panic!("{} fixture open: {error}", source.as_str())); + assert!( + hard_bounded.total_turn_count >= 2, + "{} fixture must exercise turn pagination", + source.as_str() + ); + assert!( + hard_bounded.total_event_count >= 4, + "{} fixture must normalize multiple events", + source.as_str() + ); + assert!(hard_bounded.chunks.len() <= HARD_MAX_EVENTS); + assert!(hard_bounded.turn_headers.len() <= HARD_MAX_TURNS); + assert!(hard_bounded.stats.ipc_bytes <= HARD_MAX_IPC_BYTES as u64); + assert_eq!(hard_bounded.cursor.source_id, source.as_str()); + assert_eq!(hard_bounded.cursor.session_id, session_id); + + let one_event = open_window( + &mut cache, + source, + &session_id, + ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }, + ) + .unwrap_or_else(|error| panic!("{} one-event window: {error}", source.as_str())); + assert_eq!( + one_event.turn_headers.len(), + 1, + "{} turn cap", + source.as_str() + ); + assert_eq!(one_event.chunks.len(), 1, "{} event cap", source.as_str()); + + for turn_index in 0..hard_bounded.total_turn_count { + let turn = read_turn_window_at_index( + &mut cache, + source, + &session_id, + turn_index as i64, + ReplayLimits { + max_turns: 1, + max_events: HARD_MAX_EVENTS, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }, + ) + .unwrap_or_else(|error| panic!("{} turn page {turn_index}: {error}", source.as_str())); + assert_eq!(turn.turn_headers.len(), 1); + assert_eq!(turn.turn_headers[0].turn_index, turn_index as i64); + assert!(!turn.chunks.is_empty(), "{} turn body", source.as_str()); + assert!( + turn.chunks + .iter() + .all(|event| event.turn_index == turn_index as i64), + "{} turn page does not bleed into an adjacent turn", + source.as_str() + ); + } + + let first_ids = collect_all_event_ids(&mut cache, source, &session_id); + assert_eq!( + first_ids.len() as u64, + hard_bounded.total_event_count, + "{} compact scan covers every normalized event", + source.as_str() + ); + assert_eq!( + first_ids.len(), + first_ids.iter().collect::>().len(), + "{} event ids are unique", + source.as_str() + ); + let second_ids = collect_all_event_ids(&mut cache, source, &session_id); + assert_eq!( + second_ids, + first_ids, + "{} event ids stay stable across reopen/rescan", + source.as_str() + ); +} + +fn assert_newline_terminated_bad_json_is_skipped( + spec: &FixtureSpec, + source: ImportedHistorySourceId, + root: &Path, +) { + let records = fixture_text(&spec.file) + .lines() + .filter(|line| !line.trim().is_empty()) + .collect::>(); + assert!(records.len() >= 2, "{} JSONL fixture", source.as_str()); + // Four-line chat fixtures begin their second turn at record two; Trae's + // summary format stores one complete turn per line and therefore has two. + let later_record = records[records.len() / 2]; + let clean_text = format!("{}\n{}\n", records[0], later_record); + // This malformed record is complete at the storage layer (it ends in a + // newline) but invalid JSON. The following valid line must remain an + // independent record and advance the persisted byte cursor to EOF. + let bad_text = format!("{}\n{{\"malformed\":\n{}\n", records[0], later_record); + let clean_path = root.join(format!("{}-bad-line-clean.jsonl", source.as_str())); + let bad_path = root.join(format!("{}-bad-line.jsonl", source.as_str())); + std::fs::write(&clean_path, &clean_text).expect("write clean JSONL control"); + std::fs::write(&bad_path, &bad_text).expect("write malformed JSONL fixture"); + + let clean_source_session_id = format!("{}-bad-line-clean", spec.source_session_id); + let bad_source_session_id = format!("{}-bad-line", spec.source_session_id); + let (mut clean_cache, clean_session_id) = + initialize_cache(source, &clean_source_session_id, &clean_path); + let (mut bad_cache, bad_session_id) = + initialize_cache(source, &bad_source_session_id, &bad_path); + let limits = ReplayLimits { + max_turns: HARD_MAX_TURNS, + max_events: HARD_MAX_EVENTS, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }; + let clean = open_window(&mut clean_cache, source, &clean_session_id, limits) + .unwrap_or_else(|error| panic!("{} clean control: {error}", source.as_str())); + let bad = open_window(&mut bad_cache, source, &bad_session_id, limits) + .unwrap_or_else(|error| panic!("{} malformed fixture: {error}", source.as_str())); + + assert_eq!(clean.stats.parsed_rows, 2, "{} clean rows", source.as_str()); + assert_eq!( + bad.stats.parsed_rows, + 2, + "{} bad row is skipped", + source.as_str() + ); + assert_eq!( + bad.stats.parsed_bytes, + bad_text.len() as u64, + "{} consumes the complete bad line and continues", + source.as_str() + ); + assert_eq!(bad.total_event_count, clean.total_event_count); + assert_eq!(bad.total_turn_count, clean.total_turn_count); + assert_eq!( + bad.total_turn_count, + 2, + "{} reaches the later user turn", + source.as_str() + ); + let normalized_chunks = bad + .chunks + .iter() + .map(|indexed| &indexed.chunk) + .collect::>(); + assert!( + serde_json::to_string(&normalized_chunks) + .expect("serialize malformed-line result") + .contains("second"), + "{} later valid record is normalized", + source.as_str() + ); + let unchanged = poll_delta( + &mut bad_cache, + source, + &bad_session_id, + &bad.cursor, + ReplayLimits::default(), + ) + .unwrap_or_else(|error| panic!("{} post-bad-line poll: {error}", source.as_str())); + assert_eq!(unchanged.stats.parsed_bytes, 0); + assert_eq!(unchanged.stats.parsed_rows, 0); + assert!(unchanged.chunks.is_empty()); +} + +fn large_assistant_line(source: ImportedHistorySourceId, body: &str, timestamp: i64) -> String { + match source { + ImportedHistorySourceId::CodexApp => json!({ + "timestamp": "2026-07-22T00:00:00Z", + "type": "event_msg", + "payload": {"type":"agent_message", "message":body}, + }) + .to_string(), + ImportedHistorySourceId::ClaudeCode => json!({ + "type":"assistant", + "timestamp":timestamp, + "message":{"role":"assistant","content":[{"type":"text","text":body}]}, + }) + .to_string(), + _ => unreachable!("large append test covers Codex and the shared JSONL driver"), + } +} + +fn assert_ten_mib_utf8_append_is_incremental(source: ImportedHistorySourceId, root: &Path) { + const MIB: usize = 1024 * 1024; + let source_session_id = format!("{}-ten-mib-utf8-append", source.as_str()); + let path = root.join(format!("{}-ten-mib-utf8-append.jsonl", source.as_str())); + // Make the acknowledged prefix larger than the permitted boundary + // overhead. A regression that seeks to byte zero cannot satisfy the + // completed append's parsedBytes bound below. + let old_body = "old-prefix-".repeat((3 * MIB) / "old-prefix-".len() + 1); + let old_line = large_assistant_line(source, &old_body, 1_753_142_400_000); + let old_source_bytes = old_line.len() + 1; + std::fs::write(&path, format!("{old_line}\n")).expect("write acknowledged JSONL prefix"); + let (mut cache, session_id) = initialize_cache(source, &source_session_id, &path); + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .unwrap_or_else(|error| panic!("{} open large prefix: {error}", source.as_str())); + + let utf8_unit = "中🙂x"; + let appended_body = utf8_unit.repeat((10 * MIB) / utf8_unit.len() + 1); + assert!(appended_body.len() >= 10 * MIB); + let appended_line = large_assistant_line(source, &appended_body, 1_753_142_401_000); + let appended_bytes = appended_line.len() + 1; + // End the first physical write inside the final four-byte emoji. Without + // a newline this must stay unacknowledged and must not advance parsedBytes. + let split = appended_line + .rfind('🙂') + .expect("large UTF-8 fixture contains emoji") + + 1; + assert!(std::str::from_utf8(&appended_line.as_bytes()[..split]).is_err()); + let mut writer = std::fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append large UTF-8 line"); + writer + .write_all(&appended_line.as_bytes()[..split]) + .expect("write UTF-8 torn tail"); + writer.flush().expect("flush UTF-8 torn tail"); + let partial = poll_delta( + &mut cache, + source, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .unwrap_or_else(|error| panic!("{} poll UTF-8 torn tail: {error}", source.as_str())); + assert_eq!(partial.stats.parsed_bytes, 0); + assert_eq!(partial.stats.parsed_rows, 0); + assert!(partial.chunks.is_empty()); + + writer + .write_all(&appended_line.as_bytes()[split..]) + .expect("finish UTF-8 scalar and JSONL record"); + writer + .write_all(b"\n") + .expect("terminate large JSONL record"); + writer.flush().expect("flush completed UTF-8 append"); + drop(writer); + let completed = poll_delta( + &mut cache, + source, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .unwrap_or_else(|error| panic!("{} poll 10 MiB append: {error}", source.as_str())); + assert_eq!(completed.stats.parsed_rows, 1); + assert!(completed.stats.parsed_bytes >= appended_bytes as u64); + assert!( + completed.stats.parsed_bytes <= appended_bytes.saturating_add(MIB) as u64, + "{} reads only the 10 MiB append plus at most 1 MiB boundary overhead: parsed={} append={appended_bytes}", + source.as_str(), + completed.stats.parsed_bytes + ); + assert!( + completed.stats.parsed_bytes < old_source_bytes.saturating_add(appended_bytes) as u64, + "{} must not replay the acknowledged 3 MiB prefix", + source.as_str() + ); + assert!(!completed.reset_required); + assert_eq!(completed.cursor.generation, opened.cursor.generation); + let event = completed + .chunks + .iter() + .find(|event| { + event + .payloads + .iter() + .any(|payload| payload.total_bytes == appended_body.len() as u64) + }) + .unwrap_or_else(|| panic!("{} appended payload descriptor", source.as_str())); + let payload = event + .payloads + .iter() + .find(|payload| payload.total_bytes == appended_body.len() as u64) + .expect("large appended payload"); + let cut_range = read_payload_range( + &mut cache, + source, + &session_id, + &completed.cursor.generation, + &event.chunk.chunk_id, + &payload.field_path, + 1, + Some(257), + ) + .unwrap_or_else(|error| panic!("{} UTF-8 cut range: {error}", source.as_str())); + assert!(cut_range.offset > 1, "range skips UTF-8 continuation bytes"); + assert!(!cut_range.text.is_empty()); + assert!(cut_range.text.len() <= 260); + assert!(cut_range.next_offset > cut_range.offset); +} + +#[test] +fn manifest_has_exactly_one_real_shape_fixture_for_every_registered_source() { + let manifest = manifest(); + assert_eq!(manifest.fixtures.len(), ImportedHistorySourceId::ALL.len()); + let mut by_source = BTreeMap::new(); + let mut files = BTreeSet::new(); + for fixture in manifest.fixtures { + let source = ImportedHistorySourceId::parse(&fixture.source_id) + .unwrap_or_else(|error| panic!("fixture source: {error}")); + assert_eq!(fixture.storage_family, source.descriptor().storage_family); + assert!( + by_source + .insert(source.as_str(), fixture.file.clone()) + .is_none(), + "duplicate fixture for {}", + source.as_str() + ); + assert!( + files.insert(fixture.file.clone()), + "fixture file must not stand in for two adapters: {}", + fixture.file + ); + let parsed: Value = if fixture.storage_family == ReplayStorageFamily::JsonLines { + Value::Array( + fixture_text(&fixture.file) + .lines() + .filter(|line| !line.trim().is_empty()) + .map(|line| serde_json::from_str(line).expect("fixture JSONL record")) + .collect(), + ) + } else { + serde_json::from_str(fixture_text(&fixture.file)).expect("fixture JSON document") + }; + assert!(!parsed.is_null(), "{} fixture parses", source.as_str()); + } + let expected = ImportedHistorySourceId::ALL + .into_iter() + .map(ImportedHistorySourceId::as_str) + .collect::>(); + assert_eq!(by_source.into_keys().collect::>(), expected); +} + +#[test] +fn all_fifteen_fixtures_parse_index_page_and_keep_stable_ids_with_hard_limits() { + let root = FixtureRoot::new(); + for spec in manifest().fixtures { + let source = ImportedHistorySourceId::parse(&spec.source_id).expect("fixture source"); + if source == ImportedHistorySourceId::Qoder { + crate::sources::qoder::log_enrichment::with_qoder_log_paths_for_test( + Vec::new(), + || assert_source_conformance(&spec, &root.0), + ); + } else { + assert_source_conformance(&spec, &root.0); + } + } +} + +#[test] +fn codex_and_every_shared_jsonl_adapter_skip_bad_complete_lines_and_continue() { + let root = FixtureRoot::new(); + let manifest = manifest(); + for source in [ + ImportedHistorySourceId::CodexApp, + ImportedHistorySourceId::ClaudeCode, + ImportedHistorySourceId::WorkBuddy, + ImportedHistorySourceId::Trae, + ImportedHistorySourceId::Qoder, + ImportedHistorySourceId::Omp, + ImportedHistorySourceId::QoderCli, + ] { + let spec = manifest + .fixtures + .iter() + .find(|fixture| fixture.source_id == source.as_str()) + .unwrap_or_else(|| panic!("{} manifest fixture", source.as_str())); + if source == ImportedHistorySourceId::Qoder { + crate::sources::qoder::log_enrichment::with_qoder_log_paths_for_test( + Vec::new(), + || assert_newline_terminated_bad_json_is_skipped(spec, source, &root.0), + ); + } else { + assert_newline_terminated_bad_json_is_skipped(spec, source, &root.0); + } + } +} + +#[test] +fn codex_and_shared_jsonl_ten_mib_utf8_append_reads_only_the_delta() { + let root = FixtureRoot::new(); + for source in [ + ImportedHistorySourceId::CodexApp, + ImportedHistorySourceId::ClaudeCode, + ] { + assert_ten_mib_utf8_append_is_incremental(source, &root.0); + } +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/common.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/common.rs new file mode 100644 index 000000000..e22ba3f83 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/common.rs @@ -0,0 +1,381 @@ +//! Storage-neutral replay driver primitives. +//! +//! This module contains only mechanics whose semantics are identical across +//! storage families: bounded newline reads, UTF-8 range accounting, content +//! digests, compatibility ID hashes, and compact turn folding. + +use std::io::{self, BufRead}; + +use rusqlite::{params, Transaction}; +use sha2::{Digest, Sha256}; + +use crate::sources::imported_history::replay::ReplayPayloadRange; +use crate::sources::imported_history::{self, replay::ImportedHistorySourceId}; + +pub(super) const MAX_JSONL_RECORD_BYTES: usize = 32 * 1024 * 1024; + +#[derive(Debug, PartialEq, Eq)] +pub(super) enum BoundedLine { + Eof, + Complete(Vec), + Incomplete, + TooLarge, +} + +/// Reads one newline-terminated record without ever growing the backing +/// allocation beyond `max_bytes`. +pub(super) fn read_bounded_line( + reader: &mut impl BufRead, + max_bytes: usize, +) -> io::Result { + let mut bytes = Vec::with_capacity(max_bytes.min(64 * 1024)); + loop { + let available = reader.fill_buf()?; + if available.is_empty() { + return Ok(if bytes.is_empty() { + BoundedLine::Eof + } else { + BoundedLine::Incomplete + }); + } + let take = available + .iter() + .position(|byte| *byte == b'\n') + .map_or(available.len(), |index| index + 1); + let Some(next_len) = bytes.len().checked_add(take) else { + return Ok(BoundedLine::TooLarge); + }; + if next_len > max_bytes { + return Ok(BoundedLine::TooLarge); + } + bytes + .try_reserve_exact(take) + .map_err(|error| io::Error::other(format!("reserve replay line: {error}")))?; + bytes.extend_from_slice(&available[..take]); + let complete = available[take - 1] == b'\n'; + reader.consume(take); + if complete { + return Ok(BoundedLine::Complete(bytes)); + } + } +} + +pub(super) fn trim_jsonl_line(mut bytes: &[u8]) -> &[u8] { + while bytes + .last() + .is_some_and(|byte| matches!(byte, b'\n' | b'\r')) + { + bytes = &bytes[..bytes.len() - 1]; + } + bytes +} + +pub(super) fn utf8_boundary_at_or_before(text: &str, mut offset: usize) -> usize { + offset = offset.min(text.len()); + while offset > 0 && !text.is_char_boundary(offset) { + offset -= 1; + } + offset +} + +pub(super) fn utf8_boundary_at_or_after(text: &str, mut offset: usize) -> usize { + offset = offset.min(text.len()); + while offset < text.len() && !text.is_char_boundary(offset) { + offset += 1; + } + offset +} + +/// Accumulates a byte range over one or more decoded UTF-8 payload parts. +/// +/// `part_start` is the byte offset of each part in the decoded logical +/// payload. Reporting the adjusted first boundary (rather than the requested +/// mid-scalar offset) keeps the next page aligned. +pub(super) struct Utf8RangeBuilder { + requested_start: u64, + requested_end: u64, + max_bytes: usize, + actual_start: Option, + next_offset: Option, + text: String, +} + +impl Utf8RangeBuilder { + pub(super) fn new(offset: u64, total_bytes: u64, max_bytes: usize) -> Self { + let requested_start = offset.min(total_bytes); + let requested_end = requested_start + .saturating_add(max_bytes as u64) + .min(total_bytes); + Self { + requested_start, + requested_end, + max_bytes, + actual_start: None, + next_offset: None, + text: String::with_capacity(max_bytes.min(256 * 1024).saturating_add(4)), + } + } + + pub(super) fn push_part(&mut self, part_start: u64, part: &str) { + let part_end = part_start.saturating_add(part.len() as u64); + if part_end <= self.requested_start || part_start >= self.requested_end { + return; + } + let overlap_start = self.requested_start.max(part_start); + let overlap_end = self.requested_end.min(part_end); + let start = + utf8_boundary_at_or_after(part, overlap_start.saturating_sub(part_start) as usize); + let mut end = + utf8_boundary_at_or_before(part, overlap_end.saturating_sub(part_start) as usize); + if end <= start { + end = start; + } + if end == start && start < part.len() && self.max_bytes > 0 && self.text.is_empty() { + end = part[start..] + .char_indices() + .nth(1) + .map_or(part.len(), |(next, _)| start + next); + } + let actual_start = part_start.saturating_add(start as u64); + self.actual_start.get_or_insert(actual_start); + if start < end { + self.text.push_str(&part[start..end]); + self.next_offset = Some(part_start.saturating_add(end as u64)); + } else { + self.next_offset.get_or_insert(actual_start); + } + } + + pub(super) fn finish( + self, + event_id: &str, + field_path: &str, + total_bytes: u64, + ) -> ReplayPayloadRange { + let offset = self.actual_start.unwrap_or(self.requested_start); + let next_offset = self.next_offset.unwrap_or(offset); + ReplayPayloadRange { + event_id: event_id.to_string(), + field_path: field_path.to_string(), + offset, + next_offset, + eof: next_offset >= total_bytes, + total_bytes, + text: self.text, + } + } +} + +pub(super) fn range_from_text( + event_id: &str, + field_path: &str, + text: &str, + offset: u64, + max_bytes: usize, +) -> Result { + let mut range = Utf8RangeBuilder::new(offset, text.len() as u64, max_bytes); + range.push_part(0, text); + Ok(range.finish(event_id, field_path, text.len() as u64)) +} + +#[derive(Clone, Default)] +pub(in crate::sources::imported_history::replay) struct ContentDigest(Sha256); + +impl ContentDigest { + fn update(&mut self, bytes: &[u8]) { + self.0.update(bytes); + } + + pub(in crate::sources::imported_history::replay) fn update_part(&mut self, bytes: &[u8]) { + self.update(&(bytes.len() as u64).to_le_bytes()); + self.update(bytes); + } + + pub(in crate::sources::imported_history::replay) fn finish_hex(&self) -> String { + self.0 + .clone() + .finalize() + .iter() + .map(|byte| format!("{byte:02x}")) + .collect() + } +} + +pub(super) fn content_digest(parts: &[&[u8]]) -> String { + let mut digest = ContentDigest::default(); + for part in parts { + digest.update_part(part); + } + digest.finish_hex() +} + +/// Compatibility hash for IDs that are already persisted or exposed on the +/// replay wire. New correctness/change-detection hashes must use SHA-256. +pub(super) fn legacy_stable_id_hash_concat(parts: &[&[u8]]) -> String { + legacy_fnv1a(parts, false) +} + +/// Compatibility variant used by structured and whole-JSON event IDs. +pub(super) fn legacy_stable_id_hash_delimited(parts: &[&[u8]]) -> String { + legacy_fnv1a(parts, true) +} + +fn legacy_fnv1a(parts: &[&[u8]], delimited: bool) -> String { + let mut hash = 0xcbf29ce484222325_u64; + for part in parts { + for byte in *part { + hash ^= u64::from(*byte); + hash = hash.wrapping_mul(0x100000001b3); + } + if delimited { + hash ^= 0xff; + hash = hash.wrapping_mul(0x100000001b3); + } + } + format!("{hash:016x}") +} + +#[derive(Debug)] +struct PendingTurn { + turn_id: String, + start_sequence: i64, + end_sequence: i64, + started_at: String, + ended_at: String, + event_count: u64, +} + +pub(super) fn rebuild_turns( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result<(), String> { + tx.execute( + "DELETE FROM imported_replay_turns + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + ) + .map_err(|error| format!("clear {} replay turn headers: {error}", source.as_str()))?; + let mut statement = tx + .prepare( + "SELECT sequence,event_id,function_name,created_at + FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + ORDER BY sequence", + ) + .map_err(|error| format!("prepare {} replay turn fold: {error}", source.as_str()))?; + let mut rows = statement + .query(params![source.as_str(), source_session_id, generation]) + .map_err(|error| format!("query {} replay turn fold: {error}", source.as_str()))?; + let mut turn_index = -1_i64; + let mut current = None; + while let Some(row) = rows.next().map_err(|error| error.to_string())? { + let sequence = row.get::<_, i64>(0).map_err(|error| error.to_string())?; + let event_id = row.get::<_, String>(1).map_err(|error| error.to_string())?; + let function = row.get::<_, String>(2).map_err(|error| error.to_string())?; + let created_at = row.get::<_, String>(3).map_err(|error| error.to_string())?; + if function == imported_history::FUNCTION_USER_MESSAGE || current.is_none() { + if let Some(turn) = current.take() { + insert_turn_header(tx, source, source_session_id, generation, turn_index, turn)?; + } + turn_index = turn_index + .checked_add(1) + .ok_or_else(|| "Replay turn index overflow".to_string())?; + current = Some(PendingTurn { + turn_id: event_id, + start_sequence: sequence, + end_sequence: sequence, + started_at: created_at.clone(), + ended_at: created_at, + event_count: 1, + }); + } else if let Some(turn) = current.as_mut() { + turn.end_sequence = sequence; + turn.ended_at = created_at; + turn.event_count = turn + .event_count + .checked_add(1) + .ok_or_else(|| "Replay turn event count overflow".to_string())?; + } + tx.execute( + "UPDATE imported_replay_events SET turn_index=?1 + WHERE source=?2 AND source_session_id=?3 AND generation=?4 AND sequence=?5", + params![ + turn_index, + source.as_str(), + source_session_id, + generation, + sequence + ], + ) + .map_err(|error| format!("assign {} replay turn: {error}", source.as_str()))?; + } + drop(rows); + drop(statement); + if let Some(turn) = current { + insert_turn_header(tx, source, source_session_id, generation, turn_index, turn)?; + } + Ok(()) +} + +fn insert_turn_header( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + turn_index: i64, + turn: PendingTurn, +) -> Result<(), String> { + tx.execute( + "INSERT INTO imported_replay_turns( + source,source_session_id,generation,turn_index,turn_id, + start_sequence,end_sequence,started_at,ended_at,event_count + ) VALUES (?1,?2,?3,?4,?5,?6,?7,?8,?9,?10)", + params![ + source.as_str(), + source_session_id, + generation, + turn_index, + turn.turn_id, + turn.start_sequence, + turn.end_sequence, + turn.started_at, + turn.ended_at, + turn.event_count as i64 + ], + ) + .map(|_| ()) + .map_err(|error| format!("insert {} replay turn: {error}", source.as_str())) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Cursor; + + #[test] + fn bounded_line_refuses_oversize_before_growing_past_the_cap() { + let mut input = Cursor::new(b"123456789\n".as_slice()); + assert_eq!( + read_bounded_line(&mut input, 8).expect("bounded line"), + BoundedLine::TooLarge + ); + } + + #[test] + fn utf8_range_reports_the_actual_boundary_and_makes_progress() { + let range = range_from_text("event", "result.output", "a你b", 2, 1).expect("UTF-8 range"); + assert_eq!(range.offset, 4); + assert_eq!(range.next_offset, 5); + assert_eq!(range.text, "b"); + } + + #[test] + fn content_digest_is_sha256_and_length_delimits_parts() { + let first = content_digest(&[b"ab", b"c"]); + let second = content_digest(&[b"a", b"bc"]); + assert_eq!(first.len(), 64); + assert_ne!(first, second); + } +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/codex.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/codex.rs new file mode 100644 index 000000000..836d98adf --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/codex.rs @@ -0,0 +1,1370 @@ +//! Incremental Codex JSONL replay driver. +//! +//! Complete newline-terminated records are folded directly into SQLite. The +//! driver keeps only unresolved tool calls/background handles in its cursor; +//! it never constructs a transcript-sized `Vec` and never +//! retains a complete background output string. + +use std::collections::HashMap; +use std::fs; +use std::io::{BufReader, Read, Seek, SeekFrom}; +use std::path::Path; + +use core_types::activity::ActivityChunk; +use core_types::extracted::ExtractedGitArtifactData; +use rusqlite::{params, OptionalExtension, Transaction}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +#[cfg(test)] +thread_local! { + static PAYLOAD_FALLBACK_DECODES: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +use crate::development_artifact::{ + attach_replay_git_artifacts, parse_git_artifacts, GitArtifactParseInput, +}; +use crate::sources::codex::app::{ + desktop_exec::codex_tool_output_text, + transcript::{ + background_cell_id, background_cell_key, background_session_key, codex_tool_call_chunk, + content_text_from_payload, pending_custom_tool_calls_from_payload, + pending_tool_calls_from_payload, reasoning_text_from_payload, record_stdin_event, + user_message_from_payload, wait_cell_id, web_search_call_from_payload, + }, + CodexJsonlLine, +}; +use crate::sources::imported_history::{self, ImportedToolCall}; + +use super as jsonl_driver; +use crate::sources::imported_history::replay::drivers::common::{ + content_digest, read_bounded_line, trim_jsonl_line, utf8_boundary_at_or_before, BoundedLine, + Utf8RangeBuilder, MAX_JSONL_RECORD_BYTES, +}; +use crate::sources::imported_history::replay::index::ReplayIndexState; +use crate::sources::imported_history::replay::payload_artifact; +use crate::sources::imported_history::replay::{ + replay_payload_body_projection, ReplayPayloadDescriptor, ReplayPayloadEncoding, + ReplayPayloadKind, ReplayPayloadRange, ReplaySourceSpan, ReplayStats, + NORMAL_PAYLOAD_PREVIEW_BYTES, SHELL_PAYLOAD_PREVIEW_BYTES, +}; + +#[derive(Debug, Clone)] +pub(in crate::sources::imported_history::replay) struct CodexSyncOutcome { + pub stats: ReplayStats, + pub driver_cursor_json: String, + pub indexed_size_bytes: u64, + pub total_events: u64, + pub total_turns: u64, + pub changed: bool, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct AssignedToolCall { + call: ImportedToolCall, + sequence: i64, + turn_index: i64, + #[serde(default)] + args_payload: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct PendingToolGroup { + calls: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct PendingBackgroundGroup { + calls: Vec, + spans: Vec, + output_preview: String, + output_bytes: u64, + #[serde(default)] + git_artifacts: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct CodexDriverCursor { + byte_offset: u64, + next_sequence: i64, + current_turn_index: i64, + pending_tool_calls: HashMap, + background_tool_calls: HashMap, + pending_task_turn_id: Option, + active_task_turn_id: Option, + /// Bounded fingerprint of the already-consumed prefix. File identity and + /// size alone cannot distinguish an append from a same-inode rewrite that + /// happens to grow past the previous cursor. + #[serde(default)] + boundary_fingerprint: String, + sample_fingerprint: String, + #[serde(default)] + catalog: crate::sources::imported_history::catalog::ReplayCatalogProjection, +} + +impl Default for CodexDriverCursor { + fn default() -> Self { + Self { + byte_offset: 0, + next_sequence: 0, + current_turn_index: -1, + pending_tool_calls: HashMap::new(), + background_tool_calls: HashMap::new(), + pending_task_turn_id: None, + active_task_turn_id: None, + boundary_fingerprint: String::new(), + sample_fingerprint: String::new(), + catalog: crate::sources::imported_history::catalog::ReplayCatalogProjection::default(), + } + } +} + +pub(in crate::sources::imported_history::replay) fn cursor_fingerprint( + cursor_json: &str, +) -> Option { + serde_json::from_str::(cursor_json) + .ok() + .map(|cursor| cursor.sample_fingerprint) +} + +pub(in crate::sources::imported_history::replay) fn cursor_matches_source( + path: &Path, + cursor_json: &str, +) -> bool { + let Ok(cursor) = serde_json::from_str::(cursor_json) else { + return false; + }; + if cursor.byte_offset == 0 { + return true; + } + jsonl_driver::boundary_fingerprint(path, cursor.byte_offset) + .is_ok_and(|value| value == cursor.boundary_fingerprint) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn sync( + tx: &Transaction<'_>, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + previous_state: Option<&ReplayIndexState>, + sample_fingerprint: &str, +) -> Result { + let mut cursor = match previous_state { + Some(state) => serde_json::from_str::(&state.driver_cursor_json) + .map_err(|err| format!("decode Codex replay cursor: {err}"))?, + None => CodexDriverCursor::default(), + }; + let mut stats = ReplayStats::default(); + let mut changed = false; + let mut file = fs::File::open(source_path) + .map_err(|err| format!("open Codex replay {}: {err}", source_path.display()))?; + file.seek(SeekFrom::Start(cursor.byte_offset)) + .map_err(|err| format!("seek Codex replay cursor: {err}"))?; + let mut reader = BufReader::new(file); + + loop { + let line_start = cursor.byte_offset; + let bytes = match read_bounded_line(&mut reader, MAX_JSONL_RECORD_BYTES) + .map_err(|err| format!("read Codex replay line: {err}"))? + { + BoundedLine::Eof | BoundedLine::Incomplete => break, + BoundedLine::TooLarge => { + return Err(format!( + "Codex replay record exceeds the {} MiB safety limit", + MAX_JSONL_RECORD_BYTES / (1024 * 1024) + )) + } + BoundedLine::Complete(bytes) => bytes, + }; + let read = bytes.len() as u64; + cursor.byte_offset = cursor + .byte_offset + .checked_add(read) + .ok_or_else(|| "Codex replay byte cursor overflow".to_string())?; + stats.parsed_bytes = stats.parsed_bytes.saturating_add(read); + let trimmed = trim_jsonl_line(&bytes); + if trimmed.is_empty() { + continue; + } + let parsed: CodexJsonlLine = match serde_json::from_slice(trimmed) { + Ok(parsed) => parsed, + Err(_) => continue, + }; + stats.parsed_rows = stats.parsed_rows.saturating_add(1); + cursor.catalog.observe_codex( + &parsed.line_type, + parsed.timestamp.as_deref(), + &parsed.payload, + source_session_id, + ); + let span = ReplaySourceSpan { + start: line_start, + end: cursor.byte_offset, + }; + changed |= fold_line( + tx, + display_session_id, + source_session_id, + generation, + write_revision, + source_path, + parsed, + span, + &mut cursor, + &mut stats, + )?; + } + + cursor.boundary_fingerprint = + jsonl_driver::boundary_fingerprint(source_path, cursor.byte_offset)?; + cursor.sample_fingerprint = sample_fingerprint.to_string(); + let total_events = tx + .query_row( + "SELECT COUNT(*) FROM imported_replay_events + WHERE source='codex_app' AND source_session_id=?1 AND generation=?2", + params![source_session_id, generation], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("count Codex replay events: {err}"))? + .max(0) as u64; + let total_turns = tx + .query_row( + "SELECT COUNT(*) FROM imported_replay_turns + WHERE source='codex_app' AND source_session_id=?1 AND generation=?2", + params![source_session_id, generation], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("count Codex replay turns: {err}"))? + .max(0) as u64; + Ok(CodexSyncOutcome { + stats, + driver_cursor_json: serde_json::to_string(&cursor) + .map_err(|err| format!("encode Codex replay cursor: {err}"))?, + indexed_size_bytes: cursor.byte_offset, + total_events, + total_turns, + changed, + }) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn fold_line( + tx: &Transaction<'_>, + session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + source_path: &Path, + parsed: CodexJsonlLine, + span: ReplaySourceSpan, + cursor: &mut CodexDriverCursor, + stats: &mut ReplayStats, +) -> Result { + let created_at = parsed + .timestamp + .as_deref() + .map(imported_history::normalize_created_at) + .unwrap_or_else(|| "1970-01-01T00:00:00Z".to_string()); + let Some(payload_type) = parsed.payload.get("type").and_then(Value::as_str) else { + return Ok(false); + }; + let mut changed = false; + match payload_type { + "task_started" => { + cursor.pending_task_turn_id = parsed + .payload + .get("turn_id") + .and_then(Value::as_str) + .map(str::to_string); + } + "user_message" => { + if let Some(message) = user_message_from_payload(&parsed.payload) { + start_turn(tx, source_session_id, generation, cursor, &created_at)?; + let (preview, truncated) = head_preview(&message, NORMAL_PAYLOAD_PREVIEW_BYTES); + let chunk = imported_history::user_message_chunk( + session_id, + "codex", + cursor.next_sequence as usize, + &created_at, + &preview, + ); + let payloads = payload_descriptor( + "result.message.content", + ReplayPayloadKind::UserMessage, + span, + message.len(), + truncated, + ); + changed |= insert_chunk( + tx, + source_session_id, + generation, + write_revision, + cursor.current_turn_index, + cursor.next_sequence, + &chunk, + &payloads, + span, + stats, + )?; + if truncated { + payload_artifact::store_text( + tx, + super::ImportedHistorySourceId::CodexApp, + source_session_id, + generation, + &chunk.chunk_id, + "result.message.content", + &message, + )?; + } + cursor.next_sequence += 1; + if let Some(turn_id) = cursor.pending_task_turn_id.take() { + let lifecycle = imported_history::task_lifecycle_chunk( + session_id, + "codex", + cursor.next_sequence as usize, + &created_at, + imported_history::ACTION_TYPE_TASK_START, + &turn_id, + ); + changed |= insert_chunk( + tx, + source_session_id, + generation, + write_revision, + cursor.current_turn_index, + cursor.next_sequence, + &lifecycle, + &[], + span, + stats, + )?; + cursor.next_sequence += 1; + cursor.active_task_turn_id = Some(turn_id); + } + } + } + "agent_message" => { + if let Some(message) = parsed.payload.get("message").and_then(Value::as_str) { + changed |= emit_text_chunk( + tx, + session_id, + source_session_id, + generation, + write_revision, + cursor, + stats, + &created_at, + span, + message, + ReplayPayloadKind::AgentMessage, + false, + )?; + } + } + "message" if parsed.payload.get("role").and_then(Value::as_str) == Some("assistant") => { + if let Some(message) = content_text_from_payload(&parsed.payload) { + changed |= emit_text_chunk( + tx, + session_id, + source_session_id, + generation, + write_revision, + cursor, + stats, + &created_at, + span, + &message, + ReplayPayloadKind::AssistantContent, + false, + )?; + } + } + "reasoning" | "agent_reasoning" => { + if let Some(thought) = reasoning_text_from_payload(&parsed.payload) { + changed |= emit_text_chunk( + tx, + session_id, + source_session_id, + generation, + write_revision, + cursor, + stats, + &created_at, + span, + &thought, + ReplayPayloadKind::Reasoning, + true, + )?; + } + } + "function_call" | "custom_tool_call" => { + let group = if payload_type == "function_call" { + pending_tool_calls_from_payload(&parsed.payload, &created_at) + } else { + pending_custom_tool_calls_from_payload(&parsed.payload, &created_at) + }; + if let Some((call_id, calls)) = group { + ensure_turn(tx, source_session_id, generation, cursor, &created_at)?; + let mut assigned = Vec::with_capacity(calls.len()); + for (call_ordinal, mut call) in calls.into_iter().enumerate() { + let compacted_args = compact_codex_tool_args( + &mut call, + span, + call_ordinal.min(u32::MAX as usize) as u32, + ); + let args_payload = compacted_args + .as_ref() + .map(|(descriptor, _)| descriptor.clone()); + let sequence = cursor.next_sequence; + let chunk = + codex_tool_call_chunk(session_id, sequence as usize, &call, "", None); + let payloads = args_payload.iter().cloned().collect::>(); + changed |= insert_chunk( + tx, + source_session_id, + generation, + write_revision, + cursor.current_turn_index, + sequence, + &chunk, + &payloads, + span, + stats, + )?; + if let Some((descriptor, full_args)) = compacted_args { + payload_artifact::store_text( + tx, + super::ImportedHistorySourceId::CodexApp, + source_session_id, + generation, + &chunk.chunk_id, + &descriptor.field_path, + &full_args, + )?; + } + assigned.push(AssignedToolCall { + call, + sequence, + turn_index: cursor.current_turn_index, + args_payload, + }); + cursor.next_sequence += 1; + } + cursor + .pending_tool_calls + .insert(call_id, PendingToolGroup { calls: assigned }); + } + } + "web_search_call" => { + if let Some(mut call) = web_search_call_from_payload(&parsed.payload, &created_at) { + ensure_turn(tx, source_session_id, generation, cursor, &created_at)?; + let compacted_args = compact_codex_tool_args(&mut call, span, 0); + let args_payload = compacted_args + .as_ref() + .map(|(descriptor, _)| descriptor.clone()); + let chunk = codex_tool_call_chunk( + session_id, + cursor.next_sequence as usize, + &call, + "", + None, + ); + changed |= insert_chunk( + tx, + source_session_id, + generation, + write_revision, + cursor.current_turn_index, + cursor.next_sequence, + &chunk, + &args_payload.into_iter().collect::>(), + span, + stats, + )?; + if let Some((descriptor, full_args)) = compacted_args { + payload_artifact::store_text( + tx, + super::ImportedHistorySourceId::CodexApp, + source_session_id, + generation, + &chunk.chunk_id, + &descriptor.field_path, + &full_args, + )?; + } + cursor.next_sequence += 1; + } + } + "function_call_output" | "custom_tool_call_output" => { + if let Some(call_id) = parsed.payload.get("call_id").and_then(Value::as_str) { + if let Some(group) = cursor.pending_tool_calls.remove(call_id) { + let output = codex_tool_output_text(parsed.payload.get("output")); + changed |= resolve_tool_output( + tx, + session_id, + source_session_id, + generation, + write_revision, + source_path, + group, + &output, + span, + cursor, + stats, + )?; + } + } + } + "task_complete" | "turn_aborted" => { + let action_type = if payload_type == "task_complete" { + imported_history::ACTION_TYPE_TASK_COMPLETED + } else { + imported_history::ACTION_TYPE_TASK_FAILED + }; + if let Some(turn_id) = parsed + .payload + .get("turn_id") + .and_then(Value::as_str) + .map(str::to_string) + .or_else(|| cursor.active_task_turn_id.clone()) + { + ensure_turn(tx, source_session_id, generation, cursor, &created_at)?; + let chunk = imported_history::task_lifecycle_chunk( + session_id, + "codex", + cursor.next_sequence as usize, + &created_at, + action_type, + &turn_id, + ); + changed |= insert_chunk( + tx, + source_session_id, + generation, + write_revision, + cursor.current_turn_index, + cursor.next_sequence, + &chunk, + &[], + span, + stats, + )?; + cursor.next_sequence += 1; + cursor.active_task_turn_id = None; + close_turn( + tx, + source_session_id, + generation, + cursor.current_turn_index, + cursor.next_sequence.saturating_sub(1), + &created_at, + )?; + } + } + _ => {} + } + Ok(changed) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn emit_text_chunk( + tx: &Transaction<'_>, + session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + cursor: &mut CodexDriverCursor, + stats: &mut ReplayStats, + created_at: &str, + span: ReplaySourceSpan, + text: &str, + kind: ReplayPayloadKind, + thinking: bool, +) -> Result { + ensure_turn(tx, source_session_id, generation, cursor, created_at)?; + // Compatibility chunks duplicate assistant/thinking text across several + // fields, and JSON escaping can expand Unicode. Keep the source-backed + // display preview comfortably below the per-field 8 KiB ceiling. + let (preview, truncated) = head_preview(text, 1024); + let chunk = if thinking { + imported_history::thinking_chunk( + session_id, + "codex", + cursor.next_sequence as usize, + created_at, + &preview, + ) + } else { + imported_history::assistant_message_chunk( + session_id, + "codex", + cursor.next_sequence as usize, + created_at, + &preview, + ) + }; + let payloads = payload_descriptor("result.content", kind, span, text.len(), truncated); + let changed = insert_chunk( + tx, + source_session_id, + generation, + write_revision, + cursor.current_turn_index, + cursor.next_sequence, + &chunk, + &payloads, + span, + stats, + )?; + if truncated { + payload_artifact::store_text( + tx, + super::ImportedHistorySourceId::CodexApp, + source_session_id, + generation, + &chunk.chunk_id, + "result.content", + text, + )?; + } + cursor.next_sequence += 1; + Ok(changed) +} + +fn compact_codex_tool_args( + call: &mut ImportedToolCall, + span: ReplaySourceSpan, + source_ordinal: u32, +) -> Option<(ReplayPayloadDescriptor, String)> { + let encoded = serde_json::to_string(&call.args).ok()?; + let limit = if call.canonical_name == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + if encoded.len() <= limit { + return None; + } + let body_projection = + replay_payload_body_projection("args", &call.args, Some(&encoded), limit, false); + call.args = jsonl_driver::compact_tool_args(&call.args, &call.canonical_name); + Some(( + ReplayPayloadDescriptor { + field_path: "args".to_string(), + kind: ReplayPayloadKind::ToolArguments, + encoding: ReplayPayloadEncoding::JsonValue, + body_projection, + spans: vec![span], + total_bytes: encoded.len() as u64, + source_ordinal: Some(source_ordinal), + source_key: None, + }, + encoded, + )) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn resolve_tool_output( + tx: &Transaction<'_>, + session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + source_path: &Path, + group: PendingToolGroup, + output: &str, + span: ReplaySourceSpan, + cursor: &mut CodexDriverCursor, + stats: &mut ReplayStats, +) -> Result { + if let Some(wait_key) = wait_key(&group.calls) { + if let Some(mut background) = cursor.background_tool_calls.remove(&wait_key) { + if let Some(continuation) = group.calls.first() { + let mut original_calls = background + .calls + .iter() + .map(|assigned| assigned.call.clone()) + .collect::>(); + record_stdin_event(&mut original_calls, &continuation.call); + for (assigned, call) in background.calls.iter_mut().zip(original_calls) { + assigned.call = call; + } + } + append_background_output(&mut background, output, span); + if let Some(next_key) = background_key_from_output(output) { + cursor.background_tool_calls.insert(next_key, background); + return Ok(false); + } + return finalize_tool_group( + tx, + session_id, + source_session_id, + generation, + write_revision, + source_path, + &background.calls, + &background.output_preview, + &background.spans, + background.output_bytes, + &background.git_artifacts, + stats, + ); + } + } + + if let Some(key) = background_key_from_output(output) { + let mut background = PendingBackgroundGroup { + calls: group.calls, + spans: Vec::new(), + output_preview: String::new(), + output_bytes: 0, + git_artifacts: Vec::new(), + }; + append_background_output(&mut background, output, span); + cursor.background_tool_calls.insert(key, background); + return Ok(false); + } + finalize_tool_group( + tx, + session_id, + source_session_id, + generation, + write_revision, + source_path, + &group.calls, + output, + &[span], + output.len() as u64, + &[], + stats, + ) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn finalize_tool_group( + tx: &Transaction<'_>, + session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + source_path: &Path, + calls: &[AssignedToolCall], + output: &str, + spans: &[ReplaySourceSpan], + output_bytes: u64, + precomputed_git_artifacts: &[ExtractedGitArtifactData], + stats: &mut ReplayStats, +) -> Result { + let mut changed = false; + let mut output_artifact_hash: Option = None; + for assigned in calls { + let preview_limit = + if assigned.call.canonical_name == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let (preview, preview_truncated) = tail_preview(output, preview_limit); + let truncated = preview_truncated || output_bytes > output.len() as u64; + let mut chunk = codex_tool_call_chunk( + session_id, + assigned.sequence as usize, + &assigned.call, + &preview, + None, + ); + let successful = chunk.result.get("success").and_then(Value::as_bool) != Some(false); + let command = assigned + .call + .args + .get("command") + .or_else(|| assigned.call.args.get("cmd")) + .and_then(Value::as_str) + .unwrap_or_default(); + let mut git_artifacts = if successful { + parse_git_artifacts(GitArtifactParseInput { + command, + output: Some(output), + exit_code: None, + }) + } else { + Vec::new() + }; + for artifact in precomputed_git_artifacts { + let key = serde_json::to_string(artifact).unwrap_or_default(); + if !git_artifacts + .iter() + .any(|existing| serde_json::to_string(existing).unwrap_or_default() == key) + { + git_artifacts.push(artifact.clone()); + } + } + attach_replay_git_artifacts(&mut chunk.result, &git_artifacts); + let mut payloads = assigned.args_payload.iter().cloned().collect::>(); + if truncated { + payloads.push(ReplayPayloadDescriptor { + field_path: "result.output".to_string(), + kind: ReplayPayloadKind::ToolOutput, + encoding: ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: spans.to_vec(), + total_bytes: output_bytes, + source_ordinal: None, + source_key: None, + }); + } + let source_span = spans + .last() + .copied() + .unwrap_or(ReplaySourceSpan { start: 0, end: 0 }); + changed |= insert_chunk( + tx, + source_session_id, + generation, + write_revision, + assigned.turn_index, + assigned.sequence, + &chunk, + &payloads, + source_span, + stats, + )?; + if truncated { + let content_hash = if let Some(content_hash) = output_artifact_hash.as_ref() { + payload_artifact::reference( + tx, + super::ImportedHistorySourceId::CodexApp, + source_session_id, + generation, + &chunk.chunk_id, + "result.output", + content_hash, + )?; + content_hash.clone() + } else if output.len() as u64 == output_bytes { + payload_artifact::store_text( + tx, + super::ImportedHistorySourceId::CodexApp, + source_session_id, + generation, + &chunk.chunk_id, + "result.output", + output, + )? + } else { + stream_codex_output_artifact( + tx, + source_session_id, + generation, + &chunk.chunk_id, + source_path, + spans, + output_bytes, + )? + }; + output_artifact_hash = Some(content_hash); + } + } + Ok(changed) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn stream_codex_output_artifact( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + event_id: &str, + source_path: &Path, + spans: &[ReplaySourceSpan], + output_bytes: u64, +) -> Result { + let mut file = fs::File::open(source_path).map_err(|error| { + format!( + "open Codex replay artifact source {}: {error}", + source_path.display() + ) + })?; + payload_artifact::store_streamed( + tx, + super::ImportedHistorySourceId::CodexApp, + source_session_id, + generation, + event_id, + "result.output", + output_bytes, + |writer| { + for span in spans { + file.seek(SeekFrom::Start(span.start)) + .map_err(|error| format!("seek Codex replay artifact source: {error}"))?; + let length = usize::try_from(span.end.saturating_sub(span.start)) + .map_err(|_| "Codex replay source span exceeds address space".to_string())?; + let mut bytes = vec![0_u8; length]; + file.read_exact(&mut bytes) + .map_err(|error| format!("read Codex replay artifact source: {error}"))?; + let parsed: CodexJsonlLine = serde_json::from_slice(trim_jsonl_line(&bytes)) + .map_err(|error| format!("decode Codex replay artifact line: {error}"))?; + let text = codex_tool_output_text(parsed.payload.get("output")); + writer + .write_all(text.as_bytes()) + .map_err(|error| format!("write Codex replay artifact: {error}"))?; + } + Ok(()) + }, + ) +} + +fn append_background_output( + background: &mut PendingBackgroundGroup, + output: &str, + span: ReplaySourceSpan, +) { + for assigned in &background.calls { + let command = assigned + .call + .args + .get("command") + .or_else(|| assigned.call.args.get("cmd")) + .and_then(Value::as_str) + .unwrap_or_default(); + for artifact in parse_git_artifacts(GitArtifactParseInput { + command, + output: Some(output), + exit_code: None, + }) { + let key = serde_json::to_string(&artifact).unwrap_or_default(); + if !background + .git_artifacts + .iter() + .any(|existing| serde_json::to_string(existing).unwrap_or_default() == key) + { + background.git_artifacts.push(artifact); + } + } + } + background.spans.push(span); + background.output_bytes = background.output_bytes.saturating_add(output.len() as u64); + background.output_preview.push_str(output); + if background.output_preview.len() > SHELL_PAYLOAD_PREVIEW_BYTES { + background.output_preview = + utf8_tail(&background.output_preview, SHELL_PAYLOAD_PREVIEW_BYTES).to_string(); + } +} + +fn wait_key(calls: &[AssignedToolCall]) -> Option { + let raw = calls + .iter() + .map(|call| call.call.clone()) + .collect::>(); + if let Some(cell_id) = wait_cell_id(&raw) { + return Some(background_cell_key(cell_id)); + } + let call = raw.first()?; + if call.canonical_name == imported_history::FUNCTION_AWAIT_OUTPUT { + return call + .args + .get("session_id") + .or_else(|| call.args.get("handle")) + .and_then(Value::as_str) + .filter(|id| !id.is_empty()) + .map(background_session_key); + } + None +} + +fn background_key_from_output(output: &str) -> Option { + if let Some(cell_id) = background_cell_id(output) { + return Some(background_cell_key(&cell_id)); + } + let value: Value = serde_json::from_str(output.trim()).ok()?; + let session_id = value + .get("session_id") + .or_else(|| value.get("sessionId")) + .and_then(|value| match value { + Value::String(value) => Some(value.clone()), + Value::Number(value) => Some(value.to_string()), + _ => None, + })?; + Some(background_session_key(&session_id)) +} + +fn start_turn( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + cursor: &mut CodexDriverCursor, + created_at: &str, +) -> Result<(), String> { + if cursor.current_turn_index >= 0 { + close_turn( + tx, + source_session_id, + generation, + cursor.current_turn_index, + cursor.next_sequence.saturating_sub(1), + created_at, + )?; + } + cursor.current_turn_index += 1; + insert_turn( + tx, + source_session_id, + generation, + cursor.current_turn_index, + cursor.next_sequence, + created_at, + ) +} + +fn ensure_turn( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + cursor: &mut CodexDriverCursor, + created_at: &str, +) -> Result<(), String> { + if cursor.current_turn_index < 0 { + cursor.current_turn_index = 0; + insert_turn( + tx, + source_session_id, + generation, + 0, + cursor.next_sequence, + created_at, + )?; + } + Ok(()) +} + +fn insert_turn( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + turn_index: i64, + start_sequence: i64, + started_at: &str, +) -> Result<(), String> { + tx.execute( + "INSERT OR IGNORE INTO imported_replay_turns ( + source, source_session_id, generation, turn_index, turn_id, + start_sequence, started_at, event_count + ) VALUES ('codex_app', ?1, ?2, ?3, ?4, ?5, ?6, 0)", + params![ + source_session_id, + generation, + turn_index, + format!("codex-turn-{turn_index}"), + start_sequence, + started_at, + ], + ) + .map_err(|err| format!("insert Codex replay turn: {err}"))?; + Ok(()) +} + +fn close_turn( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + turn_index: i64, + end_sequence: i64, + ended_at: &str, +) -> Result<(), String> { + tx.execute( + "UPDATE imported_replay_turns SET end_sequence=?1, ended_at=?2 + WHERE source='codex_app' AND source_session_id=?3 + AND generation=?4 AND turn_index=?5", + params![ + end_sequence, + ended_at, + source_session_id, + generation, + turn_index + ], + ) + .map_err(|err| format!("close Codex replay turn: {err}"))?; + Ok(()) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn insert_chunk( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + write_revision: u64, + turn_index: i64, + sequence: i64, + chunk: &ActivityChunk, + payloads: &[ReplayPayloadDescriptor], + span: ReplaySourceSpan, + stats: &mut ReplayStats, +) -> Result { + let args_json = serde_json::to_string(&chunk.args) + .map_err(|err| format!("encode Codex replay args: {err}"))?; + let result_json = serde_json::to_string(&chunk.result) + .map_err(|err| format!("encode Codex replay result: {err}"))?; + let payloads_json = serde_json::to_string(payloads) + .map_err(|err| format!("encode Codex replay payload locators: {err}"))?; + let content_hash = content_digest(&[ + chunk.action_type.as_bytes(), + chunk.function.as_bytes(), + args_json.as_bytes(), + result_json.as_bytes(), + payloads_json.as_bytes(), + ]); + let existing = tx + .query_row( + "SELECT content_hash FROM imported_replay_events + WHERE source='codex_app' AND source_session_id=?1 + AND generation=?2 AND sequence=?3", + params![source_session_id, generation, sequence], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|err| format!("read Codex replay content hash: {err}"))?; + if existing.as_deref() == Some(content_hash.as_str()) { + return Ok(false); + } + let inserted = existing.is_none(); + tx.execute( + "INSERT INTO imported_replay_events ( + source, source_session_id, generation, sequence, event_id, + turn_index, action_type, function_name, created_at, + args_preview_json, result_preview_json, args_size_bytes, + result_size_bytes, thread_id, process_id, source_start, source_end, + payloads_json, content_hash, event_revision + ) VALUES ( + 'codex_app', ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, + ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19 + ) ON CONFLICT(source, source_session_id, generation, sequence) DO UPDATE SET + event_id=excluded.event_id, + turn_index=excluded.turn_index, + action_type=excluded.action_type, + function_name=excluded.function_name, + created_at=excluded.created_at, + args_preview_json=excluded.args_preview_json, + result_preview_json=excluded.result_preview_json, + args_size_bytes=excluded.args_size_bytes, + result_size_bytes=excluded.result_size_bytes, + thread_id=excluded.thread_id, + process_id=excluded.process_id, + source_start=excluded.source_start, + source_end=excluded.source_end, + payloads_json=excluded.payloads_json, + content_hash=excluded.content_hash, + event_revision=excluded.event_revision", + params![ + source_session_id, + generation, + sequence, + chunk.chunk_id, + turn_index, + chunk.action_type, + chunk.function, + chunk.created_at, + args_json, + result_json, + serde_json::to_vec(&chunk.args).map_or(0, |bytes| bytes.len()) as i64, + serde_json::to_vec(&chunk.result).map_or(0, |bytes| bytes.len()) as i64, + chunk.thread_id, + chunk.process_id, + span.start as i64, + span.end as i64, + payloads_json, + content_hash, + write_revision.min(i64::MAX as u64) as i64, + ], + ) + .map_err(|err| format!("upsert Codex replay event: {err}"))?; + if inserted { + tx.execute( + "UPDATE imported_replay_turns SET event_count=event_count+1 + WHERE source='codex_app' AND source_session_id=?1 + AND generation=?2 AND turn_index=?3", + params![source_session_id, generation, turn_index], + ) + .map_err(|err| format!("increment Codex replay turn count: {err}"))?; + } + stats.normalized_events = stats.normalized_events.saturating_add(1); + stats.upserted_events = stats.upserted_events.saturating_add(1); + Ok(true) +} + +fn payload_descriptor( + field_path: &str, + kind: ReplayPayloadKind, + span: ReplaySourceSpan, + total_bytes: usize, + truncated: bool, +) -> Vec { + if !truncated { + return Vec::new(); + } + vec![ReplayPayloadDescriptor { + field_path: field_path.to_string(), + kind, + encoding: ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: vec![span], + total_bytes: total_bytes as u64, + source_ordinal: None, + source_key: None, + }] +} + +pub(in crate::sources::imported_history::replay) fn read_payload( + source_path: &Path, + payloads_json: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + #[cfg(test)] + PAYLOAD_FALLBACK_DECODES.with(|count| count.set(count.get().saturating_add(1))); + let payloads: Vec = serde_json::from_str(payloads_json) + .map_err(|err| format!("decode Codex replay payload locator: {err}"))?; + let descriptor = payloads + .into_iter() + .find(|payload| payload.field_path == field_path) + .ok_or_else(|| format!("No deferred replay payload for {field_path}"))?; + let requested_start = offset.min(descriptor.total_bytes); + let requested_end = requested_start + .saturating_add(max_bytes as u64) + .min(descriptor.total_bytes); + let mut range = Utf8RangeBuilder::new(offset, descriptor.total_bytes, max_bytes); + let mut decoded_offset = 0_u64; + let mut file = fs::File::open(source_path) + .map_err(|err| format!("open Codex replay payload {}: {err}", source_path.display()))?; + for span in &descriptor.spans { + file.seek(SeekFrom::Start(span.start)) + .map_err(|err| format!("seek Codex replay payload: {err}"))?; + let len = span.end.saturating_sub(span.start) as usize; + let mut bytes = vec![0_u8; len]; + file.read_exact(&mut bytes) + .map_err(|err| format!("read Codex replay payload: {err}"))?; + let parsed: CodexJsonlLine = serde_json::from_slice(trim_jsonl_line(&bytes)) + .map_err(|err| format!("decode Codex replay payload line: {err}"))?; + let part = payload_text(&descriptor, &parsed).ok_or_else(|| { + format!( + "Codex replay payload span no longer yields {:?} for {field_path}", + descriptor.kind + ) + })?; + let part_start = decoded_offset; + decoded_offset = decoded_offset + .checked_add(part.len() as u64) + .ok_or_else(|| "Codex decoded payload offset overflow".to_string())?; + range.push_part(part_start, &part); + if decoded_offset >= requested_end { + break; + } + } + Ok(range.finish(event_id, field_path, descriptor.total_bytes)) +} + +#[cfg(test)] +pub(in crate::sources::imported_history::replay) fn reset_payload_fallback_decodes() { + PAYLOAD_FALLBACK_DECODES.with(|count| count.set(0)); +} + +#[cfg(test)] +pub(in crate::sources::imported_history::replay) fn payload_fallback_decodes() -> usize { + PAYLOAD_FALLBACK_DECODES.with(std::cell::Cell::get) +} + +fn payload_text(descriptor: &ReplayPayloadDescriptor, parsed: &CodexJsonlLine) -> Option { + let payload = &parsed.payload; + match descriptor.kind { + ReplayPayloadKind::UserMessage => user_message_from_payload(payload), + ReplayPayloadKind::AgentMessage => payload + .get("message") + .and_then(Value::as_str) + .map(str::to_string), + ReplayPayloadKind::AssistantContent => content_text_from_payload(payload), + ReplayPayloadKind::Reasoning => reasoning_text_from_payload(payload), + ReplayPayloadKind::ToolOutput => Some(codex_tool_output_text(payload.get("output"))), + ReplayPayloadKind::ToolArguments => { + let payload_type = payload + .get("type") + .and_then(Value::as_str) + .unwrap_or_default(); + let calls = match payload_type { + "function_call" => { + pending_tool_calls_from_payload(payload, "").map(|(_, calls)| calls) + } + "custom_tool_call" => { + pending_custom_tool_calls_from_payload(payload, "").map(|(_, calls)| calls) + } + "web_search_call" => { + web_search_call_from_payload(payload, "").map(|call| vec![call]) + } + _ => None, + }?; + let ordinal = descriptor.source_ordinal.unwrap_or(0) as usize; + calls + .get(ordinal) + .and_then(|call| serde_json::to_string(&call.args).ok()) + } + ReplayPayloadKind::ToolDiff => None, + } +} + +fn head_preview(text: &str, max_bytes: usize) -> (String, bool) { + if text.len() <= max_bytes { + return (text.to_string(), false); + } + let end = utf8_boundary_at_or_before(text, max_bytes); + (format!("{}\n… [payload truncated]", &text[..end]), true) +} + +fn tail_preview(text: &str, max_bytes: usize) -> (String, bool) { + if text.len() <= max_bytes { + return (text.to_string(), false); + } + ( + format!("[payload truncated] …\n{}", utf8_tail(text, max_bytes)), + true, + ) +} + +fn utf8_tail(text: &str, max_bytes: usize) -> &str { + if text.len() <= max_bytes { + return text; + } + let mut start = text.len().saturating_sub(max_bytes); + while start < text.len() && !text.is_char_boundary(start) { + start += 1; + } + &text[start..] +} + +#[cfg(test)] +#[path = "../../tests/codex_jsonl.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/mod.rs new file mode 100644 index 000000000..4a4099d70 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/mod.rs @@ -0,0 +1,1048 @@ +//! Shared incremental driver for imported JSONL histories other than Codex. +//! +//! Storage mechanics live here (complete-line byte cursor, lineage checking, +//! compact pending tool state and atomic SQLite folding). Provider schemas are +//! normalized per source so sharing the driver does not erase replay semantics. + +use std::collections::HashMap; +use std::fs; +use std::io::{BufReader, Read, Seek, SeekFrom}; +use std::path::Path; + +use core_types::activity::ActivityChunk; +use rusqlite::{params, OptionalExtension, Transaction}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; + +#[cfg(test)] +thread_local! { + static PAYLOAD_FALLBACK_DECODES: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +use crate::development_artifact::{ + attach_replay_git_artifacts, parse_git_artifacts, GitArtifactParseInput, +}; +use crate::sources::imported_history::{self, ImportedToolCall}; + +use crate::sources::imported_history::replay::drivers::common::{ + content_digest, read_bounded_line, trim_jsonl_line, utf8_boundary_at_or_before, BoundedLine, + ContentDigest, Utf8RangeBuilder, MAX_JSONL_RECORD_BYTES, +}; +use crate::sources::imported_history::replay::index::ReplayIndexState; +use crate::sources::imported_history::replay::payload_artifact; +pub(in crate::sources::imported_history::replay) mod codex; +mod normalize; +pub(in crate::sources::imported_history::replay) mod qoder_sidecar; + +pub(in crate::sources::imported_history::replay) use normalize::compact_tool_args; +use normalize::*; + +use crate::sources::imported_history::replay::{ + replay_payload_body_projection, ImportedHistorySourceId, ReplayPayloadBodyProjection, + ReplayPayloadDescriptor, ReplayPayloadEncoding, ReplayPayloadKind, ReplayPayloadRange, + ReplaySourceSpan, ReplayStats, NORMAL_PAYLOAD_PREVIEW_BYTES, SHELL_PAYLOAD_PREVIEW_BYTES, +}; + +const CURSOR_VERSION: u32 = 1; +const BOUNDARY_BYTES: u64 = 4 * 1024; +/// Qoder's text transcript omits its tool trajectory. Reserve a wide, stable +/// sequence gap between transcript records so timestamp-ordered sidecar cards +/// can arrive later without renumbering transcript events. +pub(in crate::sources::imported_history::replay) const QODER_PRIMARY_SEQUENCE_STEP: i64 = + 1_000_000_000_000_000; + +#[derive(Debug, Clone)] +pub(in crate::sources::imported_history::replay) struct JsonlSyncOutcome { + pub stats: ReplayStats, + pub driver_cursor_json: String, + pub indexed_size_bytes: u64, + pub total_events: u64, + pub total_turns: u64, + pub changed: bool, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct PendingCall { + call: ImportedToolCall, + sequence: i64, + turn_index: i64, + call_span: ReplaySourceSpan, + payload_ordinal: u32, + args_size_bytes: usize, + args_truncated: bool, + #[serde(default)] + args_body_projection: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct JsonlCursor { + version: u32, + byte_offset: u64, + next_sequence: i64, + current_turn_index: i64, + #[serde(default)] + pending_calls: HashMap, + #[serde(default)] + boundary_fingerprint: String, + #[serde(default)] + sample_fingerprint: String, + #[serde(default)] + qoder_sidecar: qoder_sidecar::QoderSidecarCursor, + #[serde(default)] + catalog: crate::sources::imported_history::catalog::ReplayCatalogProjection, +} + +impl Default for JsonlCursor { + fn default() -> Self { + Self { + version: CURSOR_VERSION, + byte_offset: 0, + next_sequence: 0, + current_turn_index: -1, + pending_calls: HashMap::new(), + boundary_fingerprint: String::new(), + sample_fingerprint: String::new(), + qoder_sidecar: qoder_sidecar::QoderSidecarCursor::default(), + catalog: crate::sources::imported_history::catalog::ReplayCatalogProjection::default(), + } + } +} + +#[derive(Debug, Clone)] +enum NormalizedKind { + UserText(String), + AssistantText(String), + Thinking(String), + ToolUse(ImportedToolCall), + ToolResult { + call_id: String, + output: String, + failed: bool, + diff: Option, + }, +} + +#[derive(Debug, Clone)] +struct NormalizedEvent { + created_at: String, + starts_turn: bool, + kind: NormalizedKind, +} + +pub(in crate::sources::imported_history::replay) fn cursor_fingerprint( + cursor_json: &str, +) -> Option { + serde_json::from_str::(cursor_json) + .ok() + .map(|cursor| cursor.sample_fingerprint) +} + +/// Detect a same-inode rewrite that grew instead of truncating. Metadata and +/// whole-file samples cannot distinguish that from append; the compact cursor +/// therefore remembers bounded samples from the already-indexed prefix. The +/// offsets are based on the old complete-line cursor, so an ordinary append +/// preserves the fingerprint while a growing in-place rewrite resets. +pub(in crate::sources::imported_history::replay) fn cursor_matches_source( + path: &Path, + cursor_json: &str, +) -> bool { + let Ok(cursor) = serde_json::from_str::(cursor_json) else { + return false; + }; + if cursor.byte_offset == 0 { + return true; + } + boundary_fingerprint(path, cursor.byte_offset) + .is_ok_and(|value| value == cursor.boundary_fingerprint) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn sync( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + previous: Option<&ReplayIndexState>, + sample_fingerprint: &str, +) -> Result { + let mut cursor = previous + .and_then(|state| serde_json::from_str::(&state.driver_cursor_json).ok()) + .filter(|cursor| cursor.version == CURSOR_VERSION) + .unwrap_or_default(); + let mut file = fs::File::open(source_path).map_err(|err| { + format!( + "open {} replay {}: {err}", + source.as_str(), + source_path.display() + ) + })?; + file.seek(SeekFrom::Start(cursor.byte_offset)) + .map_err(|err| format!("seek {} replay cursor: {err}", source.as_str()))?; + let mut reader = BufReader::new(file); + let mut stats = ReplayStats::default(); + let mut changed = false; + + loop { + let line_start = cursor.byte_offset; + let bytes = match read_bounded_line(&mut reader, MAX_JSONL_RECORD_BYTES) + .map_err(|err| format!("read {} replay line: {err}", source.as_str()))? + { + BoundedLine::Eof | BoundedLine::Incomplete => break, + BoundedLine::TooLarge => { + return Err(format!( + "{} replay record exceeds the {} MiB safety limit", + source.as_str(), + MAX_JSONL_RECORD_BYTES / (1024 * 1024) + )) + } + BoundedLine::Complete(bytes) => bytes, + }; + let read = bytes.len() as u64; + cursor.byte_offset = cursor + .byte_offset + .checked_add(read) + .ok_or_else(|| format!("{} replay byte cursor overflow", source.as_str()))?; + stats.parsed_bytes = stats.parsed_bytes.saturating_add(read); + let trimmed = trim_jsonl_line(&bytes); + if trimmed.is_empty() { + continue; + } + let raw: Value = match serde_json::from_slice(trimmed) { + Ok(raw) => raw, + Err(_) => continue, + }; + stats.parsed_rows = stats.parsed_rows.saturating_add(1); + cursor + .catalog + .observe_jsonl(source, &raw, source_session_id); + let span = ReplaySourceSpan { + start: line_start, + end: cursor.byte_offset, + }; + let events = normalize_line(source, &raw); + for (ordinal, event) in events.into_iter().enumerate() { + changed |= fold_event( + tx, + source, + display_session_id, + source_session_id, + generation, + write_revision, + span, + ordinal as u32, + event, + &mut cursor, + &mut stats, + )?; + } + } + + if source == ImportedHistorySourceId::Qoder { + let primary_changed = changed; + let sidecar = qoder_sidecar::sync( + tx, + display_session_id, + source_session_id, + generation, + write_revision, + &cursor.qoder_sidecar, + primary_changed, + &mut stats, + )?; + cursor.qoder_sidecar = sidecar.cursor; + changed |= sidecar.changed; + } + cursor.boundary_fingerprint = boundary_fingerprint(source_path, cursor.byte_offset)?; + cursor.sample_fingerprint = sample_fingerprint.to_string(); + let total_events = count_rows( + tx, + "imported_replay_events", + source, + source_session_id, + generation, + )?; + let total_turns = count_rows( + tx, + "imported_replay_turns", + source, + source_session_id, + generation, + )?; + Ok(JsonlSyncOutcome { + stats, + driver_cursor_json: serde_json::to_string(&cursor) + .map_err(|err| format!("encode {} replay cursor: {err}", source.as_str()))?, + indexed_size_bytes: cursor.byte_offset, + total_events, + total_turns, + changed, + }) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn fold_event( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + span: ReplaySourceSpan, + payload_ordinal: u32, + event: NormalizedEvent, + cursor: &mut JsonlCursor, + stats: &mut ReplayStats, +) -> Result { + let created_at = event.created_at.as_str(); + if event.starts_turn { + start_turn( + tx, + source, + source_session_id, + generation, + cursor, + created_at, + )?; + } else { + ensure_turn( + tx, + source, + source_session_id, + generation, + cursor, + created_at, + )?; + } + let provider = provider_slug(source); + match event.kind { + NormalizedKind::UserText(text) => { + let (preview, truncated) = head_preview(&text, NORMAL_PAYLOAD_PREVIEW_BYTES); + let chunk = imported_history::user_message_chunk( + session_id, + provider, + cursor.next_sequence as usize, + created_at, + &preview, + ); + let payloads = payload_descriptor(PayloadDescriptorInput { + field_path: "result.message.content", + kind: ReplayPayloadKind::UserMessage, + encoding: ReplayPayloadEncoding::Utf8Text, + span, + source_ordinal: payload_ordinal, + total_bytes: text.len(), + truncated, + body_projection: None, + }); + let changed = insert_new_chunk( + tx, + source, + source_session_id, + generation, + write_revision, + cursor, + &chunk, + &payloads, + span, + stats, + )?; + if truncated { + payload_artifact::store_text( + tx, + source, + source_session_id, + generation, + &chunk.chunk_id, + "result.message.content", + &text, + )?; + } + Ok(changed) + } + NormalizedKind::AssistantText(text) => { + let (preview, truncated) = head_preview(&text, NORMAL_PAYLOAD_PREVIEW_BYTES); + let chunk = imported_history::assistant_message_chunk( + session_id, + provider, + cursor.next_sequence as usize, + created_at, + &preview, + ); + let payloads = payload_descriptor(PayloadDescriptorInput { + field_path: "result.content", + kind: ReplayPayloadKind::AssistantContent, + encoding: ReplayPayloadEncoding::Utf8Text, + span, + source_ordinal: payload_ordinal, + total_bytes: text.len(), + truncated, + body_projection: None, + }); + let changed = insert_new_chunk( + tx, + source, + source_session_id, + generation, + write_revision, + cursor, + &chunk, + &payloads, + span, + stats, + )?; + if truncated { + payload_artifact::store_text( + tx, + source, + source_session_id, + generation, + &chunk.chunk_id, + "result.content", + &text, + )?; + } + Ok(changed) + } + NormalizedKind::Thinking(text) => { + let (preview, truncated) = head_preview(&text, NORMAL_PAYLOAD_PREVIEW_BYTES); + let chunk = imported_history::thinking_chunk( + session_id, + provider, + cursor.next_sequence as usize, + created_at, + &preview, + ); + let payloads = payload_descriptor(PayloadDescriptorInput { + field_path: "result.content", + kind: ReplayPayloadKind::Reasoning, + encoding: ReplayPayloadEncoding::Utf8Text, + span, + source_ordinal: payload_ordinal, + total_bytes: text.len(), + truncated, + body_projection: None, + }); + let changed = insert_new_chunk( + tx, + source, + source_session_id, + generation, + write_revision, + cursor, + &chunk, + &payloads, + span, + stats, + )?; + if truncated { + payload_artifact::store_text( + tx, + source, + source_session_id, + generation, + &chunk.chunk_id, + "result.content", + &text, + )?; + } + Ok(changed) + } + NormalizedKind::ToolUse(mut call) => { + let full_args = serde_json::to_string(&call.args).unwrap_or_default(); + let args_size_bytes = full_args.len(); + let args_limit = if call.canonical_name == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let args_truncated = args_size_bytes > args_limit; + let args_body_projection = args_truncated + .then(|| { + replay_payload_body_projection( + "args", + &call.args, + Some(&full_args), + args_limit, + false, + ) + }) + .flatten(); + call.args = compact_tool_args(&call.args, &call.canonical_name); + let sequence = cursor.next_sequence; + let chunk = imported_history::tool_call_chunk( + session_id, + provider, + sequence as usize, + &call, + "", + ); + let payloads = payload_descriptor(PayloadDescriptorInput { + field_path: "args", + kind: ReplayPayloadKind::ToolArguments, + encoding: ReplayPayloadEncoding::JsonValue, + span, + source_ordinal: payload_ordinal, + total_bytes: args_size_bytes, + truncated: args_truncated, + body_projection: args_body_projection.clone(), + }); + let changed = insert_new_chunk( + tx, + source, + source_session_id, + generation, + write_revision, + cursor, + &chunk, + &payloads, + span, + stats, + )?; + if args_truncated { + payload_artifact::store_text( + tx, + source, + source_session_id, + generation, + &chunk.chunk_id, + "args", + &full_args, + )?; + } + cursor.pending_calls.insert( + call.call_id.clone(), + PendingCall { + call, + sequence, + turn_index: cursor.current_turn_index, + call_span: span, + payload_ordinal, + args_size_bytes, + args_truncated, + args_body_projection, + }, + ); + Ok(changed) + } + NormalizedKind::ToolResult { + call_id, + output, + failed, + diff, + } => { + let Some(pending) = cursor.pending_calls.remove(&call_id) else { + return Ok(false); + }; + let limit = + if pending.call.canonical_name == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let (preview, truncated) = tail_preview(&output, limit); + let mut chunk = imported_history::tool_call_chunk( + session_id, + provider, + pending.sequence as usize, + &pending.call, + &preview, + ); + let git_artifacts = if failed { + Vec::new() + } else { + let command = pending + .call + .args + .get("command") + .or_else(|| pending.call.args.get("cmd")) + .and_then(Value::as_str) + .unwrap_or_default(); + parse_git_artifacts(GitArtifactParseInput { + command, + output: Some(&output), + exit_code: None, + }) + }; + attach_replay_git_artifacts(&mut chunk.result, &git_artifacts); + if failed { + if let Some(result) = chunk.result.as_object_mut() { + result.insert("success".to_string(), Value::Bool(false)); + result.insert("status".to_string(), Value::String("failed".to_string())); + } + } + let mut deferred_bodies = Vec::new(); + let mut payloads = payload_descriptor(PayloadDescriptorInput { + field_path: "result.output", + kind: ReplayPayloadKind::ToolOutput, + encoding: ReplayPayloadEncoding::Utf8Text, + span, + source_ordinal: payload_ordinal, + total_bytes: output.len(), + truncated, + body_projection: None, + }); + if truncated { + deferred_bodies.push(("result.output".to_string(), output)); + } + payloads.extend(payload_descriptor(PayloadDescriptorInput { + field_path: "args", + kind: ReplayPayloadKind::ToolArguments, + encoding: ReplayPayloadEncoding::JsonValue, + span: pending.call_span, + source_ordinal: pending.payload_ordinal, + total_bytes: pending.args_size_bytes, + truncated: pending.args_truncated, + body_projection: pending.args_body_projection, + })); + if let Some(diff) = diff { + let (diff_preview, diff_truncated) = + head_preview(&diff, NORMAL_PAYLOAD_PREVIEW_BYTES); + if let Some(result) = chunk.result.as_object_mut() { + result.insert("diff".to_string(), Value::String(diff_preview)); + let (added, removed) = count_diff_lines(&diff); + result.insert("linesAdded".to_string(), json!(added)); + result.insert("linesRemoved".to_string(), json!(removed)); + } + payloads.extend(payload_descriptor(PayloadDescriptorInput { + field_path: "result.diff", + kind: ReplayPayloadKind::ToolDiff, + encoding: ReplayPayloadEncoding::Utf8Text, + span, + source_ordinal: payload_ordinal, + total_bytes: diff.len(), + truncated: diff_truncated, + body_projection: None, + })); + if diff_truncated { + deferred_bodies.push(("result.diff".to_string(), diff)); + } + } + let changed = upsert_chunk( + tx, + source, + source_session_id, + generation, + write_revision, + pending.turn_index, + pending.sequence, + &chunk, + &payloads, + ReplaySourceSpan { + start: pending.call_span.start, + end: span.end, + }, + stats, + )?; + for (field_path, body) in deferred_bodies { + payload_artifact::store_text( + tx, + source, + source_session_id, + generation, + &chunk.chunk_id, + &field_path, + &body, + )?; + } + Ok(changed) + } + } +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn insert_new_chunk( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + write_revision: u64, + cursor: &mut JsonlCursor, + chunk: &ActivityChunk, + payloads: &[ReplayPayloadDescriptor], + span: ReplaySourceSpan, + stats: &mut ReplayStats, +) -> Result { + let changed = upsert_chunk( + tx, + source, + source_session_id, + generation, + write_revision, + cursor.current_turn_index, + cursor.next_sequence, + chunk, + payloads, + span, + stats, + )?; + cursor.next_sequence = cursor + .next_sequence + .checked_add(primary_sequence_step(source)) + .ok_or_else(|| { + format!( + "{} replay sequence space exhausted; refusing to overwrite existing events", + source.as_str() + ) + })?; + Ok(changed) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn upsert_chunk( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + write_revision: u64, + turn_index: i64, + sequence: i64, + chunk: &ActivityChunk, + payloads: &[ReplayPayloadDescriptor], + span: ReplaySourceSpan, + stats: &mut ReplayStats, +) -> Result { + let args_json = + serde_json::to_string(&chunk.args).map_err(|err| format!("encode replay args: {err}"))?; + let result_json = serde_json::to_string(&chunk.result) + .map_err(|err| format!("encode replay result: {err}"))?; + let payloads_json = + serde_json::to_string(payloads).map_err(|err| format!("encode replay payloads: {err}"))?; + let content_hash = content_digest(&[ + chunk.action_type.as_bytes(), + chunk.function.as_bytes(), + args_json.as_bytes(), + result_json.as_bytes(), + payloads_json.as_bytes(), + ]); + let existing = tx + .query_row( + "SELECT content_hash FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND sequence=?4", + params![source.as_str(), source_session_id, generation, sequence], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|err| format!("read replay content hash: {err}"))?; + if existing.as_deref() == Some(content_hash.as_str()) { + return Ok(false); + } + let inserted = existing.is_none(); + tx.execute( + "INSERT INTO imported_replay_events ( + source, source_session_id, generation, sequence, event_id, + turn_index, action_type, function_name, created_at, + args_preview_json, result_preview_json, args_size_bytes, + result_size_bytes, thread_id, process_id, source_start, source_end, + payloads_json, content_hash, event_revision + ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, + ?14, ?15, ?16, ?17, ?18, ?19, ?20) + ON CONFLICT(source, source_session_id, generation, sequence) DO UPDATE SET + event_id=excluded.event_id, turn_index=excluded.turn_index, + action_type=excluded.action_type, function_name=excluded.function_name, + created_at=excluded.created_at, args_preview_json=excluded.args_preview_json, + result_preview_json=excluded.result_preview_json, + args_size_bytes=excluded.args_size_bytes, result_size_bytes=excluded.result_size_bytes, + thread_id=excluded.thread_id, process_id=excluded.process_id, + source_start=excluded.source_start, source_end=excluded.source_end, + payloads_json=excluded.payloads_json, content_hash=excluded.content_hash, + event_revision=excluded.event_revision", + params![ + source.as_str(), + source_session_id, + generation, + sequence, + chunk.chunk_id, + turn_index, + chunk.action_type, + chunk.function, + chunk.created_at, + args_json, + result_json, + serde_json::to_vec(&chunk.args).map_or(0, |bytes| bytes.len()) as i64, + serde_json::to_vec(&chunk.result).map_or(0, |bytes| bytes.len()) as i64, + chunk.thread_id, + chunk.process_id, + span.start as i64, + span.end as i64, + payloads_json, + content_hash, + write_revision.min(i64::MAX as u64) as i64, + ], + ) + .map_err(|err| format!("upsert {} replay event: {err}", source.as_str()))?; + if inserted { + tx.execute( + "UPDATE imported_replay_turns SET event_count=event_count+1 + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND turn_index=?4", + params![source.as_str(), source_session_id, generation, turn_index], + ) + .map_err(|err| format!("increment replay turn count: {err}"))?; + } + stats.normalized_events = stats.normalized_events.saturating_add(1); + stats.upserted_events = stats.upserted_events.saturating_add(1); + Ok(true) +} + +fn start_turn( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + cursor: &mut JsonlCursor, + created_at: &str, +) -> Result<(), String> { + // Tool results belong to the turn that issued them. Calls still unresolved + // when a new authored user turn starts remain visible with an empty result, + // but no longer need cursor state; retaining them forever would make the + // compact cursor grow with session history. + cursor.pending_calls.clear(); + if cursor.current_turn_index >= 0 { + tx.execute( + "UPDATE imported_replay_turns SET end_sequence=?1, ended_at=?2 + WHERE source=?3 AND source_session_id=?4 AND generation=?5 AND turn_index=?6", + params![ + cursor.next_sequence.saturating_sub(1), + created_at, + source.as_str(), + source_session_id, + generation, + cursor.current_turn_index + ], + ) + .map_err(|err| format!("close replay turn: {err}"))?; + } + cursor.current_turn_index = cursor.current_turn_index.saturating_add(1); + insert_turn( + tx, + source, + source_session_id, + generation, + cursor.current_turn_index, + cursor.next_sequence, + created_at, + ) +} + +fn ensure_turn( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + cursor: &mut JsonlCursor, + created_at: &str, +) -> Result<(), String> { + if cursor.current_turn_index < 0 { + cursor.current_turn_index = 0; + insert_turn( + tx, + source, + source_session_id, + generation, + 0, + cursor.next_sequence, + created_at, + )?; + } + Ok(()) +} + +fn insert_turn( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + turn_index: i64, + start_sequence: i64, + started_at: &str, +) -> Result<(), String> { + tx.execute( + "INSERT OR IGNORE INTO imported_replay_turns ( + source, source_session_id, generation, turn_index, turn_id, + start_sequence, started_at, event_count + ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0)", + params![ + source.as_str(), + source_session_id, + generation, + turn_index, + format!("{}-turn-{turn_index}", source.as_str()), + start_sequence, + started_at + ], + ) + .map_err(|err| format!("insert replay turn: {err}"))?; + Ok(()) +} + +fn count_rows( + tx: &Transaction<'_>, + table: &str, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result { + let sql = format!( + "SELECT COUNT(*) FROM {table} WHERE source=?1 AND source_session_id=?2 AND generation=?3" + ); + tx.query_row( + &sql, + params![source.as_str(), source_session_id, generation], + |row| row.get::<_, i64>(0), + ) + .map(|count| count.max(0) as u64) + .map_err(|err| format!("count {} replay rows: {err}", source.as_str())) +} + +pub(in crate::sources::imported_history::replay) fn read_payload( + source: ImportedHistorySourceId, + source_path: &Path, + payloads_json: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + #[cfg(test)] + PAYLOAD_FALLBACK_DECODES.with(|count| count.set(count.get().saturating_add(1))); + let payloads: Vec = serde_json::from_str(payloads_json) + .map_err(|err| format!("decode replay payload locator: {err}"))?; + let descriptor = payloads + .into_iter() + .find(|payload| payload.field_path == field_path) + .ok_or_else(|| format!("No deferred replay payload for {field_path}"))?; + let requested_start = offset.min(descriptor.total_bytes); + let requested_end = requested_start + .saturating_add(max_bytes as u64) + .min(descriptor.total_bytes); + let mut range = Utf8RangeBuilder::new(offset, descriptor.total_bytes, max_bytes); + let mut decoded_offset = 0_u64; + let mut file = fs::File::open(source_path) + .map_err(|err| format!("open replay payload {}: {err}", source_path.display()))?; + for span in descriptor.spans { + file.seek(SeekFrom::Start(span.start)) + .map_err(|err| format!("seek replay payload: {err}"))?; + let mut bytes = vec![0; span.end.saturating_sub(span.start) as usize]; + file.read_exact(&mut bytes) + .map_err(|err| format!("read replay payload: {err}"))?; + let raw: Value = serde_json::from_slice(trim_jsonl_line(&bytes)) + .map_err(|err| format!("decode replay payload line: {err}"))?; + let events = normalize_line(source, &raw); + let ordinal = descriptor.source_ordinal.unwrap_or_default() as usize; + let part = events + .get(ordinal) + .and_then(|event| normalized_payload_text(event, descriptor.kind)) + .ok_or_else(|| { + format!( + "Replay payload span no longer yields {:?} for {field_path}", + descriptor.kind + ) + })?; + let part_start = decoded_offset; + decoded_offset = decoded_offset + .checked_add(part.len() as u64) + .ok_or_else(|| "Replay decoded payload offset overflow".to_string())?; + range.push_part(part_start, &part); + if decoded_offset >= requested_end { + break; + } + } + Ok(range.finish(event_id, field_path, descriptor.total_bytes)) +} + +#[cfg(test)] +pub(in crate::sources::imported_history::replay) fn reset_payload_fallback_decodes() { + PAYLOAD_FALLBACK_DECODES.with(|count| count.set(0)); +} + +#[cfg(test)] +pub(in crate::sources::imported_history::replay) fn payload_fallback_decodes() -> usize { + PAYLOAD_FALLBACK_DECODES.with(std::cell::Cell::get) +} + +fn normalized_payload_text(event: &NormalizedEvent, kind: ReplayPayloadKind) -> Option { + match (&event.kind, kind) { + (NormalizedKind::ToolResult { diff, .. }, ReplayPayloadKind::ToolDiff) => diff.clone(), + (NormalizedKind::ToolUse(call), ReplayPayloadKind::ToolArguments) => { + serde_json::to_string(&call.args).ok() + } + (NormalizedKind::ToolResult { output, .. }, ReplayPayloadKind::ToolOutput) => { + Some(output.clone()) + } + (NormalizedKind::UserText(text), ReplayPayloadKind::UserMessage) + | (NormalizedKind::AssistantText(text), ReplayPayloadKind::AssistantContent) + | (NormalizedKind::Thinking(text), ReplayPayloadKind::Reasoning) => Some(text.clone()), + _ => None, + } +} + +pub(in crate::sources::imported_history::replay) fn boundary_fingerprint( + path: &Path, + offset: u64, +) -> Result { + if offset == 0 { + return Ok(String::new()); + } + let mut file = fs::File::open(path).map_err(|err| format!("open JSONL boundary: {err}"))?; + let sample_bytes = BOUNDARY_BYTES.min(offset); + let starts = [ + 0, + offset.saturating_sub(sample_bytes) / 2, + offset.saturating_sub(sample_bytes), + ]; + let mut hash = ContentDigest::default(); + for start in starts { + file.seek(SeekFrom::Start(start)) + .map_err(|err| format!("seek JSONL lineage sample: {err}"))?; + let len = sample_bytes.min(offset.saturating_sub(start)) as usize; + let mut bytes = vec![0; len]; + file.read_exact(&mut bytes) + .map_err(|err| format!("read JSONL lineage sample: {err}"))?; + hash.update_part(&start.to_le_bytes()); + hash.update_part(&bytes); + } + Ok(hash.finish_hex()) +} + +pub(in crate::sources::imported_history::replay) fn head_preview( + text: &str, + max_bytes: usize, +) -> (String, bool) { + if text.len() <= max_bytes { + return (text.to_string(), false); + } + let end = utf8_boundary_at_or_before(text, max_bytes); + (format!("{}\n… [payload truncated]", &text[..end]), true) +} + +pub(in crate::sources::imported_history::replay) fn tail_preview( + text: &str, + max_bytes: usize, +) -> (String, bool) { + if text.len() <= max_bytes { + return (text.to_string(), false); + } + let mut start = text.len().saturating_sub(max_bytes); + while start < text.len() && !text.is_char_boundary(start) { + start += 1; + } + (format!("[payload truncated] …\n{}", &text[start..]), true) +} + +#[cfg(test)] +#[path = "../../tests/jsonl_driver.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/normalize.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/normalize.rs new file mode 100644 index 000000000..58929ae69 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/normalize.rs @@ -0,0 +1,630 @@ +use super::*; + +pub(super) fn normalize_line(source: ImportedHistorySourceId, raw: &Value) -> Vec { + if source == ImportedHistorySourceId::Trae { + return normalize_trae(raw); + } + let created_at = normalized_timestamp(raw.get("timestamp").or_else(|| raw.get("createdAt"))); + let mut events = Vec::new(); + + // WorkBuddy also writes top-level function_call/result records. + if source == ImportedHistorySourceId::WorkBuddy { + if let Some(call) = workbuddy_top_level_call(raw, &created_at) { + events.push(NormalizedEvent { + created_at: created_at.clone(), + starts_turn: false, + kind: NormalizedKind::ToolUse(call), + }); + } + if let Some((call_id, output, failed)) = workbuddy_top_level_result(raw) { + events.push(NormalizedEvent { + created_at: created_at.clone(), + starts_turn: false, + kind: NormalizedKind::ToolResult { + call_id, + output, + failed, + diff: None, + }, + }); + } + if raw.get("type").and_then(Value::as_str) == Some("reasoning") { + if let Some(text) = first_text(raw.get("content").or_else(|| raw.get("rawContent"))) { + events.push(NormalizedEvent { + created_at: created_at.clone(), + starts_turn: false, + kind: NormalizedKind::AssistantText(text), + }); + } + } + } + + let message = raw + .get("message") + .filter(|message| message.is_object()) + .or_else(|| { + (raw.get("type").and_then(Value::as_str) == Some("message") + && raw.get("role").is_some()) + .then_some(raw) + }); + let Some(message) = message else { + return events; + }; + let role = message + .get("role") + .and_then(Value::as_str) + .or_else(|| raw.get("role").and_then(Value::as_str)) + .or_else(|| raw.get("type").and_then(Value::as_str)) + .unwrap_or_default(); + let content = message.get("content").unwrap_or(&Value::Null); + let claude_diff = (source == ImportedHistorySourceId::ClaudeCode) + .then(|| raw.get("toolUseResult").and_then(claude_structured_diff)) + .flatten(); + normalize_content( + source, + role, + content, + &created_at, + claude_diff.as_deref(), + &mut events, + ); + events +} + +pub(super) fn normalize_content( + source: ImportedHistorySourceId, + role: &str, + content: &Value, + created_at: &str, + tool_diff: Option<&str>, + events: &mut Vec, +) { + let items: Vec<&Value> = match content { + Value::Array(items) => items.iter().collect(), + Value::String(_) => vec![content], + _ => Vec::new(), + }; + let mut user_text_seen = false; + for item in items { + if let Some(text) = item.as_str() { + let text = normalize_user_text(source, role, text); + if text.is_empty() { + continue; + } + let is_user = role == "user"; + events.push(NormalizedEvent { + created_at: created_at.to_string(), + starts_turn: is_user && !user_text_seen, + kind: if is_user { + NormalizedKind::UserText(text) + } else { + NormalizedKind::AssistantText(text) + }, + }); + user_text_seen |= is_user; + continue; + } + match item.get("type").and_then(Value::as_str).unwrap_or("text") { + "text" => { + let text = item.get("text").and_then(Value::as_str).unwrap_or_default(); + let text = normalize_user_text(source, role, text); + if text.is_empty() { + continue; + } + let is_user = role == "user"; + events.push(NormalizedEvent { + created_at: created_at.to_string(), + starts_turn: is_user && !user_text_seen, + kind: if is_user { + NormalizedKind::UserText(text) + } else { + NormalizedKind::AssistantText(text) + }, + }); + user_text_seen |= is_user; + } + "thinking" | "reasoning" => { + let text = item + .get("thinking") + .or_else(|| item.get("text")) + .and_then(Value::as_str) + .unwrap_or_default() + .trim(); + if !text.is_empty() { + events.push(NormalizedEvent { + created_at: created_at.to_string(), + starts_turn: false, + kind: NormalizedKind::Thinking(text.to_string()), + }); + } + } + "tool_use" | "function_call" => { + if let Some(call) = block_tool_call(source, item, created_at) { + events.push(NormalizedEvent { + created_at: created_at.to_string(), + starts_turn: false, + kind: NormalizedKind::ToolUse(call), + }); + } + } + "tool_result" | "function_call_result" => { + if let Some(call_id) = item + .get("tool_use_id") + .or_else(|| item.get("callId")) + .or_else(|| item.get("call_id")) + .and_then(Value::as_str) + { + let output = value_to_text(item.get("content").or_else(|| item.get("output"))); + let failed = item.get("is_error").and_then(Value::as_bool) == Some(true); + events.push(NormalizedEvent { + created_at: created_at.to_string(), + starts_turn: false, + kind: NormalizedKind::ToolResult { + call_id: call_id.to_string(), + output, + failed, + diff: tool_diff.map(str::to_string), + }, + }); + } + } + _ => {} + } + } +} + +pub(super) fn normalize_trae(raw: &Value) -> Vec { + let created_at = raw + .get("message_summary_time") + .and_then(Value::as_str) + .and_then(trae_timestamp) + .unwrap_or_default(); + let mut events = Vec::new(); + let intent = raw + .get("intent") + .and_then(Value::as_str) + .unwrap_or_default() + .trim(); + if !intent.is_empty() { + events.push(NormalizedEvent { + created_at: created_at.clone(), + starts_turn: true, + kind: NormalizedKind::UserText(intent.to_string()), + }); + } + let mut body = String::new(); + if let Some(outcome) = raw + .get("outcome") + .and_then(Value::as_str) + .map(str::trim) + .filter(|text| !text.is_empty()) + { + body.push_str(outcome); + } + append_summary_list(&mut body, "Actions:", raw.get("actions")); + append_summary_list(&mut body, "Learned:", raw.get("learned")); + if !body.is_empty() { + events.push(NormalizedEvent { + created_at, + starts_turn: intent.is_empty(), + kind: NormalizedKind::AssistantText(body), + }); + } + events +} + +pub(super) fn append_summary_list(output: &mut String, heading: &str, value: Option<&Value>) { + let Some(items) = value + .and_then(Value::as_array) + .filter(|items| !items.is_empty()) + else { + return; + }; + if !output.is_empty() { + output.push_str("\n\n"); + } + output.push_str(heading); + for item in items.iter().filter_map(Value::as_str) { + output.push_str("\n- "); + output.push_str(item.trim()); + } +} + +pub(super) fn block_tool_call( + source: ImportedHistorySourceId, + item: &Value, + created_at: &str, +) -> Option { + let call_id = item + .get("id") + .or_else(|| item.get("callId")) + .or_else(|| item.get("call_id")) + .and_then(Value::as_str)? + .to_string(); + let raw_name = item + .get("name") + .or_else(|| item.get("tool")) + .and_then(Value::as_str) + .unwrap_or("tool") + .to_string(); + let args = item + .get("input") + .or_else(|| item.get("arguments")) + .cloned() + .unwrap_or_else(|| json!({})); + let (canonical_name, args) = normalize_tool_call(source, &raw_name, parse_argument_value(args)); + Some(ImportedToolCall { + call_id, + raw_name, + canonical_name, + args, + created_at: created_at.to_string(), + }) +} + +pub(super) fn workbuddy_top_level_call(raw: &Value, created_at: &str) -> Option { + let call = raw.get("functionCall").or_else(|| { + (raw.get("type").and_then(Value::as_str) == Some("function_call")).then_some(raw) + })?; + block_tool_call(ImportedHistorySourceId::WorkBuddy, call, created_at) +} + +pub(super) fn workbuddy_top_level_result(raw: &Value) -> Option<(String, String, bool)> { + let result = raw.get("functionCallResult").or_else(|| { + (raw.get("type").and_then(Value::as_str) == Some("function_call_result")).then_some(raw) + })?; + let call_id = result + .get("callId") + .or_else(|| result.get("call_id")) + .or_else(|| result.get("id")) + .and_then(Value::as_str)? + .to_string(); + let value = result + .get("output") + .or_else(|| result.get("result")) + .or_else(|| result.get("content")); + let output = value_to_text(value); + let failed = result + .get("status") + .and_then(Value::as_str) + .is_some_and(|status| matches!(status, "failed" | "error")); + Some((call_id, output, failed)) +} + +pub(super) fn normalize_tool_call( + source: ImportedHistorySourceId, + raw_name: &str, + args: Value, +) -> (String, Value) { + let lower = raw_name.to_ascii_lowercase(); + if matches!( + lower.as_str(), + "bash" | "shell" | "execute" | "run_command" | "terminal" | "terminal_command" + ) { + let command = args + .get("command") + .or_else(|| args.get("cmd")) + .and_then(Value::as_str) + .unwrap_or_default(); + return ( + imported_history::FUNCTION_RUN_COMMAND_LINE.to_string(), + json!({ "command": command, "cmd": command }), + ); + } + if source != ImportedHistorySourceId::Qoder + && matches!( + lower.as_str(), + "edit" + | "multiedit" + | "write" + | "edit_file" + | "edit_file_v2" + | "write_file" + | "patch" + | "apply_patch" + | "str_replace" + ) + { + let file_path = args + .get("file_path") + .or_else(|| args.get("filePath")) + .or_else(|| args.get("path")) + .or_else(|| args.get("targetFile")) + .or_else(|| args.get("relativeWorkspacePath")) + .and_then(Value::as_str) + .unwrap_or_default(); + return ( + imported_history::FUNCTION_EDIT_FILE.to_string(), + json!({ "action": raw_name, "file_path": file_path, "payload": args }), + ); + } + (raw_name.to_string(), args) +} + +pub(super) fn parse_argument_value(value: Value) -> Value { + match value { + Value::String(text) => imported_history::parse_inner_json(&text), + other => other, + } +} + +pub(super) fn normalize_user_text( + source: ImportedHistorySourceId, + role: &str, + text: &str, +) -> String { + if role != "user" { + return text.trim().to_string(); + } + let stripped = imported_history::strip_internal_context_blocks(text); + if source == ImportedHistorySourceId::Qoder { + if let Some(start) = stripped.find("") { + let rest = &stripped[start + "".len()..]; + return rest + .split("") + .next() + .unwrap_or(rest) + .trim() + .to_string(); + } + return strip_tag_blocks(stripped, "system-reminder"); + } + stripped.trim().to_string() +} + +pub(super) fn strip_tag_blocks(text: &str, tag: &str) -> String { + let open = format!("<{tag}>"); + let close = format!(""); + let mut output = String::new(); + let mut rest = text; + while let Some(start) = rest.find(&open) { + output.push_str(&rest[..start]); + let Some(end) = rest[start + open.len()..].find(&close) else { + break; + }; + rest = &rest[start + open.len() + end + close.len()..]; + } + output.push_str(rest); + output.trim().to_string() +} + +pub(super) fn first_text(value: Option<&Value>) -> Option { + match value? { + Value::String(text) => Some(text.clone()), + Value::Array(items) => { + let parts = items + .iter() + .filter_map(|item| { + item.get("text") + .or_else(|| item.get("content")) + .and_then(Value::as_str) + }) + .collect::>(); + (!parts.is_empty()).then(|| parts.join("\n")) + } + _ => None, + } +} + +pub(super) fn value_to_text(value: Option<&Value>) -> String { + let Some(value) = value else { + return String::new(); + }; + match value { + Value::String(text) => text.clone(), + Value::Array(items) => items + .iter() + .map(|item| value_to_text(Some(item))) + .filter(|text| !text.is_empty()) + .collect::>() + .join("\n"), + Value::Object(map) => map + .get("text") + .or_else(|| map.get("content")) + .map(|value| value_to_text(Some(value))) + .unwrap_or_else(|| value.to_string()), + Value::Null => String::new(), + other => other.to_string(), + } +} + +pub(super) fn claude_structured_diff(result: &Value) -> Option { + let hunks = result.get("structuredPatch").and_then(Value::as_array)?; + if hunks.is_empty() { + return None; + } + let path = result + .get("filePath") + .and_then(Value::as_str) + .unwrap_or_default(); + let mut diff = format!("--- {path}\n+++ {path}\n"); + for hunk in hunks { + let old_start = hunk + .get("oldStart") + .and_then(Value::as_i64) + .unwrap_or_default(); + let old_lines = hunk + .get("oldLines") + .and_then(Value::as_i64) + .unwrap_or_default(); + let new_start = hunk + .get("newStart") + .and_then(Value::as_i64) + .unwrap_or_default(); + let new_lines = hunk + .get("newLines") + .and_then(Value::as_i64) + .unwrap_or_default(); + diff.push_str(&format!( + "@@ -{old_start},{old_lines} +{new_start},{new_lines} @@\n" + )); + for line in hunk + .get("lines") + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter_map(Value::as_str) + { + diff.push_str(line); + diff.push('\n'); + } + } + Some(diff) +} + +pub(super) fn count_diff_lines(diff: &str) -> (i64, i64) { + let mut added = 0; + let mut removed = 0; + for line in diff.lines() { + if line.starts_with('+') && !line.starts_with("+++") { + added += 1; + } else if line.starts_with('-') && !line.starts_with("---") { + removed += 1; + } + } + (added, removed) +} + +pub(super) fn normalized_timestamp(value: Option<&Value>) -> String { + match value { + Some(Value::String(raw)) if !raw.trim().is_empty() => { + imported_history::normalize_created_at(raw) + } + Some(Value::Number(number)) => number + .as_i64() + .map(|value| { + if value < 10_000_000_000 { + value.saturating_mul(1_000) + } else { + value + } + }) + .map(imported_history::epoch_ms_to_iso) + .unwrap_or_default(), + _ => String::new(), + } +} + +pub(super) fn trae_timestamp(raw: &str) -> Option { + chrono::NaiveDateTime::parse_from_str(raw.trim(), "%Y-%m-%d %H:%M:%S") + .ok() + .and_then(|naive| naive.and_local_timezone(chrono::Local).earliest()) + .map(|timestamp| timestamp.to_rfc3339()) +} + +pub(super) fn provider_slug(source: ImportedHistorySourceId) -> &'static str { + match source { + ImportedHistorySourceId::ClaudeCode => "claudecode", + ImportedHistorySourceId::WorkBuddy => "workbuddy", + ImportedHistorySourceId::Trae => "trae", + ImportedHistorySourceId::Qoder => "qoder", + ImportedHistorySourceId::Omp => "omp", + ImportedHistorySourceId::QoderCli => "qoder_cli", + _ => source.as_str(), + } +} + +pub(super) fn primary_sequence_step(source: ImportedHistorySourceId) -> i64 { + if source == ImportedHistorySourceId::Qoder { + QODER_PRIMARY_SEQUENCE_STEP + } else { + 1 + } +} + +pub(in crate::sources::imported_history::replay) fn compact_tool_args( + args: &Value, + function: &str, +) -> Value { + let limit = if function == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let Ok(encoded) = serde_json::to_string(args) else { + return json!({}); + }; + if encoded.len() <= limit { + return args.clone(); + } + let preview_budget = (limit / 2).max(256); + let (preview, _) = head_preview(&encoded, preview_budget); + let mut compact = json!({ "payloadPreview": preview, "payloadTruncated": true }); + if let (Some(target), Some(source)) = (compact.as_object_mut(), args.as_object()) { + let semantic_keys = [ + "command", + "cmd", + "file_path", + "filePath", + "path", + "action", + "query", + "pattern", + "linesAdded", + "linesRemoved", + "operation", + "agentType", + "description", + "cell_id", + "session_id", + "chars", + "limit", + "offset", + ]; + let selected = semantic_keys + .into_iter() + .filter_map(|key| source.get(key).map(|value| (key, value))) + .collect::>(); + let per_value_budget = (limit / 2) + .checked_div(selected.len().max(1)) + .unwrap_or(256) + .max(128); + for (key, value) in selected { + target.insert( + key.to_string(), + compact_semantic_arg_value(value, per_value_budget), + ); + } + } + compact +} + +pub(super) fn compact_semantic_arg_value(value: &Value, max_bytes: usize) -> Value { + let encoded = serde_json::to_string(value).unwrap_or_default(); + if encoded.len() <= max_bytes { + return value.clone(); + } + if let Some(text) = value.as_str() { + return Value::String(head_preview(text, max_bytes).0); + } + Value::String(head_preview(&encoded, max_bytes).0) +} + +pub(super) struct PayloadDescriptorInput<'a> { + pub(super) field_path: &'a str, + pub(super) kind: ReplayPayloadKind, + pub(super) encoding: ReplayPayloadEncoding, + pub(super) span: ReplaySourceSpan, + pub(super) source_ordinal: u32, + pub(super) total_bytes: usize, + pub(super) truncated: bool, + pub(super) body_projection: Option, +} + +pub(super) fn payload_descriptor( + input: PayloadDescriptorInput<'_>, +) -> Vec { + if !input.truncated { + return Vec::new(); + } + vec![ReplayPayloadDescriptor { + field_path: input.field_path.to_string(), + kind: input.kind, + encoding: input.encoding, + body_projection: input.body_projection, + spans: vec![input.span], + total_bytes: input.total_bytes as u64, + source_ordinal: Some(input.source_ordinal), + source_key: None, + }] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/qoder_sidecar.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/qoder_sidecar.rs new file mode 100644 index 000000000..8dedc8b92 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/jsonl/qoder_sidecar.rs @@ -0,0 +1,1316 @@ +//! Incremental, compact replay index for Qoder's trajectory sidecars. +//! +//! Qoder's conversation JSONL omits most tool activity. The missing data is +//! spread across per-launch `agent.log` and exthost logs shared by every +//! session. This adapter acknowledges only complete records, persists compact +//! source locators (never whole log bodies), and replays the old conservative +//! content/window attribution rules from SQLite rows. + +use std::collections::HashMap; +use std::fs; +use std::io::{BufReader, Read, Seek, SeekFrom}; +use std::path::{Path, PathBuf}; + +use rusqlite::{params, OptionalExtension, Transaction}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; + +use crate::development_artifact::{ + attach_replay_git_artifacts, parse_git_artifacts_from_tool_payload, +}; +use crate::sources::imported_history::{self, ImportedToolCall}; +use crate::sources::qoder::log_enrichment::{ + self, ContentSignal, LogEvent, CALL_ID_PAIR_MS, WINDOW_PAD_MS, +}; + +use super::{compact_tool_args, tail_preview, upsert_chunk, QODER_PRIMARY_SEQUENCE_STEP}; +use crate::sources::imported_history::replay::drivers::common::{ + content_digest, read_bounded_line, trim_jsonl_line, BoundedLine, MAX_JSONL_RECORD_BYTES, +}; +use crate::sources::imported_history::replay::payload_artifact; +use crate::sources::imported_history::replay::{ + replay_payload_body_projection, ImportedHistorySourceId, ReplayPayloadDescriptor, + ReplayPayloadEncoding, ReplayPayloadKind, ReplaySourceSpan, ReplayStats, + NORMAL_PAYLOAD_PREVIEW_BYTES, SHELL_PAYLOAD_PREVIEW_BYTES, +}; + +const SIDECAR_CURSOR_VERSION: u32 = 2; +const BOUNDARY_BYTES: u64 = 4 * 1024; +const SEQUENCE_EPOCH_MS: i64 = 1_577_836_800_000; // 2020-01-01 UTC +const SEQUENCE_TIE_SLOTS: i64 = 256; +const RAW_TABLE: &str = "imported_replay_qoder_log_events"; + +pub(in crate::sources::imported_history::replay) fn watch_paths( + source_session_id: &str, + transcript_path: &Path, +) -> Vec { + let task_dir_name = source_session_id + .split_once('/') + .map_or(source_session_id, |(_, task)| task); + let mut paths = log_enrichment::replay_sidecar_watch_paths(task_dir_name); + if let Some(project_dir) = transcript_path + .parent() + .and_then(Path::parent) + .and_then(Path::parent) + { + let spill_root = project_dir.join("agent-tools"); + if spill_root.is_dir() { + paths.push(spill_root); + } + } + paths.sort(); + paths.dedup(); + paths +} + +#[derive(Debug, Clone)] +pub(in crate::sources::imported_history::replay) struct SidecarProbe { + files: Vec, + pub(in crate::sources::imported_history::replay) signature: String, + edit_signature: String, +} + +#[derive(Debug, Clone)] +struct ProbedFile { + path: PathBuf, + identity: String, + size_bytes: u64, + mtime_ns: i64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub(in crate::sources::imported_history::replay) struct QoderSidecarCursor { + #[serde(default)] + version: u32, + #[serde(default)] + signature: String, + #[serde(default)] + edit_signature: String, + #[serde(default)] + files: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct SidecarFileCursor { + path: String, + identity: String, + byte_offset: u64, + boundary_fingerprint: String, +} + +#[derive(Debug)] +pub(in crate::sources::imported_history::replay) struct SidecarSyncOutcome { + pub cursor: QoderSidecarCursor, + pub changed: bool, +} + +#[derive(Debug)] +struct RawRow { + source_key: String, + source_path: PathBuf, + source_start: u64, + source_end: u64, +} + +#[derive(Debug)] +struct PendingTool { + ts_ms: i64, + source_key: String, + source_span: ReplaySourceSpan, + call_id: String, + name: String, + args: Value, + output: String, +} + +/// A bounded stat-only probe. It reads no log contents and is therefore safe +/// on the unchanged 60-second integrity path. +pub(in crate::sources::imported_history::replay) fn probe( + source_session_id: &str, +) -> Result { + let task_dir_name = source_session_id + .split_once('/') + .map_or(source_session_id, |(_, task)| task); + let mut files = Vec::new(); + for path in log_enrichment::qoder_launch_log_paths() { + let metadata = match fs::metadata(&path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => continue, + Err(error) => { + return Err(format!( + "stat Qoder replay sidecar {}: {error}", + path.display() + )) + } + }; + let canonical = fs::canonicalize(&path).unwrap_or(path); + files.push(ProbedFile { + identity: file_identity(&canonical, &metadata), + size_bytes: metadata.len(), + mtime_ns: metadata_mtime_ns(&metadata), + path: canonical, + }); + } + files.sort_by(|left, right| left.path.cmp(&right.path)); + files.dedup_by(|left, right| left.path == right.path); + let edit_signature = log_enrichment::edit_store_signature(task_dir_name, None); + let mut parts = Vec::with_capacity(files.len().saturating_mul(4).saturating_add(1)); + for file in &files { + parts.push(file.path.to_string_lossy().into_owned()); + parts.push(file.identity.clone()); + parts.push(file.size_bytes.to_string()); + parts.push(file.mtime_ns.to_string()); + } + parts.push(edit_signature.clone()); + let refs = parts.iter().map(String::as_bytes).collect::>(); + Ok(SidecarProbe { + signature: content_digest(&refs), + edit_signature, + files, + }) +} + +pub(in crate::sources::imported_history::replay) fn cursor_signature( + cursor_json: &str, +) -> Option { + #[derive(Deserialize)] + struct Wrapper { + #[serde(default)] + qoder_sidecar: QoderSidecarCursor, + } + + serde_json::from_str::(cursor_json) + .ok() + .map(|wrapper| wrapper.qoder_sidecar.signature) +} + +/// Replacement/truncation/removal invalidates source locators and requires a +/// generation reset. Ordinary append and newly-created launch logs do not. +pub(in crate::sources::imported_history::replay) fn cursor_lineage_matches( + cursor_json: &str, + probe: &SidecarProbe, +) -> bool { + #[derive(Deserialize)] + struct Wrapper { + #[serde(default)] + qoder_sidecar: QoderSidecarCursor, + } + + let Ok(wrapper) = serde_json::from_str::(cursor_json) else { + return false; + }; + if wrapper.qoder_sidecar.version != SIDECAR_CURSOR_VERSION { + return false; + } + wrapper.qoder_sidecar.files.iter().all(|cursor| { + let Some(current) = probe + .files + .iter() + .find(|file| file.path.to_string_lossy() == cursor.path) + else { + return false; + }; + current.identity == cursor.identity + && current.size_bytes >= cursor.byte_offset + && boundary_fingerprint(¤t.path, cursor.byte_offset) + .is_ok_and(|fingerprint| fingerprint == cursor.boundary_fingerprint) + }) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn sync( + tx: &Transaction<'_>, + display_session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + previous: &QoderSidecarCursor, + primary_changed: bool, + stats: &mut ReplayStats, +) -> Result { + ensure_raw_table(tx)?; + let probe = probe(source_session_id)?; + if !primary_changed + && previous.version == SIDECAR_CURSOR_VERSION + && previous.signature == probe.signature + && previous.edit_signature == probe.edit_signature + { + return Ok(SidecarSyncOutcome { + cursor: previous.clone(), + changed: false, + }); + } + + tx.execute( + &format!("DELETE FROM {RAW_TABLE} WHERE source_session_id=?1 AND generation<>?2"), + params![source_session_id, generation], + ) + .map_err(|error| format!("retire Qoder sidecar raw generation: {error}"))?; + + let previous_by_path = previous + .files + .iter() + .map(|cursor| (cursor.path.as_str(), cursor)) + .collect::>(); + let mut file_cursors = Vec::with_capacity(probe.files.len()); + let mut raw_changed = false; + for file in &probe.files { + let path_text = file.path.to_string_lossy(); + let byte_offset = previous_by_path + .get(path_text.as_ref()) + .filter(|cursor| cursor.identity == file.identity) + .map_or(0, |cursor| cursor.byte_offset); + let (new_offset, inserted) = + ingest_file(tx, source_session_id, generation, file, byte_offset, stats)?; + raw_changed |= inserted; + file_cursors.push(SidecarFileCursor { + path: path_text.into_owned(), + identity: file.identity.clone(), + byte_offset: new_offset, + boundary_fingerprint: boundary_fingerprint(&file.path, new_offset)?, + }); + } + + let rendered_changed = + if raw_changed || primary_changed || previous.edit_signature != probe.edit_signature { + fold_sidecar_events( + tx, + display_session_id, + source_session_id, + generation, + write_revision, + stats, + )? + } else { + false + }; + Ok(SidecarSyncOutcome { + cursor: QoderSidecarCursor { + version: SIDECAR_CURSOR_VERSION, + signature: probe.signature, + edit_signature: probe.edit_signature, + files: file_cursors, + }, + // Raw locator progress is persisted in the cursor, but only visible + // replay mutations advance the public revision. + changed: rendered_changed, + }) +} + +fn ensure_raw_table(tx: &Transaction<'_>) -> Result<(), String> { + tx.execute_batch(&format!( + "CREATE TABLE IF NOT EXISTS {RAW_TABLE} ( + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + source_key TEXT NOT NULL, + source_path TEXT NOT NULL, + source_start INTEGER NOT NULL, + source_end INTEGER NOT NULL, + ts_ms INTEGER NOT NULL, + kind TEXT NOT NULL, + task_id TEXT NOT NULL DEFAULT '', + call_id TEXT NOT NULL DEFAULT '', + PRIMARY KEY(source_session_id,generation,source_key) + ); + CREATE INDEX IF NOT EXISTS idx_qoder_replay_log_time + ON {RAW_TABLE}(source_session_id,generation,ts_ms,source_key); + CREATE INDEX IF NOT EXISTS idx_qoder_replay_log_task + ON {RAW_TABLE}(source_session_id,generation,task_id,kind,ts_ms);" + )) + .map_err(|error| format!("initialize Qoder sidecar compact index: {error}")) +} + +fn ingest_file( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + file: &ProbedFile, + start_offset: u64, + stats: &mut ReplayStats, +) -> Result<(u64, bool), String> { + let opened = fs::File::open(&file.path) + .map_err(|error| format!("open Qoder sidecar {}: {error}", file.path.display()))?; + let mut reader = BufReader::new(opened); + reader + .seek(SeekFrom::Start(start_offset)) + .map_err(|error| format!("seek Qoder sidecar {}: {error}", file.path.display()))?; + let mut acknowledged = start_offset; + let mut inserted_any = false; + loop { + let record_start = acknowledged; + let Some((line, line_bytes)) = read_complete_line(&mut reader)? else { + break; + }; + let line_text = String::from_utf8_lossy(trim_line(&line)); + let needs_following = line_text.contains(" ToolInvoke : "); + let mut record_bytes = line_bytes; + let following_storage = if needs_following { + let Some((next, next_bytes)) = read_complete_line(&mut reader)? else { + break; + }; + if trim_line(&next).starts_with(b"{") { + record_bytes = record_bytes + .checked_add(next_bytes) + .filter(|bytes| *bytes <= MAX_JSONL_RECORD_BYTES) + .ok_or_else(|| { + format!( + "Qoder sidecar record exceeds the {} MiB safety limit", + MAX_JSONL_RECORD_BYTES / (1024 * 1024) + ) + })?; + Some(next) + } else { + let rewind = i64::try_from(next_bytes) + .map_err(|_| "Qoder sidecar lookahead length overflow".to_string())?; + reader + .seek(SeekFrom::Current(-rewind)) + .map_err(|error| format!("rewind Qoder exthost lookahead: {error}"))?; + None + } + } else { + None + }; + let following = following_storage + .as_deref() + .map(trim_line) + .map(String::from_utf8_lossy); + acknowledged = acknowledged + .checked_add(record_bytes as u64) + .ok_or_else(|| "Qoder sidecar byte cursor overflow".to_string())?; + stats.parsed_bytes = stats.parsed_bytes.saturating_add(record_bytes as u64); + let (event, _consumed_following) = + log_enrichment::parse_launch_log_record(&line_text, following.as_deref()); + let Some(event) = event else { + continue; + }; + stats.parsed_rows = stats.parsed_rows.saturating_add(1); + let (ts_ms, kind, task_id, call_id) = event_index_fields(&event); + let source_key = content_digest(&[ + file.identity.as_bytes(), + &record_start.to_le_bytes(), + &acknowledged.to_le_bytes(), + ]); + let inserted = tx + .execute( + &format!( + "INSERT OR IGNORE INTO {RAW_TABLE}( + source_session_id,generation,source_key,source_path, + source_start,source_end,ts_ms,kind,task_id,call_id + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10)" + ), + params![ + source_session_id, + generation, + source_key, + file.path.to_string_lossy(), + record_start as i64, + acknowledged as i64, + ts_ms, + kind, + task_id, + call_id, + ], + ) + .map_err(|error| format!("index Qoder sidecar locator: {error}"))?; + inserted_any |= inserted > 0; + } + Ok((acknowledged, inserted_any)) +} + +fn read_complete_line( + reader: &mut impl std::io::BufRead, +) -> Result, usize)>, String> { + match read_bounded_line(reader, MAX_JSONL_RECORD_BYTES) + .map_err(|error| format!("read Qoder sidecar line: {error}"))? + { + BoundedLine::Eof | BoundedLine::Incomplete => Ok(None), + BoundedLine::TooLarge => Err(format!( + "Qoder sidecar line exceeds the {} MiB safety limit", + MAX_JSONL_RECORD_BYTES / (1024 * 1024) + )), + BoundedLine::Complete(bytes) => { + let length = bytes.len(); + Ok(Some((bytes, length))) + } + } +} + +fn trim_line(bytes: &[u8]) -> &[u8] { + trim_jsonl_line(bytes) +} + +fn event_index_fields(event: &LogEvent) -> (i64, &'static str, &str, &str) { + match event { + LogEvent::Acp { + ts_ms, + session_task_id, + tool_call_id, + } => ( + *ts_ms, + "acp", + session_task_id, + tool_call_id.as_deref().unwrap_or_default(), + ), + LogEvent::Subagent { + ts_ms, + session_task_id, + tool_call_id, + .. + } => (*ts_ms, "subagent", session_task_id, tool_call_id), + LogEvent::ToolInvoke { ts_ms, .. } => (*ts_ms, "invoke", "", ""), + LogEvent::FileEdit { + ts_ms, + session_dir_name, + .. + } => (*ts_ms, "file_edit", session_dir_name, ""), + } +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn fold_sidecar_events( + tx: &Transaction<'_>, + display_session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + stats: &mut ReplayStats, +) -> Result { + tx.execute_batch( + "CREATE TEMP TABLE IF NOT EXISTS imported_qoder_desired_events( + event_id TEXT PRIMARY KEY + ); + DELETE FROM imported_qoder_desired_events;", + ) + .map_err(|error| format!("prepare Qoder desired sidecar set: {error}"))?; + let (project_dir_name, task_dir_name) = source_session_id + .split_once('/') + .unwrap_or(("", source_session_id)); + let Some(task_id) = resolve_full_task_id(tx, source_session_id, generation, task_dir_name)? + else { + let changed = + remove_stale_events(tx, source_session_id, generation, write_revision, stats)?; + recompute_turn_counts(tx, source_session_id, generation)?; + return Ok(changed); + }; + let workspace_path = tx + .query_row( + "SELECT repo_path FROM imported_history_session_cache + WHERE source=?1 AND source_session_id=?2 LIMIT 1", + params![ImportedHistorySourceId::Qoder.as_str(), source_session_id], + |row| row.get::<_, Option>(0), + ) + .optional() + .map_err(|error| format!("load Qoder replay workspace: {error}"))? + .flatten(); + let Some((our_lo, our_hi)) = activity_window(tx, source_session_id, generation, &task_id)? + else { + let changed = + remove_stale_events(tx, source_session_id, generation, write_revision, stats)?; + recompute_turn_counts(tx, source_session_id, generation)?; + return Ok(changed); + }; + let (base_sequence, turn_index) = final_user_anchor(tx, source_session_id, generation)?; + let last_edit_keys = last_edit_keys(tx, source_session_id, generation, task_dir_name)?; + let snapshots = if last_edit_keys.is_empty() { + HashMap::new() + } else { + log_enrichment::edit_snapshots_for_task(&task_id) + }; + + let sql = format!( + "SELECT source_key,source_path,source_start,source_end + FROM {RAW_TABLE} + WHERE source_session_id=?1 AND generation=?2 + AND (kind='invoke' OR (kind='subagent' AND task_id=?3) + OR (kind='file_edit' AND task_id=?4)) + ORDER BY ts_ms,source_key" + ); + let mut statement = tx + .prepare(&sql) + .map_err(|error| format!("prepare Qoder sidecar fold: {error}"))?; + let mut rows = statement + .query(params![ + source_session_id, + generation, + task_id, + task_dir_name + ]) + .map_err(|error| format!("query Qoder sidecar fold: {error}"))?; + let mut previous_signature: Option<(i64, String)> = None; + let mut changed = false; + while let Some(row) = rows + .next() + .map_err(|error| format!("read Qoder sidecar row: {error}"))? + { + let raw = RawRow { + source_key: row.get(0).map_err(|error| error.to_string())?, + source_path: PathBuf::from(row.get::<_, String>(1).map_err(|error| error.to_string())?), + source_start: row + .get::<_, i64>(2) + .map_err(|error| error.to_string())? + .max(0) as u64, + source_end: row + .get::<_, i64>(3) + .map_err(|error| error.to_string())? + .max(0) as u64, + }; + let Some(event) = read_raw_event(&raw)? else { + continue; + }; + let pending = match event { + LogEvent::Subagent { + ts_ms, + tool_call_id, + agent_type, + description, + prompt, + .. + } => PendingTool { + ts_ms, + source_key: raw.source_key.clone(), + source_span: ReplaySourceSpan { + start: raw.source_start, + end: raw.source_end, + }, + call_id: tool_call_id, + name: "subagent".to_string(), + args: json!({ + "agentType": agent_type, + "description": description, + "prompt": prompt, + }), + output: String::new(), + }, + LogEvent::ToolInvoke { ts_ms, name, args } => { + let owned = match log_enrichment::invoke_content_signal( + &args, + project_dir_name, + workspace_path.as_deref(), + ) { + ContentSignal::Ours => true, + ContentSignal::Theirs => false, + ContentSignal::Silent => { + ts_ms >= our_lo.saturating_sub(WINDOW_PAD_MS) + && ts_ms <= our_hi.saturating_add(WINDOW_PAD_MS) + && !overlaps_other_window( + tx, + source_session_id, + generation, + &task_id, + ts_ms, + )? + } + }; + if !owned { + continue; + } + PendingTool { + ts_ms, + source_key: raw.source_key.clone(), + source_span: ReplaySourceSpan { + start: raw.source_start, + end: raw.source_end, + }, + call_id: paired_call_id(tx, source_session_id, generation, &task_id, ts_ms)? + .unwrap_or_else(|| format!("invoke-{ts_ms}")), + name, + output: log_enrichment::spill_file_output(&args), + args, + } + } + LogEvent::FileEdit { + ts_ms, + path, + operation, + .. + } => { + let mut args = json!({ "file_path": path, "operation": operation }); + if last_edit_keys.get(&path) == Some(&raw.source_key) { + if let Some((old_content, new_content)) = snapshots.get(&path) { + if let Some(map) = args.as_object_mut() { + map.insert("old_string".to_string(), json!(old_content)); + map.insert("new_string".to_string(), json!(new_content)); + } + } + } + PendingTool { + ts_ms, + source_key: raw.source_key.clone(), + source_span: ReplaySourceSpan { + start: raw.source_start, + end: raw.source_end, + }, + call_id: paired_call_id(tx, source_session_id, generation, &task_id, ts_ms)? + .unwrap_or_else(|| format!("edit-{ts_ms}")), + name: format!("file_{operation}"), + args, + output: String::new(), + } + } + LogEvent::Acp { .. } => continue, + }; + let signature = serde_json::to_string(&(&pending.name, &pending.args)) + .unwrap_or_else(|_| pending.name.clone()); + if previous_signature + .as_ref() + .is_some_and(|(previous_ts, previous)| { + previous == &signature && (pending.ts_ms - *previous_ts).abs() <= 2_000 + }) + { + continue; + } + previous_signature = Some((pending.ts_ms, signature)); + changed |= upsert_pending( + tx, + display_session_id, + source_session_id, + generation, + write_revision, + base_sequence, + turn_index, + pending, + stats, + )?; + } + drop(rows); + drop(statement); + changed |= remove_stale_events(tx, source_session_id, generation, write_revision, stats)?; + recompute_turn_counts(tx, source_session_id, generation)?; + Ok(changed) +} + +fn resolve_full_task_id( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + task_dir_name: &str, +) -> Result, String> { + let sql = format!( + "SELECT DISTINCT task_id FROM {RAW_TABLE} + WHERE source_session_id=?1 AND generation=?2 + AND kind IN ('acp','subagent') AND task_id<>''" + ); + let mut statement = tx + .prepare(&sql) + .map_err(|error| format!("prepare Qoder task attribution: {error}"))?; + let mut rows = statement + .query(params![source_session_id, generation]) + .map_err(|error| format!("query Qoder task attribution: {error}"))?; + let mut matched = None; + while let Some(row) = rows + .next() + .map_err(|error| format!("read Qoder task attribution: {error}"))? + { + let candidate: String = row.get(0).map_err(|error| error.to_string())?; + if !candidate.starts_with(task_dir_name) { + continue; + } + match matched.as_deref() { + None => matched = Some(candidate), + Some(existing) if existing == candidate => {} + Some(_) => return Ok(None), + } + } + Ok(matched) +} + +fn activity_window( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + task_id: &str, +) -> Result, String> { + let sql = format!( + "SELECT MIN(ts_ms),MAX(ts_ms) FROM {RAW_TABLE} + WHERE source_session_id=?1 AND generation=?2 AND kind='acp' AND task_id=?3" + ); + tx.query_row( + &sql, + params![source_session_id, generation, task_id], + |row| Ok((row.get::<_, Option>(0)?, row.get::<_, Option>(1)?)), + ) + .map_err(|error| format!("read Qoder activity window: {error}")) + .map(|(lo, hi)| lo.zip(hi)) +} + +fn overlaps_other_window( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + task_id: &str, + ts_ms: i64, +) -> Result { + let sql = format!( + "SELECT EXISTS( + SELECT 1 FROM ( + SELECT task_id,MIN(ts_ms) AS lo,MAX(ts_ms) AS hi + FROM {RAW_TABLE} + WHERE source_session_id=?1 AND generation=?2 AND kind='acp' + AND task_id<>?3 + GROUP BY task_id + ) WHERE ?4 BETWEEN lo-?5 AND hi+?5 + )" + ); + tx.query_row( + &sql, + params![source_session_id, generation, task_id, ts_ms, WINDOW_PAD_MS], + |row| row.get::<_, bool>(0), + ) + .map_err(|error| format!("check Qoder overlapping activity: {error}")) +} + +fn paired_call_id( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + task_id: &str, + ts_ms: i64, +) -> Result, String> { + let sql = format!( + "SELECT call_id FROM {RAW_TABLE} + WHERE source_session_id=?1 AND generation=?2 AND kind='acp' + AND task_id=?3 AND call_id<>'' AND ts_ms<=?4 AND ts_ms>=?5 + ORDER BY ts_ms DESC,source_key DESC LIMIT 1" + ); + tx.query_row( + &sql, + params![ + source_session_id, + generation, + task_id, + ts_ms, + ts_ms.saturating_sub(CALL_ID_PAIR_MS) + ], + |row| row.get(0), + ) + .optional() + .map_err(|error| format!("pair Qoder tool call id: {error}")) +} + +fn final_user_anchor( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, +) -> Result<(i64, i64), String> { + tx.query_row( + "SELECT sequence,turn_index FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND function_name=?4 + ORDER BY sequence DESC LIMIT 1", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation, + imported_history::FUNCTION_USER_MESSAGE + ], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .optional() + .map_err(|error| format!("read Qoder final user anchor: {error}")) + .map(|anchor| anchor.unwrap_or((0, 0))) +} + +fn last_edit_keys( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + task_dir_name: &str, +) -> Result, String> { + let sql = format!( + "SELECT source_key,source_path,source_start,source_end FROM {RAW_TABLE} + WHERE source_session_id=?1 AND generation=?2 + AND kind='file_edit' AND task_id=?3 + ORDER BY ts_ms,source_key" + ); + let mut statement = tx + .prepare(&sql) + .map_err(|error| format!("prepare Qoder last edits: {error}"))?; + let mut rows = statement + .query(params![source_session_id, generation, task_dir_name]) + .map_err(|error| format!("query Qoder last edits: {error}"))?; + let mut last = HashMap::new(); + while let Some(row) = rows + .next() + .map_err(|error| format!("read Qoder last edit: {error}"))? + { + let raw = RawRow { + source_key: row.get(0).map_err(|error| error.to_string())?, + source_path: PathBuf::from(row.get::<_, String>(1).map_err(|error| error.to_string())?), + source_start: row + .get::<_, i64>(2) + .map_err(|error| error.to_string())? + .max(0) as u64, + source_end: row + .get::<_, i64>(3) + .map_err(|error| error.to_string())? + .max(0) as u64, + }; + if let Some(LogEvent::FileEdit { path, .. }) = read_raw_event(&raw)? { + last.insert(path, raw.source_key); + } + } + Ok(last) +} + +fn read_raw_event(raw: &RawRow) -> Result, String> { + let length = raw.source_end.saturating_sub(raw.source_start) as usize; + if length == 0 || length > MAX_JSONL_RECORD_BYTES { + return Ok(None); + } + let mut file = fs::File::open(&raw.source_path).map_err(|error| { + format!( + "open Qoder indexed log {}: {error}", + raw.source_path.display() + ) + })?; + file.seek(SeekFrom::Start(raw.source_start)) + .map_err(|error| format!("seek Qoder indexed log: {error}"))?; + let mut bytes = vec![0; length]; + file.read_exact(&mut bytes) + .map_err(|error| format!("read Qoder indexed log record: {error}"))?; + let mut lines = bytes.split_inclusive(|byte| *byte == b'\n'); + let first = lines.next().unwrap_or_default(); + let second = lines.next(); + let first = String::from_utf8_lossy(trim_line(first)); + let second = second.map(|line| String::from_utf8_lossy(trim_line(line))); + Ok(log_enrichment::parse_launch_log_record(&first, second.as_deref()).0) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn upsert_pending( + tx: &Transaction<'_>, + display_session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + base_sequence: i64, + turn_index: i64, + pending: PendingTool, + stats: &mut ReplayStats, +) -> Result { + let normalized = log_enrichment::normalized_args(&pending.name, &pending.args); + let call = ImportedToolCall { + call_id: pending.call_id.clone(), + raw_name: pending.name.clone(), + canonical_name: log_enrichment::canonical_tool_name(&pending.name), + args: normalized.clone(), + created_at: imported_history::epoch_ms_to_iso(pending.ts_ms), + }; + let semantic_hash = content_digest(&[ + pending.call_id.as_bytes(), + pending.name.as_bytes(), + serde_json::to_string(&normalized) + .unwrap_or_default() + .as_bytes(), + ]); + let event_id = format!("qoder-log-{semantic_hash}"); + tx.execute( + "INSERT OR IGNORE INTO imported_qoder_desired_events(event_id) VALUES(?1)", + params![event_id], + ) + .map_err(|error| format!("mark desired Qoder replay event: {error}"))?; + + let mut chunk = imported_history::tool_call_chunk( + display_session_id, + "qoder-log", + 0, + &call, + &pending.output, + ); + chunk.chunk_id = event_id.clone(); + if let Some(result) = chunk.result.as_object_mut() { + result.insert("recovered_from".to_string(), json!("agent_log")); + } + if call.canonical_name == imported_history::FUNCTION_RUN_COMMAND_LINE { + let args_json = serde_json::to_string(&chunk.args).unwrap_or_else(|_| "null".to_string()); + let result_json = + serde_json::to_string(&chunk.result).unwrap_or_else(|_| "null".to_string()); + let git_artifacts = parse_git_artifacts_from_tool_payload(&args_json, &result_json); + attach_replay_git_artifacts(&mut chunk.result, &git_artifacts); + } + let args_text = serde_json::to_string(&normalized).unwrap_or_else(|_| "{}".to_string()); + let args_limit = if call.canonical_name == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let mut payloads = Vec::new(); + if args_text.len() > args_limit { + let body_projection = replay_payload_body_projection( + "args", + &normalized, + Some(&args_text), + args_limit, + false, + ); + chunk.args = compact_tool_args(&normalized, &call.canonical_name); + store_artifact( + tx, + source_session_id, + generation, + &event_id, + "args", + args_text.as_bytes(), + )?; + payloads.push(ReplayPayloadDescriptor { + field_path: "args".to_string(), + kind: ReplayPayloadKind::ToolArguments, + encoding: ReplayPayloadEncoding::JsonValue, + body_projection, + spans: Vec::new(), + total_bytes: args_text.len() as u64, + source_ordinal: None, + source_key: Some(pending.source_key.clone()), + }); + } else { + delete_artifact_ref(tx, source_session_id, generation, &event_id, "args")?; + } + let output_limit = if call.canonical_name == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + if pending.output.len() > output_limit { + let (preview, _) = tail_preview(&pending.output, output_limit); + if let Some(result) = chunk.result.as_object_mut() { + result.insert("output".to_string(), json!(preview)); + result.insert("observation".to_string(), json!(preview)); + } + store_artifact( + tx, + source_session_id, + generation, + &event_id, + "result.output", + pending.output.as_bytes(), + )?; + payloads.push(ReplayPayloadDescriptor { + field_path: "result.output".to_string(), + kind: ReplayPayloadKind::ToolOutput, + encoding: ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: Vec::new(), + total_bytes: pending.output.len() as u64, + source_ordinal: None, + source_key: Some(pending.source_key.clone()), + }); + } else { + delete_artifact_ref( + tx, + source_session_id, + generation, + &event_id, + "result.output", + )?; + } + let sequence = sidecar_sequence( + tx, + source_session_id, + generation, + base_sequence, + pending.ts_ms, + &event_id, + )?; + upsert_chunk( + tx, + ImportedHistorySourceId::Qoder, + source_session_id, + generation, + write_revision, + turn_index, + sequence, + &chunk, + &payloads, + pending.source_span, + stats, + ) +} + +fn sidecar_sequence( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + base_sequence: i64, + ts_ms: i64, + event_id: &str, +) -> Result { + let elapsed = ts_ms + .checked_sub(SEQUENCE_EPOCH_MS) + .ok_or_else(|| "Qoder sidecar timestamp arithmetic overflow".to_string())? + .max(0); + let hash = content_digest(&[event_id.as_bytes()]); + let tie = u8::from_str_radix(hash.get(..2).unwrap_or("00"), 16).unwrap_or_default() as i64; + let mut offset = elapsed + .checked_mul(SEQUENCE_TIE_SLOTS) + .and_then(|offset| offset.checked_add(tie)) + .and_then(|offset| offset.checked_add(1)) + .ok_or_else(|| "Qoder sidecar sequence offset overflow".to_string())?; + if offset >= QODER_PRIMARY_SEQUENCE_STEP { + return Err("Qoder sidecar timestamp exceeds reserved replay sequence range".to_string()); + } + loop { + let sequence = base_sequence + .checked_add(offset) + .ok_or_else(|| "Qoder sidecar sequence space exhausted".to_string())?; + let occupied = tx + .query_row( + "SELECT event_id FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND sequence=?4", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation, + sequence + ], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|error| format!("allocate Qoder sidecar sequence: {error}"))?; + if occupied + .as_deref() + .is_none_or(|occupied| occupied == event_id) + { + return Ok(sequence); + } + offset = offset + .checked_add(1) + .ok_or_else(|| "Qoder sidecar tie-break sequence overflow".to_string())?; + if offset % SEQUENCE_TIE_SLOTS == 0 || offset >= QODER_PRIMARY_SEQUENCE_STEP { + return Err("Too many Qoder sidecar events share one timestamp".to_string()); + } + } +} + +fn store_artifact( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + payload: &[u8], +) -> Result<(), String> { + payload_artifact::store_bytes( + tx, + ImportedHistorySourceId::Qoder, + source_session_id, + generation, + event_id, + field_path, + payload, + ) + .map_err(|error| format!("store Qoder replay payload artifact: {error}"))?; + Ok(()) +} + +fn delete_artifact_ref( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, +) -> Result<(), String> { + tx.execute( + "DELETE FROM imported_replay_payload_artifact_refs + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND event_id=?4 AND field_path=?5", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation, + event_id, + field_path + ], + ) + .map_err(|error| format!("delete stale Qoder payload reference: {error}"))?; + Ok(()) +} + +fn remove_stale_events( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, + write_revision: u64, + stats: &mut ReplayStats, +) -> Result { + // Upserts are published immediately after the driver returns. Reserve the + // following revisions for tombstones now, one row at a time, so an + // attribution back-off never builds a session-sized removal vector. + let base_revision = write_revision.saturating_sub(1); + let pending_upserts = tx + .query_row( + "SELECT COUNT(*) FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND event_revision=?4", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation, + write_revision.min(i64::MAX as u64) as i64 + ], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("count pending Qoder replay upserts: {error}"))? + .max(0) as u64; + let mut next_revision = base_revision.saturating_add(pending_upserts); + let mut removed = 0_u64; + loop { + let event_id = tx + .query_row( + "SELECT event_id FROM imported_replay_events AS event + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND event_id LIKE 'qoder-log-%' + AND NOT EXISTS( + SELECT 1 FROM imported_qoder_desired_events AS desired + WHERE desired.event_id=event.event_id + ) + ORDER BY sequence,event_id LIMIT 1", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation + ], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|error| format!("read stale Qoder replay event: {error}"))?; + let Some(event_id) = event_id else { + break; + }; + next_revision = next_revision.saturating_add(1); + tx.execute( + "INSERT INTO imported_replay_changes( + source,source_session_id,generation,change_revision, + event_id,change_kind,sequence + ) VALUES(?1,?2,?3,?4,?5,'remove',NULL)", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation, + next_revision.min(i64::MAX as u64) as i64, + event_id + ], + ) + .map_err(|error| format!("stage Qoder replay tombstone: {error}"))?; + tx.execute( + "DELETE FROM imported_replay_payload_artifact_refs + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation, + event_id + ], + ) + .map_err(|error| format!("delete stale Qoder payload refs: {error}"))?; + tx.execute( + "DELETE FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation, + event_id + ], + ) + .map_err(|error| format!("delete stale Qoder replay event: {error}"))?; + removed = removed.saturating_add(1); + } + tx.execute( + "DELETE FROM imported_replay_payload_artifacts AS artifact + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND NOT EXISTS( + SELECT 1 FROM imported_replay_payload_artifact_refs AS ref + WHERE ref.source=artifact.source + AND ref.source_session_id=artifact.source_session_id + AND ref.generation=artifact.generation + AND ref.content_hash=artifact.content_hash + ) + AND NOT EXISTS( + SELECT 1 FROM imported_replay_shell_segments AS shell + WHERE shell.source=artifact.source + AND shell.source_session_id=artifact.source_session_id + AND shell.generation=artifact.generation + AND shell.content_hash=artifact.content_hash + )", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation + ], + ) + .map_err(|error| format!("delete orphan Qoder payload artifacts: {error}"))?; + stats.removed_events = stats.removed_events.saturating_add(removed); + Ok(removed > 0) +} + +fn recompute_turn_counts( + tx: &Transaction<'_>, + source_session_id: &str, + generation: &str, +) -> Result<(), String> { + tx.execute( + "UPDATE imported_replay_turns AS turn SET event_count=( + SELECT COUNT(*) FROM imported_replay_events AS event + WHERE event.source=turn.source + AND event.source_session_id=turn.source_session_id + AND event.generation=turn.generation + AND event.turn_index=turn.turn_index + ) WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + generation + ], + ) + .map_err(|error| format!("recount Qoder replay turns: {error}"))?; + Ok(()) +} + +fn boundary_fingerprint(path: &Path, offset: u64) -> Result { + if offset == 0 { + return Ok(content_digest(&[b"empty"])); + } + let start = offset.saturating_sub(BOUNDARY_BYTES); + let mut file = fs::File::open(path) + .map_err(|error| format!("open Qoder sidecar boundary {}: {error}", path.display()))?; + file.seek(SeekFrom::Start(start)) + .map_err(|error| format!("seek Qoder sidecar boundary: {error}"))?; + let mut bytes = vec![0; offset.saturating_sub(start) as usize]; + file.read_exact(&mut bytes) + .map_err(|error| format!("read Qoder sidecar boundary: {error}"))?; + Ok(content_digest(&[&start.to_le_bytes(), &bytes])) +} + +fn file_identity(path: &Path, metadata: &fs::Metadata) -> String { + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + format!("{}:{}:{}", path.display(), metadata.dev(), metadata.ino()) + } + #[cfg(not(unix))] + { + path.to_string_lossy().into_owned() + } +} + +fn metadata_mtime_ns(metadata: &fs::Metadata) -> i64 { + metadata + .modified() + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map(|duration| duration.as_secs() as i64 * 1_000_000_000 + duration.subsec_nanos() as i64) + .unwrap_or_default() +} + +#[cfg(test)] +#[path = "../../tests/qoder_sidecar.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/mod.rs new file mode 100644 index 000000000..acaebceb7 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/mod.rs @@ -0,0 +1,5 @@ +pub(super) mod common; +pub(super) mod jsonl; +pub(super) mod sqlite; +pub(super) mod structured; +pub(super) mod whole_json; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/common.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/common.rs new file mode 100644 index 000000000..6e81bd606 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/common.rs @@ -0,0 +1,36 @@ +use super::*; +use crate::sources::imported_history::replay::drivers::common::{ + legacy_stable_id_hash_concat, utf8_boundary_at_or_before, +}; + +pub(super) fn field_path_is_under(field_path: &str, root: &str) -> bool { + field_path == root + || field_path + .strip_prefix(root) + .is_some_and(|suffix| suffix.starts_with('.')) +} + +pub(super) fn value_at_path_mut<'a>(value: &'a mut Value, path: &str) -> Option<&'a mut String> { + let mut current = value; + for segment in path.split('.') { + current = current.as_object_mut()?.get_mut(segment)?; + } + current.as_str()?; + match current { + Value::String(text) => Some(text), + _ => None, + } +} + +pub(super) fn head_preview(text: &str, max_bytes: usize) -> String { + let end = utf8_boundary_at_or_before(text, max_bytes); + format!("{}\n… [payload truncated]", &text[..end]) +} + +pub(super) fn stable_event_id(source: ImportedHistorySourceId, source_key: &str) -> String { + format!( + "{}-sqlite-{}", + source.as_str(), + legacy_stable_id_hash_concat(&[source_key.as_bytes()]) + ) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/kv_store.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/kv_store.rs new file mode 100644 index 000000000..e4bd0000e --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/kv_store.rs @@ -0,0 +1,574 @@ +use super::*; +use std::collections::HashSet; + +const KV_TURN_PREVIEW_MAX_CHARS: usize = 160; +// Read a little beyond the rendered preview so whitespace normalization can +// still find useful words without ever copying a provider-sized user bubble. +const KV_TURN_PREVIEW_SCAN_CHARS: usize = KV_TURN_PREVIEW_MAX_CHARS * 16; + +/// Read only the user-bubble fields needed by the virtual turn catalog. +/// +/// `imported_replay_turns` already retains the provider user-bubble id for +/// each Cursor/Windsurf turn. Exact primary-key lookups avoid scanning the +/// composer or hydrating assistant/tool bubbles, and SQL `substr` bounds the +/// text copied from the provider database before it reaches Rust. +pub(in crate::sources::imported_history::replay) fn read_kv_turn_previews( + source_path: &Path, + source: ImportedHistorySourceId, + source_session_id: &str, + requested_turns: &[(i64, String)], +) -> Result, String> { + if !matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + return Err(format!( + "{} is not a SQLite/KV replay source", + source.as_str() + )); + } + if requested_turns.is_empty() { + return Ok(Vec::new()); + } + + let source_conn = open_source_db(source_path)?; + validate_schema(&source_conn, source)?; + let mut statement = source_conn + .prepare( + "SELECT + CASE WHEN json_valid(value) + THEN COALESCE(CAST(json_extract(value, '$.createdAt') AS TEXT), '') + ELSE '' END, + CASE WHEN json_valid(value) + THEN COALESCE( + substr( + trim(CAST(json_extract(value, '$.text') AS TEXT)), + 1, + ?2 + ), + '' + ) + ELSE '' END + FROM cursorDiskKV + WHERE key=?1", + ) + .map_err(|err| format!("prepare compact {} turn previews: {err}", source.as_str()))?; + let mut previews = Vec::with_capacity(requested_turns.len()); + for (turn_index, user_bubble_id) in requested_turns { + if !is_valid_kv_bubble_id(user_bubble_id) { + continue; + } + let key = format!("bubbleId:{source_session_id}:{user_bubble_id}"); + let selected = statement + .query_row(params![key, KV_TURN_PREVIEW_SCAN_CHARS as i64], |row| { + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + }) + .optional() + .map_err(|err| { + format!( + "read compact {} turn preview {turn_index}: {err}", + source.as_str() + ) + })?; + let Some((created_at, text)) = selected else { + continue; + }; + let user_preview = text + .split_whitespace() + .collect::>() + .join(" ") + .chars() + .take(KV_TURN_PREVIEW_MAX_CHARS) + .collect(); + previews.push(KvTurnPreview { + turn_index: *turn_index, + created_at, + user_preview, + }); + } + Ok(previews) +} + +#[allow( + clippy::too_many_arguments, + reason = "KV hydration boundary keeps the pinned generation and turn range explicit" +)] +pub(in crate::sources::imported_history::replay) fn hydrate_kv_turn( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + start_sequence: i64, + end_sequence: i64, +) -> Result { + if !matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + return Err(format!( + "{} is not a SQLite/KV replay source", + source.as_str() + )); + } + ensure_source_row_table(tx)?; + tx.execute_batch( + "CREATE TEMP TABLE IF NOT EXISTS imported_replay_seen_rows( + source_key TEXT PRIMARY KEY + ) WITHOUT ROWID; + DELETE FROM imported_replay_seen_rows;", + ) + .map_err(|err| format!("prepare lazy KV replay row set: {err}"))?; + let source_conn = open_source_db(source_path)?; + validate_schema(&source_conn, source)?; + let composer_json = load_composer_json(&source_conn, source_session_id)?; + let mut stats = ReplayStats::default(); + let mut changed = false; + let mut removed = Vec::new(); + stream_kv_bubbles( + &source_conn, + source_session_id, + &composer_json, + start_sequence.max(0) as u64, + Some(end_sequence.max(start_sequence).max(0) as u64), + |row| { + fold_source_row( + tx, + source, + display_session_id, + source_session_id, + generation, + write_revision, + row, + &mut stats, + &mut changed, + &mut removed, + &source_conn, + Some(&composer_json), + ) + }, + )?; + if changed { + delete_stale_payload_artifacts(tx, source, source_session_id, generation)?; + } + Ok(stats) +} + +#[derive(Debug, Default, Deserialize)] +#[serde(default, rename_all = "camelCase")] +pub(super) struct KvComposerOrder { + full_conversation_headers_only: Vec, +} + +#[derive(Debug, Default, Deserialize)] +#[serde(default, rename_all = "camelCase")] +pub(super) struct KvHeader { + bubble_id: String, + #[serde(rename = "type")] + bubble_type: i64, +} + +#[derive(Debug)] +struct KvBubbleOrderEntry { + key: String, + bubble_id: String, + bubble_type: i64, + ordinal: i64, + turn_index: i64, +} + +pub(super) fn kv_source_summary( + conn: &Connection, + composer_id: &str, + composer_json: &str, +) -> Result { + let mut order_hash = ContentDigest::default(); + let mut last_source_key = String::new(); + let mut row_count = 0_u64; + stream_kv_bubble_order(conn, composer_id, composer_json, |entry| { + order_hash.update_part(entry.bubble_id.as_bytes()); + order_hash.update_part(&entry.bubble_type.to_le_bytes()); + last_source_key = entry.key; + row_count = row_count.saturating_add(1); + Ok(true) + })?; + Ok(SqliteSourceSummary { + row_count, + max_time_created: 0, + max_source_key: String::new(), + source_signal: content_digest(&[composer_json.as_bytes()]), + last_source_key, + order_signal: order_hash.finish_hex(), + }) +} + +pub(super) fn kv_sync_plan( + previous: &KvStoreReplayCursor, + current: &SqliteSourceSummary, +) -> SyncPlan { + if previous.source_signal.is_empty() { + return SyncPlan::Reconcile; + } + if current.row_count == previous.total_source_rows + && current.source_signal == previous.source_signal + { + return SyncPlan::Skip; + } + if current.row_count > previous.total_source_rows + && (previous.total_source_rows == 0 + || (!previous.last_source_key.is_empty() + && previous.last_source_key != current.last_source_key)) + { + SyncPlan::Append + } else { + SyncPlan::Reconcile + } +} + +pub(super) fn kv_recent_turn_start( + conn: &Connection, + composer_id: &str, + composer_json: &str, +) -> Result { + let mut latest_start = 0_u64; + let mut current_turn = None; + stream_kv_bubble_order(conn, composer_id, composer_json, |entry| { + if current_turn != Some(entry.turn_index) { + latest_start = entry.ordinal.max(0) as u64; + current_turn = Some(entry.turn_index); + } + Ok(true) + })?; + Ok(latest_start) +} + +pub(super) fn replace_kv_turn_headers( + tx: &Transaction<'_>, + source_conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + composer_json: &str, +) -> Result<(), String> { + tx.execute( + "DELETE FROM imported_replay_turns + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + ) + .map_err(|err| format!("clear {} compact turn headers: {err}", source.as_str()))?; + let mut pending = None; + stream_kv_bubble_order(source_conn, source_session_id, composer_json, |entry| { + if pending + .as_ref() + .is_some_and(|turn: &PendingKvTurn| turn.turn_index != entry.turn_index) + { + if let Some(turn) = pending.take() { + publish_pending_kv_turn(tx, source, source_session_id, generation, turn)?; + } + } + let turn = pending.get_or_insert_with(|| PendingKvTurn { + turn_index: entry.turn_index, + turn_id: if entry.bubble_type == 1 { + entry.bubble_id.clone() + } else { + format!("{}-turn-{}", source.as_str(), entry.turn_index) + }, + start_sequence: entry.ordinal, + end_sequence: entry.ordinal, + event_count: 0, + }); + turn.end_sequence = entry.ordinal; + turn.event_count = turn.event_count.saturating_add(1); + Ok(true) + })?; + if let Some(turn) = pending { + publish_pending_kv_turn(tx, source, source_session_id, generation, turn)?; + } + Ok(()) +} + +#[derive(Debug)] +struct PendingKvTurn { + turn_index: i64, + turn_id: String, + start_sequence: i64, + end_sequence: i64, + event_count: u64, +} + +fn publish_pending_kv_turn( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + turn: PendingKvTurn, +) -> Result<(), String> { + insert_turn_header( + tx, + source, + source_session_id, + generation, + turn.turn_index, + ( + turn.turn_id, + turn.start_sequence, + turn.end_sequence, + "1970-01-01T00:00:00Z".to_string(), + "1970-01-01T00:00:00Z".to_string(), + turn.event_count, + ), + ) +} + +pub(super) fn load_composer_json(conn: &Connection, composer_id: &str) -> Result { + let key = format!("composerData:{composer_id}"); + conn.query_row( + "SELECT value FROM cursorDiskKV WHERE key=?1", + [key], + |row| row.get::<_, Option>(0), + ) + .optional() + .map_err(|err| format!("read replay composer {composer_id}: {err}"))? + .flatten() + .ok_or_else(|| format!("Replay composer {composer_id} is missing")) +} + +pub(super) fn stream_kv_bubbles( + conn: &Connection, + composer_id: &str, + composer_json: &str, + start_ordinal: u64, + end_ordinal: Option, + mut visit: impl FnMut(SourceRow) -> Result<(), String>, +) -> Result<(), String> { + stream_kv_bubble_order(conn, composer_id, composer_json, |entry| { + if (entry.ordinal as u64) < start_ordinal { + return Ok(true); + } + if end_ordinal.is_some_and(|end| entry.ordinal as u64 > end) { + return Ok(false); + } + let raw_json = conn + .query_row( + "SELECT value FROM cursorDiskKV WHERE key=?1", + [&entry.key], + |row| row.get::<_, Option>(0), + ) + .map_err(|err| format!("read replay bubble {}: {err}", entry.bubble_id))? + .unwrap_or_default(); + visit(SourceRow { + key: entry.key, + message_id: String::new(), + role: String::new(), + raw_json, + time_created: 0, + header_type: entry.bubble_type, + ordinal: entry.ordinal, + turn_index: entry.turn_index, + })?; + Ok(true) + })?; + Ok(()) +} + +fn stream_kv_bubble_order( + conn: &Connection, + composer_id: &str, + composer_json: &str, + mut visit: impl FnMut(KvBubbleOrderEntry) -> Result, +) -> Result<(), String> { + let composer = serde_json::from_str::(composer_json) + .map_err(|err| format!("parse replay composer {composer_id}: {err}"))?; + let mut seen = HashSet::new(); + let mut ordinal = 0_i64; + let mut turn_index = -1_i64; + for header in composer.full_conversation_headers_only { + if !is_valid_kv_bubble_id(&header.bubble_id) || !seen.insert(header.bubble_id.clone()) { + continue; + } + let key = format!("bubbleId:{composer_id}:{}", header.bubble_id); + let exists = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM cursorDiskKV WHERE key=?1)", + [&key], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("check replay bubble {}: {err}", header.bubble_id))? + != 0; + if !exists { + continue; + } + if header.bubble_type == 1 || turn_index < 0 { + turn_index += 1; + } + let entry = KvBubbleOrderEntry { + key, + bubble_id: header.bubble_id, + bubble_type: header.bubble_type, + ordinal, + turn_index: turn_index.max(0), + }; + ordinal += 1; + if !visit(entry)? { + return Ok(()); + } + } + + // Cursor can persist bubbles before updating composerData. Preserve the + // existing reader's fallback by streaming those rows after header entries. + let prefix = format!("bubbleId:{composer_id}:"); + let upper = format!("bubbleId:{composer_id};"); + let mut stmt = conn + .prepare( + "SELECT key, COALESCE(json_extract(value, '$.type'), 0) + FROM cursorDiskKV WHERE key>=?1 AND key>(1) + .map_err(|err| err.to_string())? + .unwrap_or_default(); + if bubble_type == 1 || turn_index < 0 { + turn_index += 1; + } + let entry = KvBubbleOrderEntry { + key, + bubble_id, + bubble_type, + ordinal, + turn_index: turn_index.max(0), + }; + ordinal += 1; + if !visit(entry)? { + return Ok(()); + } + } + Ok(()) +} + +fn is_valid_kv_bubble_id(bubble_id: &str) -> bool { + !bubble_id.trim().is_empty() && bubble_id != "undefined" +} + +pub(super) fn stage_and_clear_kv_order( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result<(), String> { + tx.execute_batch( + "CREATE TEMP TABLE IF NOT EXISTS imported_replay_previous_kv_events( + event_id TEXT PRIMARY KEY + ) WITHOUT ROWID; + DELETE FROM imported_replay_previous_kv_events;", + ) + .map_err(|err| format!("prepare reordered KV replay staging: {err}"))?; + tx.execute( + "INSERT OR IGNORE INTO imported_replay_previous_kv_events(event_id) + SELECT event_id FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + ) + .map_err(|err| format!("stage reordered KV replay ids: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + ) + .map_err(|err| format!("clear reordered KV replay events: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_source_rows + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + ) + .map_err(|err| format!("clear reordered KV replay row hashes: {err}"))?; + Ok(()) +} + +pub(super) fn kv_order_removals( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result, String> { + let mut stmt = tx + .prepare( + "SELECT previous.event_id + FROM imported_replay_previous_kv_events AS previous + WHERE NOT EXISTS ( + SELECT 1 FROM imported_replay_events AS current + WHERE current.source=?1 AND current.source_session_id=?2 + AND current.generation=?3 AND current.event_id=previous.event_id + ) ORDER BY previous.event_id", + ) + .map_err(|err| format!("prepare reordered KV replay removals: {err}"))?; + let mut rows = stmt + .query(params![source.as_str(), source_session_id, generation]) + .map_err(|err| format!("query reordered KV replay removals: {err}"))?; + let mut removed = Vec::new(); + while let Some(row) = rows.next().map_err(|err| err.to_string())? { + removed.push(row.get::<_, String>(0).map_err(|err| err.to_string())?); + } + Ok(removed) +} + +pub(super) fn remove_missing_rows( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + _write_revision: u64, +) -> Result, String> { + let mut removed = Vec::new(); + { + let mut stmt = tx + .prepare( + "SELECT event_id FROM imported_replay_source_rows r + WHERE r.source=?1 AND r.source_session_id=?2 AND r.generation=?3 + AND NOT EXISTS ( + SELECT 1 FROM imported_replay_seen_rows s + WHERE s.source_key=r.source_key + ) AND event_id IS NOT NULL", + ) + .map_err(|err| format!("prepare removed SQLite replay rows: {err}"))?; + let mut rows = stmt + .query(params![source.as_str(), source_session_id, generation]) + .map_err(|err| format!("query removed SQLite replay rows: {err}"))?; + while let Some(row) = rows.next().map_err(|err| err.to_string())? { + removed.push(row.get::<_, String>(0).map_err(|err| err.to_string())?); + } + } + for event_id in &removed { + tx.execute( + "DELETE FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![source.as_str(), source_session_id, generation, event_id], + ) + .map_err(|err| format!("delete removed SQLite replay event: {err}"))?; + } + tx.execute( + "DELETE FROM imported_replay_source_rows AS r + WHERE r.source=?1 AND r.source_session_id=?2 AND r.generation=?3 + AND NOT EXISTS ( + SELECT 1 FROM imported_replay_seen_rows s WHERE s.source_key=r.source_key + )", + params![source.as_str(), source_session_id, generation], + ) + .map_err(|err| format!("delete removed SQLite source rows: {err}"))?; + Ok(removed) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/mod.rs new file mode 100644 index 000000000..55efddda3 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/mod.rs @@ -0,0 +1,1159 @@ +//! Incremental SQLite replay drivers for imported history stores. +//! +//! Provider databases are opened read-only (including their WAL view). Rows +//! are folded one at a time into ORGII's compact replay index. A persistent +//! source-row hash table lets an unchanged poll avoid JSON parsing and event +//! upserts even when a WAL checkpoint changed the physical files. + +use std::path::Path; + +use core_types::activity::ActivityChunk; +use rusqlite::{params, Connection, OpenFlags, OptionalExtension, Transaction}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use crate::development_artifact::{ + attach_replay_git_artifacts, parse_git_artifacts_from_tool_payload, +}; +use crate::sources::imported_history::{self, replay::ImportedHistorySourceId}; + +use crate::sources::imported_history::replay::drivers::common::{ + content_digest, rebuild_turns, ContentDigest, +}; +use crate::sources::imported_history::replay::index::ReplayIndexState; +use crate::sources::imported_history::replay::payload_artifact; +mod common; +mod kv_store; +mod row_store; + +use common::*; +use kv_store::*; +pub(in crate::sources::imported_history::replay) use kv_store::{ + hydrate_kv_turn, read_kv_turn_previews, +}; +use row_store::*; + +use crate::sources::imported_history::replay::{ + replay_payload_body_projection, ReplayPayloadBodyProjection, ReplayPayloadDescriptor, + ReplayPayloadEncoding, ReplayPayloadKind, ReplayPayloadRange, ReplayStats, + NORMAL_PAYLOAD_PREVIEW_BYTES, SHELL_PAYLOAD_PREVIEW_BYTES, +}; + +#[derive(Debug, Clone)] +pub(in crate::sources::imported_history::replay) struct SqliteSyncOutcome { + pub stats: ReplayStats, + pub driver_cursor_json: String, + pub total_events: u64, + pub total_turns: u64, + pub changed: bool, + pub removed_event_ids: Vec, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +struct RowStoreReplayCursor { + schema_version: i64, + total_source_rows: u64, + max_time_created: i64, + max_source_key: String, + source_signal: String, + last_source_key: String, + #[serde(default)] + order_signal: String, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +struct KvStoreReplayCursor { + schema_version: i64, + total_source_rows: u64, + max_time_created: i64, + max_source_key: String, + source_signal: String, + last_source_key: String, + #[serde(default)] + order_signal: String, +} + +#[derive(Debug, Deserialize)] +struct SqliteCursorVersion { + schema_version: i64, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum SyncPlan { + Skip, + Append, + Reconcile, +} + +#[derive(Debug)] +struct SqliteSourceSummary { + row_count: u64, + max_time_created: i64, + max_source_key: String, + source_signal: String, + last_source_key: String, + order_signal: String, +} + +#[derive(Debug)] +struct SourceRow { + key: String, + message_id: String, + role: String, + raw_json: String, + time_created: i64, + header_type: i64, + ordinal: i64, + turn_index: i64, +} + +#[derive(Debug)] +struct DeferredPayloadBody { + field_path: String, + text: String, +} + +#[derive(Debug, Clone)] +pub(in crate::sources::imported_history::replay) struct KvTurnPreview { + pub turn_index: i64, + pub created_at: String, + pub user_preview: String, +} + +pub(in crate::sources::imported_history::replay) fn cursor_schema_version( + cursor_json: &str, +) -> Option { + serde_json::from_str::(cursor_json) + .ok() + .map(|cursor| cursor.schema_version) +} + +pub(in crate::sources::imported_history::replay) fn database_schema_version( + path: &Path, +) -> Result { + let conn = open_source_db(path)?; + conn.query_row("PRAGMA schema_version", [], |row| row.get(0)) + .map_err(|err| { + format!( + "read replay SQLite schema version {}: {err}", + path.display() + ) + }) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn sync( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + previous_state: Option<&ReplayIndexState>, +) -> Result { + ensure_source_row_table(tx)?; + let source_conn = open_source_db(source_path)?; + validate_schema(&source_conn, source)?; + let schema_version = source_conn + .query_row("PRAGMA schema_version", [], |row| row.get::<_, i64>(0)) + .map_err(|err| format!("read {} schema version: {err}", source.as_str()))?; + tx.execute_batch( + "CREATE TEMP TABLE IF NOT EXISTS imported_replay_seen_rows ( + source_key TEXT PRIMARY KEY + ) WITHOUT ROWID; + DELETE FROM imported_replay_seen_rows;", + ) + .map_err(|err| format!("prepare SQLite replay seen-row set: {err}"))?; + + let mut stats = ReplayStats::default(); + let mut changed = false; + let mut removed_event_ids = Vec::new(); + let mut kv_order_rebuilt = false; + let (summary, plan) = match source { + ImportedHistorySourceId::OpenCode | ImportedHistorySourceId::MimoCode => { + let previous_cursor = previous_row_store_cursor(previous_state, source)?; + let summary = opencode_source_summary(&source_conn, source_session_id)?; + let plan = opencode_sync_plan(&previous_cursor, &summary); + if plan != SyncPlan::Skip { + stream_opencode_family( + &source_conn, + source, + source_session_id, + (plan == SyncPlan::Append).then_some(( + previous_cursor.max_time_created, + previous_cursor.max_source_key.as_str(), + )), + |row| { + fold_source_row( + tx, + source, + display_session_id, + source_session_id, + generation, + write_revision, + row, + &mut stats, + &mut changed, + &mut removed_event_ids, + &source_conn, + None, + ) + }, + )?; + } + (summary, plan) + } + ImportedHistorySourceId::ZCode => { + let previous_cursor = previous_row_store_cursor(previous_state, source)?; + let summary = opencode_source_summary(&source_conn, source_session_id)?; + let plan = opencode_sync_plan(&previous_cursor, &summary); + if plan != SyncPlan::Skip { + stream_opencode_family( + &source_conn, + source, + source_session_id, + (plan == SyncPlan::Append).then_some(( + previous_cursor.max_time_created, + previous_cursor.max_source_key.as_str(), + )), + |row| { + fold_source_row( + tx, + source, + display_session_id, + source_session_id, + generation, + write_revision, + row, + &mut stats, + &mut changed, + &mut removed_event_ids, + &source_conn, + None, + ) + }, + )?; + } + (summary, plan) + } + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf => { + let previous_cursor = previous_kv_store_cursor(previous_state, source)?; + let composer_json = load_composer_json(&source_conn, source_session_id)?; + let summary = kv_source_summary(&source_conn, source_session_id, &composer_json)?; + let plan = kv_sync_plan(&previous_cursor, &summary); + if plan != SyncPlan::Skip { + let order_changed = plan == SyncPlan::Reconcile + && !previous_cursor.order_signal.is_empty() + && previous_cursor.order_signal != summary.order_signal; + if order_changed { + stage_and_clear_kv_order(tx, source, source_session_id, generation)?; + kv_order_rebuilt = true; + } + let start_ordinal = if previous_cursor.source_signal.is_empty() { + kv_recent_turn_start(&source_conn, source_session_id, &composer_json)? + } else if plan == SyncPlan::Append { + previous_cursor.total_source_rows + } else { + 0 + }; + stream_kv_bubbles( + &source_conn, + source_session_id, + &composer_json, + start_ordinal, + None, + |row| { + fold_source_row( + tx, + source, + display_session_id, + source_session_id, + generation, + write_revision, + row, + &mut stats, + &mut changed, + &mut removed_event_ids, + &source_conn, + Some(&composer_json), + ) + }, + )?; + replace_kv_turn_headers( + tx, + &source_conn, + source, + source_session_id, + generation, + &composer_json, + )?; + } + (summary, plan) + } + _ => { + return Err(format!( + "{} is not a SQLite replay adapter", + source.as_str() + )) + } + }; + + if kv_order_rebuilt { + removed_event_ids.extend(kv_order_removals( + tx, + source, + source_session_id, + generation, + )?); + } + if plan == SyncPlan::Reconcile { + removed_event_ids.extend(remove_missing_rows( + tx, + source, + source_session_id, + generation, + write_revision, + )?); + } + if !removed_event_ids.is_empty() { + stats.removed_events = removed_event_ids.len() as u64; + changed = true; + } + if changed + && !matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) + { + rebuild_turns(tx, source, source_session_id, generation)?; + } + if changed { + delete_stale_payload_artifacts(tx, source, source_session_id, generation)?; + } + + let total_events = if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + summary.row_count + } else { + count_index_rows(tx, source, source_session_id, generation)? + }; + let total_turns = count_turns(tx, source, source_session_id, generation)?; + let driver_cursor_json = if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + serde_json::to_string(&KvStoreReplayCursor { + schema_version, + total_source_rows: summary.row_count, + max_time_created: summary.max_time_created, + max_source_key: summary.max_source_key, + source_signal: summary.source_signal, + last_source_key: summary.last_source_key, + order_signal: summary.order_signal, + }) + } else { + serde_json::to_string(&RowStoreReplayCursor { + schema_version, + total_source_rows: summary.row_count, + max_time_created: summary.max_time_created, + max_source_key: summary.max_source_key, + source_signal: summary.source_signal, + last_source_key: summary.last_source_key, + order_signal: summary.order_signal, + }) + } + .map_err(|err| format!("encode {} replay cursor: {err}", source.as_str()))?; + Ok(SqliteSyncOutcome { + stats, + driver_cursor_json, + // This is a logical row watermark for SQLite, never a byte offset. + total_events, + total_turns, + changed, + removed_event_ids, + }) +} + +fn previous_row_store_cursor( + previous_state: Option<&ReplayIndexState>, + source: ImportedHistorySourceId, +) -> Result { + previous_state + .map(|state| { + serde_json::from_str(&state.driver_cursor_json) + .map_err(|err| format!("decode {} replay cursor: {err}", source.as_str())) + }) + .transpose() + .map(Option::unwrap_or_default) +} + +fn previous_kv_store_cursor( + previous_state: Option<&ReplayIndexState>, + source: ImportedHistorySourceId, +) -> Result { + previous_state + .map(|state| { + serde_json::from_str(&state.driver_cursor_json) + .map_err(|err| format!("decode {} replay cursor: {err}", source.as_str())) + }) + .transpose() + .map(Option::unwrap_or_default) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn delete_stale_payload_artifacts( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result<(), String> { + tx.execute( + "DELETE FROM imported_replay_payload_artifact_refs AS ref + WHERE ref.source=?1 AND ref.source_session_id=?2 AND ref.generation=?3 + AND NOT EXISTS ( + SELECT 1 FROM imported_replay_events AS event + WHERE event.source=ref.source + AND event.source_session_id=ref.source_session_id + AND event.generation=ref.generation + AND event.event_id=ref.event_id + )", + params![source.as_str(), source_session_id, generation], + ) + .map_err(|err| format!("delete stale SQLite replay payload refs: {err}"))?; + payload_artifact::delete_orphans(tx, source, source_session_id, generation) +} + +fn open_source_db(path: &Path) -> Result { + Connection::open_with_flags( + path, + OpenFlags::SQLITE_OPEN_READ_ONLY + | OpenFlags::SQLITE_OPEN_URI + | OpenFlags::SQLITE_OPEN_NO_MUTEX, + ) + .map_err(|err| format!("open replay SQLite source {}: {err}", path.display())) +} + +fn validate_schema(conn: &Connection, source: ImportedHistorySourceId) -> Result<(), String> { + let sql = match source { + ImportedHistorySourceId::OpenCode + | ImportedHistorySourceId::MimoCode + | ImportedHistorySourceId::ZCode => { + "SELECT p.id, p.message_id, p.session_id, p.time_created, p.data, m.data + FROM part p JOIN message m ON m.id=p.message_id LIMIT 0" + } + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf => { + "SELECT key, value FROM cursorDiskKV LIMIT 0" + } + _ => return Err(format!("{} has no SQLite schema", source.as_str())), + }; + conn.prepare(sql).map(|_| ()).map_err(|err| { + format!( + "Unsupported {} replay schema (bounded adapter will not fall back): {err}", + source.as_str() + ) + }) +} + +#[allow( + clippy::too_many_arguments, + reason = "SQLite row projection keeps provider identity, ordering, and compact payload inputs explicit" +)] +fn fold_source_row( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + generation: &str, + write_revision: u64, + row: SourceRow, + stats: &mut ReplayStats, + changed: &mut bool, + removed_event_ids: &mut Vec, + source_conn: &Connection, + composer_json: Option<&str>, +) -> Result<(), String> { + if row.key.trim().is_empty() { + return Err(format!( + "{} replay row has no stable key; refusing rowid fallback", + source.as_str() + )); + } + let raw_hash = content_digest(&[ + row.key.as_bytes(), + row.message_id.as_bytes(), + row.role.as_bytes(), + row.raw_json.as_bytes(), + &row.time_created.to_le_bytes(), + &row.header_type.to_le_bytes(), + ]); + tx.execute( + "INSERT OR IGNORE INTO imported_replay_seen_rows(source_key) VALUES (?1)", + [&row.key], + ) + .map_err(|err| format!("mark SQLite replay row seen: {err}"))?; + let previous = tx + .query_row( + "SELECT content_hash, event_id, sequence FROM imported_replay_source_rows + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND source_key=?4", + params![source.as_str(), source_session_id, generation, row.key], + |db_row| { + Ok(( + db_row.get::<_, String>(0)?, + db_row.get::<_, Option>(1)?, + db_row.get::<_, Option>(2)?, + )) + }, + ) + .optional() + .map_err(|err| format!("read SQLite replay source-row hash: {err}"))?; + if previous + .as_ref() + .is_some_and(|(hash, _, _)| hash == &raw_hash) + { + return Ok(()); + } + + stats.parsed_rows = stats.parsed_rows.saturating_add(1); + stats.parsed_bytes = stats.parsed_bytes.saturating_add(row.raw_json.len() as u64); + let sequence = if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + row.ordinal + } else { + match previous.as_ref().and_then(|(_, _, sequence)| *sequence) { + Some(sequence) => sequence, + None => next_sequence(tx, source, source_session_id, generation)?, + } + }; + let bubble_id = row.key.rsplit(':').next().unwrap_or(row.key.as_str()); + let normalized = match source { + ImportedHistorySourceId::OpenCode => { + crate::sources::opencode::history::replay_chunk_from_part_json( + display_session_id, + "opencode", + sequence.max(0) as usize, + row.key.clone(), + row.message_id.clone(), + row.role.clone(), + &row.raw_json, + row.time_created, + )? + } + ImportedHistorySourceId::MimoCode => { + crate::sources::opencode::history::replay_chunk_from_part_json( + display_session_id, + "mimo_code", + sequence.max(0) as usize, + row.key.clone(), + row.message_id.clone(), + row.role.clone(), + &row.raw_json, + row.time_created, + )? + } + ImportedHistorySourceId::ZCode => { + crate::sources::zcode::history::replay_chunk_from_part_json( + display_session_id, + sequence.max(0) as usize, + row.key.clone(), + row.message_id.clone(), + row.role.clone(), + &row.raw_json, + row.time_created, + )? + } + ImportedHistorySourceId::CursorIde => { + crate::sources::cursor_ide::history::replay_chunk_from_bubble_json( + source_conn, + display_session_id, + bubble_id, + row.header_type, + &row.raw_json, + composer_json.unwrap_or("{}"), + )? + } + ImportedHistorySourceId::Windsurf => { + crate::sources::windsurf::history::replay_chunk_from_bubble_json( + source_conn, + display_session_id, + sequence.max(0) as usize, + bubble_id, + row.header_type, + &row.raw_json, + )? + } + _ => None, + }; + + let old_event_id = previous + .as_ref() + .and_then(|(_, event_id, _)| event_id.clone()); + let mut event_id = None; + if let Some(mut chunk) = normalized { + chunk.chunk_id = stable_event_id(source, &row.key); + payload_artifact::delete_event_refs( + tx, + source, + source_session_id, + generation, + &chunk.chunk_id, + )?; + let (payloads, deferred_bodies) = compact_chunk_with_bodies(&mut chunk, &row.key); + upsert_event( + tx, + source, + source_session_id, + generation, + write_revision, + sequence, + row.turn_index, + row.ordinal, + &chunk, + &payloads, + &raw_hash, + )?; + for body in deferred_bodies { + payload_artifact::store_text( + tx, + source, + source_session_id, + generation, + &chunk.chunk_id, + &body.field_path, + &body.text, + )?; + } + event_id = Some(chunk.chunk_id.clone()); + stats.normalized_events = stats.normalized_events.saturating_add(1); + stats.upserted_events = stats.upserted_events.saturating_add(1); + *changed = true; + } else if let Some(old_event_id) = old_event_id { + tx.execute( + "DELETE FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![source.as_str(), source_session_id, generation, old_event_id], + ) + .map_err(|err| format!("remove no-longer-renderable replay event: {err}"))?; + stats.removed_events = stats.removed_events.saturating_add(1); + removed_event_ids.push(old_event_id); + *changed = true; + } + tx.execute( + "INSERT INTO imported_replay_source_rows( + source, source_session_id, generation, source_key, content_hash, + event_id, sequence, source_order, seen_revision + ) VALUES (?1,?2,?3,?4,?5,?6,?7,?8,?9) + ON CONFLICT(source,source_session_id,generation,source_key) DO UPDATE SET + content_hash=excluded.content_hash,event_id=excluded.event_id, + sequence=excluded.sequence,source_order=excluded.source_order, + seen_revision=excluded.seen_revision", + params![ + source.as_str(), + source_session_id, + generation, + row.key, + raw_hash, + event_id, + sequence, + row.ordinal, + write_revision.min(i64::MAX as u64) as i64, + ], + ) + .map_err(|err| format!("publish SQLite replay source-row hash: {err}"))?; + Ok(()) +} + +fn ensure_source_row_table(tx: &Transaction<'_>) -> Result<(), String> { + tx.execute_batch( + "CREATE TABLE IF NOT EXISTS imported_replay_source_rows ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + source_key TEXT NOT NULL, + content_hash TEXT NOT NULL, + event_id TEXT, + sequence INTEGER, + source_order INTEGER NOT NULL, + seen_revision INTEGER NOT NULL, + PRIMARY KEY(source, source_session_id, generation, source_key) + ); + CREATE INDEX IF NOT EXISTS idx_imported_replay_source_rows_seen + ON imported_replay_source_rows( + source, source_session_id, generation, seen_revision + );", + ) + .map_err(|err| format!("create SQLite replay source-row index: {err}")) +} + +fn next_sequence( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result { + tx.query_row( + "SELECT COALESCE(MAX(sequence), -1)+1 FROM imported_replay_source_rows + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + |row| row.get(0), + ) + .map_err(|err| format!("allocate SQLite replay sequence: {err}")) +} + +/// Reordering stable KV keys can swap two occupied `sequence` primary keys. +/// Stage the previous ids in SQLite, then rebuild rows inside the same ORGII +/// transaction so no session-sized Rust vector or half-reordered index is +/// observable. +#[cfg(test)] +fn compact_chunk(chunk: &mut ActivityChunk, source_key: &str) -> Vec { + compact_chunk_with_bodies(chunk, source_key).0 +} + +fn compact_chunk_with_bodies( + chunk: &mut ActivityChunk, + source_key: &str, +) -> (Vec, Vec) { + let mut payloads = Vec::new(); + let mut deferred_bodies = Vec::new(); + let encoded_args = serde_json::to_string(&chunk.args).unwrap_or_else(|_| "null".to_string()); + let (git_artifacts, exact_result_body) = + if chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE { + let encoded_result = + serde_json::to_string(&chunk.result).unwrap_or_else(|_| "null".to_string()); + let artifacts = parse_git_artifacts_from_tool_payload(&encoded_args, &encoded_result); + let exact_result_body = (!artifacts.is_empty()).then_some(encoded_result); + (artifacts, exact_result_body) + } else { + (Vec::new(), None) + }; + let result_limit = if chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let result_body_projection = exact_result_body.as_deref().and_then(|encoded| { + replay_payload_body_projection( + "result", + &chunk.result, + Some(encoded), + result_limit, + chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE, + ) + }); + let result_fields: &[(&str, ReplayPayloadKind)] = match chunk.function.as_str() { + imported_history::FUNCTION_USER_MESSAGE => { + &[("result.message.content", ReplayPayloadKind::UserMessage)] + } + imported_history::FUNCTION_ASSISTANT => &[ + ("result.content", ReplayPayloadKind::AssistantContent), + ("result.observation", ReplayPayloadKind::AssistantContent), + ], + imported_history::FUNCTION_THINKING => &[ + ("result.thought", ReplayPayloadKind::Reasoning), + ("result.content", ReplayPayloadKind::Reasoning), + ("result.observation", ReplayPayloadKind::Reasoning), + ], + _ => &[ + ("result.output", ReplayPayloadKind::ToolOutput), + ("result.observation", ReplayPayloadKind::ToolOutput), + ("result.old_content", ReplayPayloadKind::ToolDiff), + ("result.new_content", ReplayPayloadKind::ToolDiff), + ], + }; + for &(path, kind) in result_fields { + if let Some(text) = value_at_path_mut(&mut chunk.result, path.trim_start_matches("result.")) + { + if text.len() > result_limit { + let total_bytes = text.len() as u64; + *text = head_preview(text, result_limit); + payloads.push(sqlite_payload_descriptor( + path, + kind, + ReplayPayloadEncoding::Utf8Text, + None, + source_key, + total_bytes, + )); + } + } + } + let args_size = encoded_args.len(); + let args_limit = if chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + if args_size > args_limit { + let args_body_projection = replay_payload_body_projection( + "args", + &chunk.args, + Some(&encoded_args), + args_limit, + false, + ); + let mut original = std::mem::take(&mut chunk.args); + let mut compact = serde_json::Map::new(); + compact.insert("_replayTruncated".to_string(), Value::Bool(true)); + for key in [ + "command", + "cmd", + "path", + "file_path", + "filePath", + "action", + "query", + "pattern", + "linesAdded", + "linesRemoved", + "operation", + ] { + let Some(value) = original.get_mut(key) else { + continue; + }; + if let Value::String(text) = value { + if text.len() > args_limit { + let full_text = std::mem::take(text); + compact.insert( + key.to_string(), + Value::String(head_preview(&full_text, args_limit)), + ); + } else { + compact.insert(key.to_string(), Value::String(text.clone())); + } + } else if value.is_number() || value.is_boolean() || value.is_null() { + compact.insert(key.to_string(), value.clone()); + } + } + if compact.len() == 1 { + compact.insert( + "_preview".to_string(), + Value::String(head_preview(&encoded_args, args_limit)), + ); + } + chunk.args = Value::Object(compact); + // The compact object keeps cheap semantic scalars, but it is not a + // lossless copy of the normalized arguments. One root descriptor is + // canonical so hydration cannot leave an omitted sibling or compact + // marker behind after applying nested fields. + payloads.retain(|payload| !field_path_is_under(&payload.field_path, "args")); + payloads.push(sqlite_artifact_payload_descriptor( + "args", + ReplayPayloadKind::ToolArguments, + args_body_projection, + args_size as u64, + )); + deferred_bodies.push(DeferredPayloadBody { + field_path: "args".to_string(), + text: encoded_args, + }); + } + attach_replay_git_artifacts(&mut chunk.result, &git_artifacts); + if let Some(exact_result_body) = exact_result_body.filter(|_| chunk.result.is_object()) { + // `_replayGitArtifacts` is compact projection metadata. A canonical + // root result keeps it available to metadata projection without + // leaking it into a hydrated legacy transcript or export. + payloads.retain(|payload| !field_path_is_under(&payload.field_path, "result")); + payloads.push(sqlite_artifact_payload_descriptor( + "result", + ReplayPayloadKind::ToolOutput, + result_body_projection, + exact_result_body.len() as u64, + )); + deferred_bodies.retain(|body| !field_path_is_under(&body.field_path, "result")); + deferred_bodies.push(DeferredPayloadBody { + field_path: "result".to_string(), + text: exact_result_body, + }); + } + (payloads, deferred_bodies) +} + +fn sqlite_payload_descriptor( + field_path: &str, + kind: ReplayPayloadKind, + encoding: ReplayPayloadEncoding, + body_projection: Option, + source_key: &str, + total_bytes: u64, +) -> ReplayPayloadDescriptor { + ReplayPayloadDescriptor { + field_path: field_path.to_string(), + kind, + encoding, + body_projection, + spans: Vec::new(), + total_bytes, + source_ordinal: None, + source_key: Some(source_key.to_string()), + } +} + +fn sqlite_artifact_payload_descriptor( + field_path: &str, + kind: ReplayPayloadKind, + body_projection: Option, + total_bytes: u64, +) -> ReplayPayloadDescriptor { + ReplayPayloadDescriptor { + field_path: field_path.to_string(), + kind, + encoding: ReplayPayloadEncoding::JsonValue, + body_projection, + spans: Vec::new(), + total_bytes, + source_ordinal: None, + // Provider normalization can wrap/rename raw SQLite JSON. The exact + // root body therefore lives only in the generation-scoped artifact. + source_key: None, + } +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn upsert_event( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + write_revision: u64, + sequence: i64, + turn_index: i64, + source_order: i64, + chunk: &ActivityChunk, + payloads: &[ReplayPayloadDescriptor], + content_hash: &str, +) -> Result<(), String> { + let args = serde_json::to_string(&chunk.args).map_err(|err| err.to_string())?; + let result = serde_json::to_string(&chunk.result).map_err(|err| err.to_string())?; + let payloads = serde_json::to_string(payloads).map_err(|err| err.to_string())?; + tx.execute( + "INSERT INTO imported_replay_events( + source,source_session_id,generation,sequence,event_id,turn_index, + action_type,function_name,created_at,args_preview_json,result_preview_json, + args_size_bytes,result_size_bytes,thread_id,process_id,source_start, + source_end,payloads_json,content_hash,event_revision + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13,?14,?15,?16,?16,?17,?18,?19) + ON CONFLICT(source,source_session_id,generation,event_id) DO UPDATE SET + sequence=excluded.sequence,turn_index=excluded.turn_index, + action_type=excluded.action_type, + function_name=excluded.function_name,created_at=excluded.created_at, + args_preview_json=excluded.args_preview_json, + result_preview_json=excluded.result_preview_json, + args_size_bytes=excluded.args_size_bytes, + result_size_bytes=excluded.result_size_bytes, + thread_id=excluded.thread_id,process_id=excluded.process_id, + source_start=excluded.source_start,source_end=excluded.source_end, + payloads_json=excluded.payloads_json,content_hash=excluded.content_hash, + event_revision=excluded.event_revision", + params![ + source.as_str(), + source_session_id, + generation, + sequence, + chunk.chunk_id, + turn_index, + chunk.action_type, + chunk.function, + chunk.created_at, + args, + result, + serde_json::to_vec(&chunk.args).map_or(0, |v| v.len()) as i64, + serde_json::to_vec(&chunk.result).map_or(0, |v| v.len()) as i64, + chunk.thread_id, + chunk.process_id, + source_order, + payloads, + content_hash, + write_revision.min(i64::MAX as u64) as i64, + ], + ) + .map_err(|err| format!("upsert {} replay event: {err}", source.as_str()))?; + Ok(()) +} + +fn insert_turn_header( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + turn_index: i64, + header: (String, i64, i64, String, String, u64), +) -> Result<(), String> { + tx.execute( + "INSERT INTO imported_replay_turns(source,source_session_id,generation,turn_index, + turn_id,start_sequence,end_sequence,started_at,ended_at,event_count) + VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10)", + params![ + source.as_str(), + source_session_id, + generation, + turn_index, + header.0, + header.1, + header.2, + header.3, + header.4, + header.5 as i64 + ], + ) + .map_err(|err| format!("insert {} replay turn header: {err}", source.as_str()))?; + Ok(()) +} + +fn count_index_rows( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + session: &str, + generation: &str, +) -> Result { + tx.query_row( + "SELECT COUNT(*) FROM imported_replay_events WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), session, generation], |row| row.get::<_, i64>(0), + ).map(|count| count.max(0) as u64).map_err(|err| err.to_string()) +} + +fn count_turns( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + session: &str, + generation: &str, +) -> Result { + tx.query_row( + "SELECT COUNT(*) FROM imported_replay_turns WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), session, generation], |row| row.get::<_, i64>(0), + ).map(|count| count.max(0) as u64).map_err(|err| err.to_string()) +} + +pub(in crate::sources::imported_history::replay) fn read_payload( + source: ImportedHistorySourceId, + source_path: &Path, + payloads_json: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + let payloads: Vec = serde_json::from_str(payloads_json) + .map_err(|err| format!("decode {} payload locator: {err}", source.as_str()))?; + let descriptor = payloads + .into_iter() + .find(|payload| payload.field_path == field_path) + .ok_or_else(|| format!("No deferred replay payload for {field_path}"))?; + let source_key = descriptor + .source_key + .ok_or_else(|| "SQLite replay payload has no stable source key".to_string())?; + let conn = open_source_db(source_path)?; + let (table, key_column, expression) = + payload_sql(source, descriptor.kind, &descriptor.field_path)?; + let sql = format!( + "SELECT length(CAST(({expression}) AS BLOB)), + substr(CAST(({expression}) AS BLOB), ?2, ?3) + FROM {table} AS source_row WHERE source_row.{key_column}=?1" + ); + let requested = offset.min(descriptor.total_bytes); + let (total, bytes): (i64, Vec) = conn + .query_row( + &sql, + params![ + source_key, + requested.saturating_add(1) as i64, + max_bytes as i64 + 4 + ], + |row| { + Ok(( + row.get::<_, Option>(0)?.unwrap_or_default(), + row.get::<_, Option>>(1)?.unwrap_or_default(), + )) + }, + ) + .map_err(|err| format!("read {} replay payload range: {err}", source.as_str()))?; + let mut leading = 0_usize; + while leading < bytes.len() && (bytes[leading] & 0b1100_0000) == 0b1000_0000 { + leading += 1; + } + let actual_offset = requested.saturating_add(leading as u64); + let available = &bytes[leading..]; + let mut usable = available.len().min(max_bytes); + while usable > 0 && std::str::from_utf8(&available[..usable]).is_err() { + usable -= 1; + } + let text = String::from_utf8(available[..usable].to_vec()) + .map_err(|err| format!("decode SQLite replay payload UTF-8: {err}"))?; + let total_bytes = total.max(0) as u64; + let next_offset = actual_offset.saturating_add(usable as u64).min(total_bytes); + Ok(ReplayPayloadRange { + event_id: event_id.to_string(), + field_path: field_path.to_string(), + offset: actual_offset, + next_offset, + eof: next_offset >= total_bytes, + total_bytes, + text, + }) +} + +fn payload_sql( + source: ImportedHistorySourceId, + kind: ReplayPayloadKind, + field_path: &str, +) -> Result<(&'static str, &'static str, &'static str), String> { + let expression = match (source, kind) { + (ImportedHistorySourceId::OpenCode | ImportedHistorySourceId::MimoCode | ImportedHistorySourceId::ZCode, ReplayPayloadKind::UserMessage | ReplayPayloadKind::AgentMessage | ReplayPayloadKind::AssistantContent | ReplayPayloadKind::Reasoning) => "COALESCE(json_extract(data, '$.text'), '')", + (ImportedHistorySourceId::OpenCode | ImportedHistorySourceId::MimoCode | ImportedHistorySourceId::ZCode, ReplayPayloadKind::ToolOutput) => "COALESCE(NULLIF(json_extract(data, '$.state.output'), ''), json_extract(data, '$.state.metadata.output'), '')", + (ImportedHistorySourceId::OpenCode | ImportedHistorySourceId::MimoCode | ImportedHistorySourceId::ZCode, ReplayPayloadKind::ToolArguments) => match field_path.rsplit('.').next().unwrap_or_default() { + "command" | "cmd" => "COALESCE(json_extract(data, '$.state.input.command'), json_extract(data, '$.state.input.cmd'), '')", + "path" => "COALESCE(json_extract(data, '$.state.input.path'), '')", + "file_path" | "filePath" => "COALESCE(json_extract(data, '$.state.input.file_path'), json_extract(data, '$.state.input.filePath'), json_extract(data, '$.state.input.path'), '')", + "action" => "COALESCE(json_extract(data, '$.state.input.action'), '')", + "query" => "COALESCE(json_extract(data, '$.state.input.query'), '')", + "pattern" => "COALESCE(json_extract(data, '$.state.input.pattern'), '')", + _ => "COALESCE(json_extract(data, '$.state.input'), '{}')", + }, + (ImportedHistorySourceId::OpenCode | ImportedHistorySourceId::MimoCode | ImportedHistorySourceId::ZCode, ReplayPayloadKind::ToolDiff) if field_path.ends_with("old_content") => "COALESCE(json_extract(data, '$.state.metadata.old_content'), json_extract(data, '$.state.metadata.before'), '')", + (ImportedHistorySourceId::OpenCode | ImportedHistorySourceId::MimoCode | ImportedHistorySourceId::ZCode, ReplayPayloadKind::ToolDiff) => "COALESCE(json_extract(data, '$.state.metadata.new_content'), json_extract(data, '$.state.metadata.after'), '')", + (ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf, ReplayPayloadKind::UserMessage | ReplayPayloadKind::AgentMessage | ReplayPayloadKind::AssistantContent | ReplayPayloadKind::Reasoning) => "COALESCE(json_extract(value, '$.text'), '')", + (ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf, ReplayPayloadKind::ToolOutput) => "CASE WHEN json_valid(json_extract(value, '$.toolFormerData.result')) THEN COALESCE(json_extract(json_extract(value, '$.toolFormerData.result'), '$.output'), json_extract(json_extract(value, '$.toolFormerData.result'), '$.observation'), json_extract(json_extract(value, '$.toolFormerData.result'), '$.content'), json_extract(value, '$.toolFormerData.result'), '') ELSE COALESCE(json_extract(value, '$.toolFormerData.result'), '') END", + (ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf, ReplayPayloadKind::ToolArguments) => match field_path.rsplit('.').next().unwrap_or_default() { + "command" | "cmd" => "CASE WHEN json_valid(json_extract(value, '$.toolFormerData.params')) THEN COALESCE(json_extract(json_extract(value, '$.toolFormerData.params'), '$.command'), json_extract(json_extract(value, '$.toolFormerData.params'), '$.cmd'), '') ELSE '' END", + "path" => "CASE WHEN json_valid(json_extract(value, '$.toolFormerData.params')) THEN COALESCE(json_extract(json_extract(value, '$.toolFormerData.params'), '$.path'), '') ELSE '' END", + "file_path" | "filePath" => "CASE WHEN json_valid(json_extract(value, '$.toolFormerData.params')) THEN COALESCE(json_extract(json_extract(value, '$.toolFormerData.params'), '$.file_path'), json_extract(json_extract(value, '$.toolFormerData.params'), '$.filePath'), json_extract(json_extract(value, '$.toolFormerData.params'), '$.targetFile'), '') ELSE '' END", + "action" => "CASE WHEN json_valid(json_extract(value, '$.toolFormerData.params')) THEN COALESCE(json_extract(json_extract(value, '$.toolFormerData.params'), '$.action'), '') ELSE '' END", + "query" => "CASE WHEN json_valid(json_extract(value, '$.toolFormerData.params')) THEN COALESCE(json_extract(json_extract(value, '$.toolFormerData.params'), '$.query'), '') ELSE '' END", + "pattern" => "CASE WHEN json_valid(json_extract(value, '$.toolFormerData.params')) THEN COALESCE(json_extract(json_extract(value, '$.toolFormerData.params'), '$.pattern'), json_extract(json_extract(value, '$.toolFormerData.params'), '$.globPattern'), '') ELSE '' END", + _ => "COALESCE(json_extract(value, '$.toolFormerData.params'), '{}')", + }, + (ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf, ReplayPayloadKind::ToolDiff) if field_path.ends_with("old_content") => "CASE WHEN json_valid(json_extract(source_row.value, '$.toolFormerData.result')) THEN COALESCE((SELECT blob.value FROM cursorDiskKV AS blob WHERE blob.key=json_extract(json_extract(source_row.value, '$.toolFormerData.result'), '$.beforeContentId')), json_extract(json_extract(source_row.value, '$.toolFormerData.result'), '$.old_content'), '') ELSE '' END", + (ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf, ReplayPayloadKind::ToolDiff) => "CASE WHEN json_valid(json_extract(source_row.value, '$.toolFormerData.result')) THEN COALESCE((SELECT blob.value FROM cursorDiskKV AS blob WHERE blob.key=json_extract(json_extract(source_row.value, '$.toolFormerData.result'), '$.afterContentId')), json_extract(json_extract(source_row.value, '$.toolFormerData.result'), '$.new_content'), '') ELSE '' END", + _ => return Err(format!("Unsupported {} SQLite payload kind", source.as_str())), + }; + Ok(match source { + ImportedHistorySourceId::OpenCode + | ImportedHistorySourceId::MimoCode + | ImportedHistorySourceId::ZCode => ("part", "id", expression), + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf => { + ("cursorDiskKV", "key", expression) + } + _ => { + return Err(format!( + "{} is not a SQLite payload source", + source.as_str() + )) + } + }) +} + +#[cfg(test)] +#[path = "../../tests/sqlite_driver.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/row_store.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/row_store.rs new file mode 100644 index 000000000..a7f2acd3f --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/sqlite/row_store.rs @@ -0,0 +1,125 @@ +use super::*; + +pub(super) fn opencode_source_summary( + conn: &Connection, + source_session_id: &str, +) -> Result { + let row_count = conn + .query_row( + "SELECT COUNT(*) FROM part WHERE session_id=?1", + [source_session_id], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("count SQLite replay parts: {err}"))? + .max(0) as u64; + let (max_time_created, max_source_key) = conn + .query_row( + "SELECT COALESCE(time_created,0), COALESCE(id,'') FROM part + WHERE session_id=?1 ORDER BY time_created DESC,id DESC LIMIT 1", + [source_session_id], + |row| Ok((row.get::<_, i64>(0)?, row.get::<_, String>(1)?)), + ) + .optional() + .map_err(|err| format!("read SQLite replay part watermark: {err}"))? + .unwrap_or_default(); + let session_signal = conn + .query_row( + "SELECT COALESCE(time_updated, time_created, 0) FROM session WHERE id=?1", + [source_session_id], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|err| format!("read SQLite replay session signal: {err}"))? + .unwrap_or_default() + .to_string(); + Ok(SqliteSourceSummary { + row_count, + max_time_created, + max_source_key: max_source_key.clone(), + source_signal: session_signal, + last_source_key: max_source_key, + order_signal: String::new(), + }) +} + +pub(super) fn opencode_sync_plan( + previous: &RowStoreReplayCursor, + current: &SqliteSourceSummary, +) -> SyncPlan { + if previous.source_signal.is_empty() { + return SyncPlan::Reconcile; + } + if current.row_count == previous.total_source_rows + && current.max_time_created == previous.max_time_created + && current.max_source_key == previous.max_source_key + && current.source_signal == previous.source_signal + { + return SyncPlan::Skip; + } + let watermark_advanced = (current.max_time_created, current.max_source_key.as_str()) + > (previous.max_time_created, previous.max_source_key.as_str()); + if current.row_count > previous.total_source_rows && watermark_advanced { + SyncPlan::Append + } else { + SyncPlan::Reconcile + } +} + +pub(super) fn stream_opencode_family( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + after: Option<(i64, &str)>, + mut visit: impl FnMut(SourceRow) -> Result<(), String>, +) -> Result<(), String> { + let (after_time, after_key) = after.unwrap_or((i64::MIN, "")); + let mut stmt = conn + .prepare( + "SELECT p.id, p.message_id, COALESCE(json_extract(m.data, '$.role'), ''), + p.data, COALESCE(p.time_created, 0) + FROM part p JOIN message m ON m.id=p.message_id + WHERE p.session_id=?1 + AND (p.time_created>?2 OR (p.time_created=?2 AND p.id>?3)) + ORDER BY p.time_created ASC, p.id ASC", + ) + .map_err(|err| format!("prepare {} replay row stream: {err}", source.as_str()))?; + let mut rows = stmt + .query(params![source_session_id, after_time, after_key]) + .map_err(|err| format!("query {} replay rows: {err}", source.as_str()))?; + let mut ordinal = 0_i64; + while let Some(row) = rows + .next() + .map_err(|err| format!("stream {} replay row: {err}", source.as_str()))? + { + let Some(raw_json) = row + .get::<_, Option>(3) + .map_err(|err| format!("read {} replay JSON: {err}", source.as_str()))? + else { + continue; + }; + visit(SourceRow { + key: row + .get::<_, Option>(0) + .map_err(|e| e.to_string())? + .unwrap_or_default(), + message_id: row + .get::<_, Option>(1) + .map_err(|e| e.to_string())? + .unwrap_or_default(), + role: row + .get::<_, Option>(2) + .map_err(|e| e.to_string())? + .unwrap_or_default(), + raw_json, + time_created: row + .get::<_, Option>(4) + .map_err(|e| e.to_string())? + .unwrap_or_default(), + header_type: 0, + ordinal, + turn_index: 0, + })?; + ordinal += 1; + } + Ok(()) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/common.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/common.rs new file mode 100644 index 000000000..cfb6e7991 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/common.rs @@ -0,0 +1,129 @@ +use super::*; +use crate::sources::imported_history::replay::drivers::common::{ + legacy_stable_id_hash_delimited, utf8_boundary_at_or_before, +}; + +pub(super) fn open_source_db(path: &Path) -> Result { + Connection::open_with_flags( + path, + OpenFlags::SQLITE_OPEN_READ_ONLY + | OpenFlags::SQLITE_OPEN_URI + | OpenFlags::SQLITE_OPEN_NO_MUTEX, + ) + .map_err(|err| format!("open structured replay source {}: {err}", path.display())) +} + +pub(super) fn field<'a>(value: &'a Value, names: &[&str]) -> Option<&'a Value> { + let object = value.as_object()?; + names.iter().find_map(|name| object.get(*name)) +} + +pub(super) fn field_str<'a>(value: &'a Value, names: &[&str]) -> Option<&'a str> { + field(value, names).and_then(Value::as_str) +} + +pub(super) fn timestamp_value_to_iso(value: &Value) -> Option { + if let Some(raw) = value.as_str() { + return Some(imported_history::normalize_created_at(raw)); + } + let seconds = field(value, &["seconds"])?; + let seconds = seconds + .as_i64() + .or_else(|| seconds.as_str().and_then(|raw| raw.parse().ok()))?; + let nanos = field(value, &["nanos"]) + .and_then(Value::as_i64) + .unwrap_or_default(); + chrono::DateTime::from_timestamp(seconds, nanos.max(0) as u32).map(|dt| dt.to_rfc3339()) +} + +pub(super) fn parse_warp_timestamp_ms(value: &str) -> Option { + imported_history::parse_iso_to_epoch_ms_opt(value).or_else(|| { + ["%Y-%m-%d %H:%M:%S%.f", "%Y-%m-%d %H:%M:%S"] + .iter() + .find_map(|format| chrono::NaiveDateTime::parse_from_str(value, format).ok()) + .map(|timestamp| timestamp.and_utc().timestamp_millis()) + }) +} + +pub(super) fn camel_to_snake(value: &str) -> String { + let mut output = String::with_capacity(value.len() + 4); + for (index, ch) in value.chars().enumerate() { + if ch.is_ascii_uppercase() { + if index > 0 { + output.push('_'); + } + output.push(ch.to_ascii_lowercase()); + } else { + output.push(ch); + } + } + output +} + +pub(super) fn chunk_field_text(chunk: &ActivityChunk, field_path: &str) -> Result { + let (root, path) = field_path + .split_once('.') + .map_or((field_path, ""), |parts| parts); + let value = match root { + "args" => &chunk.args, + "result" => &chunk.result, + _ => return Err("Replay payload field must be under args or result".to_string()), + }; + let target = if path.is_empty() { + value + } else { + path.split('.') + .try_fold(value, |current, key| current.get(key)) + .ok_or_else(|| "Replay payload field no longer exists".to_string())? + }; + Ok(target + .as_str() + .map(str::to_string) + .unwrap_or_else(|| target.to_string())) +} + +pub(super) fn value_at_path_mut<'a>(value: &'a mut Value, path: &str) -> Option<&'a mut String> { + let mut current = value; + for segment in path.split('.') { + current = current.as_object_mut()?.get_mut(segment)?; + } + match current { + Value::String(text) => Some(text), + _ => None, + } +} + +pub(super) fn head_preview(text: &str, max_bytes: usize) -> String { + let end = utf8_boundary_at_or_before(text, max_bytes); + format!("{}\n… [payload truncated]", &text[..end]) +} + +pub(super) fn stable_event_id( + source: ImportedHistorySourceId, + source_session_id: &str, + event_key: &str, +) -> String { + format!( + "replay-{}-{}", + source.as_str(), + legacy_stable_id_hash_delimited(&[source_session_id.as_bytes(), event_key.as_bytes()]) + ) +} + +pub(super) fn hex_encode(bytes: &[u8]) -> String { + bytes.iter().map(|byte| format!("{byte:02x}")).collect() +} + +pub(super) fn hex_decode(text: &str) -> Option> { + let text = text.trim(); + if text.is_empty() || text.len() % 2 != 0 { + return None; + } + text.as_bytes() + .chunks_exact(2) + .map(|pair| { + let pair = std::str::from_utf8(pair).ok()?; + u8::from_str_radix(pair, 16).ok() + }) + .collect() +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/cursor_cli.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/cursor_cli.rs new file mode 100644 index 000000000..192366410 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/cursor_cli.rs @@ -0,0 +1,654 @@ +use super::*; + +pub(super) fn sync_cursor_cli( + tx: &Transaction<'_>, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + previous_state: Option<&ReplayIndexState>, +) -> Result { + let source = ImportedHistorySourceId::CursorCli; + let source_conn = open_source_db(source_path)?; + validate_cursor_schema(&source_conn)?; + let schema_version = source_conn + .query_row("PRAGMA schema_version", [], |row| row.get::<_, i64>(0)) + .map_err(|err| format!("read Cursor CLI schema version: {err}"))?; + let meta = read_cursor_meta(&source_conn)?; + let mut cursor = previous_state + .map(|state| serde_json::from_str::(&state.driver_cursor_json)) + .transpose() + .map_err(|err| format!("decode Cursor CLI replay cursor: {err}"))? + .filter(|cursor| cursor.driver == "cursor_cli") + .unwrap_or_else(|| CursorCliReplayCursor { + schema_version, + driver: "cursor_cli".to_string(), + cursor_turn_index: -1, + ..CursorCliReplayCursor::default() + }); + + // Content-addressed roots never mutate. This remains a true zero-parse, + // zero-upsert poll even after the coordinator's 60 second integrity tick. + if cursor.root_blob_id == meta.latest_root_blob_id && previous_state.is_some() { + return unchanged_outcome(tx, source, source_session_id, generation, cursor); + } + + let root = read_blob(&source_conn, &meta.latest_root_blob_id)? + .ok_or_else(|| "Cursor CLI root blob is missing".to_string())?; + if previous_state.is_some() + && cursor.message_count > 0 + && cursor.root_blob_id != meta.latest_root_blob_id + { + let (current_count, current_prefix_hash) = + manifest_prefix_hash(&root, cursor.message_count)?; + if current_count < cursor.message_count + || current_prefix_hash != cursor.manifest_prefix_hash + { + // The coordinator checked lineage before opening this source + // snapshot. A concurrent fork/reorder must roll the ORGII index + // transaction back and retry as a new generation, never append + // the new suffix onto the old conversation. + return Err("Cursor CLI replay lineage changed during synchronization".to_string()); + } + } + let created_at = imported_history::epoch_ms_to_iso(meta.created_at); + let mut stats = ReplayStats::default(); + let mut changed = false; + let mut message_count = 0_u64; + let mut prefix_hash = ContentDigest::default(); + visit_manifest_message_ids(&root, |blob_id| { + prefix_hash.update_part(blob_id.as_bytes()); + let ordinal = message_count; + message_count = message_count.saturating_add(1); + if ordinal < cursor.message_count { + return Ok(()); + } + let Some(data) = read_blob(&source_conn, blob_id)? else { + return Ok(()); + }; + stats.parsed_rows = stats.parsed_rows.saturating_add(1); + stats.parsed_bytes = stats.parsed_bytes.saturating_add(data.len() as u64); + let Ok(message) = serde_json::from_slice::(&data) else { + return Ok(()); + }; + let emitted = fold_cursor_message( + display_session_id, + source_session_id, + blob_id, + ordinal, + &message, + &created_at, + &source_conn, + &mut cursor, + )?; + for emitted in emitted { + upsert_emitted( + tx, + source, + source_session_id, + generation, + write_revision, + emitted, + &mut stats, + )?; + changed = true; + } + Ok(()) + })?; + + cursor.schema_version = schema_version; + cursor.driver = "cursor_cli".to_string(); + cursor.root_blob_id = meta.latest_root_blob_id; + cursor.message_count = message_count; + cursor.manifest_prefix_hash = prefix_hash.finish_hex(); + if changed { + payload_artifact::delete_orphans(tx, source, source_session_id, generation)?; + rebuild_turns(tx, source, source_session_id, generation)?; + } + finish_outcome( + tx, + source, + source_session_id, + generation, + cursor, + stats, + changed, + Vec::new(), + ) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn fold_cursor_message( + display_session_id: &str, + source_session_id: &str, + blob_id: &str, + manifest_ordinal: u64, + message: &Value, + created_at: &str, + source_conn: &Connection, + cursor: &mut CursorCliReplayCursor, +) -> Result, String> { + let mut emitted = Vec::new(); + match message.get("role").and_then(Value::as_str) { + Some("user") => { + let text = cursor_message_text(message.get("content")); + let Some(text) = clean_cursor_user_text(&text) else { + return Ok(emitted); + }; + if cursor.last_user_text.as_deref() == Some(text.as_str()) { + return Ok(emitted); + } + cursor.last_user_text = Some(text.clone()); + cursor.cursor_turn_index = cursor.cursor_turn_index.saturating_add(1).max(0); + let chunk = imported_history::user_message_chunk( + display_session_id, + CURSOR_PROVIDER, + manifest_ordinal as usize, + created_at, + &text, + ); + emitted.push(EmittedChunk { + event_key: format!("message:{manifest_ordinal}:user"), + turn_index: cursor.cursor_turn_index, + chunk, + locator: StructuredPayloadLocator::Cursor { + call_blob_id: blob_id.to_string(), + result_blob_id: None, + item_index: 0, + result_item_index: None, + event_kind: CursorEventKind::User, + segment_index: 0, + }, + }); + } + Some("assistant") => { + for (item_index, item) in cursor_message_items(message.get("content")).enumerate() { + match item.get("type").and_then(Value::as_str) { + Some("text") => { + let text = item.get("text").and_then(Value::as_str).unwrap_or_default(); + let (thoughts, visible) = split_think_blocks(text); + for (segment_index, thought) in thoughts.into_iter().enumerate() { + let chunk = imported_history::thinking_chunk( + display_session_id, + CURSOR_PROVIDER, + manifest_ordinal as usize, + created_at, + &thought, + ); + emitted.push(EmittedChunk { + event_key: format!( + "message:{manifest_ordinal}:item:{item_index}:thought:{segment_index}" + ), + turn_index: cursor.cursor_turn_index.max(0), + chunk, + locator: StructuredPayloadLocator::Cursor { + call_blob_id: blob_id.to_string(), + result_blob_id: None, + item_index, + result_item_index: None, + event_kind: CursorEventKind::Thinking, + segment_index, + }, + }); + } + let visible = visible.trim(); + if !visible.is_empty() { + let chunk = imported_history::assistant_message_chunk( + display_session_id, + CURSOR_PROVIDER, + manifest_ordinal as usize, + created_at, + visible, + ); + emitted.push(EmittedChunk { + event_key: format!( + "message:{manifest_ordinal}:item:{item_index}:assistant" + ), + turn_index: cursor.cursor_turn_index.max(0), + chunk, + locator: StructuredPayloadLocator::Cursor { + call_blob_id: blob_id.to_string(), + result_blob_id: None, + item_index, + result_item_index: None, + event_kind: CursorEventKind::AssistantVisible, + segment_index: 0, + }, + }); + } + } + Some("tool-call") => { + let Some(call) = cursor_tool_call(item, created_at) else { + continue; + }; + let event_id = stable_event_id( + ImportedHistorySourceId::CursorCli, + source_session_id, + &format!("call:{manifest_ordinal}:{}", call.call_id), + ); + let chunk = imported_history::tool_call_chunk( + display_session_id, + CURSOR_PROVIDER, + manifest_ordinal as usize, + &call, + "", + ); + emitted.push(EmittedChunk { + event_key: format!("call:{manifest_ordinal}:{}", call.call_id), + turn_index: cursor.cursor_turn_index.max(0), + chunk, + locator: StructuredPayloadLocator::Cursor { + call_blob_id: blob_id.to_string(), + result_blob_id: None, + item_index, + result_item_index: None, + event_kind: CursorEventKind::Tool, + segment_index: 0, + }, + }); + cursor.pending_cursor_calls.insert( + call.call_id.clone(), + PendingCursorCall { + call_blob_id: blob_id.to_string(), + item_index, + manifest_ordinal, + event_id, + turn_index: cursor.cursor_turn_index.max(0), + }, + ); + } + _ => {} + } + } + } + Some("tool") => { + for (result_item_index, item) in + cursor_message_items(message.get("content")).enumerate() + { + if item.get("type").and_then(Value::as_str) != Some("tool-result") { + continue; + } + let Some(call_id) = item.get("toolCallId").and_then(Value::as_str) else { + continue; + }; + let Some(pending) = cursor.pending_cursor_calls.remove(call_id) else { + continue; + }; + let Some(call_blob) = read_blob(source_conn, &pending.call_blob_id)? else { + continue; + }; + let Ok(call_message) = serde_json::from_slice::(&call_blob) else { + continue; + }; + let Some(call_item) = + cursor_message_items(call_message.get("content")).nth(pending.item_index) + else { + continue; + }; + let Some(call) = cursor_tool_call(call_item, created_at) else { + continue; + }; + let output = cursor_tool_result_text(item.get("result")); + let mut chunk = imported_history::tool_call_chunk( + display_session_id, + CURSOR_PROVIDER, + pending.manifest_ordinal as usize, + &call, + &output, + ); + chunk.chunk_id = pending.event_id; + emitted.push(EmittedChunk { + event_key: format!("call:{}:{call_id}", pending.manifest_ordinal), + turn_index: pending.turn_index, + chunk, + locator: StructuredPayloadLocator::Cursor { + call_blob_id: pending.call_blob_id, + result_blob_id: Some(blob_id.to_string()), + item_index: pending.item_index, + result_item_index: Some(result_item_index), + event_kind: CursorEventKind::Tool, + segment_index: 0, + }, + }); + } + } + _ => {} + } + Ok(emitted) +} + +pub(super) fn reconstruct_cursor_chunk( + conn: &Connection, + event_id: &str, + locator: &StructuredPayloadLocator, +) -> Result { + let StructuredPayloadLocator::Cursor { + call_blob_id, + result_blob_id, + item_index, + result_item_index, + event_kind, + segment_index, + } = locator + else { + return Err("not a Cursor replay locator".to_string()); + }; + let data = read_blob(conn, call_blob_id)? + .ok_or_else(|| "Cursor replay payload blob no longer exists".to_string())?; + let message: Value = serde_json::from_slice(&data) + .map_err(|err| format!("decode Cursor replay payload message: {err}"))?; + let created_at = String::new(); + let mut chunk = match event_kind { + CursorEventKind::User => { + let text = clean_cursor_user_text(&cursor_message_text(message.get("content"))) + .unwrap_or_default(); + imported_history::user_message_chunk( + "cursorcliapp-payload", + CURSOR_PROVIDER, + 0, + &created_at, + &text, + ) + } + CursorEventKind::AssistantVisible | CursorEventKind::Thinking => { + let item = cursor_message_items(message.get("content")) + .nth(*item_index) + .ok_or_else(|| "Cursor replay text item no longer exists".to_string())?; + let text = item.get("text").and_then(Value::as_str).unwrap_or_default(); + let (thoughts, visible) = split_think_blocks(text); + if matches!(event_kind, CursorEventKind::Thinking) { + let thought = thoughts.get(*segment_index).cloned().unwrap_or_default(); + imported_history::thinking_chunk( + "cursorcliapp-payload", + CURSOR_PROVIDER, + 0, + &created_at, + &thought, + ) + } else { + imported_history::assistant_message_chunk( + "cursorcliapp-payload", + CURSOR_PROVIDER, + 0, + &created_at, + visible.trim(), + ) + } + } + CursorEventKind::Tool => { + let item = cursor_message_items(message.get("content")) + .nth(*item_index) + .ok_or_else(|| "Cursor replay tool call no longer exists".to_string())?; + let call = cursor_tool_call(item, &created_at) + .ok_or_else(|| "Cursor replay tool call is invalid".to_string())?; + let output = match (result_blob_id, result_item_index) { + (Some(result_blob_id), Some(result_item_index)) => { + let result_data = read_blob(conn, result_blob_id)? + .ok_or_else(|| "Cursor replay result blob no longer exists".to_string())?; + let result_message: Value = serde_json::from_slice(&result_data) + .map_err(|err| format!("decode Cursor replay result: {err}"))?; + let output = cursor_message_items(result_message.get("content")) + .nth(*result_item_index) + .map(|item| cursor_tool_result_text(item.get("result"))) + .unwrap_or_default(); + output + } + _ => String::new(), + }; + imported_history::tool_call_chunk( + "cursorcliapp-payload", + CURSOR_PROVIDER, + 0, + &call, + &output, + ) + } + }; + chunk.chunk_id = event_id.to_string(); + Ok(chunk) +} + +pub(super) fn validate_cursor_schema(conn: &Connection) -> Result<(), String> { + conn.prepare("SELECT value FROM meta WHERE key='0' LIMIT 0") + .and_then(|_| conn.prepare("SELECT id,data FROM blobs LIMIT 0")) + .map(|_| ()) + .map_err(|err| format!("unsupported Cursor CLI replay schema: {err}")) +} + +pub(super) fn read_cursor_meta(conn: &Connection) -> Result { + let raw = conn + .query_row("SELECT value FROM meta WHERE key='0'", [], |row| { + row.get::<_, rusqlite::types::Value>(0) + }) + .optional() + .map_err(|err| format!("read Cursor CLI replay meta: {err}"))? + .ok_or_else(|| "Cursor CLI replay meta is missing".to_string())?; + let bytes = match raw { + rusqlite::types::Value::Text(text) => text.into_bytes(), + rusqlite::types::Value::Blob(bytes) => bytes, + _ => return Err("Cursor CLI replay meta has unsupported type".to_string()), + }; + let decoded = if bytes.first() == Some(&b'{') { + bytes + } else { + hex_decode(std::str::from_utf8(&bytes).unwrap_or_default()) + .ok_or_else(|| "Cursor CLI replay meta is not valid hex JSON".to_string())? + }; + serde_json::from_slice(&decoded).map_err(|err| format!("decode Cursor CLI replay meta: {err}")) +} + +pub(super) fn read_blob(conn: &Connection, blob_id: &str) -> Result>, String> { + conn.query_row("SELECT data FROM blobs WHERE id=?1", [blob_id], |row| { + row.get::<_, Vec>(0) + }) + .optional() + .map_err(|err| format!("read structured replay blob {blob_id}: {err}")) +} + +pub(super) fn visit_manifest_message_ids( + data: &[u8], + mut visit: impl FnMut(&str) -> Result<(), String>, +) -> Result<(), String> { + let mut offset = 0usize; + while offset < data.len() { + let (tag, next) = read_varint(data, offset) + .ok_or_else(|| "Cursor CLI manifest has a truncated tag".to_string())?; + offset = next; + match tag & 7 { + 0 => { + offset = read_varint(data, offset) + .ok_or_else(|| "Cursor CLI manifest has a truncated varint".to_string())? + .1; + } + 1 => offset = offset.saturating_add(8), + 2 => { + let (length, next) = read_varint(data, offset) + .ok_or_else(|| "Cursor CLI manifest has a truncated length".to_string())?; + offset = next; + let end = offset + .checked_add(length as usize) + .filter(|end| *end <= data.len()) + .ok_or_else(|| "Cursor CLI manifest field exceeds root blob".to_string())?; + if tag >> 3 == 1 && length == 32 { + let id = hex_encode(&data[offset..end]); + visit(&id)?; + } + offset = end; + } + 5 => offset = offset.saturating_add(4), + _ => return Err("Cursor CLI manifest uses an unsupported wire type".to_string()), + } + if offset > data.len() { + return Err("Cursor CLI manifest is truncated".to_string()); + } + } + Ok(()) +} + +pub(super) fn manifest_prefix_hash( + data: &[u8], + prefix_count: u64, +) -> Result<(u64, String), String> { + let mut count = 0_u64; + let mut hash = ContentDigest::default(); + visit_manifest_message_ids(data, |id| { + if count < prefix_count { + hash.update_part(id.as_bytes()); + } + count = count.saturating_add(1); + Ok(()) + })?; + Ok((count, hash.finish_hex())) +} + +pub(super) fn read_varint(data: &[u8], mut offset: usize) -> Option<(u64, usize)> { + let mut value = 0_u64; + let mut shift = 0_u32; + loop { + let byte = *data.get(offset)?; + offset += 1; + value |= u64::from(byte & 0x7f) << shift; + if byte & 0x80 == 0 { + return Some((value, offset)); + } + shift += 7; + if shift >= 64 { + return None; + } + } +} + +pub(super) fn cursor_message_text(content: Option<&Value>) -> String { + match content { + Some(Value::String(text)) => text.clone(), + Some(Value::Array(items)) => items + .iter() + .filter(|item| item.get("type").and_then(Value::as_str) == Some("text")) + .filter_map(|item| item.get("text").and_then(Value::as_str)) + .collect::>() + .join("\n"), + _ => String::new(), + } +} + +pub(super) fn cursor_message_items(content: Option<&Value>) -> impl Iterator { + content.and_then(Value::as_array).into_iter().flatten() +} + +pub(super) fn clean_cursor_user_text(raw: &str) -> Option { + let trimmed = raw.trim(); + if trimmed.is_empty() || trimmed.starts_with("") { + return None; + } + let inner = if let Some(start) = trimmed.find("") { + let rest = &trimmed[start + "".len()..]; + rest.split_once("") + .map_or(rest, |(inner, _)| inner) + } else { + trimmed + }; + let mut clean = trim_cursor_edges(inner); + if let Some(request) = clean.strip_prefix("USER REQUEST:") { + let cut = [request.find("\n---"), request.find("\\n---")] + .into_iter() + .flatten() + .min() + .unwrap_or(request.len()); + clean = trim_cursor_edges(&request[..cut]); + } + (!clean.is_empty()).then(|| clean.to_string()) +} + +pub(super) fn trim_cursor_edges(mut text: &str) -> &str { + loop { + let before = text; + text = text.trim(); + text = text.strip_prefix("\\n").unwrap_or(text); + text = text.strip_suffix("\\n").unwrap_or(text); + if before == text { + return text; + } + } +} + +pub(super) fn split_think_blocks(text: &str) -> (Vec, String) { + let mut thoughts = Vec::new(); + let mut visible = String::new(); + let mut rest = text; + while let Some(start) = rest.find("") { + visible.push_str(&rest[..start]); + let after = &rest[start + "".len()..]; + if let Some(end) = after.find("") { + let thought = after[..end].trim(); + if !thought.is_empty() { + thoughts.push(thought.to_string()); + } + rest = &after[end + "".len()..]; + } else { + let thought = after.trim(); + if !thought.is_empty() { + thoughts.push(thought.to_string()); + } + rest = ""; + } + } + visible.push_str(rest); + (thoughts, visible) +} + +pub(super) fn cursor_tool_call(item: &Value, created_at: &str) -> Option { + let call_id = item.get("toolCallId")?.as_str()?.to_string(); + let raw_name = item.get("toolName")?.as_str()?.to_string(); + let args = item.get("args").cloned().unwrap_or_else(|| json!({})); + let (canonical_name, args) = normalize_cursor_tool(&raw_name, args); + Some(ImportedToolCall { + call_id, + raw_name, + canonical_name, + args, + created_at: created_at.to_string(), + }) +} + +pub(super) fn normalize_cursor_tool(raw_name: &str, args: Value) -> (String, Value) { + match raw_name { + "shell" | "bash" | "run_terminal_cmd" => { + let command = args + .get("command") + .and_then(Value::as_str) + .or_else(|| args.get("cmd").and_then(Value::as_str)) + .unwrap_or_default(); + ( + imported_history::FUNCTION_RUN_COMMAND_LINE.to_string(), + json!({"command":command,"cmd":command,"payload":args}), + ) + } + "search_replace" | "edit_file" | "write" | "write_file" | "create_file" | "multi_edit" + | "MultiEdit" | "apply_patch" => { + let file_path = args + .get("file_path") + .and_then(Value::as_str) + .or_else(|| args.get("filePath").and_then(Value::as_str)) + .or_else(|| args.get("target_file").and_then(Value::as_str)) + .or_else(|| args.get("path").and_then(Value::as_str)) + .unwrap_or_default(); + ( + imported_history::FUNCTION_EDIT_FILE.to_string(), + json!({"action":raw_name,"file_path":file_path,"payload":args}), + ) + } + _ => (raw_name.to_string(), args), + } +} + +pub(super) fn cursor_tool_result_text(result: Option<&Value>) -> String { + match result { + Some(Value::String(text)) => text.clone(), + Some(Value::Null) | None => String::new(), + Some(value) => value.to_string(), + } +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/mod.rs new file mode 100644 index 000000000..c865a6ba8 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/mod.rs @@ -0,0 +1,1051 @@ +//! Bounded replay adapters for structured SQLite stores. +//! +//! Cursor CLI stores an append-oriented message manifest backed by content +//! addressed blobs. Warp stores mutable protobuf task rows. They share the +//! ORGII compact replay tables, but deliberately do not share a fake rowid +//! cursor: their lineage and reconciliation rules are different. + +use std::collections::{HashMap, HashSet}; +use std::path::Path; +use std::sync::LazyLock; + +use core_types::activity::ActivityChunk; +use prost_reflect::{DescriptorPool, DynamicMessage}; +use rusqlite::{params, Connection, OpenFlags, OptionalExtension, Transaction}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; + +#[cfg(test)] +thread_local! { + static PAYLOAD_FALLBACK_DECODES: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +use crate::development_artifact::{ + attach_replay_git_artifacts, parse_git_artifacts_from_tool_payload, +}; +use crate::sources::imported_history::{self, ImportedToolCall}; + +use crate::sources::imported_history::replay::drivers::common::{ + content_digest, range_from_text, rebuild_turns, ContentDigest, +}; +use crate::sources::imported_history::replay::index::ReplayIndexState; +use crate::sources::imported_history::replay::payload_artifact; +mod common; +mod cursor_cli; +mod warp; + +use common::{ + camel_to_snake, chunk_field_text, field, field_str, head_preview, hex_decode, hex_encode, + open_source_db, parse_warp_timestamp_ms, stable_event_id, timestamp_value_to_iso, + value_at_path_mut, +}; +use cursor_cli::*; +use warp::*; + +use crate::sources::imported_history::replay::{ + replay_payload_body_projection, ImportedHistorySourceId, ReplayPayloadBodyProjection, + ReplayPayloadDescriptor, ReplayPayloadEncoding, ReplayPayloadKind, ReplayPayloadRange, + ReplayStats, NORMAL_PAYLOAD_PREVIEW_BYTES, SHELL_PAYLOAD_PREVIEW_BYTES, +}; + +const CURSOR_PROVIDER: &str = "cursorcli"; +const WARP_PROVIDER: &str = "warp"; +const WARP_TASK_PROTO_NAME: &str = "warp.multi_agent.v1.Task"; +const WARP_FILE_DESCRIPTOR_SET: &[u8] = include_bytes!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../proto/warp_multi_agent_v1.descriptor.pb" +)); + +static WARP_DESCRIPTOR_POOL: LazyLock> = LazyLock::new(|| { + DescriptorPool::decode(WARP_FILE_DESCRIPTOR_SET) + .map_err(|err| format!("load Warp protobuf descriptor: {err}")) +}); + +#[derive(Debug, Clone)] +pub(in crate::sources::imported_history::replay) struct StructuredSyncOutcome { + pub stats: ReplayStats, + pub driver_cursor_json: String, + pub total_events: u64, + pub total_turns: u64, + pub changed: bool, + pub removed_event_ids: Vec, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(default)] +struct CursorCliReplayCursor { + schema_version: i64, + driver: String, + root_blob_id: String, + message_count: u64, + manifest_prefix_hash: String, + cursor_turn_index: i64, + last_user_text: Option, + pending_cursor_calls: HashMap, + // Warp logical row summary. + source_row_count: u64, + source_signal: String, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(default)] +struct WarpReplayCursor { + schema_version: i64, + driver: String, + root_blob_id: String, + message_count: u64, + manifest_prefix_hash: String, + cursor_turn_index: i64, + last_user_text: Option, + pending_cursor_calls: HashMap, + source_row_count: u64, + source_signal: String, +} + +#[derive(Debug, Deserialize)] +struct StructuredCursorVersion { + schema_version: i64, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct PendingCursorCall { + call_blob_id: String, + item_index: usize, + manifest_ordinal: u64, + event_id: String, + turn_index: i64, +} + +#[derive(Debug, Clone, Default, Deserialize)] +#[serde(default, rename_all = "camelCase")] +struct CursorMeta { + latest_root_blob_id: String, + created_at: i64, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "driver", rename_all = "snake_case")] +enum StructuredPayloadLocator { + Cursor { + call_blob_id: String, + result_blob_id: Option, + item_index: usize, + result_item_index: Option, + event_kind: CursorEventKind, + segment_index: usize, + }, + Warp { + task_row_id: String, + task_id: String, + local_event_index: usize, + fallback_ms: i64, + }, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +enum CursorEventKind { + User, + AssistantVisible, + Thinking, + Tool, +} + +#[derive(Debug)] +struct EmittedChunk { + event_key: String, + turn_index: i64, + chunk: ActivityChunk, + locator: StructuredPayloadLocator, +} + +#[derive(Debug)] +struct DeferredPayloadBody { + field_path: String, + text: String, +} + +#[derive(Debug)] +struct WarpSummary { + row_count: u64, + signal: String, + fallback_ms: i64, +} + +pub(in crate::sources::imported_history::replay) fn cursor_schema_version( + cursor_json: &str, +) -> Option { + serde_json::from_str::(cursor_json) + .ok() + .map(|cursor| cursor.schema_version) +} + +pub(in crate::sources::imported_history::replay) fn database_schema_version( + path: &Path, +) -> Result { + let conn = open_source_db(path)?; + conn.query_row("PRAGMA schema_version", [], |row| row.get(0)) + .map_err(|err| format!("read structured replay schema version: {err}")) +} + +/// Cursor roots are immutable content-addressed blobs. A new root is an +/// append only when every previously indexed message id remains an identical +/// prefix. Forks, reordered roots and root replacement return `false`, which +/// makes the replay coordinator publish a new generation atomically. +pub(in crate::sources::imported_history::replay) fn cursor_lineage_matches( + path: &Path, + cursor_json: &str, +) -> Result { + let cursor = serde_json::from_str::(cursor_json) + .map_err(|err| format!("decode Cursor CLI replay cursor: {err}"))?; + if cursor.driver != "cursor_cli" || cursor.message_count == 0 { + return Ok(true); + } + let conn = open_source_db(path)?; + let meta = read_cursor_meta(&conn)?; + if meta.latest_root_blob_id == cursor.root_blob_id { + return Ok(true); + } + let root = read_blob(&conn, &meta.latest_root_blob_id)? + .ok_or_else(|| "Cursor CLI root blob is missing".to_string())?; + let (count, prefix_hash) = manifest_prefix_hash(&root, cursor.message_count)?; + Ok(count >= cursor.message_count && prefix_hash == cursor.manifest_prefix_hash) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn sync( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + previous_state: Option<&ReplayIndexState>, +) -> Result { + ensure_structured_tables(tx)?; + match source { + ImportedHistorySourceId::CursorCli => sync_cursor_cli( + tx, + display_session_id, + source_session_id, + source_path, + generation, + write_revision, + previous_state, + ), + ImportedHistorySourceId::Warp => sync_warp( + tx, + display_session_id, + source_session_id, + source_path, + generation, + write_revision, + previous_state, + ), + _ => Err(format!( + "{} is not a structured replay adapter", + source.as_str() + )), + } +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn reconcile_structured_row( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + write_revision: u64, + row_id: &str, + task_id: &str, + chunks: Vec, + fallback_ms: i64, + stats: &mut ReplayStats, + removed_event_ids: &mut Vec, +) -> Result<(), String> { + let mut previous = HashMap::::new(); + { + let mut stmt = tx + .prepare( + "SELECT local_key,event_id,sequence FROM imported_replay_structured_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND source_key=?4", + ) + .map_err(|err| format!("prepare prior Warp replay events: {err}"))?; + let mut rows = stmt + .query(params![ + source.as_str(), + source_session_id, + generation, + row_id + ]) + .map_err(|err| format!("query prior Warp replay events: {err}"))?; + while let Some(row) = rows.next().map_err(|err| err.to_string())? { + previous.insert( + row.get::<_, String>(0).map_err(|err| err.to_string())?, + ( + row.get::<_, String>(1).map_err(|err| err.to_string())?, + row.get::<_, i64>(2).map_err(|err| err.to_string())?, + ), + ); + } + } + let mut seen = HashSet::new(); + for (local_event_index, chunk) in chunks.into_iter().enumerate() { + let local_key = format!("event:{local_event_index}"); + seen.insert(local_key.clone()); + let sequence = previous + .get(&local_key) + .map(|(_, sequence)| *sequence) + .unwrap_or(next_sequence(tx, source, source_session_id, generation)?); + let event_id = stable_event_id( + source, + source_session_id, + &format!("task-row:{row_id}:{local_key}"), + ); + if let Some((old_event_id, _)) = previous.get(&local_key) { + if old_event_id != &event_id { + delete_event(tx, source, source_session_id, generation, old_event_id)?; + removed_event_ids.push(old_event_id.clone()); + } + } + let emitted = EmittedChunk { + event_key: format!("task-row:{row_id}:{local_key}"), + turn_index: 0, + chunk, + locator: StructuredPayloadLocator::Warp { + task_row_id: row_id.to_string(), + task_id: task_id.to_string(), + local_event_index, + fallback_ms, + }, + }; + upsert_emitted_at_sequence( + tx, + source, + source_session_id, + generation, + write_revision, + sequence, + event_id.clone(), + emitted, + stats, + )?; + tx.execute( + "INSERT INTO imported_replay_structured_events( + source,source_session_id,generation,source_key,local_key,event_id,sequence + ) VALUES (?1,?2,?3,?4,?5,?6,?7) + ON CONFLICT(source,source_session_id,generation,source_key,local_key) DO UPDATE SET + event_id=excluded.event_id,sequence=excluded.sequence", + params![ + source.as_str(), + source_session_id, + generation, + row_id, + local_key, + event_id, + sequence + ], + ) + .map_err(|err| format!("publish Warp replay event mapping: {err}"))?; + } + for (local_key, (event_id, _)) in previous { + if seen.contains(&local_key) { + continue; + } + delete_event(tx, source, source_session_id, generation, &event_id)?; + tx.execute( + "DELETE FROM imported_replay_structured_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND source_key=?4 AND local_key=?5", + params![ + source.as_str(), + source_session_id, + generation, + row_id, + local_key + ], + ) + .map_err(|err| format!("delete stale Warp event mapping: {err}"))?; + removed_event_ids.push(event_id); + } + Ok(()) +} + +fn remove_missing_structured_rows( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + removed_event_ids: &mut Vec, +) -> Result { + let mut missing = Vec::new(); + { + let mut stmt = tx + .prepare( + "SELECT source_key FROM imported_replay_structured_rows r + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND NOT EXISTS ( + SELECT 1 FROM imported_structured_seen_rows s WHERE s.source_key=r.source_key + )", + ) + .map_err(|err| format!("prepare deleted structured rows: {err}"))?; + let mut rows = stmt + .query(params![source.as_str(), source_session_id, generation]) + .map_err(|err| format!("query deleted structured rows: {err}"))?; + while let Some(row) = rows.next().map_err(|err| err.to_string())? { + missing.push(row.get::<_, String>(0).map_err(|err| err.to_string())?); + } + } + for source_key in &missing { + let mut event_ids = Vec::new(); + { + let mut stmt = tx + .prepare( + "SELECT event_id FROM imported_replay_structured_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND source_key=?4", + ) + .map_err(|err| format!("prepare deleted structured events: {err}"))?; + let mut rows = stmt + .query(params![ + source.as_str(), + source_session_id, + generation, + source_key + ]) + .map_err(|err| format!("query deleted structured events: {err}"))?; + while let Some(row) = rows.next().map_err(|err| err.to_string())? { + event_ids.push(row.get::<_, String>(0).map_err(|err| err.to_string())?); + } + } + for event_id in event_ids { + delete_event(tx, source, source_session_id, generation, &event_id)?; + removed_event_ids.push(event_id); + } + tx.execute( + "DELETE FROM imported_replay_structured_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND source_key=?4", + params![source.as_str(), source_session_id, generation, source_key], + ) + .map_err(|err| format!("delete structured event mappings: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_structured_rows + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND source_key=?4", + params![source.as_str(), source_session_id, generation, source_key], + ) + .map_err(|err| format!("delete structured row hash: {err}"))?; + } + Ok(!missing.is_empty()) +} + +pub(in crate::sources::imported_history::replay) fn read_payload( + source: ImportedHistorySourceId, + source_path: &Path, + payloads_json: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + #[cfg(test)] + PAYLOAD_FALLBACK_DECODES.with(|count| count.set(count.get().saturating_add(1))); + let payloads: Vec = serde_json::from_str(payloads_json) + .map_err(|err| format!("decode structured payload locators: {err}"))?; + let payload = payloads + .iter() + .find(|payload| payload.field_path == field_path) + .ok_or_else(|| "Replay payload field is not range-backed".to_string())?; + let locator: StructuredPayloadLocator = serde_json::from_str( + payload + .source_key + .as_deref() + .ok_or_else(|| "Structured replay payload has no locator".to_string())?, + ) + .map_err(|err| format!("decode structured replay locator: {err}"))?; + let conn = open_source_db(source_path)?; + let chunk = match (&locator, source) { + (StructuredPayloadLocator::Cursor { .. }, ImportedHistorySourceId::CursorCli) => { + reconstruct_cursor_chunk(&conn, event_id, &locator)? + } + ( + StructuredPayloadLocator::Warp { + task_row_id, + task_id, + local_event_index, + fallback_ms, + }, + ImportedHistorySourceId::Warp, + ) => { + let blob = conn + .query_row( + "SELECT task FROM agent_tasks + WHERE CAST(id AS TEXT)=?1 AND task_id=?2 LIMIT 1", + params![task_row_id, task_id], + |row| row.get::<_, Vec>(0), + ) + .optional() + .map_err(|err| format!("read Warp payload task: {err}"))? + .ok_or_else(|| "Warp payload task no longer exists".to_string())?; + normalize_warp_task("warpapp-payload", &blob, *fallback_ms)? + .into_iter() + .nth(*local_event_index) + .ok_or_else(|| "Warp payload event no longer exists".to_string())? + } + _ => return Err("Structured replay payload/source mismatch".to_string()), + }; + let text = chunk_field_text(&chunk, field_path)?; + range_from_text(event_id, field_path, &text, offset, max_bytes) +} + +#[cfg(test)] +pub(in crate::sources::imported_history::replay) fn reset_payload_fallback_decodes() { + PAYLOAD_FALLBACK_DECODES.with(|count| count.set(0)); +} + +#[cfg(test)] +pub(in crate::sources::imported_history::replay) fn payload_fallback_decodes() -> usize { + PAYLOAD_FALLBACK_DECODES.with(std::cell::Cell::get) +} + +fn upsert_emitted( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + write_revision: u64, + emitted: EmittedChunk, + stats: &mut ReplayStats, +) -> Result<(), String> { + let event_id = if emitted.chunk.chunk_id.trim().is_empty() { + stable_event_id(source, source_session_id, &emitted.event_key) + } else { + // Imported chunk builders create ids from transient ordinals. Only a + // previously persisted tool-call id is authoritative here. + let expected = stable_event_id(source, source_session_id, &emitted.event_key); + if emitted.chunk.chunk_id.starts_with("replay-") { + emitted.chunk.chunk_id.clone() + } else { + expected + } + }; + let sequence = tx + .query_row( + "SELECT sequence FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![source.as_str(), source_session_id, generation, event_id], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|err| format!("resolve structured replay sequence: {err}"))? + .unwrap_or(next_sequence(tx, source, source_session_id, generation)?); + upsert_emitted_at_sequence( + tx, + source, + source_session_id, + generation, + write_revision, + sequence, + event_id, + emitted, + stats, + ) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn upsert_emitted_at_sequence( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + write_revision: u64, + sequence: i64, + event_id: String, + mut emitted: EmittedChunk, + stats: &mut ReplayStats, +) -> Result<(), String> { + emitted.chunk.chunk_id = event_id.clone(); + let content_hash = content_digest(&[serde_json::to_string(&emitted.chunk) + .unwrap_or_default() + .as_bytes()]); + let previous_hash = tx + .query_row( + "SELECT content_hash FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![source.as_str(), source_session_id, generation, event_id], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|err| format!("read structured replay event hash: {err}"))?; + if previous_hash.as_deref() == Some(content_hash.as_str()) { + return Ok(()); + } + let locator_json = serde_json::to_string(&emitted.locator) + .map_err(|err| format!("encode structured payload locator: {err}"))?; + payload_artifact::delete_event_refs(tx, source, source_session_id, generation, &event_id)?; + let (payloads, deferred_bodies) = compact_chunk_with_bodies(&mut emitted.chunk, &locator_json); + let args_json = serde_json::to_string(&emitted.chunk.args) + .map_err(|err| format!("encode structured replay args: {err}"))?; + let result_json = serde_json::to_string(&emitted.chunk.result) + .map_err(|err| format!("encode structured replay result: {err}"))?; + let payloads_json = serde_json::to_string(&payloads) + .map_err(|err| format!("encode structured payload descriptors: {err}"))?; + tx.execute( + "INSERT INTO imported_replay_events( + source,source_session_id,generation,sequence,event_id,turn_index, + action_type,function_name,created_at,args_preview_json,result_preview_json, + args_size_bytes,result_size_bytes,thread_id,process_id,source_start,source_end, + payloads_json,content_hash,event_revision + ) VALUES (?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13,?14,?15,0,0,?16,?17,?18) + ON CONFLICT(source,source_session_id,generation,event_id) DO UPDATE SET + turn_index=excluded.turn_index,action_type=excluded.action_type, + function_name=excluded.function_name,created_at=excluded.created_at, + args_preview_json=excluded.args_preview_json,result_preview_json=excluded.result_preview_json, + args_size_bytes=excluded.args_size_bytes,result_size_bytes=excluded.result_size_bytes, + thread_id=excluded.thread_id,process_id=excluded.process_id, + payloads_json=excluded.payloads_json,content_hash=excluded.content_hash, + event_revision=excluded.event_revision", + params![ + source.as_str(), + source_session_id, + generation, + sequence, + event_id, + emitted.turn_index, + emitted.chunk.action_type, + emitted.chunk.function, + emitted.chunk.created_at, + args_json, + result_json, + emitted.chunk.args.to_string().len() as i64, + emitted.chunk.result.to_string().len() as i64, + emitted.chunk.thread_id, + emitted.chunk.process_id, + payloads_json, + content_hash, + write_revision.min(i64::MAX as u64) as i64, + ], + ) + .map_err(|err| format!("upsert structured replay event: {err}"))?; + for body in deferred_bodies { + payload_artifact::store_text( + tx, + source, + source_session_id, + generation, + &event_id, + &body.field_path, + &body.text, + )?; + } + stats.normalized_events = stats.normalized_events.saturating_add(1); + stats.upserted_events = stats.upserted_events.saturating_add(1); + Ok(()) +} + +pub(in crate::sources::imported_history::replay) fn compact_chunk( + chunk: &mut ActivityChunk, + locator_json: &str, +) -> Vec { + compact_chunk_with_bodies(chunk, locator_json).0 +} + +fn compact_chunk_with_bodies( + chunk: &mut ActivityChunk, + locator_json: &str, +) -> (Vec, Vec) { + let mut payloads = Vec::new(); + let mut deferred_bodies = Vec::new(); + // Metadata projection runs over the compact rows, not the source-backed + // payload. Preserve the cheap scalar edit summary and extract Git data + // before any large args/result field is replaced by a preview. + let edit_summary_will_change = compact_edit_line_summary_will_change(chunk); + let mut exact_result_body = edit_summary_will_change + .then(|| serde_json::to_string(&chunk.result).unwrap_or_else(|_| "null".to_string())); + attach_compact_edit_line_summary(chunk); + let git_artifacts = if chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE { + let args = serde_json::to_string(&chunk.args).unwrap_or_else(|_| "null".to_string()); + let result = serde_json::to_string(&chunk.result).unwrap_or_else(|_| "null".to_string()); + let artifacts = parse_git_artifacts_from_tool_payload(&args, &result); + if !artifacts.is_empty() { + exact_result_body = Some(result); + } + artifacts + } else { + Vec::new() + }; + let result_limit = if chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let result_body_projection = exact_result_body.as_deref().and_then(|encoded| { + replay_payload_body_projection( + "result", + &chunk.result, + Some(encoded), + result_limit, + chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE, + ) + }); + let fields: &[(&str, ReplayPayloadKind)] = match chunk.function.as_str() { + imported_history::FUNCTION_USER_MESSAGE => { + &[("result.message.content", ReplayPayloadKind::UserMessage)] + } + imported_history::FUNCTION_ASSISTANT => &[ + ("result.content", ReplayPayloadKind::AssistantContent), + ("result.observation", ReplayPayloadKind::AssistantContent), + ], + imported_history::FUNCTION_THINKING => &[ + ("result.thought", ReplayPayloadKind::Reasoning), + ("result.content", ReplayPayloadKind::Reasoning), + ("result.observation", ReplayPayloadKind::Reasoning), + ], + _ => &[ + ("result.output", ReplayPayloadKind::ToolOutput), + ("result.observation", ReplayPayloadKind::ToolOutput), + ("result.old_content", ReplayPayloadKind::ToolDiff), + ("result.new_content", ReplayPayloadKind::ToolDiff), + ], + }; + for &(path, kind) in fields { + let Some(text) = value_at_path_mut(&mut chunk.result, path.trim_start_matches("result.")) + else { + continue; + }; + if text.len() > result_limit { + let total_bytes = text.len() as u64; + let full_text = std::mem::take(text); + *text = head_preview(&full_text, result_limit); + payloads.push(payload_descriptor( + path, + kind, + ReplayPayloadEncoding::Utf8Text, + None, + locator_json, + total_bytes, + )); + deferred_bodies.push(DeferredPayloadBody { + field_path: path.to_string(), + text: full_text, + }); + } + } + let encoded_args = serde_json::to_string(&chunk.args).unwrap_or_default(); + let args_size = encoded_args.len(); + let args_limit = if chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE { + SHELL_PAYLOAD_PREVIEW_BYTES + } else { + NORMAL_PAYLOAD_PREVIEW_BYTES + }; + if args_size > args_limit { + let args_body_projection = replay_payload_body_projection( + "args", + &chunk.args, + Some(&encoded_args), + args_limit, + false, + ); + let mut original = std::mem::take(&mut chunk.args); + let mut preview = serde_json::Map::new(); + preview.insert("_replayTruncated".to_string(), Value::Bool(true)); + for key in [ + "command", + "cmd", + "path", + "file_path", + "filePath", + "action", + "query", + "pattern", + "linesAdded", + "linesRemoved", + "operation", + ] { + let Some(value) = original.get_mut(key) else { + continue; + }; + if let Value::String(text) = value { + if text.len() > args_limit { + let full_text = std::mem::take(text); + preview.insert( + key.to_string(), + Value::String(head_preview(&full_text, args_limit)), + ); + } else { + preview.insert(key.to_string(), Value::String(text.clone())); + } + } else if value.is_number() || value.is_boolean() || value.is_null() { + preview.insert(key.to_string(), value.clone()); + } + } + if preview.len() == 1 { + preview.insert( + "_preview".to_string(), + Value::String(head_preview(&encoded_args, args_limit)), + ); + } + chunk.args = Value::Object(preview); + // A semantic preview is necessarily incomplete even when it retained + // one or more recognized scalars (for example `path`). Use one root + // body so hydration/export atomically restores every omitted sibling + // and removes compact-only markers. Nested descriptors would conflict + // with that root replacement and are deliberately discarded. + payloads.retain(|payload| !field_path_is_under(&payload.field_path, "args")); + deferred_bodies.retain(|body| !field_path_is_under(&body.field_path, "args")); + payloads.push(payload_descriptor( + "args", + ReplayPayloadKind::ToolArguments, + ReplayPayloadEncoding::JsonValue, + args_body_projection, + locator_json, + args_size as u64, + )); + deferred_bodies.push(DeferredPayloadBody { + field_path: "args".to_string(), + text: encoded_args, + }); + } + attach_replay_git_artifacts(&mut chunk.result, &git_artifacts); + if let Some(exact_result_body) = exact_result_body { + // Derived edit counts and `_replayGitArtifacts` stay in the compact + // row for metadata projection, but were never provider result fields. + // A canonical root result makes compatibility hydration/export exact. + payloads.retain(|payload| !field_path_is_under(&payload.field_path, "result")); + deferred_bodies.retain(|body| !field_path_is_under(&body.field_path, "result")); + payloads.push(payload_descriptor( + "result", + ReplayPayloadKind::ToolOutput, + ReplayPayloadEncoding::JsonValue, + result_body_projection, + locator_json, + exact_result_body.len() as u64, + )); + deferred_bodies.push(DeferredPayloadBody { + field_path: "result".to_string(), + text: exact_result_body, + }); + } + (payloads, deferred_bodies) +} + +fn field_path_is_under(field_path: &str, root: &str) -> bool { + field_path == root + || field_path + .strip_prefix(root) + .is_some_and(|suffix| suffix.starts_with('.')) +} + +fn compact_edit_line_summary_will_change(chunk: &ActivityChunk) -> bool { + if chunk.function != imported_history::FUNCTION_EDIT_FILE { + return false; + } + let Some(result) = chunk.result.as_object() else { + return false; + }; + let scalar = |key: &str| chunk.args.get(key).and_then(Value::as_u64); + let line_count = |keys: &[&str]| { + keys.iter() + .find_map(|key| chunk.args.get(*key).and_then(Value::as_str)) + .map(|text| { + if text.is_empty() { + 0 + } else { + text.lines().count() as u64 + } + }) + }; + let additions = scalar("linesAdded") + .or_else(|| line_count(&["new_string", "content", "insert_text", "file_text"])); + let removals = scalar("linesRemoved").or_else(|| line_count(&["old_string", "old_str"])); + (additions.is_some() && !result.contains_key("linesAdded")) + || (removals.is_some() && !result.contains_key("linesRemoved")) +} + +fn attach_compact_edit_line_summary(chunk: &mut ActivityChunk) { + if chunk.function != imported_history::FUNCTION_EDIT_FILE { + return; + } + let scalar = |key: &str| chunk.args.get(key).and_then(Value::as_u64); + let line_count = |keys: &[&str]| { + keys.iter() + .find_map(|key| chunk.args.get(*key).and_then(Value::as_str)) + .map(|text| { + if text.is_empty() { + 0 + } else { + text.lines().count() as u64 + } + }) + }; + let additions = scalar("linesAdded") + .or_else(|| line_count(&["new_string", "content", "insert_text", "file_text"])); + let removals = scalar("linesRemoved").or_else(|| line_count(&["old_string", "old_str"])); + let Some(result) = chunk.result.as_object_mut() else { + return; + }; + if let Some(additions) = additions { + result + .entry("linesAdded".to_string()) + .or_insert_with(|| json!(additions)); + } + if let Some(removals) = removals { + result + .entry("linesRemoved".to_string()) + .or_insert_with(|| json!(removals)); + } +} + +fn payload_descriptor( + field_path: &str, + kind: ReplayPayloadKind, + encoding: ReplayPayloadEncoding, + body_projection: Option, + locator_json: &str, + total_bytes: u64, +) -> ReplayPayloadDescriptor { + ReplayPayloadDescriptor { + field_path: field_path.to_string(), + kind, + encoding, + body_projection, + spans: Vec::new(), + total_bytes, + source_ordinal: None, + source_key: Some(locator_json.to_string()), + } +} + +fn ensure_structured_tables(tx: &Transaction<'_>) -> Result<(), String> { + tx.execute_batch( + "CREATE TABLE IF NOT EXISTS imported_replay_structured_rows( + source TEXT NOT NULL,source_session_id TEXT NOT NULL,generation TEXT NOT NULL, + source_key TEXT NOT NULL,content_hash TEXT NOT NULL,seen_revision INTEGER NOT NULL, + PRIMARY KEY(source,source_session_id,generation,source_key) + ); + CREATE TABLE IF NOT EXISTS imported_replay_structured_events( + source TEXT NOT NULL,source_session_id TEXT NOT NULL,generation TEXT NOT NULL, + source_key TEXT NOT NULL,local_key TEXT NOT NULL,event_id TEXT NOT NULL,sequence INTEGER NOT NULL, + PRIMARY KEY(source,source_session_id,generation,source_key,local_key) + );", + ) + .map_err(|err| format!("create structured replay tables: {err}")) +} + +fn unchanged_outcome( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + cursor: C, +) -> Result { + finish_outcome( + tx, + source, + source_session_id, + generation, + cursor, + ReplayStats::default(), + false, + Vec::new(), + ) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn finish_outcome( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + cursor: C, + stats: ReplayStats, + changed: bool, + removed_event_ids: Vec, +) -> Result { + Ok(StructuredSyncOutcome { + driver_cursor_json: serde_json::to_string(&cursor) + .map_err(|err| format!("encode structured replay cursor: {err}"))?, + total_events: count_rows( + tx, + "imported_replay_events", + source, + source_session_id, + generation, + )?, + total_turns: count_rows( + tx, + "imported_replay_turns", + source, + source_session_id, + generation, + )?, + stats, + changed, + removed_event_ids, + }) +} + +fn count_rows( + tx: &Transaction<'_>, + table: &str, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result { + tx.query_row( + &format!( + "SELECT COUNT(*) FROM {table} WHERE source=?1 AND source_session_id=?2 AND generation=?3" + ), + params![source.as_str(), source_session_id, generation], + |row| row.get::<_, i64>(0), + ) + .map(|count| count.max(0) as u64) + .map_err(|err| format!("count structured replay {table}: {err}")) +} + +fn next_sequence( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result { + tx.query_row( + "SELECT COALESCE(MAX(sequence),-1)+1 FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![source.as_str(), source_session_id, generation], + |row| row.get(0), + ) + .map_err(|err| format!("allocate structured replay sequence: {err}")) +} + +fn delete_event( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + event_id: &str, +) -> Result<(), String> { + payload_artifact::delete_event_refs(tx, source, source_session_id, generation, event_id)?; + tx.execute( + "DELETE FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![source.as_str(), source_session_id, generation, event_id], + ) + .map(|_| ()) + .map_err(|err| format!("delete structured replay event: {err}")) +} + +#[cfg(test)] +#[path = "../../tests/structured_driver.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/warp.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/warp.rs new file mode 100644 index 000000000..d8d0925f5 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/structured/warp.rs @@ -0,0 +1,435 @@ +use super::*; + +pub(super) fn sync_warp( + tx: &Transaction<'_>, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + previous_state: Option<&ReplayIndexState>, +) -> Result { + let source = ImportedHistorySourceId::Warp; + let source_conn = open_source_db(source_path)?; + validate_warp_schema(&source_conn)?; + let schema_version = source_conn + .query_row("PRAGMA schema_version", [], |row| row.get::<_, i64>(0)) + .map_err(|err| format!("read Warp schema version: {err}"))?; + let summary = warp_summary(&source_conn, source_session_id, source_path)?; + let previous_cursor = previous_state + .map(|state| serde_json::from_str::(&state.driver_cursor_json)) + .transpose() + .map_err(|err| format!("decode Warp replay cursor: {err}"))? + .filter(|cursor| cursor.driver == "warp") + .unwrap_or_default(); + if previous_state.is_some() + && previous_cursor.source_row_count == summary.row_count + && previous_cursor.source_signal == summary.signal + { + return unchanged_outcome(tx, source, source_session_id, generation, previous_cursor); + } + + tx.execute_batch( + "CREATE TEMP TABLE IF NOT EXISTS imported_structured_seen_rows( + source_key TEXT PRIMARY KEY + ) WITHOUT ROWID; + DELETE FROM imported_structured_seen_rows;", + ) + .map_err(|err| format!("prepare Warp replay seen rows: {err}"))?; + + let mut stats = ReplayStats::default(); + let mut changed = false; + let mut removed_event_ids = Vec::new(); + let mut stmt = source_conn + .prepare( + "SELECT CAST(id AS TEXT), COALESCE(task_id, CAST(id AS TEXT)), task + FROM agent_tasks WHERE conversation_id=?1 ORDER BY id ASC", + ) + .map_err(|err| format!("prepare Warp task replay stream: {err}"))?; + let mut rows = stmt + .query([source_session_id]) + .map_err(|err| format!("query Warp task replay stream: {err}"))?; + while let Some(row) = rows + .next() + .map_err(|err| format!("stream Warp task replay row: {err}"))? + { + let row_id: String = row.get(0).map_err(|err| err.to_string())?; + let task_id: String = row.get(1).map_err(|err| err.to_string())?; + let blob: Vec = row.get(2).map_err(|err| err.to_string())?; + tx.execute( + "INSERT OR IGNORE INTO imported_structured_seen_rows(source_key) VALUES (?1)", + [&row_id], + ) + .map_err(|err| format!("mark Warp replay row seen: {err}"))?; + let content_hash = content_digest(&[task_id.as_bytes(), &blob]); + let previous_hash = tx + .query_row( + "SELECT content_hash FROM imported_replay_structured_rows + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND source_key=?4", + params![source.as_str(), source_session_id, generation, row_id], + |db_row| db_row.get::<_, String>(0), + ) + .optional() + .map_err(|err| format!("read Warp replay row hash: {err}"))?; + if previous_hash.as_deref() == Some(content_hash.as_str()) { + continue; + } + + stats.parsed_rows = stats.parsed_rows.saturating_add(1); + stats.parsed_bytes = stats.parsed_bytes.saturating_add(blob.len() as u64); + let chunks = normalize_warp_task(display_session_id, &blob, summary.fallback_ms)?; + let upserts_before = stats.upserted_events; + let removals_before = removed_event_ids.len(); + reconcile_structured_row( + tx, + source, + source_session_id, + generation, + write_revision, + &row_id, + &task_id, + chunks, + summary.fallback_ms, + &mut stats, + &mut removed_event_ids, + )?; + tx.execute( + "INSERT INTO imported_replay_structured_rows( + source,source_session_id,generation,source_key,content_hash,seen_revision + ) VALUES (?1,?2,?3,?4,?5,?6) + ON CONFLICT(source,source_session_id,generation,source_key) DO UPDATE SET + content_hash=excluded.content_hash,seen_revision=excluded.seen_revision", + params![ + source.as_str(), + source_session_id, + generation, + row_id, + content_hash, + write_revision.min(i64::MAX as u64) as i64 + ], + ) + .map_err(|err| format!("publish Warp replay row hash: {err}"))?; + changed |= + stats.upserted_events > upserts_before || removed_event_ids.len() > removals_before; + } + + let deleted = remove_missing_structured_rows( + tx, + source, + source_session_id, + generation, + &mut removed_event_ids, + )?; + changed |= deleted; + if changed { + payload_artifact::delete_orphans(tx, source, source_session_id, generation)?; + rebuild_turns(tx, source, source_session_id, generation)?; + } + stats.removed_events = removed_event_ids.len() as u64; + let cursor = WarpReplayCursor { + schema_version, + driver: "warp".to_string(), + source_row_count: summary.row_count, + source_signal: summary.signal, + ..WarpReplayCursor::default() + }; + finish_outcome( + tx, + source, + source_session_id, + generation, + cursor, + stats, + changed, + removed_event_ids, + ) +} + +pub(super) fn warp_summary( + conn: &Connection, + source_session_id: &str, + source_path: &Path, +) -> Result { + let (row_count, total_bytes, max_id, max_modified) = conn + .query_row( + "SELECT COUNT(*), COALESCE(SUM(LENGTH(task)),0), + COALESCE(MAX(id),0), COALESCE(MAX(CAST(last_modified_at AS TEXT)),'') + FROM agent_tasks WHERE conversation_id=?1", + [source_session_id], + |row| { + Ok(( + row.get::<_, i64>(0)?.max(0) as u64, + row.get::<_, i64>(1)?.max(0), + row.get::<_, i64>(2)?.max(0), + row.get::<_, String>(3)?, + )) + }, + ) + .map_err(|err| format!("summarize Warp task rows: {err}"))?; + let conversation_modified = conn + .query_row( + "SELECT COALESCE(CAST(last_modified_at AS TEXT),'') FROM agent_conversations + WHERE conversation_id=?1", + [source_session_id], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|err| format!("read Warp conversation watermark: {err}"))? + .unwrap_or_default(); + let fallback_ms = parse_warp_timestamp_ms(&conversation_modified).unwrap_or_default(); + let physical_signal = sqlite_physical_signal(source_path)?; + Ok(WarpSummary { + row_count, + signal: format!( + "{row_count}:{total_bytes}:{max_id}:{max_modified}:{conversation_modified}:{physical_signal}" + ), + fallback_ms, + }) +} + +pub(super) fn sqlite_physical_signal(path: &Path) -> Result { + let mut hash = ContentDigest::default(); + for candidate in [ + path.to_path_buf(), + std::path::PathBuf::from(format!("{}-wal", path.to_string_lossy())), + ] { + hash.update_part(candidate.to_string_lossy().as_bytes()); + match std::fs::metadata(&candidate) { + Ok(metadata) => { + hash.update_part(&metadata.len().to_le_bytes()); + let modified = metadata + .modified() + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map(|duration| { + duration.as_secs() as i64 * 1_000_000_000 + duration.subsec_nanos() as i64 + }) + .unwrap_or_default(); + hash.update_part(&modified.to_le_bytes()); + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + hash.update_part(b"missing") + } + Err(error) => { + return Err(format!( + "stat Warp replay source {}: {error}", + candidate.display() + )); + } + } + } + Ok(hash.finish_hex()) +} + +pub(super) fn normalize_warp_task( + session_id: &str, + blob: &[u8], + fallback_ms: i64, +) -> Result, String> { + let task = decode_warp_task(blob)?; + let fallback_created_at = imported_history::epoch_ms_to_iso(fallback_ms); + let messages = field(&task, &["messages"]) + .and_then(Value::as_array) + .map(Vec::as_slice) + .unwrap_or_default(); + let mut tool_results = HashMap::new(); + for message in messages { + let Some(result) = field(message, &["toolCallResult", "tool_call_result"]) else { + continue; + }; + if let Some(call_id) = field_str(result, &["toolCallId", "tool_call_id"]) { + tool_results.insert(call_id.to_string(), result); + } + } + let mut chunks = Vec::new(); + for (ordinal, message) in messages.iter().enumerate() { + let created_at = field(message, &["timestamp"]) + .and_then(timestamp_value_to_iso) + .unwrap_or_else(|| fallback_created_at.clone()); + if let Some(user_query) = field(message, &["userQuery", "user_query"]) { + if let Some(query) = + field_str(user_query, &["query"]).filter(|text| !text.trim().is_empty()) + { + chunks.push(imported_history::user_message_chunk( + session_id, + WARP_PROVIDER, + ordinal, + &created_at, + query.trim(), + )); + } + continue; + } + if let Some(agent_output) = field(message, &["agentOutput", "agent_output"]) { + if let Some(text) = + field_str(agent_output, &["text"]).filter(|text| !text.trim().is_empty()) + { + chunks.push(imported_history::assistant_message_chunk( + session_id, + WARP_PROVIDER, + ordinal, + &created_at, + text.trim(), + )); + } + continue; + } + if let Some(reasoning) = field(message, &["agentReasoning", "agent_reasoning"]) { + if let Some(text) = + field_str(reasoning, &["reasoning"]).filter(|text| !text.trim().is_empty()) + { + chunks.push(imported_history::thinking_chunk( + session_id, + WARP_PROVIDER, + ordinal, + &created_at, + text.trim(), + )); + } + continue; + } + let Some(tool_call) = field(message, &["toolCall", "tool_call"]) else { + continue; + }; + let Some((raw_name, payload)) = warp_tool_variant(tool_call) else { + continue; + }; + let call_id = field_str(tool_call, &["toolCallId", "tool_call_id"]) + .filter(|id| !id.trim().is_empty()) + .map(str::to_string) + .unwrap_or_else(|| format!("warp-{ordinal}")); + let (canonical_name, args) = normalize_warp_tool_call(raw_name, payload.clone()); + let output = tool_results + .get(&call_id) + .map(|result| warp_tool_result_text(result)) + .unwrap_or_default(); + let call = ImportedToolCall { + call_id, + raw_name: camel_to_snake(raw_name), + canonical_name, + args, + created_at: created_at.clone(), + }; + chunks.push(imported_history::tool_call_chunk( + session_id, + WARP_PROVIDER, + ordinal, + &call, + &output, + )); + } + Ok(chunks) +} + +pub(super) fn decode_warp_task(blob: &[u8]) -> Result { + let pool = WARP_DESCRIPTOR_POOL.as_ref().map_err(Clone::clone)?; + let descriptor = pool + .get_message_by_name(WARP_TASK_PROTO_NAME) + .ok_or_else(|| format!("missing Warp descriptor {WARP_TASK_PROTO_NAME}"))?; + let message = DynamicMessage::decode(descriptor, blob) + .map_err(|err| format!("decode Warp task protobuf: {err}"))?; + serde_json::to_value(message).map_err(|err| format!("project Warp task JSON: {err}")) +} + +pub(super) fn validate_warp_schema(conn: &Connection) -> Result<(), String> { + conn.prepare( + "SELECT id,task_id,task,last_modified_at FROM agent_tasks + WHERE conversation_id='' LIMIT 0", + ) + .and_then(|_| { + conn.prepare("SELECT conversation_id,last_modified_at FROM agent_conversations LIMIT 0") + }) + .map(|_| ()) + .map_err(|err| format!("unsupported Warp replay schema: {err}")) +} + +pub(super) fn warp_tool_variant(tool_call: &Value) -> Option<(&str, &Value)> { + tool_call.as_object()?.iter().find_map(|(key, value)| { + (!matches!(key.as_str(), "toolCallId" | "tool_call_id")).then_some((key.as_str(), value)) + }) +} + +pub(super) fn normalize_warp_tool_call(raw_name: &str, payload: Value) -> (String, Value) { + match raw_name { + "runShellCommand" | "run_shell_command" => { + let command = field_str(&payload, &["command"]).unwrap_or_default(); + ( + imported_history::FUNCTION_RUN_COMMAND_LINE.to_string(), + json!({"command":command,"cmd":command,"payload":payload}), + ) + } + "readFiles" | "read_files" => (imported_history::FUNCTION_READ_FILE.to_string(), payload), + "applyFileDiffs" | "apply_file_diffs" | "editDocuments" | "edit_documents" + | "createDocuments" | "create_documents" => { + let file_path = first_warp_edited_file_path(&payload).unwrap_or_default(); + ( + imported_history::FUNCTION_EDIT_FILE.to_string(), + json!({ + "action":camel_to_snake(raw_name), + "file_path":file_path, + "payload":payload, + }), + ) + } + "grep" | "searchCodebase" | "search_codebase" => { + (imported_history::FUNCTION_CODE_SEARCH.to_string(), payload) + } + "fileGlob" | "file_glob" | "fileGlobV2" | "file_glob_v2" => ( + imported_history::FUNCTION_GLOB_FILE_SEARCH.to_string(), + payload, + ), + _ => (camel_to_snake(raw_name), payload), + } +} + +pub(super) fn first_warp_edited_file_path(payload: &Value) -> Option { + [ + "diffs", + "newFiles", + "new_files", + "deletedFiles", + "deleted_files", + "v4aUpdates", + "v4a_updates", + ] + .iter() + .find_map(|key| { + field(payload, &[*key]) + .and_then(Value::as_array) + .and_then(|rows| rows.first()) + .and_then(|row| field_str(row, &["filePath", "file_path", "documentId", "document_id"])) + .map(str::trim) + .filter(|path| !path.is_empty()) + .map(str::to_string) + }) +} + +pub(super) fn warp_tool_result_text(result: &Value) -> String { + let payload = result + .as_object() + .and_then(|object| { + object.iter().find_map(|(key, value)| { + (!matches!(key.as_str(), "toolCallId" | "tool_call_id")).then_some(value) + }) + }) + .unwrap_or(result); + if let Some(output) = find_warp_output_text(payload) { + return output.to_string(); + } + serde_json::to_string(payload).unwrap_or_default() +} + +pub(super) fn find_warp_output_text(value: &Value) -> Option<&str> { + let object = value.as_object()?; + for key in [ + "output", + "stdout", + "interleavedOutput", + "interleaved_output", + ] { + if let Some(output) = object.get(key).and_then(Value::as_str) { + return Some(output); + } + } + object.values().find_map(find_warp_output_text) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/whole_json.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/whole_json.rs new file mode 100644 index 000000000..606a82c54 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/drivers/whole_json.rs @@ -0,0 +1,969 @@ +//! Streaming whole-document replay adapter for Cline. +//! +//! Cline rewrites one `{ "messages": [...] }` JSON document. A rewrite is a +//! new replay generation, never a byte-offset append. The custom serde seed +//! holds at most one message plus outstanding tool calls, so cold indexing is +//! bounded by the largest message rather than the transcript size. + +use std::collections::HashMap; +use std::fs::File; +use std::io::BufReader; +use std::path::Path; + +use core_types::activity::ActivityChunk; +use rusqlite::{params, Transaction}; +use serde::de::{DeserializeSeed, IgnoredAny, MapAccess, SeqAccess, Visitor}; +use serde::{Deserialize, Deserializer, Serialize}; +use serde_json::{json, Value}; + +use crate::sources::imported_history::{self, ImportedToolCall}; + +use crate::sources::imported_history::replay::drivers::common::{ + content_digest, legacy_stable_id_hash_delimited, range_from_text, rebuild_turns, +}; +use crate::sources::imported_history::replay::drivers::structured::compact_chunk; +use crate::sources::imported_history::replay::index::ReplayIndexState; +use crate::sources::imported_history::replay::payload_artifact; +use crate::sources::imported_history::replay::{ + ImportedHistorySourceId, ReplayPayloadDescriptor, ReplayPayloadRange, ReplayStats, +}; + +const CLINE_PROVIDER: &str = "cline"; + +#[cfg(test)] +thread_local! { + static SYNC_ATTEMPTS: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +#[derive(Debug, Clone)] +pub(in crate::sources::imported_history::replay) struct WholeJsonSyncOutcome { + pub stats: ReplayStats, + pub driver_cursor_json: String, + pub indexed_size_bytes: u64, + pub total_events: u64, + pub total_turns: u64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(default)] +struct WholeJsonCursor { + message_count: u64, + source_bytes: u64, + sample_fingerprint: String, +} + +pub(in crate::sources::imported_history::replay) fn cursor_fingerprint( + cursor_json: &str, +) -> Option { + serde_json::from_str::(cursor_json) + .ok() + .map(|cursor| cursor.sample_fingerprint) +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(default)] +struct ClineMessage { + role: String, + content: Value, + ts: Option, +} + +impl Default for ClineMessage { + fn default() -> Self { + Self { + role: String::new(), + content: Value::Null, + ts: None, + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +enum ClinePayloadLocator { + Text { + message_ordinal: u64, + block_ordinal: usize, + user: bool, + }, + Tool { + call_message_ordinal: u64, + call_block_ordinal: usize, + sub_index: usize, + result_message_ordinal: Option, + result_block_ordinal: Option, + }, +} + +#[derive(Debug)] +struct PendingCall { + raw_name: String, + created_at: String, + call_message_ordinal: u64, + call_block_ordinal: usize, + sub_calls: Vec<(String, Value)>, + sequences: Vec, + turn_index: i64, +} + +struct IndexFold<'tx, 'conn, 'data> { + tx: &'tx Transaction<'conn>, + display_session_id: &'data str, + source_session_id: &'data str, + generation: &'data str, + write_revision: u64, + message_ordinal: u64, + next_sequence: i64, + turn_index: i64, + pending: HashMap, + stats: ReplayStats, +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn sync( + tx: &Transaction<'_>, + display_session_id: &str, + source_session_id: &str, + source_path: &Path, + generation: &str, + write_revision: u64, + _previous_state: Option<&ReplayIndexState>, + sample_fingerprint: &str, +) -> Result { + #[cfg(test)] + SYNC_ATTEMPTS.with(|attempts| attempts.set(attempts.get().saturating_add(1))); + let source_bytes = std::fs::metadata(source_path) + .map_err(|err| format!("stat Cline replay source {}: {err}", source_path.display()))? + .len(); + let mut fold = IndexFold { + tx, + display_session_id, + source_session_id, + generation, + write_revision, + message_ordinal: 0, + next_sequence: 0, + turn_index: -1, + pending: HashMap::new(), + stats: ReplayStats::default(), + }; + visit_messages(source_path, |message| fold_message(&mut fold, message))?; + flush_pending(&mut fold)?; + rebuild_turns( + tx, + ImportedHistorySourceId::Cline, + source_session_id, + generation, + )?; + let total_events = count_rows(tx, "imported_replay_events", source_session_id, generation)?; + let total_turns = count_rows(tx, "imported_replay_turns", source_session_id, generation)?; + Ok(WholeJsonSyncOutcome { + driver_cursor_json: serde_json::to_string(&WholeJsonCursor { + message_count: fold.message_ordinal, + source_bytes, + sample_fingerprint: sample_fingerprint.to_string(), + }) + .map_err(|err| format!("encode Cline replay cursor: {err}"))?, + indexed_size_bytes: source_bytes, + total_events, + total_turns, + stats: fold.stats, + }) +} + +#[cfg(test)] +fn take_sync_attempts() -> u64 { + SYNC_ATTEMPTS.with(|attempts| attempts.replace(0)) +} + +fn fold_message(fold: &mut IndexFold<'_, '_, '_>, message: ClineMessage) -> Result<(), String> { + let message_ordinal = fold.message_ordinal; + fold.message_ordinal = fold.message_ordinal.saturating_add(1); + fold.stats.parsed_rows = fold.stats.parsed_rows.saturating_add(1); + fold.stats.parsed_bytes = fold + .stats + .parsed_bytes + .saturating_add(serde_json::to_vec(&message).map_or(0, |bytes| bytes.len()) as u64); + let created_at = message + .ts + .filter(|timestamp| *timestamp > 0) + .map(imported_history::epoch_ms_to_iso) + .unwrap_or_default(); + let user = message.role == "user"; + for (block_ordinal, block) in content_blocks(&message.content).enumerate() { + match block_type(block) { + "text" => { + let raw = block + .get("text") + .and_then(Value::as_str) + .unwrap_or_default(); + let text = if user { + strip_user_input_wrapper(raw) + } else { + raw.trim() + }; + if text.is_empty() { + continue; + } + if user { + fold.turn_index = fold.turn_index.saturating_add(1).max(0); + } + let chunk = if user { + imported_history::user_message_chunk( + fold.display_session_id, + CLINE_PROVIDER, + fold.next_sequence.max(0) as usize, + &created_at, + text, + ) + } else { + imported_history::assistant_message_chunk( + fold.display_session_id, + CLINE_PROVIDER, + fold.next_sequence.max(0) as usize, + &created_at, + text, + ) + }; + let locator = ClinePayloadLocator::Text { + message_ordinal, + block_ordinal, + user, + }; + let sequence = take_sequence(fold); + let turn_index = fold.turn_index.max(0); + upsert_cline_event( + fold, + &format!("message:{message_ordinal}:block:{block_ordinal}:text"), + sequence, + turn_index, + chunk, + &locator, + )?; + } + "tool_use" => { + let call_id = block + .get("id") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(); + if call_id.is_empty() { + continue; + } + let raw_name = block + .get("name") + .and_then(Value::as_str) + .unwrap_or("tool") + .to_string(); + let input = block.get("input").cloned().unwrap_or(Value::Null); + let (sub_calls, _) = expand_cline_tool_call(&raw_name, &input); + let sequences = (0..sub_calls.len()).map(|_| take_sequence(fold)).collect(); + fold.pending.insert( + call_id, + PendingCall { + raw_name, + created_at: created_at.clone(), + call_message_ordinal: message_ordinal, + call_block_ordinal: block_ordinal, + sub_calls, + sequences, + turn_index: fold.turn_index.max(0), + }, + ); + } + "tool_result" => { + let Some(call_id) = block + .get("tool_use_id") + .and_then(Value::as_str) + .filter(|call_id| !call_id.is_empty()) + else { + continue; + }; + let Some(pending) = fold.pending.remove(call_id) else { + continue; + }; + emit_cline_tool( + fold, + call_id, + pending, + Some((message_ordinal, block_ordinal, block)), + )?; + } + _ => {} + } + } + Ok(()) +} + +fn flush_pending(fold: &mut IndexFold<'_, '_, '_>) -> Result<(), String> { + let pending = std::mem::take(&mut fold.pending); + for (call_id, pending) in pending { + emit_cline_tool(fold, &call_id, pending, None)?; + } + Ok(()) +} + +fn emit_cline_tool( + fold: &mut IndexFold<'_, '_, '_>, + call_id: &str, + pending: PendingCall, + result: Option<(u64, usize, &Value)>, +) -> Result<(), String> { + let batched = pending.sub_calls.len() > 1 + || matches!( + pending.raw_name.as_str(), + "run_commands" | "read_files" | "search_codebase" + ); + let result_value = result.and_then(|(_, _, block)| block.get("content")); + let failed = result + .map(|(_, _, block)| { + block.get("is_error").and_then(Value::as_bool) == Some(true) + || block.get("success").and_then(Value::as_bool) == Some(false) + }) + .unwrap_or(false); + for (sub_index, ((canonical_name, args), sequence)) in pending + .sub_calls + .into_iter() + .zip(pending.sequences) + .enumerate() + { + let mut output = cline_sub_output(result_value, sub_index, batched); + if pending.raw_name == "read_files" { + output = strip_cline_read_gutter(&output); + } + let call = ImportedToolCall { + call_id: format!("{call_id}#{sub_index}"), + raw_name: pending.raw_name.clone(), + canonical_name, + args, + created_at: pending.created_at.clone(), + }; + let mut chunk = imported_history::tool_call_chunk( + fold.display_session_id, + CLINE_PROVIDER, + sequence.max(0) as usize, + &call, + &output, + ); + if failed || cline_sub_success(result_value, sub_index, batched) == Some(false) { + if let Some(object) = chunk.result.as_object_mut() { + object.insert("success".to_string(), Value::Bool(false)); + object.insert("status".to_string(), Value::String("failed".to_string())); + } + } + let locator = ClinePayloadLocator::Tool { + call_message_ordinal: pending.call_message_ordinal, + call_block_ordinal: pending.call_block_ordinal, + sub_index, + result_message_ordinal: result.map(|(ordinal, _, _)| ordinal), + result_block_ordinal: result.map(|(_, block, _)| block), + }; + upsert_cline_event( + fold, + &format!( + "message:{}:block:{}:tool:{call_id}:{sub_index}", + pending.call_message_ordinal, pending.call_block_ordinal + ), + sequence, + pending.turn_index, + chunk, + &locator, + )?; + } + Ok(()) +} + +fn take_sequence(fold: &mut IndexFold<'_, '_, '_>) -> i64 { + let sequence = fold.next_sequence; + fold.next_sequence = fold.next_sequence.saturating_add(1); + sequence +} + +fn upsert_cline_event( + fold: &mut IndexFold<'_, '_, '_>, + event_key: &str, + sequence: i64, + turn_index: i64, + mut chunk: ActivityChunk, + locator: &ClinePayloadLocator, +) -> Result<(), String> { + let source = ImportedHistorySourceId::Cline; + let event_id = stable_event_id(fold.source_session_id, event_key); + chunk.chunk_id = event_id.clone(); + let content_hash = + content_digest(&[serde_json::to_string(&chunk).unwrap_or_default().as_bytes()]); + let locator_json = serde_json::to_string(locator) + .map_err(|err| format!("encode Cline replay locator: {err}"))?; + let full_chunk = chunk.clone(); + let payloads = compact_chunk(&mut chunk, &locator_json); + for payload in &payloads { + let text = chunk_field_text(&full_chunk, &payload.field_path)?; + payload_artifact::store_text( + fold.tx, + source, + fold.source_session_id, + fold.generation, + &event_id, + &payload.field_path, + &text, + ) + .map_err(|err| format!("store Cline replay payload artifact: {err}"))?; + } + let args_json = serde_json::to_string(&chunk.args) + .map_err(|err| format!("encode Cline replay args: {err}"))?; + let result_json = serde_json::to_string(&chunk.result) + .map_err(|err| format!("encode Cline replay result: {err}"))?; + let payloads_json = serde_json::to_string(&payloads) + .map_err(|err| format!("encode Cline replay payloads: {err}"))?; + fold.tx + .execute( + "INSERT INTO imported_replay_events( + source,source_session_id,generation,sequence,event_id,turn_index, + action_type,function_name,created_at,args_preview_json,result_preview_json, + args_size_bytes,result_size_bytes,thread_id,process_id,source_start,source_end, + payloads_json,content_hash,event_revision + ) VALUES (?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13,?14,?15,0,0,?16,?17,?18) + ON CONFLICT(source,source_session_id,generation,event_id) DO UPDATE SET + sequence=excluded.sequence,turn_index=excluded.turn_index, + action_type=excluded.action_type,function_name=excluded.function_name, + created_at=excluded.created_at,args_preview_json=excluded.args_preview_json, + result_preview_json=excluded.result_preview_json,args_size_bytes=excluded.args_size_bytes, + result_size_bytes=excluded.result_size_bytes,thread_id=excluded.thread_id, + process_id=excluded.process_id,payloads_json=excluded.payloads_json, + content_hash=excluded.content_hash,event_revision=excluded.event_revision", + params![ + source.as_str(), + fold.source_session_id, + fold.generation, + sequence, + event_id, + turn_index, + chunk.action_type, + chunk.function, + chunk.created_at, + args_json, + result_json, + chunk.args.to_string().len() as i64, + chunk.result.to_string().len() as i64, + chunk.thread_id, + chunk.process_id, + payloads_json, + content_hash, + fold.write_revision.min(i64::MAX as u64) as i64, + ], + ) + .map_err(|err| format!("upsert Cline replay event: {err}"))?; + fold.stats.normalized_events = fold.stats.normalized_events.saturating_add(1); + fold.stats.upserted_events = fold.stats.upserted_events.saturating_add(1); + Ok(()) +} + +pub(in crate::sources::imported_history::replay) fn read_payload( + source_path: &Path, + payloads_json: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + let payloads: Vec = serde_json::from_str(payloads_json) + .map_err(|err| format!("decode Cline payload locators: {err}"))?; + let payload = payloads + .iter() + .find(|payload| payload.field_path == field_path) + .ok_or_else(|| "Cline replay payload field is not range-backed".to_string())?; + let locator: ClinePayloadLocator = serde_json::from_str( + payload + .source_key + .as_deref() + .ok_or_else(|| "Cline replay payload locator is missing".to_string())?, + ) + .map_err(|err| format!("decode Cline payload locator: {err}"))?; + let chunk = reconstruct_chunk(source_path, &locator)?; + let text = chunk_field_text(&chunk, field_path)?; + range_from_text(event_id, field_path, &text, offset, max_bytes) +} + +fn reconstruct_chunk(path: &Path, locator: &ClinePayloadLocator) -> Result { + let mut call_block = None; + let mut result_block = None; + let mut text_block = None; + let mut text_created_at = String::new(); + let mut ordinal = 0_u64; + visit_messages(path, |message| { + let current = ordinal; + ordinal = ordinal.saturating_add(1); + match locator { + ClinePayloadLocator::Text { + message_ordinal, + block_ordinal, + .. + } if current == *message_ordinal => { + text_block = content_blocks(&message.content) + .nth(*block_ordinal) + .cloned(); + text_created_at = message + .ts + .filter(|timestamp| *timestamp > 0) + .map(imported_history::epoch_ms_to_iso) + .unwrap_or_default(); + } + ClinePayloadLocator::Tool { + call_message_ordinal, + call_block_ordinal, + result_message_ordinal, + result_block_ordinal, + .. + } => { + if current == *call_message_ordinal { + call_block = content_blocks(&message.content) + .nth(*call_block_ordinal) + .cloned(); + text_created_at = message + .ts + .filter(|timestamp| *timestamp > 0) + .map(imported_history::epoch_ms_to_iso) + .unwrap_or_default(); + } + if result_message_ordinal == &Some(current) { + if let Some(block_ordinal) = result_block_ordinal { + result_block = content_blocks(&message.content) + .nth(*block_ordinal) + .cloned(); + } + } + } + _ => {} + } + Ok(()) + })?; + match locator { + ClinePayloadLocator::Text { user, .. } => { + let block = text_block.ok_or_else(|| "Cline text payload moved".to_string())?; + let raw = block + .get("text") + .and_then(Value::as_str) + .unwrap_or_default(); + let text = if *user { + strip_user_input_wrapper(raw) + } else { + raw.trim() + }; + Ok(if *user { + imported_history::user_message_chunk( + "clineapp-payload", + CLINE_PROVIDER, + 0, + &text_created_at, + text, + ) + } else { + imported_history::assistant_message_chunk( + "clineapp-payload", + CLINE_PROVIDER, + 0, + &text_created_at, + text, + ) + }) + } + ClinePayloadLocator::Tool { sub_index, .. } => { + let call_block = call_block.ok_or_else(|| "Cline tool payload moved".to_string())?; + let call_id = call_block + .get("id") + .and_then(Value::as_str) + .unwrap_or_default(); + let raw_name = call_block + .get("name") + .and_then(Value::as_str) + .unwrap_or("tool"); + let input = call_block.get("input").cloned().unwrap_or(Value::Null); + let (sub_calls, batched) = expand_cline_tool_call(raw_name, &input); + let (canonical_name, args) = sub_calls + .into_iter() + .nth(*sub_index) + .ok_or_else(|| "Cline tool sub-call moved".to_string())?; + let result_value = result_block.as_ref().and_then(|block| block.get("content")); + let mut output = cline_sub_output(result_value, *sub_index, batched); + if raw_name == "read_files" { + output = strip_cline_read_gutter(&output); + } + let call = ImportedToolCall { + call_id: format!("{call_id}#{sub_index}"), + raw_name: raw_name.to_string(), + canonical_name, + args, + created_at: text_created_at, + }; + Ok(imported_history::tool_call_chunk( + "clineapp-payload", + CLINE_PROVIDER, + 0, + &call, + &output, + )) + } + } +} + +fn visit_messages( + path: &Path, + mut visit: impl FnMut(ClineMessage) -> Result<(), String>, +) -> Result<(), String> { + let file = File::open(path) + .map_err(|err| format!("open Cline replay source {}: {err}", path.display()))?; + let mut deserializer = serde_json::Deserializer::from_reader(BufReader::new(file)); + let mut sink_error = None; + let parse_result = { + let mut capture_sink_error = |message| match visit(message) { + Ok(()) => Ok(()), + Err(error) => { + sink_error = Some(error); + Err("Cline replay sink rejected a message".to_string()) + } + }; + TranscriptSeed { + visit: &mut capture_sink_error, + } + .deserialize(&mut deserializer) + }; + if let Some(error) = sink_error { + return Err(format!( + "process Cline replay source {}: {error}", + path.display() + )); + } + parse_result.map_err(|err| format!("parse Cline replay source {}: {err}", path.display()))?; + deserializer + .end() + .map_err(|err| format!("parse Cline replay source {}: {err}", path.display())) +} + +struct TranscriptSeed<'a, F> { + visit: &'a mut F, +} + +impl<'de, F> DeserializeSeed<'de> for TranscriptSeed<'_, F> +where + F: FnMut(ClineMessage) -> Result<(), String>, +{ + type Value = (); + + fn deserialize(self, deserializer: D) -> Result + where + D: Deserializer<'de>, + { + deserializer.deserialize_any(TranscriptVisitor { visit: self.visit }) + } +} + +struct TranscriptVisitor<'a, F> { + visit: &'a mut F, +} + +impl<'de, F> Visitor<'de> for TranscriptVisitor<'_, F> +where + F: FnMut(ClineMessage) -> Result<(), String>, +{ + type Value = (); + + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("a Cline transcript object or message array") + } + + fn visit_map(self, mut map: A) -> Result + where + A: MapAccess<'de>, + { + let mut found = false; + while let Some(key) = map.next_key::()? { + if key == "messages" { + found = true; + map.next_value_seed(MessagesSeed { + visit: &mut *self.visit, + })?; + } else { + map.next_value::()?; + } + } + if !found { + return Err(serde::de::Error::custom( + "Cline transcript has no messages array", + )); + } + Ok(()) + } + + fn visit_seq(self, sequence: A) -> Result + where + A: SeqAccess<'de>, + { + MessagesVisitor { + visit: &mut *self.visit, + } + .visit_seq(sequence) + } +} + +struct MessagesSeed<'a, F> { + visit: &'a mut F, +} + +impl<'de, F> DeserializeSeed<'de> for MessagesSeed<'_, F> +where + F: FnMut(ClineMessage) -> Result<(), String>, +{ + type Value = (); + + fn deserialize(self, deserializer: D) -> Result + where + D: Deserializer<'de>, + { + deserializer.deserialize_seq(MessagesVisitor { visit: self.visit }) + } +} + +struct MessagesVisitor<'a, F> { + visit: &'a mut F, +} + +impl<'de, F> Visitor<'de> for MessagesVisitor<'_, F> +where + F: FnMut(ClineMessage) -> Result<(), String>, +{ + type Value = (); + + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("a Cline messages array") + } + + fn visit_seq(self, mut sequence: A) -> Result + where + A: SeqAccess<'de>, + { + while let Some(message) = sequence.next_element::()? { + (self.visit)(message).map_err(serde::de::Error::custom)?; + } + Ok(()) + } +} + +fn content_blocks(content: &Value) -> impl Iterator { + content.as_array().into_iter().flatten() +} + +fn block_type(block: &Value) -> &str { + block + .get("type") + .and_then(Value::as_str) + .unwrap_or_default() +} + +fn strip_user_input_wrapper(text: &str) -> &str { + let trimmed = text.trim(); + let Some(after_open) = trimmed.strip_prefix("') else { + return trimmed; + }; + after_open[end + 1..] + .strip_suffix("") + .unwrap_or(&after_open[end + 1..]) + .trim() +} + +fn expand_cline_tool_call(name: &str, input: &Value) -> (Vec<(String, Value)>, bool) { + let sub_calls = match name { + "run_commands" => input_array(input, "commands") + .map(|command| { + ( + imported_history::FUNCTION_RUN_COMMAND_LINE.to_string(), + json!({"command":command,"cmd":command}), + ) + }) + .collect::>(), + "read_files" => input_array(input, "files") + .map(|file| { + ( + imported_history::FUNCTION_READ_FILE.to_string(), + json!({"file_path":file.get("path").cloned().unwrap_or(Value::Null)}), + ) + }) + .collect(), + "search_codebase" => input_array(input, "queries") + .map(|query| { + ( + imported_history::FUNCTION_CODE_SEARCH.to_string(), + json!({"query":query}), + ) + }) + .collect(), + "editor" => { + return ( + vec![( + imported_history::FUNCTION_EDIT_FILE.to_string(), + json!({ + "file_path":input.get("path").cloned().unwrap_or(Value::Null), + "old_string":input.get("old_text").cloned().filter(|value| !value.is_null()).unwrap_or_else(|| json!("")), + "new_string":input.get("new_text").cloned().unwrap_or_else(|| json!("")), + }), + )], + false, + ); + } + _ => Vec::new(), + }; + if sub_calls.is_empty() { + (vec![(name.to_string(), input.clone())], false) + } else { + (sub_calls, true) + } +} + +fn input_array<'a>(input: &'a Value, key: &str) -> impl Iterator { + input + .get(key) + .and_then(Value::as_array) + .into_iter() + .flatten() +} + +fn cline_sub_output(results: Option<&Value>, index: usize, batched: bool) -> String { + if batched { + return results + .and_then(Value::as_array) + .and_then(|items| items.get(index)) + .map(|item| value_to_text(item.get("result").unwrap_or(item))) + .unwrap_or_default(); + } + results.map(value_to_text).unwrap_or_default() +} + +fn cline_sub_success(results: Option<&Value>, index: usize, batched: bool) -> Option { + let result = if batched { + results?.as_array()?.get(index)? + } else if let Some(first) = results?.as_array().and_then(|items| items.first()) { + first + } else { + results? + }; + result.get("success").and_then(Value::as_bool) +} + +fn value_to_text(value: &Value) -> String { + let mut output = String::new(); + append_value_text(value, &mut output); + output.trim().to_string() +} + +fn append_value_text(value: &Value, output: &mut String) { + match value { + Value::String(text) => push_line(output, text), + Value::Array(items) => { + for item in items { + append_value_text(item, output); + } + } + Value::Object(map) => { + if let Some(Value::String(text)) = map.get("text").or_else(|| map.get("result")) { + push_line(output, text); + } else { + push_line(output, &value.to_string()); + } + } + Value::Null => {} + value => push_line(output, &value.to_string()), + } +} + +fn push_line(output: &mut String, text: &str) { + let text = text.trim(); + if text.is_empty() { + return; + } + if !output.is_empty() { + output.push('\n'); + } + output.push_str(text); +} + +fn strip_cline_read_gutter(text: &str) -> String { + if text + .lines() + .find(|line| !line.trim().is_empty()) + .is_none_or(|line| gutter_body(line).is_none()) + { + return text.to_string(); + } + text.lines() + .map(|line| gutter_body(line).unwrap_or(line)) + .collect::>() + .join("\n") +} + +fn gutter_body(line: &str) -> Option<&str> { + let trimmed = line.trim_start(); + let digits_end = trimmed.find(|character: char| !character.is_ascii_digit())?; + if digits_end == 0 { + return None; + } + let rest = trimmed[digits_end..] + .strip_prefix(' ') + .unwrap_or(&trimmed[digits_end..]); + let rest = rest.strip_prefix('|')?; + Some(rest.strip_prefix(' ').unwrap_or(rest)) +} + +fn chunk_field_text(chunk: &ActivityChunk, field_path: &str) -> Result { + let (root, path) = field_path + .split_once('.') + .map_or((field_path, ""), |parts| parts); + let value = match root { + "args" => &chunk.args, + "result" => &chunk.result, + _ => return Err("Cline replay field must be under args or result".to_string()), + }; + let target = if path.is_empty() { + value + } else { + path.split('.') + .try_fold(value, |current, key| current.get(key)) + .ok_or_else(|| "Cline replay payload field moved".to_string())? + }; + Ok(target + .as_str() + .map(str::to_string) + .unwrap_or_else(|| target.to_string())) +} + +fn count_rows( + tx: &Transaction<'_>, + table: &str, + source_session_id: &str, + generation: &str, +) -> Result { + tx.query_row( + &format!( + "SELECT COUNT(*) FROM {table} WHERE source='cline' AND source_session_id=?1 AND generation=?2" + ), + params![source_session_id, generation], + |row| row.get::<_, i64>(0), + ) + .map(|count| count.max(0) as u64) + .map_err(|err| format!("count Cline replay {table}: {err}")) +} + +fn stable_event_id(source_session_id: &str, event_key: &str) -> String { + format!( + "replay-cline-{}", + legacy_stable_id_hash_delimited(&[source_session_id.as_bytes(), event_key.as_bytes()]) + ) +} + +#[cfg(test)] +#[path = "../tests/whole_json_driver.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/claude_code.jsonl b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/claude_code.jsonl new file mode 100644 index 000000000..84466424f --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/claude_code.jsonl @@ -0,0 +1,4 @@ +{"type":"user","timestamp":"2026-07-22T00:00:00Z","message":{"role":"user","content":"first Claude Code question"}} +{"type":"assistant","timestamp":"2026-07-22T00:00:01Z","message":{"role":"assistant","content":[{"type":"text","text":"first Claude Code answer"}]}} +{"type":"user","timestamp":"2026-07-22T00:00:02Z","message":{"role":"user","content":"second Claude Code question"}} +{"type":"assistant","timestamp":"2026-07-22T00:00:03Z","message":{"role":"assistant","content":[{"type":"text","text":"second Claude Code answer"}]}} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cline.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cline.json new file mode 100644 index 000000000..afece59a6 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cline.json @@ -0,0 +1,34 @@ +{ + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "first Cline question" + } + ], + "ts": 1753142400000 + }, + { + "role": "assistant", + "content": [{ "type": "text", "text": "first Cline answer" }], + "ts": 1753142401000 + }, + { + "role": "user", + "content": [ + { + "type": "text", + "text": "second Cline question" + } + ], + "ts": 1753142402000 + }, + { + "role": "assistant", + "content": [{ "type": "text", "text": "second Cline answer" }], + "ts": 1753142403000 + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/codex_app.jsonl b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/codex_app.jsonl new file mode 100644 index 000000000..49882d807 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/codex_app.jsonl @@ -0,0 +1,4 @@ +{"timestamp":"2026-07-22T00:00:00Z","type":"event_msg","payload":{"type":"user_message","message":"first Codex question"}} +{"timestamp":"2026-07-22T00:00:01Z","type":"event_msg","payload":{"type":"agent_message","message":"first Codex answer"}} +{"timestamp":"2026-07-22T00:00:02Z","type":"event_msg","payload":{"type":"user_message","message":"second Codex question"}} +{"timestamp":"2026-07-22T00:00:03Z","type":"event_msg","payload":{"type":"agent_message","message":"second Codex answer"}} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cursor_cli.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cursor_cli.json new file mode 100644 index 000000000..24d509c5f --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cursor_cli.json @@ -0,0 +1,22 @@ +{ + "agentId": "fixture-cursor-cli", + "createdAt": 1753142400000, + "messages": [ + { + "role": "user", + "content": "first Cursor CLI question" + }, + { + "role": "assistant", + "content": [{ "type": "text", "text": "first Cursor CLI answer" }] + }, + { + "role": "user", + "content": "second Cursor CLI question" + }, + { + "role": "assistant", + "content": [{ "type": "text", "text": "second Cursor CLI answer" }] + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cursor_ide.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cursor_ide.json new file mode 100644 index 000000000..d1ce8778f --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/cursor_ide.json @@ -0,0 +1,39 @@ +{ + "composer": { + "composerId": "fixture-cursor-ide", + "createdAt": 1753142400000, + "lastUpdatedAt": 1753142403000, + "fullConversationHeadersOnly": [ + { "bubbleId": "cursor-ide-u1", "type": 1 }, + { "bubbleId": "cursor-ide-a1", "type": 2 }, + { "bubbleId": "cursor-ide-u2", "type": 1 }, + { "bubbleId": "cursor-ide-a2", "type": 2 } + ] + }, + "bubbles": [ + { + "bubbleId": "cursor-ide-u1", + "type": 1, + "createdAt": "2026-07-22T00:00:00Z", + "text": "first Cursor IDE question" + }, + { + "bubbleId": "cursor-ide-a1", + "type": 2, + "createdAt": "2026-07-22T00:00:01Z", + "text": "first Cursor IDE answer" + }, + { + "bubbleId": "cursor-ide-u2", + "type": 1, + "createdAt": "2026-07-22T00:00:02Z", + "text": "second Cursor IDE question" + }, + { + "bubbleId": "cursor-ide-a2", + "type": 2, + "createdAt": "2026-07-22T00:00:03Z", + "text": "second Cursor IDE answer" + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/manifest.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/manifest.json new file mode 100644 index 000000000..0848fed58 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/manifest.json @@ -0,0 +1,94 @@ +{ + "fixtures": [ + { + "sourceId": "claude_code", + "storageFamily": "json_lines", + "sourceSessionId": "fixture-claude", + "file": "claude_code.jsonl" + }, + { + "sourceId": "codex_app", + "storageFamily": "json_lines", + "sourceSessionId": "fixture-codex", + "file": "codex_app.jsonl" + }, + { + "sourceId": "cursor_ide", + "storageFamily": "sqlite_key_value", + "sourceSessionId": "fixture-cursor-ide", + "file": "cursor_ide.json" + }, + { + "sourceId": "cursor_cli", + "storageFamily": "sqlite_manifest_blob", + "sourceSessionId": "fixture-cursor-cli", + "file": "cursor_cli.json" + }, + { + "sourceId": "opencode", + "storageFamily": "sqlite_wal", + "sourceSessionId": "fixture-opencode", + "file": "opencode.json" + }, + { + "sourceId": "windsurf", + "storageFamily": "sqlite_key_value", + "sourceSessionId": "fixture-windsurf", + "file": "windsurf.json" + }, + { + "sourceId": "workbuddy", + "storageFamily": "json_lines", + "sourceSessionId": "fixture-workbuddy", + "file": "workbuddy.jsonl" + }, + { + "sourceId": "trae", + "storageFamily": "json_lines", + "sourceSessionId": "fixture-trae", + "file": "trae.jsonl" + }, + { + "sourceId": "cline", + "storageFamily": "whole_json", + "sourceSessionId": "fixture-cline", + "file": "cline.json" + }, + { + "sourceId": "warp", + "storageFamily": "sqlite_task_blob", + "sourceSessionId": "fixture-warp", + "file": "warp.json" + }, + { + "sourceId": "zcode", + "storageFamily": "sqlite_wal", + "sourceSessionId": "fixture-zcode", + "file": "zcode.json" + }, + { + "sourceId": "qoder", + "storageFamily": "json_lines", + "sourceSessionId": "fixture-project/fixture-qoder", + "file": "qoder.jsonl" + }, + { + "sourceId": "mimo_code", + "storageFamily": "sqlite_wal", + "sourceSessionId": "fixture-mimo", + "file": "mimo_code.json" + }, + { + "sourceId": "omp", + "storageFamily": "json_lines", + "sourceSessionId": "fixture-omp", + "file": "omp.jsonl" + }, + { + "sourceId": "qoder_cli", + "storageFamily": "json_lines", + "sourceSessionId": "fixture-qoder-cli", + "file": "qoder_cli.jsonl" + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/mimo_code.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/mimo_code.json new file mode 100644 index 000000000..d4e6dda71 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/mimo_code.json @@ -0,0 +1,21 @@ +{ + "sessionId": "fixture-mimo", + "parts": [ + { + "role": "user", + "data": { "type": "text", "text": "first Mimo Code question" } + }, + { + "role": "assistant", + "data": { "type": "text", "text": "first Mimo Code answer" } + }, + { + "role": "user", + "data": { "type": "text", "text": "second Mimo Code question" } + }, + { + "role": "assistant", + "data": { "type": "text", "text": "second Mimo Code answer" } + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/omp.jsonl b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/omp.jsonl new file mode 100644 index 000000000..ddea2457e --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/omp.jsonl @@ -0,0 +1,4 @@ +{"type":"user","timestamp":1753142400000,"message":{"role":"user","content":[{"type":"text","text":"first OMP question"}]}} +{"type":"assistant","timestamp":1753142401000,"message":{"role":"assistant","content":[{"type":"text","text":"first OMP answer"}]}} +{"type":"user","timestamp":1753142402000,"message":{"role":"user","content":[{"type":"text","text":"second OMP question"}]}} +{"type":"assistant","timestamp":1753142403000,"message":{"role":"assistant","content":[{"type":"text","text":"second OMP answer"}]}} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/opencode.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/opencode.json new file mode 100644 index 000000000..093fee319 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/opencode.json @@ -0,0 +1,21 @@ +{ + "sessionId": "fixture-opencode", + "parts": [ + { + "role": "user", + "data": { "type": "text", "text": "first OpenCode question" } + }, + { + "role": "assistant", + "data": { "type": "text", "text": "first OpenCode answer" } + }, + { + "role": "user", + "data": { "type": "text", "text": "second OpenCode question" } + }, + { + "role": "assistant", + "data": { "type": "text", "text": "second OpenCode answer" } + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/qoder.jsonl b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/qoder.jsonl new file mode 100644 index 000000000..3d13ceaa7 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/qoder.jsonl @@ -0,0 +1,4 @@ +{"role":"user","message":{"content":[{"type":"text","text":"first Qoder question"}]}} +{"role":"assistant","message":{"content":[{"type":"text","text":"first Qoder answer"}]}} +{"role":"user","message":{"content":[{"type":"text","text":"second Qoder question"}]}} +{"role":"assistant","message":{"content":[{"type":"text","text":"second Qoder answer"}]}} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/qoder_cli.jsonl b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/qoder_cli.jsonl new file mode 100644 index 000000000..1dd9b57a7 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/qoder_cli.jsonl @@ -0,0 +1,4 @@ +{"type":"user","timestamp":1753142400000,"message":{"role":"user","content":[{"type":"text","text":"first Qoder CLI question"}]}} +{"type":"assistant","timestamp":1753142401000,"message":{"role":"assistant","content":[{"type":"text","text":"first Qoder CLI answer"}]}} +{"type":"user","timestamp":1753142402000,"message":{"role":"user","content":[{"type":"text","text":"second Qoder CLI question"}]}} +{"type":"assistant","timestamp":1753142403000,"message":{"role":"assistant","content":[{"type":"text","text":"second Qoder CLI answer"}]}} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/trae.jsonl b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/trae.jsonl new file mode 100644 index 000000000..4f0204329 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/trae.jsonl @@ -0,0 +1,2 @@ +{"intent":"first Trae question","outcome":"first Trae answer","actions":[],"learned":[],"message_summary_time":"2026-07-22 08:00:00"} +{"intent":"second Trae question","outcome":"second Trae answer","actions":[],"learned":[],"message_summary_time":"2026-07-22 08:00:01"} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/warp.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/warp.json new file mode 100644 index 000000000..22088e930 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/warp.json @@ -0,0 +1,37 @@ +{ + "conversationId": "fixture-warp", + "tasks": [ + { + "id": "fixture-warp-task-1", + "description": "First fixture turn", + "messages": [ + { + "id": "warp-u1", + "timestamp": "2026-07-22T00:00:00Z", + "userQuery": { "query": "first Warp question" } + }, + { + "id": "warp-a1", + "timestamp": "2026-07-22T00:00:01Z", + "agentOutput": { "text": "first Warp answer" } + } + ] + }, + { + "id": "fixture-warp-task-2", + "description": "Second fixture turn", + "messages": [ + { + "id": "warp-u2", + "timestamp": "2026-07-22T00:00:02Z", + "userQuery": { "query": "second Warp question" } + }, + { + "id": "warp-a2", + "timestamp": "2026-07-22T00:00:03Z", + "agentOutput": { "text": "second Warp answer" } + } + ] + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/windsurf.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/windsurf.json new file mode 100644 index 000000000..c8788b267 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/windsurf.json @@ -0,0 +1,39 @@ +{ + "composer": { + "composerId": "fixture-windsurf", + "createdAt": 1753142400000, + "lastUpdatedAt": 1753142403000, + "fullConversationHeadersOnly": [ + { "bubbleId": "windsurf-u1", "type": 1 }, + { "bubbleId": "windsurf-a1", "type": 2 }, + { "bubbleId": "windsurf-u2", "type": 1 }, + { "bubbleId": "windsurf-a2", "type": 2 } + ] + }, + "bubbles": [ + { + "bubbleId": "windsurf-u1", + "type": 1, + "createdAt": "2026-07-22T00:00:00Z", + "text": "first Windsurf question" + }, + { + "bubbleId": "windsurf-a1", + "type": 2, + "createdAt": "2026-07-22T00:00:01Z", + "text": "first Windsurf answer" + }, + { + "bubbleId": "windsurf-u2", + "type": 1, + "createdAt": "2026-07-22T00:00:02Z", + "text": "second Windsurf question" + }, + { + "bubbleId": "windsurf-a2", + "type": 2, + "createdAt": "2026-07-22T00:00:03Z", + "text": "second Windsurf answer" + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/workbuddy.jsonl b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/workbuddy.jsonl new file mode 100644 index 000000000..9785014be --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/workbuddy.jsonl @@ -0,0 +1,4 @@ +{"type":"message","timestamp":"2026-07-22T00:00:00Z","role":"user","content":"first WorkBuddy question"} +{"type":"message","timestamp":"2026-07-22T00:00:01Z","role":"assistant","content":"first WorkBuddy answer"} +{"type":"message","timestamp":"2026-07-22T00:00:02Z","role":"user","content":"second WorkBuddy question"} +{"type":"message","timestamp":"2026-07-22T00:00:03Z","role":"assistant","content":"second WorkBuddy answer"} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/zcode.json b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/zcode.json new file mode 100644 index 000000000..991dea39e --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/fixtures/zcode.json @@ -0,0 +1,21 @@ +{ + "sessionId": "fixture-zcode", + "parts": [ + { + "role": "user", + "data": { "type": "text", "text": "first ZCode question" } + }, + { + "role": "assistant", + "data": { "type": "text", "text": "first ZCode answer" } + }, + { + "role": "user", + "data": { "type": "text", "text": "second ZCode question" } + }, + { + "role": "assistant", + "data": { "type": "text", "text": "second ZCode answer" } + } + ] +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/mod.rs new file mode 100644 index 000000000..ff4ea61d6 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/mod.rs @@ -0,0 +1,110 @@ +//! ORGII-owned compact replay index synchronization and bounded queries. + +use std::collections::HashSet; +use std::fs; +use std::io::{Read, Seek, SeekFrom}; +use std::path::{Path, PathBuf}; + +use chrono::Utc; +use core_types::activity::ActivityChunk; +use rusqlite::{params, Connection, OptionalExtension, Row, Transaction, TransactionBehavior}; + +use super::{ + codex_jsonl, jsonl_driver, payload_artifact, qoder_sidecar, sqlite_driver, structured_driver, + whole_json_driver, ImportedHistorySourceId, ReplayChunkDelta, ReplayChunkScan, + ReplayChunkWindow, ReplayCursor, ReplayIndexedChunk, ReplayLimits, + ReplayPayloadArtifactLocator, ReplayPayloadDescriptor, ReplayPayloadRange, ReplayStats, + ReplayTurnHeader, HARD_MAX_PAYLOAD_RANGE_BYTES, +}; + +/// Replay indexing can touch hundreds of MiB inside one atomic generation +/// transaction. The application-wide SQLite default permits a 64 MiB page +/// cache, which lets dirty replay-index pages alone exceed the #443 process +/// growth budget. This connection-local cap keeps the atomic transaction but +/// makes its resident cache explicitly byte-bounded. +const REPLAY_INDEX_CACHE_KIB: i64 = 16 * 1024; + +fn begin_replay_write_transaction<'conn>( + conn: &'conn mut Connection, + context: &str, +) -> Result, String> { + conn.transaction_with_behavior(TransactionBehavior::Immediate) + .map_err(|err| format!("start {context} transaction: {err}")) +} + +#[derive(Debug, Clone)] +pub(super) struct ReplayIndexState { + pub generation: String, + pub revision: u64, + pub parser_version: u32, + pub source_identity: String, + pub driver_cursor_json: String, + pub indexed_size_bytes: u64, + pub indexed_mtime_ns: i64, + pub total_events: u64, + pub total_turns: u64, + pub state_updated_at_ms: i64, +} + +#[derive(Debug, Clone, Default)] +pub(super) struct ReplaySyncResult { + pub stats: ReplayStats, + pub generation_changed: bool, +} + +#[derive(Debug)] +pub(super) struct ResolvedSource { + pub(super) source_session_id: String, + pub(super) path: PathBuf, +} + +#[derive(Debug, Clone)] +struct SourcePhysicalSnapshot { + identity: String, + size_bytes: u64, + mtime_ns: i64, + sample_fingerprint: String, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum RejectedSnapshotKind { + ClineInvalidDocument, + CursorCliLineageChanged, +} + +impl RejectedSnapshotKind { + const fn as_str(self) -> &'static str { + match self { + Self::ClineInvalidDocument => "cline_invalid_document", + Self::CursorCliLineageChanged => "cursor_cli_lineage_changed", + } + } +} + +struct DriverSyncOutcome { + stats: ReplayStats, + driver_cursor_json: String, + indexed_size_bytes: u64, + total_events: u64, + total_turns: u64, + removed_event_ids: Vec, +} + +mod payload; +mod query; +mod source_identity; +mod sync; + +pub(super) use payload::{materialize_payload_artifact, read_payload_range}; +pub(super) use query::{ + hydrate_turn_if_needed, read_delta, read_recent_window, read_scan_after, read_window_before, + read_window_for_turn, read_window_for_turn_index, +}; +#[cfg(test)] +pub(super) use source_identity::take_file_sample_count; +pub(super) use source_identity::{load_state, resolve_source}; +pub(super) use sync::sync_index; + +#[cfg(test)] +#[path = "../tests/index.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/payload.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/payload.rs new file mode 100644 index 000000000..060c3fc9a --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/payload.rs @@ -0,0 +1,329 @@ +use super::source_identity::*; +use super::*; + +#[allow( + clippy::too_many_arguments, + reason = "Payload range API mirrors the stable source, generation, event, field, and byte-range contract" +)] +pub(in crate::sources::imported_history::replay) fn read_payload_range( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + let resolved = resolve_source(conn, source, session_id)?; + let payloads_json = conn + .query_row( + "SELECT payloads_json FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![ + source.as_str(), + resolved.source_session_id, + generation, + event_id + ], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|err| format!("query replay payload locator: {err}"))? + .ok_or_else(|| "Replay event/generation is no longer available".to_string())?; + if let Some(range) = read_payload_artifact_range( + conn, + source, + &resolved.source_session_id, + generation, + event_id, + field_path, + offset, + max_bytes, + )? { + return Ok(range); + } + read_provider_payload_range( + source, + &resolved.path, + &payloads_json, + event_id, + field_path, + offset, + max_bytes, + ) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn read_provider_payload_range( + source: ImportedHistorySourceId, + source_path: &Path, + payloads_json: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + match replay_driver_kind(source) { + ReplayDriverKind::CodexJsonl => codex_jsonl::read_payload( + source_path, + payloads_json, + event_id, + field_path, + offset, + max_bytes, + ), + ReplayDriverKind::SharedJsonl => jsonl_driver::read_payload( + source, + source_path, + payloads_json, + event_id, + field_path, + offset, + max_bytes, + ), + ReplayDriverKind::Sqlite => sqlite_driver::read_payload( + source, + source_path, + payloads_json, + event_id, + field_path, + offset, + max_bytes, + ), + ReplayDriverKind::StructuredSqlite => structured_driver::read_payload( + source, + source_path, + payloads_json, + event_id, + field_path, + offset, + max_bytes, + ), + ReplayDriverKind::WholeJson => whole_json_driver::read_payload( + source_path, + payloads_json, + event_id, + field_path, + offset, + max_bytes, + ), + } +} + +/// Materialize one direct provider locator into the same immutable replay +/// artifact table used by adapters that decode payloads while indexing. +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(in crate::sources::imported_history::replay) fn materialize_payload_artifact( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, +) -> Result { + let resolved = resolve_source(tx, source, session_id)?; + let payloads_json = tx + .query_row( + "SELECT payloads_json FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![ + source.as_str(), + resolved.source_session_id, + generation, + event_id + ], + |row| row.get::<_, String>(0), + ) + .optional() + .map_err(|error| format!("query replay payload for materialization: {error}"))? + .ok_or_else(|| "Replay event/generation is no longer available".to_string())?; + let descriptor = serde_json::from_str::>(&payloads_json) + .map_err(|error| format!("decode replay payload locator: {error}"))? + .into_iter() + .find(|descriptor| descriptor.field_path == field_path) + .ok_or_else(|| format!("No deferred replay payload for {field_path}"))?; + + if let Some((content_hash, total_bytes)) = tx + .query_row( + "SELECT ref.content_hash, LENGTH(artifact.payload) + FROM imported_replay_payload_artifact_refs AS ref + JOIN imported_replay_payload_artifacts AS artifact + ON artifact.source=ref.source + AND artifact.source_session_id=ref.source_session_id + AND artifact.generation=ref.generation + AND artifact.content_hash=ref.content_hash + WHERE ref.source=?1 AND ref.source_session_id=?2 AND ref.generation=?3 + AND ref.event_id=?4 AND ref.field_path=?5", + params![ + source.as_str(), + resolved.source_session_id, + generation, + event_id, + field_path + ], + |row| Ok((row.get::<_, String>(0)?, row.get::<_, i64>(1)?)), + ) + .optional() + .map_err(|error| format!("locate replay payload artifact: {error}"))? + { + return Ok(ReplayPayloadArtifactLocator { + source_id: source.as_str().to_string(), + source_session_id: resolved.source_session_id, + generation: generation.to_string(), + content_hash, + total_bytes: total_bytes.max(0) as u64, + }); + } + if descriptor.source_key.is_none() && descriptor.spans.is_empty() { + return Err(format!( + "Replay payload artifact for {event_id}/{field_path} is missing and has no direct provider locator" + )); + } + + let total_bytes = descriptor.total_bytes; + let content_hash = payload_artifact::store_streamed( + tx, + source, + &resolved.source_session_id, + generation, + event_id, + field_path, + total_bytes, + |writer| { + let mut offset = 0_u64; + while offset < total_bytes { + let range = read_provider_payload_range( + source, + &resolved.path, + &payloads_json, + event_id, + field_path, + offset, + HARD_MAX_PAYLOAD_RANGE_BYTES, + )?; + if range.offset != offset || range.next_offset <= offset { + return Err(format!( + "Replay payload materialization made no exact progress at byte {offset}: returned {}..{}", + range.offset, range.next_offset + )); + } + if range.total_bytes != total_bytes { + return Err(format!( + "Replay payload changed during materialization: expected {total_bytes} bytes, found {}", + range.total_bytes + )); + } + writer + .write_all(range.text.as_bytes()) + .map_err(|error| format!("write replay payload artifact page: {error}"))?; + offset = range.next_offset; + } + Ok(()) + }, + )?; + Ok(ReplayPayloadArtifactLocator { + source_id: source.as_str().to_string(), + source_session_id: resolved.source_session_id, + generation: generation.to_string(), + content_hash, + total_bytes, + }) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn read_payload_artifact_range( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result, String> { + let offset_i64 = offset.min(i64::MAX as u64) as i64; + let read_bytes = max_bytes.saturating_add(4).min(i64::MAX as usize) as i64; + let row = conn + .query_row( + "SELECT LENGTH(artifact.payload), SUBSTR(artifact.payload, ?6 + 1, ?7) + FROM imported_replay_payload_artifact_refs AS ref + JOIN imported_replay_payload_artifacts AS artifact + ON artifact.source=ref.source + AND artifact.source_session_id=ref.source_session_id + AND artifact.generation=ref.generation + AND artifact.content_hash=ref.content_hash + WHERE ref.source=?1 AND ref.source_session_id=?2 AND ref.generation=?3 + AND ref.event_id=?4 AND ref.field_path=?5", + params![ + source.as_str(), + source_session_id, + generation, + event_id, + field_path, + offset_i64, + read_bytes + ], + |row| { + Ok(( + row.get::<_, i64>(0)?.max(0) as u64, + row.get::<_, Vec>(1)?, + )) + }, + ) + .optional() + .map_err(|err| format!("read replay payload artifact: {err}"))?; + let Some((total_bytes, bytes)) = row else { + return Ok(None); + }; + if offset >= total_bytes { + return Ok(Some(ReplayPayloadRange { + event_id: event_id.to_string(), + field_path: field_path.to_string(), + offset: total_bytes, + next_offset: total_bytes, + eof: true, + total_bytes, + text: String::new(), + })); + } + let mut start = 0_usize; + while start < bytes.len() && bytes[start] & 0b1100_0000 == 0b1000_0000 { + start += 1; + } + let mut end = start.saturating_add(max_bytes).min(bytes.len()); + while end > start && std::str::from_utf8(&bytes[start..end]).is_err() { + end -= 1; + } + // Avoid an empty, non-EOF page when the caller's byte budget is smaller + // than the next UTF-8 scalar. The public hard cap still bounds this to at + // most three extra bytes. + if end == start && start < bytes.len() && max_bytes > 0 { + end = (start + 1..=bytes.len()) + .find(|candidate| std::str::from_utf8(&bytes[start..*candidate]).is_ok()) + .unwrap_or(bytes.len()); + } + let text = std::str::from_utf8(&bytes[start..end]) + .map_err(|err| format!("decode replay payload artifact UTF-8: {err}"))? + .to_string(); + let actual_offset = offset.saturating_add(start as u64); + let next_offset = actual_offset.saturating_add((end - start) as u64); + Ok(Some(ReplayPayloadRange { + event_id: event_id.to_string(), + field_path: field_path.to_string(), + offset: actual_offset, + next_offset, + eof: next_offset >= total_bytes, + total_bytes, + text, + })) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/query.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/query.rs new file mode 100644 index 000000000..1b542ca9c --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/query.rs @@ -0,0 +1,961 @@ +use super::source_identity::*; +use super::sync::publish_change_log; +use super::*; +use std::io::{self, Write}; + +use serde::Serialize; + +#[derive(Default)] +struct CountingWriter { + bytes: usize, +} + +impl Write for CountingWriter { + fn write(&mut self, buffer: &[u8]) -> io::Result { + self.bytes = self + .bytes + .checked_add(buffer.len()) + .ok_or_else(|| io::Error::other("replay JSON byte count overflow"))?; + Ok(buffer.len()) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } +} + +fn serialized_json_bytes(value: &T) -> usize { + let mut writer = CountingWriter::default(); + serde_json::to_writer(&mut writer, value).map_or(0, |()| writer.bytes) +} + +pub(in crate::sources::imported_history::replay) fn read_recent_window( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + limits: ReplayLimits, +) -> Result { + read_window_before(conn, source, session_id, None, limits) +} + +pub(in crate::sources::imported_history::replay) fn read_window_before( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + before_sequence: Option, + limits: ReplayLimits, +) -> Result { + let resolved = resolve_source(conn, source, session_id)?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after synchronization".to_string())?; + let ceiling = before_sequence.unwrap_or(i64::MAX); + let newest_turn = conn + .query_row( + "SELECT MAX(turn_index) FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND sequence < ?4", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + ceiling + ], + |row| row.get::<_, Option>(0), + ) + .map_err(|err| format!("resolve replay window turn: {err}"))? + .unwrap_or(0); + let oldest_turn = newest_turn.saturating_sub(limits.max_turns.saturating_sub(1) as i64); + let mut chunks = read_chunks( + conn, + source, + session_id, + &resolved.source_session_id, + &state.generation, + "turn_index >= ?4 AND turn_index <= ?5 AND sequence < ?6", + &[oldest_turn, newest_turn, ceiling], + limits, + QueryDirection::NewestFirst, + )?; + chunks.reverse(); + let ipc_bytes = chunks + .iter() + .map(serialized_indexed_chunk_bytes) + .sum::(); + let min_sequence = chunks.first().map_or(-1, |chunk| chunk.sequence); + let through_sequence = chunks.last().map_or(-1, |chunk| chunk.sequence); + let has_older = conn + .query_row( + "SELECT EXISTS( + SELECT 1 FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND sequence < ?4 + )", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + min_sequence + ], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("query older replay events: {err}"))? + != 0; + let turn_headers = read_turn_headers( + conn, + source, + &resolved.source_session_id, + &state.generation, + oldest_turn, + newest_turn, + )?; + Ok(ReplayChunkWindow { + cursor: ReplayCursor { + source_id: source.as_str().to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence, + }, + window_start_sequence: chunks.first().map(|chunk| chunk.sequence), + chunks, + turn_headers, + total_turn_count: state.total_turns, + total_event_count: state.total_events, + has_older, + stats: ReplayStats { + ipc_bytes: ipc_bytes as u64, + ..ReplayStats::default() + }, + }) +} + +pub(in crate::sources::imported_history::replay) fn read_window_for_turn( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + turn_id: &str, + limits: ReplayLimits, +) -> Result { + let resolved = resolve_source(conn, source, session_id)?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after synchronization".to_string())?; + let turn_index = conn + .query_row( + "SELECT turn_index FROM imported_replay_turns + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND turn_id=?4", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + turn_id + ], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|err| format!("query replay turn {turn_id}: {err}"))? + .ok_or_else(|| format!("Replay turn is no longer available: {turn_id}"))?; + read_window_for_turn_index(conn, source, session_id, turn_index, limits) +} + +pub(in crate::sources::imported_history::replay) fn read_window_for_turn_index( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + requested_turn_index: i64, + limits: ReplayLimits, +) -> Result { + let resolved = resolve_source(conn, source, session_id)?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after synchronization".to_string())?; + let turn = conn + .query_row( + "SELECT turn_id,turn_index,start_sequence,end_sequence,started_at,ended_at,event_count + FROM imported_replay_turns WHERE source=?1 AND source_session_id=?2 + AND generation=?3 AND turn_index=?4", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + requested_turn_index + ], + |row| { + Ok(ReplayTurnHeader { + turn_id: row.get(0)?, + turn_index: row.get(1)?, + start_sequence: row.get(2)?, + end_sequence: row.get(3)?, + started_at: row.get(4)?, + ended_at: row.get(5)?, + event_count: row.get::<_, i64>(6)?.max(0) as u64, + }) + }, + ) + .optional() + .map_err(|err| format!("query replay turn index {requested_turn_index}: {err}"))? + .ok_or_else(|| { + format!("Replay turn index is no longer available: {requested_turn_index}") + })?; + let hydrate_stats = hydrate_turn_if_needed( + conn, + source, + session_id, + &resolved.source_session_id, + &state, + &turn, + )?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after turn hydration".to_string())?; + let (chunks, window_start_sequence) = read_anchored_turn_chunks( + conn, + source, + session_id, + &resolved.source_session_id, + &state.generation, + turn.turn_index, + limits, + )?; + let ipc_bytes = chunks + .iter() + .map(serialized_indexed_chunk_bytes) + .sum::(); + let through_sequence = chunks.last().map_or(-1, |chunk| chunk.sequence); + let has_older = if let Some(continuation_sequence) = window_start_sequence { + conn.query_row( + "SELECT EXISTS( + SELECT 1 FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND sequence(0), + ) + .map_err(|err| format!("query events before replay turn slice: {err}"))? + != 0 + } else { + turn.turn_index > 0 + }; + Ok(ReplayChunkWindow { + cursor: ReplayCursor { + source_id: source.as_str().to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence, + }, + window_start_sequence, + chunks, + turn_headers: vec![turn], + total_turn_count: state.total_turns, + total_event_count: state.total_events, + has_older, + stats: ReplayStats { + ipc_bytes: ipc_bytes as u64, + ..hydrate_stats + }, + }) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn read_anchored_turn_chunks( + conn: &Connection, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + generation: &str, + turn_index: i64, + limits: ReplayLimits, +) -> Result<(Vec, Option), String> { + let anchor_limits = ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: limits.max_ipc_bytes, + }; + let mut chunks = read_chunks( + conn, + source, + display_session_id, + source_session_id, + generation, + "turn_index=?4", + &[turn_index], + anchor_limits, + QueryDirection::OldestFirst, + )?; + let Some(anchor_sequence) = chunks.first().map(|anchor| anchor.sequence) else { + return Ok((chunks, None)); + }; + let anchor_bytes = chunks + .first() + .map(serialized_indexed_chunk_bytes) + .unwrap_or_default(); + let Some(tail_limits) = limits.after_exact_turn_anchor(anchor_bytes) else { + return Ok((chunks, Some(anchor_sequence))); + }; + let mut tail = read_chunks( + conn, + source, + display_session_id, + source_session_id, + generation, + "turn_index=?4 AND sequence>?5", + &[turn_index, anchor_sequence], + tail_limits, + QueryDirection::NewestFirst, + )?; + tail.reverse(); + let tail_start_sequence = tail.first().map(|chunk| chunk.sequence); + let has_gap = if let Some(tail_start_sequence) = tail_start_sequence { + conn.query_row( + "SELECT EXISTS( + SELECT 1 FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND turn_index=?4 AND sequence>?5 AND sequence(0), + ) + .map_err(|err| format!("query exact replay turn gap: {err}"))? + != 0 + } else { + false + }; + let window_start_sequence = if has_gap { + tail_start_sequence + } else { + Some(anchor_sequence) + }; + chunks.extend(tail); + Ok((chunks, window_start_sequence)) +} + +pub(in crate::sources::imported_history::replay) fn hydrate_turn_if_needed( + conn: &mut Connection, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + state: &ReplayIndexState, + turn: &ReplayTurnHeader, +) -> Result { + if !matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + return Ok(ReplayStats::default()); + } + let indexed_count = conn + .query_row( + "SELECT COUNT(*) FROM imported_replay_source_rows WHERE source=?1 + AND source_session_id=?2 AND generation=?3 + AND source_order>=?4 AND source_order<=?5", + params![ + source.as_str(), + source_session_id, + state.generation, + turn.start_sequence, + turn.end_sequence.unwrap_or(turn.start_sequence) + ], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("count indexed replay turn: {err}"))? + .max(0) as u64; + if indexed_count >= turn.event_count { + return Ok(ReplayStats::default()); + } + let resolved = resolve_source(conn, source, display_session_id)?; + let write_revision = state.revision.saturating_add(1); + let tx = begin_replay_write_transaction(conn, "lazy replay turn")?; + let stats = sqlite_driver::hydrate_kv_turn( + &tx, + source, + display_session_id, + source_session_id, + &resolved.path, + &state.generation, + write_revision, + turn.start_sequence, + turn.end_sequence.unwrap_or(turn.start_sequence), + )?; + if stats.upserted_events > 0 { + let revision = publish_change_log( + &tx, + source, + source_session_id, + &state.generation, + state.revision, + write_revision, + &[], + )?; + tx.execute( + "UPDATE imported_replay_state SET revision=?1,updated_at=?2 + WHERE source=?3 AND source_session_id=?4 AND generation=?5", + params![ + revision.min(i64::MAX as u64) as i64, + Utc::now().to_rfc3339(), + source.as_str(), + source_session_id, + state.generation + ], + ) + .map_err(|err| format!("publish lazy replay turn revision: {err}"))?; + } + tx.commit() + .map_err(|err| format!("commit lazy replay turn: {err}"))?; + Ok(stats) +} + +pub(in crate::sources::imported_history::replay) fn read_scan_after( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + after_sequence: i64, + limits: ReplayLimits, + sync: ReplaySyncResult, +) -> Result { + if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + return read_lazy_kv_scan_after(conn, source, session_id, after_sequence, limits, sync); + } + let resolved = resolve_source(conn, source, session_id)?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after synchronization".to_string())?; + let chunks = read_chunks( + conn, + source, + session_id, + &resolved.source_session_id, + &state.generation, + "sequence > ?4", + &[after_sequence], + limits, + QueryDirection::OldestFirst, + )?; + let through_sequence = chunks.last().map_or(after_sequence, |chunk| chunk.sequence); + let has_more = conn + .query_row( + "SELECT EXISTS( + SELECT 1 FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND sequence > ?4 + )", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + through_sequence + ], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("query remaining replay events: {err}"))? + != 0; + let ipc_bytes = chunks + .iter() + .map(serialized_indexed_chunk_bytes) + .sum::(); + let mut stats = sync.stats; + stats.ipc_bytes = ipc_bytes as u64; + Ok(ReplayChunkScan { + cursor: ReplayCursor { + source_id: source.as_str().to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence, + }, + chunks, + has_more, + stats, + }) +} + +/// Cursor/Windsurf cold indexes intentionally materialize only the newest +/// turn body. A source-neutral forward scan must therefore hydrate exactly the +/// next logical turn before reading it; querying all indexed events directly +/// would jump across the unmaterialized prefix and silently truncate exports. +pub(super) fn read_lazy_kv_scan_after( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + after_sequence: i64, + limits: ReplayLimits, + sync: ReplaySyncResult, +) -> Result { + let resolved = resolve_source(conn, source, session_id)?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after synchronization".to_string())?; + let turn = conn + .query_row( + "SELECT turn_id,turn_index,start_sequence,end_sequence,started_at,ended_at,event_count + FROM imported_replay_turns + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND COALESCE(end_sequence,start_sequence)>?4 + ORDER BY turn_index ASC LIMIT 1", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + after_sequence + ], + |row| { + Ok(ReplayTurnHeader { + turn_id: row.get(0)?, + turn_index: row.get(1)?, + start_sequence: row.get(2)?, + end_sequence: row.get(3)?, + started_at: row.get(4)?, + ended_at: row.get(5)?, + event_count: row.get::<_, i64>(6)?.max(0) as u64, + }) + }, + ) + .optional() + .map_err(|err| format!("resolve next lazy replay scan turn: {err}"))?; + + let Some(turn) = turn else { + return Ok(ReplayChunkScan { + cursor: ReplayCursor { + source_id: source.as_str().to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence: after_sequence, + }, + chunks: Vec::new(), + has_more: false, + stats: sync.stats, + }); + }; + + let hydrate_stats = hydrate_turn_if_needed( + conn, + source, + session_id, + &resolved.source_session_id, + &state, + &turn, + )?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after scan hydration".to_string())?; + let chunks = read_chunks( + conn, + source, + session_id, + &resolved.source_session_id, + &state.generation, + "turn_index=?4 AND sequence>?5", + &[turn.turn_index, after_sequence], + limits, + QueryDirection::OldestFirst, + )?; + let last_chunk_sequence = chunks.last().map(|chunk| chunk.sequence); + let mut through_sequence = last_chunk_sequence.unwrap_or(after_sequence); + let has_indexed_more_in_turn = conn + .query_row( + "SELECT EXISTS( + SELECT 1 FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND turn_index=?4 AND sequence>?5 + )", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + turn.turn_index, + through_sequence + ], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("query remaining lazy replay turn events: {err}"))? + != 0; + // A source row may intentionally normalize to no visible event. Once all + // indexed events in this turn are consumed, advance over those positions + // so the next call can reach the following turn without looping. + if !has_indexed_more_in_turn { + through_sequence = through_sequence.max( + turn.end_sequence + .unwrap_or(turn.start_sequence) + .max(turn.start_sequence), + ); + } + let has_later_turn = conn + .query_row( + "SELECT EXISTS( + SELECT 1 FROM imported_replay_turns + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND turn_index>?4 + )", + params![ + source.as_str(), + resolved.source_session_id, + state.generation, + turn.turn_index + ], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("query later lazy replay turns: {err}"))? + != 0; + let ipc_bytes = chunks + .iter() + .map(serialized_indexed_chunk_bytes) + .sum::(); + let mut stats = sync.stats; + merge_stats(&mut stats, hydrate_stats); + stats.ipc_bytes = ipc_bytes as u64; + Ok(ReplayChunkScan { + cursor: ReplayCursor { + source_id: source.as_str().to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence, + }, + chunks, + has_more: has_indexed_more_in_turn || has_later_turn, + stats, + }) +} + +pub(super) fn merge_stats(target: &mut ReplayStats, extra: ReplayStats) { + target.parsed_bytes = target.parsed_bytes.saturating_add(extra.parsed_bytes); + target.parsed_rows = target.parsed_rows.saturating_add(extra.parsed_rows); + target.normalized_events = target + .normalized_events + .saturating_add(extra.normalized_events); + target.upserted_events = target.upserted_events.saturating_add(extra.upserted_events); + target.removed_events = target.removed_events.saturating_add(extra.removed_events); + target.ipc_bytes = target.ipc_bytes.saturating_add(extra.ipc_bytes); + target.not_ready |= extra.not_ready; +} + +pub(in crate::sources::imported_history::replay) fn read_delta( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + cursor: &ReplayCursor, + limits: ReplayLimits, + sync: ReplaySyncResult, +) -> Result { + let resolved = resolve_source(conn, source, session_id)?; + let state = load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after synchronization".to_string())?; + if cursor.generation != state.generation { + let mut replacement = read_recent_window(conn, source, session_id, limits)?; + replacement.stats.parsed_bytes = sync.stats.parsed_bytes; + replacement.stats.parsed_rows = sync.stats.parsed_rows; + replacement.stats.normalized_events = sync.stats.normalized_events; + replacement.stats.upserted_events = sync.stats.upserted_events; + return Ok(ReplayChunkDelta { + cursor: replacement.cursor, + chunks: replacement.chunks, + removed_event_ids: Vec::new(), + reset_required: true, + stats: replacement.stats, + }); + } + let (chunks, removed_event_ids, through_revision) = read_changes( + conn, + source, + session_id, + &resolved.source_session_id, + &state.generation, + limits, + cursor.revision, + )?; + let through_sequence = chunks + .iter() + .map(|chunk| chunk.sequence) + .max() + .map_or(cursor.through_sequence, |sequence| { + cursor.through_sequence.max(sequence) + }); + let ipc_bytes = chunks + .iter() + .map(serialized_indexed_chunk_bytes) + .sum::() + .saturating_add( + removed_event_ids + .iter() + .map(|event_id| serde_json::to_vec(event_id).map_or(0, |bytes| bytes.len())) + .sum::(), + ); + let mut stats = sync.stats; + stats.ipc_bytes = ipc_bytes as u64; + Ok(ReplayChunkDelta { + cursor: ReplayCursor { + source_id: source.as_str().to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: through_revision, + through_sequence, + }, + chunks, + removed_event_ids, + reset_required: sync.generation_changed, + stats, + }) +} + +pub(super) fn read_changes( + conn: &Connection, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + generation: &str, + limits: ReplayLimits, + after_revision: u64, +) -> Result<(Vec, Vec, u64), String> { + let mut stmt = conn + .prepare( + "SELECT event.sequence,event.event_id,event.turn_index,event.action_type, + event.function_name,event.created_at,event.args_preview_json, + event.result_preview_json,event.thread_id,event.process_id, + event.payloads_json,change.change_revision,change.change_kind, + change.event_id + FROM imported_replay_changes AS change + LEFT JOIN imported_replay_events AS event + ON event.source=change.source + AND event.source_session_id=change.source_session_id + AND event.generation=change.generation + AND event.event_id=change.event_id + WHERE change.source=?1 AND change.source_session_id=?2 + AND change.generation=?3 AND change.change_revision>?4 + ORDER BY change.change_revision ASC LIMIT ?5", + ) + .map_err(|err| format!("prepare replay change batch: {err}"))?; + let mut rows = stmt + .query(params![ + source.as_str(), + source_session_id, + generation, + after_revision.min(i64::MAX as u64) as i64, + limits.max_events as i64 + ]) + .map_err(|err| format!("query replay change batch: {err}"))?; + let mut chunks = Vec::new(); + let mut removed = Vec::new(); + let mut ipc_bytes = 0_usize; + let mut through_revision = after_revision; + let mut included_turns = HashSet::new(); + while let Some(row) = rows + .next() + .map_err(|err| format!("read replay change row: {err}"))? + { + let revision = row.get::<_, i64>(11).map_err(|err| err.to_string())?.max(0) as u64; + let kind: String = row.get(12).map_err(|err| err.to_string())?; + if kind == "remove" { + let event_id = row.get::<_, String>(13).map_err(|err| err.to_string())?; + let next_bytes = serde_json::to_vec(&event_id).map_or(0, |bytes| bytes.len()); + if ipc_bytes.saturating_add(next_bytes) > limits.max_ipc_bytes { + if removed.is_empty() && chunks.is_empty() { + return Err(format!( + "Replay removal {event_id} exceeds the {} byte compact delta budget", + limits.max_ipc_bytes + )); + } + break; + } + ipc_bytes = ipc_bytes.saturating_add(next_bytes); + removed.push(event_id); + through_revision = revision; + continue; + } + // A lagging consumer may encounter an upsert whose event was removed + // by a later, not-yet-consumed change. The later tombstone is the + // authoritative final state; advance past this obsolete snapshot. + if row + .get::<_, Option>(0) + .map_err(|err| err.to_string())? + .is_none() + { + through_revision = revision; + continue; + } + let turn_index = row.get::<_, i64>(2).map_err(|err| err.to_string())?; + if !included_turns.contains(&turn_index) && included_turns.len() >= limits.max_turns { + break; + } + let chunk = decode_indexed_chunk(row, display_session_id)?; + let next_bytes = serialized_indexed_chunk_bytes(&chunk); + if ipc_bytes.saturating_add(next_bytes) > limits.max_ipc_bytes { + if removed.is_empty() && chunks.is_empty() { + return Err(format!( + "Replay event {} exceeds the {} byte compact delta budget", + chunk.chunk.chunk_id, limits.max_ipc_bytes + )); + } + break; + } + ipc_bytes = ipc_bytes.saturating_add(next_bytes); + included_turns.insert(turn_index); + chunks.push(chunk); + through_revision = revision; + } + Ok((chunks, removed, through_revision)) +} + +#[derive(Clone, Copy)] +pub(super) enum QueryDirection { + NewestFirst, + OldestFirst, +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn read_chunks( + conn: &Connection, + source: ImportedHistorySourceId, + display_session_id: &str, + source_session_id: &str, + generation: &str, + predicate: &str, + extra_values: &[i64], + limits: ReplayLimits, + direction: QueryDirection, +) -> Result, String> { + let order = match direction { + QueryDirection::NewestFirst => "DESC", + QueryDirection::OldestFirst => "ASC", + }; + let sql = format!( + "SELECT sequence, event_id, turn_index, action_type, function_name, + created_at, args_preview_json, result_preview_json, + thread_id, process_id, payloads_json + FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND {predicate} + ORDER BY sequence {order} LIMIT {}", + limits.max_events + ); + let mut values = vec![ + rusqlite::types::Value::Text(source.as_str().to_string()), + rusqlite::types::Value::Text(source_session_id.to_string()), + rusqlite::types::Value::Text(generation.to_string()), + ]; + values.extend( + extra_values + .iter() + .copied() + .map(rusqlite::types::Value::Integer), + ); + let mut stmt = conn + .prepare(&sql) + .map_err(|err| format!("prepare bounded replay query: {err}"))?; + let mut rows = stmt + .query(rusqlite::params_from_iter(values)) + .map_err(|err| format!("query bounded replay rows: {err}"))?; + let mut chunks = Vec::new(); + let mut ipc_bytes = 0usize; + while let Some(row) = rows + .next() + .map_err(|err| format!("read bounded replay row: {err}"))? + { + let chunk = decode_indexed_chunk(row, display_session_id)?; + let next_bytes = serialized_indexed_chunk_bytes(&chunk); + if ipc_bytes.saturating_add(next_bytes) > limits.max_ipc_bytes { + if chunks.is_empty() { + return Err(format!( + "Replay event {} exceeds the {} byte compact window budget", + chunk.chunk.chunk_id, limits.max_ipc_bytes + )); + } + break; + } + ipc_bytes = ipc_bytes.saturating_add(next_bytes); + chunks.push(chunk); + } + Ok(chunks) +} + +pub(super) fn decode_indexed_chunk( + row: &Row<'_>, + display_session_id: &str, +) -> Result { + let args_json: String = row.get(6).map_err(|err| err.to_string())?; + let result_json: String = row.get(7).map_err(|err| err.to_string())?; + let payloads_json: String = row.get(10).map_err(|err| err.to_string())?; + Ok(ReplayIndexedChunk { + sequence: row.get(0).map_err(|err| err.to_string())?, + turn_index: row.get(2).map_err(|err| err.to_string())?, + chunk: ActivityChunk { + chunk_id: row.get(1).map_err(|err| err.to_string())?, + session_id: display_session_id.to_string(), + action_type: row.get(3).map_err(|err| err.to_string())?, + function: row.get(4).map_err(|err| err.to_string())?, + args: serde_json::from_str(&args_json) + .map_err(|err| format!("decode replay args preview: {err}"))?, + result: serde_json::from_str(&result_json) + .map_err(|err| format!("decode replay result preview: {err}"))?, + created_at: row.get(5).map_err(|err| err.to_string())?, + thread_id: row.get(8).map_err(|err| err.to_string())?, + process_id: row.get(9).map_err(|err| err.to_string())?, + broadcast_only: false, + }, + payloads: serde_json::from_str::>(&payloads_json) + .map_err(|err| format!("decode replay payload locators: {err}"))?, + }) +} + +pub(super) fn read_turn_headers( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + oldest_turn: i64, + newest_turn: i64, +) -> Result, String> { + let mut stmt = conn + .prepare( + "SELECT turn_id, turn_index, start_sequence, end_sequence, + started_at, ended_at, event_count + FROM imported_replay_turns + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND turn_index BETWEEN ?4 AND ?5 + ORDER BY turn_index ASC", + ) + .map_err(|err| format!("prepare replay turn headers: {err}"))?; + let rows = stmt + .query_map( + params![ + source.as_str(), + source_session_id, + generation, + oldest_turn, + newest_turn + ], + |row| { + Ok(ReplayTurnHeader { + turn_id: row.get(0)?, + turn_index: row.get(1)?, + start_sequence: row.get(2)?, + end_sequence: row.get(3)?, + started_at: row.get(4)?, + ended_at: row.get(5)?, + event_count: row.get::<_, i64>(6)?.max(0) as u64, + }) + }, + ) + .map_err(|err| format!("query replay turn headers: {err}"))?; + rows.collect::, _>>() + .map_err(|err| format!("read replay turn header: {err}")) +} + +pub(super) fn serialized_indexed_chunk_bytes(chunk: &ReplayIndexedChunk) -> usize { + // Payload descriptors cross the Rust/JS boundary after normalization too; + // omitting them here let descriptor-heavy events bypass maxIpcBytes. + serialized_json_bytes(&chunk.chunk).saturating_add(serialized_json_bytes(&chunk.payloads)) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/source_identity.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/source_identity.rs new file mode 100644 index 000000000..b9fc1cc7b --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/source_identity.rs @@ -0,0 +1,256 @@ +use super::*; +use crate::sources::imported_history::replay::drivers::common::ContentDigest; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum ReplayDriverKind { + CodexJsonl, + SharedJsonl, + Sqlite, + StructuredSqlite, + WholeJson, +} + +/// Exhaustive storage-driver routing for every built-in source. Adding a new +/// [`ImportedHistorySourceId`] without choosing a replay driver is a compile +/// error; synchronization and range reads cannot drift onto different paths. +pub(super) const fn replay_driver_kind(source: ImportedHistorySourceId) -> ReplayDriverKind { + match source { + ImportedHistorySourceId::CodexApp => ReplayDriverKind::CodexJsonl, + ImportedHistorySourceId::ClaudeCode + | ImportedHistorySourceId::WorkBuddy + | ImportedHistorySourceId::Trae + | ImportedHistorySourceId::Qoder + | ImportedHistorySourceId::Omp + | ImportedHistorySourceId::QoderCli => ReplayDriverKind::SharedJsonl, + ImportedHistorySourceId::OpenCode + | ImportedHistorySourceId::MimoCode + | ImportedHistorySourceId::ZCode + | ImportedHistorySourceId::CursorIde + | ImportedHistorySourceId::Windsurf => ReplayDriverKind::Sqlite, + ImportedHistorySourceId::CursorCli | ImportedHistorySourceId::Warp => { + ReplayDriverKind::StructuredSqlite + } + ImportedHistorySourceId::Cline => ReplayDriverKind::WholeJson, + } +} + +pub(super) fn is_shared_jsonl(source: ImportedHistorySourceId) -> bool { + replay_driver_kind(source) == ReplayDriverKind::SharedJsonl +} + +pub(super) fn is_sqlite_replay(source: ImportedHistorySourceId) -> bool { + replay_driver_kind(source) == ReplayDriverKind::Sqlite +} + +pub(super) fn is_structured_sqlite(source: ImportedHistorySourceId) -> bool { + replay_driver_kind(source) == ReplayDriverKind::StructuredSqlite +} + +pub(super) fn is_physical_sqlite(source: ImportedHistorySourceId) -> bool { + is_sqlite_replay(source) || is_structured_sqlite(source) +} + +pub(in crate::sources::imported_history::replay) fn resolve_source( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, +) -> Result { + let requested_key = source.source_session_id(session_id)?; + conn.query_row( + "SELECT source_session_id, source_path + FROM imported_history_session_cache + WHERE source=?1 + AND ( + source_session_id=?2 + OR substr(source_session_id, -(length(?2) + 1))=('-' || ?2) + ) + ORDER BY CASE WHEN source_session_id=?2 THEN 0 ELSE 1 END, + updated_at_ms DESC + LIMIT 1", + params![source.as_str(), requested_key], + |row| { + Ok(ResolvedSource { + source_session_id: row.get(0)?, + path: PathBuf::from(row.get::<_, String>(1)?), + }) + }, + ) + .optional() + .map_err(|err| format!("resolve replay source path: {err}"))? + .ok_or_else(|| { + format!( + "Imported replay source is not indexed yet: {} {session_id}", + source.as_str() + ) + }) +} + +pub(in crate::sources::imported_history::replay) fn load_state( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, +) -> Result, String> { + conn.query_row( + "SELECT source_session_id, generation, revision, parser_version, + source_identity, driver_cursor_json, indexed_size_bytes, + indexed_mtime_ns, total_events, total_turns, updated_at + FROM imported_replay_state + WHERE source=?1 AND source_session_id=?2 AND valid=1", + params![source.as_str(), source_session_id], + |row| { + Ok(ReplayIndexState { + generation: row.get(1)?, + revision: row.get::<_, i64>(2)?.max(0) as u64, + parser_version: row.get::<_, i64>(3)?.max(0) as u32, + source_identity: row.get(4)?, + driver_cursor_json: row.get(5)?, + indexed_size_bytes: row.get::<_, i64>(6)?.max(0) as u64, + indexed_mtime_ns: row.get(7)?, + total_events: row.get::<_, i64>(8)?.max(0) as u64, + total_turns: row.get::<_, i64>(9)?.max(0) as u64, + state_updated_at_ms: row + .get::<_, String>(10)? + .parse::>() + .map(|timestamp| timestamp.timestamp_millis()) + .unwrap_or_default(), + }) + }, + ) + .optional() + .map_err(|err| format!("load imported replay state: {err}")) +} + +pub(super) fn source_snapshot( + path: &Path, + source: ImportedHistorySourceId, +) -> Result { + let metadata = fs::metadata(path) + .map_err(|err| format!("stat replay source {}: {err}", path.display()))?; + let mut size_bytes = metadata.len(); + let mut mtime_ns = metadata + .modified() + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map(|duration| duration.as_secs() as i64 * 1_000_000_000 + duration.subsec_nanos() as i64) + .unwrap_or_default(); + if is_physical_sqlite(source) { + // WAL contains committed logical changes. SHM is a transient lock and + // reader coordination file; including it makes our own SQLite reads + // look like source mutations and self-trigger replay scans. + let wal = PathBuf::from(format!("{}-wal", path.to_string_lossy())); + if let Ok(wal_metadata) = fs::metadata(wal) { + size_bytes = size_bytes.saturating_add(wal_metadata.len()); + let wal_mtime = wal_metadata + .modified() + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map(|duration| { + duration.as_secs() as i64 * 1_000_000_000 + duration.subsec_nanos() as i64 + }) + .unwrap_or_default(); + mtime_ns = mtime_ns.max(wal_mtime); + } + } + let canonical = fs::canonicalize(path).unwrap_or_else(|_| path.to_path_buf()); + #[cfg(unix)] + let identity = { + use std::os::unix::fs::MetadataExt; + format!( + "{}:{}:{}", + canonical.display(), + metadata.dev(), + metadata.ino() + ) + }; + #[cfg(not(unix))] + let identity = canonical.to_string_lossy().to_string(); + Ok(SourcePhysicalSnapshot { + identity, + size_bytes, + mtime_ns, + sample_fingerprint: String::new(), + }) +} + +pub(super) fn sqlite_physical_fingerprint(path: &Path) -> Result { + let mut hash = ContentDigest::default(); + for candidate in [ + path.to_path_buf(), + PathBuf::from(format!("{}-wal", path.to_string_lossy())), + ] { + hash.update_part(candidate.to_string_lossy().as_bytes()); + match fs::metadata(&candidate) { + Ok(metadata) => { + hash.update_part(&metadata.len().to_le_bytes()); + let modified = metadata + .modified() + .ok() + .and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok()) + .map(|duration| { + duration.as_secs() as i64 * 1_000_000_000 + duration.subsec_nanos() as i64 + }) + .unwrap_or_default(); + hash.update_part(&modified.to_le_bytes()); + } + Err(err) if err.kind() == std::io::ErrorKind::NotFound => hash.update_part(b"missing"), + Err(err) => { + return Err(format!( + "stat replay SQLite sidecar {}: {err}", + candidate.display() + )) + } + } + } + Ok(hash.finish_hex()) +} + +pub(super) fn sampled_file_fingerprint(path: &Path, size: u64) -> Result { + #[cfg(test)] + FILE_SAMPLE_COUNT.with(|count| count.set(count.get().saturating_add(1))); + const SAMPLE: usize = 4096; + let mut file = fs::File::open(path) + .map_err(|err| format!("open replay source sample {}: {err}", path.display()))?; + let mut hash = ContentDigest::default(); + hash.update_part(&size.to_le_bytes()); + for offset in [ + 0, + size.saturating_sub(SAMPLE as u64) / 2, + size.saturating_sub(SAMPLE as u64), + ] { + file.seek(SeekFrom::Start(offset)) + .map_err(|err| format!("seek replay source sample: {err}"))?; + let mut buffer = vec![0_u8; SAMPLE.min(size.saturating_sub(offset) as usize)]; + file.read_exact(&mut buffer) + .map_err(|err| format!("read replay source sample: {err}"))?; + hash.update_part(&offset.to_le_bytes()); + hash.update_part(&buffer); + } + Ok(hash.finish_hex()) +} + +#[cfg(test)] +std::thread_local! { + static FILE_SAMPLE_COUNT: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +#[cfg(test)] +pub(in crate::sources::imported_history::replay) fn take_file_sample_count() -> usize { + FILE_SAMPLE_COUNT.with(|count| count.replace(0)) +} + +pub(super) fn make_generation( + source: ImportedHistorySourceId, + source_session_id: &str, + parser_version: u32, + snapshot: &SourcePhysicalSnapshot, +) -> String { + let mut hash = ContentDigest::default(); + hash.update_part(source.as_str().as_bytes()); + hash.update_part(source_session_id.as_bytes()); + hash.update_part(&parser_version.to_le_bytes()); + hash.update_part(snapshot.identity.as_bytes()); + hash.update_part(&snapshot.size_bytes.to_le_bytes()); + hash.update_part(&snapshot.mtime_ns.to_le_bytes()); + hash.update_part(snapshot.sample_fingerprint.as_bytes()); + format!("r{parser_version}-{}", hash.finish_hex()) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/sync.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/sync.rs new file mode 100644 index 000000000..6fac816be --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/index/sync.rs @@ -0,0 +1,710 @@ +use super::source_identity::*; +use super::*; + +pub(in crate::sources::imported_history::replay) fn sync_index( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, +) -> Result { + let resolved = resolve_source(conn, source, session_id)?; + let mut snapshot = source_snapshot(&resolved.path, source)?; + let old_state = load_state(conn, source, &resolved.source_session_id)?; + let qoder_probe = if source == ImportedHistorySourceId::Qoder { + Some(qoder_sidecar::probe(&resolved.source_session_id)?) + } else { + None + }; + let parser_version = source.descriptor().parser_version; + // True unchanged fast path: metadata only. No file samples, parser rows, + // normalization, EventStore upserts, or hidden I/O counters move. + let now_ms = Utc::now().timestamp_millis(); + if old_state.as_ref().is_some_and(|state| { + state.parser_version == parser_version + && state.source_identity == snapshot.identity + && state.indexed_size_bytes == snapshot.size_bytes + && state.indexed_mtime_ns == snapshot.mtime_ns + && qoder_probe.as_ref().is_none_or(|probe| { + qoder_sidecar::cursor_signature(&state.driver_cursor_json).as_deref() + == Some(probe.signature.as_str()) + }) + && now_ms.saturating_sub(state.state_updated_at_ms) < 60_000 + }) { + return Ok(ReplaySyncResult::default()); + } + snapshot.sample_fingerprint = if is_physical_sqlite(source) { + sqlite_physical_fingerprint(&resolved.path)? + } else { + sampled_file_fingerprint(&resolved.path, snapshot.size_bytes)? + }; + if let Some(rejection_kind) = rejected_snapshot_kind(source) { + if old_state.is_some() + && rejected_snapshot_matches( + conn, + source, + &resolved.source_session_id, + parser_version, + &snapshot, + rejection_kind, + )? + { + return Ok(not_ready_sync_result()); + } + } + let cursor_fingerprint = old_state.as_ref().and_then(|state| match source { + ImportedHistorySourceId::CodexApp => { + codex_jsonl::cursor_fingerprint(&state.driver_cursor_json) + } + source if is_shared_jsonl(source) => { + jsonl_driver::cursor_fingerprint(&state.driver_cursor_json) + } + ImportedHistorySourceId::Cline => { + whole_json_driver::cursor_fingerprint(&state.driver_cursor_json) + } + _ => None, + }); + let lineage_matches = old_state.as_ref().is_none_or(|state| { + if source == ImportedHistorySourceId::CodexApp { + snapshot.size_bytes < state.indexed_size_bytes + || codex_jsonl::cursor_matches_source(&resolved.path, &state.driver_cursor_json) + } else if is_shared_jsonl(source) { + (snapshot.size_bytes < state.indexed_size_bytes + || jsonl_driver::cursor_matches_source(&resolved.path, &state.driver_cursor_json)) + && qoder_probe.as_ref().is_none_or(|probe| { + qoder_sidecar::cursor_lineage_matches(&state.driver_cursor_json, probe) + }) + } else if source == ImportedHistorySourceId::CursorCli { + structured_driver::cursor_lineage_matches(&resolved.path, &state.driver_cursor_json) + .unwrap_or(false) + } else { + true + } + }); + let sqlite_schema_changed = if is_sqlite_replay(source) { + let current = sqlite_driver::database_schema_version(&resolved.path)?; + old_state.as_ref().is_some_and(|state| { + sqlite_driver::cursor_schema_version(&state.driver_cursor_json) + .is_some_and(|previous| previous != current) + }) + } else if is_structured_sqlite(source) { + let current = structured_driver::database_schema_version(&resolved.path)?; + old_state.as_ref().is_some_and(|state| { + structured_driver::cursor_schema_version(&state.driver_cursor_json) + .is_some_and(|previous| previous != current) + }) + } else { + false + }; + let generation_changed = old_state.as_ref().is_none_or(|state| { + state.parser_version != parser_version + || state.source_identity != snapshot.identity + || sqlite_schema_changed + || !lineage_matches + || (source == ImportedHistorySourceId::Cline + && (snapshot.size_bytes != state.indexed_size_bytes + || snapshot.mtime_ns != state.indexed_mtime_ns + || cursor_fingerprint.as_deref() != Some(snapshot.sample_fingerprint.as_str()))) + || (!is_physical_sqlite(source) + && source != ImportedHistorySourceId::Cline + && (snapshot.size_bytes < state.indexed_size_bytes + || (snapshot.size_bytes == state.indexed_size_bytes + && cursor_fingerprint.as_deref() + != Some(snapshot.sample_fingerprint.as_str())))) + }); + + if let Some(state) = old_state.as_ref() { + if !is_physical_sqlite(source) + && !generation_changed + && state.indexed_size_bytes == snapshot.size_bytes + && cursor_fingerprint.as_deref() == Some(snapshot.sample_fingerprint.as_str()) + && qoder_probe.as_ref().is_none_or(|probe| { + qoder_sidecar::cursor_signature(&state.driver_cursor_json).as_deref() + == Some(probe.signature.as_str()) + }) + { + conn.execute( + "UPDATE imported_replay_state SET updated_at=?3 + WHERE source=?1 AND source_session_id=?2 AND valid=1", + params![ + source.as_str(), + resolved.source_session_id, + Utc::now().to_rfc3339() + ], + ) + .map_err(|err| format!("touch replay integrity watermark: {err}"))?; + return Ok(ReplaySyncResult::default()); + } + } + + let generation = if generation_changed { + let base = make_generation( + source, + &resolved.source_session_id, + parser_version, + &snapshot, + ); + qoder_probe.as_ref().map_or(base.clone(), |probe| { + format!( + "{base}-q{}", + &probe.signature[..probe.signature.len().min(12)] + ) + }) + } else { + old_state + .as_ref() + .expect("unchanged generation has state") + .generation + .clone() + }; + let previous_generation = old_state.as_ref().map(|state| state.generation.clone()); + let write_revision = if generation_changed { + 1 + } else { + old_state + .as_ref() + .map_or(1, |state| state.revision.saturating_add(1)) + }; + conn.pragma_update(None, "cache_size", -REPLAY_INDEX_CACHE_KIB) + .map_err(|err| format!("bound imported replay SQLite page cache: {err}"))?; + // Reserve the SQLite writer before streaming/parsing the external source. + // A deferred transaction can hold an old read snapshot for many seconds + // and then fail with SQLITE_BUSY_SNAPSHOT on its first artifact/index + // write if another startup writer committed in the meantime. IMMEDIATE + // makes that contention wait at the transaction boundary instead. + let tx = begin_replay_write_transaction(conn, "imported replay")?; + let previous = (!generation_changed) + .then_some(old_state.as_ref()) + .flatten(); + let outcome = match replay_driver_kind(source) { + ReplayDriverKind::CodexJsonl => { + let outcome = codex_jsonl::sync( + &tx, + session_id, + &resolved.source_session_id, + &resolved.path, + &generation, + write_revision, + previous, + &snapshot.sample_fingerprint, + )?; + debug_assert!(!outcome.changed || outcome.stats.upserted_events > 0); + DriverSyncOutcome { + stats: outcome.stats, + driver_cursor_json: outcome.driver_cursor_json, + indexed_size_bytes: outcome.indexed_size_bytes, + total_events: outcome.total_events, + total_turns: outcome.total_turns, + removed_event_ids: Vec::new(), + } + } + ReplayDriverKind::SharedJsonl => { + let outcome = jsonl_driver::sync( + &tx, + source, + session_id, + &resolved.source_session_id, + &resolved.path, + &generation, + write_revision, + previous, + &snapshot.sample_fingerprint, + )?; + debug_assert!( + !outcome.changed + || outcome.stats.upserted_events > 0 + || outcome.stats.removed_events > 0 + ); + DriverSyncOutcome { + stats: outcome.stats, + driver_cursor_json: outcome.driver_cursor_json, + indexed_size_bytes: outcome.indexed_size_bytes, + total_events: outcome.total_events, + total_turns: outcome.total_turns, + // Qoder stages compact tombstones directly in the same + // transaction; other JSONL adapters are append/reset only. + removed_event_ids: Vec::new(), + } + } + ReplayDriverKind::Sqlite => { + let outcome = sqlite_driver::sync( + &tx, + source, + session_id, + &resolved.source_session_id, + &resolved.path, + &generation, + write_revision, + previous, + )?; + debug_assert!( + !outcome.changed + || outcome.stats.upserted_events > 0 + || outcome.stats.removed_events > 0 + ); + DriverSyncOutcome { + stats: outcome.stats, + driver_cursor_json: outcome.driver_cursor_json, + indexed_size_bytes: snapshot.size_bytes, + total_events: outcome.total_events, + total_turns: outcome.total_turns, + removed_event_ids: outcome.removed_event_ids, + } + } + ReplayDriverKind::StructuredSqlite => { + let outcome = match structured_driver::sync( + &tx, + source, + session_id, + &resolved.source_session_id, + &resolved.path, + &generation, + write_revision, + previous, + ) { + Ok(outcome) => outcome, + Err(error) + if old_state.is_some() + && error.starts_with( + "Cursor CLI replay lineage changed during synchronization", + ) => + { + drop(tx); + record_rejected_snapshot( + conn, + source, + &resolved.source_session_id, + parser_version, + &snapshot, + RejectedSnapshotKind::CursorCliLineageChanged, + )?; + return Ok(not_ready_sync_result()); + } + Err(error) => return Err(error), + }; + debug_assert!( + !outcome.changed + || outcome.stats.upserted_events > 0 + || outcome.stats.removed_events > 0 + ); + DriverSyncOutcome { + stats: outcome.stats, + driver_cursor_json: outcome.driver_cursor_json, + indexed_size_bytes: snapshot.size_bytes, + total_events: outcome.total_events, + total_turns: outcome.total_turns, + removed_event_ids: outcome.removed_event_ids, + } + } + ReplayDriverKind::WholeJson => { + let outcome = match whole_json_driver::sync( + &tx, + session_id, + &resolved.source_session_id, + &resolved.path, + &generation, + write_revision, + previous, + &snapshot.sample_fingerprint, + ) { + Ok(outcome) => outcome, + Err(error) + if old_state.is_some() && error.starts_with("parse Cline replay source") => + { + drop(tx); + record_rejected_snapshot( + conn, + source, + &resolved.source_session_id, + parser_version, + &snapshot, + RejectedSnapshotKind::ClineInvalidDocument, + )?; + return Ok(not_ready_sync_result()); + } + Err(error) => return Err(error), + }; + DriverSyncOutcome { + stats: outcome.stats, + driver_cursor_json: outcome.driver_cursor_json, + indexed_size_bytes: outcome.indexed_size_bytes, + total_events: outcome.total_events, + total_turns: outcome.total_turns, + removed_event_ids: Vec::new(), + } + } + }; + // A whole-document writer can replace the file while the streaming + // visitor is running. Publish only when the exact snapshot parsed above + // is still current; otherwise roll the transaction back and keep the + // previous valid generation visible. This also avoids a separate 30 MiB + // preflight parse on every complete rewrite. + if source == ImportedHistorySourceId::Cline { + let mut after_parse = source_snapshot(&resolved.path, source)?; + after_parse.sample_fingerprint = + sampled_file_fingerprint(&resolved.path, after_parse.size_bytes)?; + if after_parse.identity != snapshot.identity + || after_parse.size_bytes != snapshot.size_bytes + || after_parse.mtime_ns != snapshot.mtime_ns + || after_parse.sample_fingerprint != snapshot.sample_fingerprint + { + drop(tx); + if old_state.is_some() { + return Ok(ReplaySyncResult { + stats: ReplayStats { + not_ready: true, + ..ReplayStats::default() + }, + generation_changed: false, + }); + } + return Err( + "Cline replay source changed while it was being indexed; retry".to_string(), + ); + } + } + let revision = if generation_changed { + 1 + } else { + publish_change_log( + &tx, + source, + &resolved.source_session_id, + &generation, + old_state.as_ref().map_or(0, |state| state.revision), + write_revision, + &outcome.removed_event_ids, + )? + }; + tx.execute( + "INSERT INTO imported_replay_state ( + source, source_session_id, generation, revision, parser_version, + source_identity, driver_cursor_json, indexed_size_bytes, + indexed_mtime_ns, total_events, total_turns, valid, updated_at + ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, 1, ?12) + ON CONFLICT(source, source_session_id) DO UPDATE SET + generation=excluded.generation, + revision=excluded.revision, + parser_version=excluded.parser_version, + source_identity=excluded.source_identity, + driver_cursor_json=excluded.driver_cursor_json, + indexed_size_bytes=excluded.indexed_size_bytes, + indexed_mtime_ns=excluded.indexed_mtime_ns, + total_events=excluded.total_events, + total_turns=excluded.total_turns, + valid=1, + updated_at=excluded.updated_at", + params![ + source.as_str(), + resolved.source_session_id, + generation, + revision as i64, + parser_version as i64, + snapshot.identity, + outcome.driver_cursor_json, + outcome.indexed_size_bytes as i64, + snapshot.mtime_ns, + outcome.total_events as i64, + outcome.total_turns as i64, + Utc::now().to_rfc3339(), + ], + ) + .map_err(|err| format!("publish imported replay state: {err}"))?; + clear_rejected_snapshot(&tx, source, &resolved.source_session_id)?; + + crate::sources::imported_history::catalog::publish_from_replay_tx( + &tx, + source, + &resolved.source_session_id, + &generation, + outcome.total_events, + generation_changed || outcome.stats.upserted_events > 0 || outcome.stats.removed_events > 0, + snapshot.mtime_ns, + &outcome.driver_cursor_json, + )?; + + if generation_changed { + if let Some(previous_generation) = previous_generation { + if previous_generation != generation { + tx.execute( + "DELETE FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced replay events: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_turns + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced replay turns: {err}"))?; + // SQLite drivers keep compact hashes for ignored as well as + // rendered rows; a replaced generation must retire both. + tx.execute( + "DELETE FROM imported_replay_source_rows + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced replay source rows: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_changes + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced replay changes: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_structured_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced structured replay events: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_structured_rows + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced structured replay rows: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_payload_artifact_refs + WHERE source=?1 AND source_session_id=?2 AND generation=?3", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced replay payload artifact refs: {err}"))?; + tx.execute( + "DELETE FROM imported_replay_payload_artifacts + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND NOT EXISTS( + SELECT 1 FROM imported_replay_shell_segments AS shell + WHERE shell.source=imported_replay_payload_artifacts.source + AND shell.source_session_id=imported_replay_payload_artifacts.source_session_id + AND shell.generation=imported_replay_payload_artifacts.generation + AND shell.content_hash=imported_replay_payload_artifacts.content_hash + )", + params![ + source.as_str(), + resolved.source_session_id, + previous_generation + ], + ) + .map_err(|err| format!("remove replaced replay payload artifacts: {err}"))?; + } + } + } + tx.commit() + .map_err(|err| format!("commit imported replay index: {err}"))?; + conn.execute_batch("PRAGMA shrink_memory") + .map_err(|err| format!("release imported replay SQLite page cache: {err}"))?; + Ok(ReplaySyncResult { + stats: outcome.stats, + generation_changed, + }) +} + +pub(super) const fn rejected_snapshot_kind( + source: ImportedHistorySourceId, +) -> Option { + match source { + ImportedHistorySourceId::Cline => Some(RejectedSnapshotKind::ClineInvalidDocument), + ImportedHistorySourceId::CursorCli => Some(RejectedSnapshotKind::CursorCliLineageChanged), + _ => None, + } +} + +pub(super) fn not_ready_sync_result() -> ReplaySyncResult { + ReplaySyncResult { + stats: ReplayStats { + not_ready: true, + ..ReplayStats::default() + }, + generation_changed: false, + } +} + +pub(super) fn rejected_snapshot_matches( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + parser_version: u32, + snapshot: &SourcePhysicalSnapshot, + rejection_kind: RejectedSnapshotKind, +) -> Result { + conn.query_row( + "SELECT EXISTS( + SELECT 1 FROM imported_replay_rejected_snapshots + WHERE source=?1 AND source_session_id=?2 AND parser_version=?3 + AND source_identity=?4 AND source_size_bytes=?5 + AND source_mtime_ns=?6 AND sample_fingerprint=?7 + AND rejection_kind=?8 + )", + params![ + source.as_str(), + source_session_id, + i64::from(parser_version), + snapshot.identity, + snapshot.size_bytes.min(i64::MAX as u64) as i64, + snapshot.mtime_ns, + snapshot.sample_fingerprint, + rejection_kind.as_str(), + ], + |row| row.get::<_, i64>(0), + ) + .map(|exists| exists != 0) + .map_err(|err| format!("query rejected replay snapshot: {err}")) +} + +pub(super) fn record_rejected_snapshot( + conn: &mut Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + parser_version: u32, + snapshot: &SourcePhysicalSnapshot, + rejection_kind: RejectedSnapshotKind, +) -> Result<(), String> { + // The failed generation transaction has already rolled back. Publish the + // physical rejection watermark in a separate short transaction so a crash + // can never make it visible by partially overwriting the last valid state. + let tx = begin_replay_write_transaction(conn, "rejected replay snapshot")?; + tx.execute( + "INSERT INTO imported_replay_rejected_snapshots( + source,source_session_id,parser_version,source_identity, + source_size_bytes,source_mtime_ns,sample_fingerprint, + rejection_kind,rejected_at + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9) + ON CONFLICT(source,source_session_id) DO UPDATE SET + parser_version=excluded.parser_version, + source_identity=excluded.source_identity, + source_size_bytes=excluded.source_size_bytes, + source_mtime_ns=excluded.source_mtime_ns, + sample_fingerprint=excluded.sample_fingerprint, + rejection_kind=excluded.rejection_kind, + rejected_at=excluded.rejected_at", + params![ + source.as_str(), + source_session_id, + i64::from(parser_version), + snapshot.identity, + snapshot.size_bytes.min(i64::MAX as u64) as i64, + snapshot.mtime_ns, + snapshot.sample_fingerprint, + rejection_kind.as_str(), + Utc::now().to_rfc3339(), + ], + ) + .map_err(|err| format!("record rejected replay snapshot: {err}"))?; + tx.commit() + .map_err(|err| format!("commit rejected replay snapshot: {err}")) +} + +pub(super) fn clear_rejected_snapshot( + tx: &rusqlite::Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, +) -> Result<(), String> { + tx.execute( + "DELETE FROM imported_replay_rejected_snapshots + WHERE source=?1 AND source_session_id=?2", + params![source.as_str(), source_session_id], + ) + .map(|_| ()) + .map_err(|err| format!("clear rejected replay snapshot: {err}")) +} + +pub(super) fn publish_change_log( + tx: &rusqlite::Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + base_revision: u64, + temporary_revision: u64, + removed_event_ids: &[String], +) -> Result { + let base = base_revision.min(i64::MAX as u64) as i64; + let temporary = temporary_revision.min(i64::MAX as u64) as i64; + tx.execute( + "INSERT INTO imported_replay_changes( + source,source_session_id,generation,change_revision,event_id,change_kind,sequence + ) + SELECT source,source_session_id,generation, + ?1 + ROW_NUMBER() OVER (ORDER BY sequence,event_id), + event_id,'upsert',sequence + FROM imported_replay_events + WHERE source=?2 AND source_session_id=?3 AND generation=?4 + AND event_revision=?5", + params![ + base, + source.as_str(), + source_session_id, + generation, + temporary + ], + ) + .map_err(|err| format!("append replay upsert changes: {err}"))?; + tx.execute( + "UPDATE imported_replay_events AS event SET event_revision=( + SELECT change_revision FROM imported_replay_changes AS change + WHERE change.source=event.source + AND change.source_session_id=event.source_session_id + AND change.generation=event.generation + AND change.event_id=event.event_id + AND change.change_kind='upsert' + ORDER BY change.change_revision DESC LIMIT 1 + ) WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND event_revision=?4", + params![source.as_str(), source_session_id, generation, temporary], + ) + .map_err(|err| format!("assign per-change replay revisions: {err}"))?; + let upsert_count = tx + .query_row( + "SELECT COUNT(*) FROM imported_replay_changes + WHERE source=?1 AND source_session_id=?2 AND generation=?3 + AND change_revision>?4", + params![source.as_str(), source_session_id, generation, base], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("count replay upsert changes: {err}"))? + .max(0) as u64; + let mut revision = base_revision.saturating_add(upsert_count); + for event_id in removed_event_ids { + revision = revision.saturating_add(1); + tx.execute( + "INSERT INTO imported_replay_changes( + source,source_session_id,generation,change_revision,event_id,change_kind,sequence + ) VALUES(?1,?2,?3,?4,?5,'remove',NULL)", + params![ + source.as_str(), + source_session_id, + generation, + revision.min(i64::MAX as u64) as i64, + event_id + ], + ) + .map_err(|err| format!("append replay removal change: {err}"))?; + } + Ok(revision) +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/metadata_projection.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/metadata_projection.rs new file mode 100644 index 000000000..8506b170d --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/metadata_projection.rs @@ -0,0 +1,484 @@ +//! Bounded turn-metadata projection over the compact imported replay index. +//! +//! This intentionally does not use [`super::read_turn_window`]. A rendered +//! replay window is capped at 200 events and 4 MiB, while metadata must fold +//! every event in the requested turn. Rows are therefore streamed directly +//! from SQLite and only payload columns required by the tool classifier are +//! copied out of SQLite. + +use std::collections::HashSet; + +use rusqlite::{params, Connection, OptionalExtension}; +use serde_json::Value; + +use crate::projectors::turn_metadata::{ + metadata_projection_requirements, ProjectedTurnMetadata, TurnMetadataAccumulator, +}; +use crate::sources::imported_history; + +use super::{index, ImportedHistorySourceId, ReplayTurnHeader}; + +const MAX_REQUESTED_TURNS: usize = 500; +const TURN_INDEX_SELECTOR_PREFIX: &str = "__external_replay_turn_index__:"; + +#[derive(Debug, Clone)] +struct ProjectionTurn { + header: ReplayTurnHeader, + next_turn_started_at: Option, + source_turn_id: String, +} + +/// Synchronize the compact replay index and project only the requested turns. +/// +/// `None` selects at most the newest turn, which keeps legacy callers that do +/// not yet send visible turn ids bounded. `Some(&[])` performs no source read. +/// The returned turns are ordered by their source turn index, independently of +/// the caller's id order. +pub(super) fn project_turn_metadata( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + requested_turn_ids: Option<&[String]>, +) -> Result, String> { + source.validate_session_id(session_id)?; + if requested_turn_ids.is_some_and(|ids| ids.is_empty()) { + return Ok(Vec::new()); + } + if requested_turn_ids.is_some_and(|ids| ids.len() > MAX_REQUESTED_TURNS) { + return Err(format!( + "At most {MAX_REQUESTED_TURNS} turn summaries can be loaded at once" + )); + } + + index::sync_index(conn, source, session_id)?; + let resolved = index::resolve_source(conn, source, session_id)?; + let state = index::load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared after synchronization".to_string())?; + let projection_generation = state.generation.clone(); + let turns = select_projection_turns( + conn, + source, + &resolved.source_session_id, + &projection_generation, + requested_turn_ids, + )?; + + // Cursor IDE and Windsurf cold indexes retain compact headers for all + // turns but materialize only a recent body. Hydrate exactly the selected + // body before projection; re-read state each time because hydration may + // advance the replay revision. The generation itself is immutable for + // this projection: mixing new event rows with old turn headers would + // silently attach metadata to the wrong turn after a replace/truncate. + for turn in &turns { + let current_state = load_projection_state_for_generation( + conn, + source, + &resolved.source_session_id, + &projection_generation, + "hydrating replay turn metadata", + )?; + index::hydrate_turn_if_needed( + conn, + source, + session_id, + &resolved.source_session_id, + ¤t_state, + &turn.header, + )?; + load_projection_state_for_generation( + conn, + source, + &resolved.source_session_id, + &projection_generation, + "finishing replay turn hydration", + )?; + } + + load_projection_state_for_generation( + conn, + source, + &resolved.source_session_id, + &projection_generation, + "projecting replay turn metadata", + )?; + project_indexed_turns( + conn, + source, + &resolved.source_session_id, + &projection_generation, + &turns, + ) +} + +/// Project metadata from the last atomically published compact index without +/// synchronizing the provider or taking a write transaction. +/// +/// Foreground ChatHistory calls this after a bounded replay window has +/// already supplied a generation. JSONL, row-store SQLite, structured SQLite, +/// and whole-JSON adapters project their already-indexed rows. Cursor IDE and +/// Windsurf retain every turn header but lazily materialize old bodies, so +/// their storage-specific path performs exact, read-only user-bubble lookups +/// and never indexes assistant/tool content. +pub(super) fn project_cached_turn_metadata( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + requested_turn_ids: Option<&[String]>, +) -> Result>, String> { + source.validate_session_id(session_id)?; + if requested_turn_ids.is_some_and(|ids| ids.is_empty()) { + return Ok(Some(Vec::new())); + } + if requested_turn_ids.is_some_and(|ids| ids.len() > MAX_REQUESTED_TURNS) { + return Err(format!( + "At most {MAX_REQUESTED_TURNS} turn summaries can be loaded at once" + )); + } + + let resolved = index::resolve_source(conn, source, session_id)?; + let Some(state) = index::load_state(conn, source, &resolved.source_session_id)? else { + return Ok(None); + }; + let turns = select_projection_turns( + conn, + source, + &resolved.source_session_id, + &state.generation, + requested_turn_ids, + )?; + let mut projected = project_indexed_turns( + conn, + source, + &resolved.source_session_id, + &state.generation, + &turns, + )?; + if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + apply_compact_kv_turn_previews( + &resolved.path, + source, + &resolved.source_session_id, + &turns, + &mut projected, + )?; + } + load_projection_state_for_generation( + conn, + source, + &resolved.source_session_id, + &state.generation, + "finishing cached replay metadata projection", + )?; + Ok(Some(projected)) +} + +fn apply_compact_kv_turn_previews( + source_path: &std::path::Path, + source: ImportedHistorySourceId, + source_session_id: &str, + turns: &[ProjectionTurn], + projected: &mut [ProjectedTurnMetadata], +) -> Result<(), String> { + let requested = turns + .iter() + .map(|turn| (turn.header.turn_index, turn.source_turn_id.clone())) + .collect::>(); + let previews = super::sqlite_driver::read_kv_turn_previews( + source_path, + source, + source_session_id, + &requested, + )? + .into_iter() + .map(|preview| (preview.turn_index, preview)) + .collect::>(); + + for (turn, metadata) in turns.iter().zip(projected) { + if metadata.event_count == 0 { + metadata.event_count = turn.header.event_count.min(i64::MAX as u64) as i64; + metadata.body_event_count = turn + .header + .event_count + .saturating_sub(1) + .min(i64::MAX as u64) as i64; + } + let Some(preview) = previews.get(&turn.header.turn_index) else { + continue; + }; + if metadata.user_preview.is_empty() { + metadata.user_preview = preview.user_preview.clone(); + } + if !preview.created_at.is_empty() + && chrono::DateTime::parse_from_rfc3339(&preview.created_at).is_ok() + { + metadata.started_at = preview.created_at.clone(); + if metadata.ended_at.as_deref() == Some(turn.header.started_at.as_str()) { + metadata.ended_at = Some(preview.created_at.clone()); + } + } + } + Ok(()) +} + +fn load_projection_state_for_generation( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + expected_generation: &str, + context: &str, +) -> Result { + let state = index::load_state(conn, source, source_session_id)? + .ok_or_else(|| format!("Replay index state disappeared while {context}"))?; + if state.generation != expected_generation { + return Err(format!( + "Replay generation changed while {context}: expected {expected_generation}, found {}. Retry against the new generation.", + state.generation + )); + } + Ok(state) +} + +fn select_projection_turns( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + requested_turn_ids: Option<&[String]>, +) -> Result, String> { + let select = + "SELECT turn_id,turn_index,start_sequence,end_sequence,started_at,ended_at,event_count, + (SELECT next.started_at FROM imported_replay_turns AS next + WHERE next.source=current.source + AND next.source_session_id=current.source_session_id + AND next.generation=current.generation + AND next.turn_index=current.turn_index+1) + FROM imported_replay_turns AS current + WHERE source=?1 AND source_session_id=?2 AND generation=?3"; + let decode = |row: &rusqlite::Row<'_>| { + let source_turn_id: String = row.get(0)?; + Ok(ProjectionTurn { + header: ReplayTurnHeader { + turn_id: source_turn_id.clone(), + turn_index: row.get(1)?, + start_sequence: row.get(2)?, + end_sequence: row.get(3)?, + started_at: row.get(4)?, + ended_at: row.get(5)?, + event_count: row.get::<_, i64>(6)?.max(0) as u64, + }, + next_turn_started_at: row.get(7)?, + source_turn_id, + }) + }; + + let mut turns = Vec::new(); + if let Some(requested) = requested_turn_ids { + let requested = requested.iter().collect::>(); + let mut id_statement = conn + .prepare(&format!("{select} AND turn_id=?4")) + .map_err(|err| format!("prepare requested replay metadata turn: {err}"))?; + let mut index_statement = conn + .prepare(&format!("{select} AND turn_index=?4")) + .map_err(|err| format!("prepare requested replay metadata index: {err}"))?; + for requested_turn_id in requested { + let turn_index = requested_turn_id + .strip_prefix(TURN_INDEX_SELECTOR_PREFIX) + .and_then(|value| value.parse::().ok()) + .filter(|value| *value >= 0); + let selected = if let Some(turn_index) = turn_index { + index_statement + .query_row( + params![source.as_str(), source_session_id, generation, turn_index], + decode, + ) + .optional() + .map_err(|err| format!("query requested replay metadata index: {err}"))? + .map(|mut turn| { + // The selector is a renderer catalog locator, not the + // provider's turn id. Echo it so a batched response can + // be joined to virtual rows without hydrating bodies. + turn.header.turn_id = requested_turn_id.clone(); + turn + }) + } else { + id_statement + .query_row( + params![ + source.as_str(), + source_session_id, + generation, + requested_turn_id + ], + decode, + ) + .optional() + .map_err(|err| format!("query requested replay metadata turn: {err}"))? + }; + if let Some(turn) = selected { + turns.push(turn); + } + } + } else if let Some(turn) = conn + .query_row( + &format!("{select} ORDER BY turn_index DESC LIMIT 1"), + params![source.as_str(), source_session_id, generation], + decode, + ) + .optional() + .map_err(|err| format!("query newest replay metadata turn: {err}"))? + { + turns.push(turn); + } + turns.sort_by_key(|turn| turn.header.turn_index); + Ok(turns) +} + +fn project_indexed_turns( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + turns: &[ProjectionTurn], +) -> Result, String> { + let mut statement = conn + .prepare( + "SELECT action_type,function_name,created_at, + args_preview_json,result_preview_json + FROM imported_replay_events + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND turn_index=?4 + ORDER BY sequence ASC", + ) + .map_err(|err| format!("prepare replay metadata row stream: {err}"))?; + let mut projected = Vec::with_capacity(turns.len()); + for turn in turns { + let mut rows = statement + .query(params![ + source.as_str(), + source_session_id, + generation, + turn.header.turn_index + ]) + .map_err(|err| format!("query replay metadata rows: {err}"))?; + let mut metadata = TurnMetadataAccumulator::new(); + let mut status = "completed".to_string(); + let mut ended_at = Some(turn.header.started_at.clone()); + let mut user_preview = String::new(); + let mut event_count = 0_i64; + let mut body_event_count = 0_i64; + + while let Some(row) = rows + .next() + .map_err(|err| format!("read replay metadata row: {err}"))? + { + // Read the light classification fields first. For assistant, + // thinking, Node REPL, and other known no-metadata rows, the two + // JSON text columns below are never copied into Rust Strings. + let action_type: String = row.get(0).map_err(|err| err.to_string())?; + let function_name: String = row.get(1).map_err(|err| err.to_string())?; + let created_at: String = row.get(2).map_err(|err| err.to_string())?; + + if function_name == imported_history::FUNCTION_USER_MESSAGE { + let args_json: String = row.get(3).map_err(|err| err.to_string())?; + let result_json: String = row.get(4).map_err(|err| err.to_string())?; + if user_preview.is_empty() { + user_preview = user_preview_from_json(&args_json, &result_json); + } + event_count = event_count.saturating_add(1); + continue; + } + + match action_type.as_str() { + imported_history::ACTION_TYPE_TASK_START => { + status = "pending".to_string(); + ended_at = None; + continue; + } + imported_history::ACTION_TYPE_TASK_COMPLETED => { + status = "completed".to_string(); + ended_at = Some(created_at); + continue; + } + imported_history::ACTION_TYPE_TASK_FAILED => { + status = "failed".to_string(); + ended_at = Some(created_at); + continue; + } + _ => {} + } + + event_count = event_count.saturating_add(1); + body_event_count = body_event_count.saturating_add(1); + if status != "pending" + && !created_at.is_empty() + && ended_at + .as_deref() + .is_none_or(|ended| created_at.as_str() > ended) + { + ended_at = Some(created_at.clone()); + } + + let requirements = metadata_projection_requirements(Some(&function_name)); + if requirements.is_empty() { + continue; + } + let args_json = if requirements.needs_args_json() { + row.get::<_, String>(3).map_err(|err| err.to_string())? + } else { + String::new() + }; + let result_json = if requirements.needs_result_json() { + row.get::<_, String>(4).map_err(|err| err.to_string())? + } else { + String::new() + }; + metadata.add_event_at(Some(&function_name), &args_json, &result_json, &created_at); + } + + if status == "pending" { + if let Some(next_started_at) = turn.next_turn_started_at.as_ref() { + status = "interrupted".to_string(); + ended_at = Some(next_started_at.clone()); + } + } + projected.push(ProjectedTurnMetadata { + turn_id: turn.header.turn_id.clone(), + start_sequence: turn.header.start_sequence, + started_at: turn.header.started_at.clone(), + ended_at, + status, + user_preview, + event_count, + body_event_count, + modified_files: metadata.modified_files().to_vec(), + resource_interactions: metadata.resource_interactions().to_vec(), + git_artifacts: metadata.git_artifacts().to_vec(), + }); + } + Ok(projected) +} + +fn user_preview_from_json(args_json: &str, result_json: &str) -> String { + let args = serde_json::from_str::(args_json).unwrap_or(Value::Null); + for field in ["content", "message", "prompt", "text", "query"] { + if let Some(text) = args.get(field).and_then(Value::as_str) { + return text.to_string(); + } + } + if let Some(text) = args.as_str() { + return text.to_string(); + } + let result = serde_json::from_str::(result_json).unwrap_or(Value::Null); + result + .pointer("/message/content") + .or_else(|| result.get("content")) + .or_else(|| result.get("message")) + .and_then(Value::as_str) + .unwrap_or_default() + .to_string() +} + +#[cfg(test)] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/metadata_projection/tests.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/metadata_projection/tests.rs new file mode 100644 index 000000000..ac0094877 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/metadata_projection/tests.rs @@ -0,0 +1,413 @@ +use core_types::activity::ActivityChunk; +use rusqlite::Connection; +use serde_json::json; + +use crate::projectors::turn_metadata::project_activity_chunks; +use crate::store::sqlite::SqliteRecordStore; + +use super::*; + +const SOURCE_SESSION_ID: &str = "fixture"; +const GENERATION: &str = "generation-1"; + +fn setup() -> Connection { + let conn = Connection::open_in_memory().expect("in-memory replay DB"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("replay schema"); + conn +} + +fn insert_state(conn: &Connection, generation: &str) { + conn.execute( + "INSERT INTO imported_replay_state( + source,source_session_id,generation,revision,parser_version, + source_identity,driver_cursor_json,valid,updated_at + ) VALUES(?1,?2,?3,1,1,'fixture','{}',1,'2026-07-22T00:00:00Z') + ON CONFLICT(source,source_session_id) DO UPDATE SET + generation=excluded.generation,revision=excluded.revision, + updated_at=excluded.updated_at", + params![ + ImportedHistorySourceId::CodexApp.as_str(), + SOURCE_SESSION_ID, + generation + ], + ) + .expect("insert replay state"); +} + +fn insert_turn(conn: &Connection, turn_index: i64, chunks: &[ActivityChunk]) { + let source = ImportedHistorySourceId::CodexApp; + let first = chunks.first().expect("turn must have an event"); + let last = chunks.last().expect("turn must have an event"); + let start_sequence = if turn_index == 0 { 0 } else { 10_000 }; + conn.execute( + "INSERT INTO imported_replay_turns( + source,source_session_id,generation,turn_index,turn_id,start_sequence, + end_sequence,started_at,ended_at,event_count + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10)", + params![ + source.as_str(), + SOURCE_SESSION_ID, + GENERATION, + turn_index, + first.chunk_id, + start_sequence, + start_sequence + chunks.len() as i64 - 1, + first.created_at, + last.created_at, + chunks.len() as i64 + ], + ) + .expect("insert turn header"); + for (offset, chunk) in chunks.iter().enumerate() { + conn.execute( + "INSERT INTO imported_replay_events( + source,source_session_id,generation,sequence,event_id,turn_index, + action_type,function_name,created_at,args_preview_json,result_preview_json, + args_size_bytes,result_size_bytes,content_hash + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13,?14)", + params![ + source.as_str(), + SOURCE_SESSION_ID, + GENERATION, + start_sequence + offset as i64, + chunk.chunk_id, + turn_index, + chunk.action_type, + chunk.function, + chunk.created_at, + serde_json::to_string(&chunk.args).unwrap(), + serde_json::to_string(&chunk.result).unwrap(), + chunk.args.to_string().len() as i64, + chunk.result.to_string().len() as i64, + format!("hash-{turn_index}-{offset}") + ], + ) + .expect("insert replay event"); + } +} + +fn chunk( + id: &str, + action_type: &str, + function: &str, + timestamp: usize, + args: Value, + result: Value, +) -> ActivityChunk { + let mut chunk = ActivityChunk::new("codexapp-fixture", action_type, function); + chunk.chunk_id = id.to_string(); + chunk.created_at = format!("2026-07-15T00:00:{timestamp:02}Z"); + chunk.args = args; + chunk.result = result; + chunk +} + +fn representative_turn() -> Vec { + vec![ + chunk( + "user-1", + "raw", + imported_history::FUNCTION_USER_MESSAGE, + 0, + json!({}), + json!({"message":{"content":"fix metadata"}}), + ), + chunk( + "assistant-1", + "assistant", + "assistant", + 1, + json!({"impossiblePath":"assistant/should/not/project"}), + json!({"content":"x".repeat(64 * 1024)}), + ), + chunk( + "thinking-1", + "thinking", + "thinking", + 2, + json!({}), + json!({"thought":"x".repeat(64 * 1024)}), + ), + chunk( + "edit-1", + "tool_call", + "edit_file", + 3, + json!({"file_path":"src/lib.rs","new_string":"one\ntwo"}), + json!({"success":{"linesAdded":2,"linesRemoved":1}}), + ), + chunk( + "edit-2", + "tool_call", + "edit_file", + 4, + json!({"file_path":"src/lib.rs","new_string":"three\nfour\nfive"}), + json!({"success":{"linesAdded":3,"linesRemoved":0}}), + ), + chunk( + "grep-1", + "tool_call", + "Grep", + 5, + json!({"path":"src","pattern":"metadata"}), + json!({"matches":"x".repeat(128 * 1024)}), + ), + chunk( + "unknown-1", + "tool_call", + "future_provider_tool", + 6, + json!({"command":"gh pr create"}), + json!({"output":"https://github.com/acme/repo/pull/42"}), + ), + chunk( + "git-1", + "tool_call", + "Bash", + 7, + json!({"command":"git commit -m metadata"}), + json!({"success":{"command":"git commit -m metadata","stdout":"[feature abc1234] metadata","exitCode":0}}), + ), + ] +} + +#[test] +fn compact_row_projection_matches_activity_projection() { + let conn = setup(); + let chunks = representative_turn(); + insert_turn(&conn, 0, &chunks); + let turns = vec![ProjectionTurn { + header: ReplayTurnHeader { + turn_id: "user-1".to_string(), + turn_index: 0, + start_sequence: 0, + end_sequence: Some(chunks.len() as i64 - 1), + started_at: chunks[0].created_at.clone(), + ended_at: chunks.last().map(|chunk| chunk.created_at.clone()), + event_count: chunks.len() as u64, + }, + next_turn_started_at: None, + source_turn_id: "user-1".to_string(), + }]; + + let expected = project_activity_chunks(&chunks); + let actual = project_indexed_turns( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + &turns, + ) + .expect("project replay rows"); + + assert_eq!( + serde_json::to_value(&actual).unwrap(), + serde_json::to_value(&expected).unwrap() + ); + assert_eq!(actual[0].modified_files[0].additions, 5); + assert_eq!(actual[0].modified_files[0].deletions, 1); + assert_eq!(actual[0].git_artifacts.len(), 2); +} + +#[test] +fn one_turn_over_wire_limit_projects_every_row() { + let conn = setup(); + let mut chunks = vec![chunk( + "user-large", + "raw", + imported_history::FUNCTION_USER_MESSAGE, + 0, + json!({"content":"large turn"}), + json!({}), + )]; + for index in 0..250 { + chunks.push(chunk( + &format!("edit-{index}"), + "tool_call", + "edit_file", + 1, + json!({"file_path":"src/large.rs"}), + json!({"linesAdded":1,"linesRemoved":0}), + )); + } + insert_turn(&conn, 0, &chunks); + let turns = vec![ProjectionTurn { + header: ReplayTurnHeader { + turn_id: "user-large".to_string(), + turn_index: 0, + start_sequence: 0, + end_sequence: Some(250), + started_at: chunks[0].created_at.clone(), + ended_at: chunks.last().map(|chunk| chunk.created_at.clone()), + event_count: 251, + }, + next_turn_started_at: None, + source_turn_id: "user-large".to_string(), + }]; + + let projected = project_indexed_turns( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + &turns, + ) + .expect("project full large turn"); + + assert_eq!(projected[0].event_count, 251); + assert_eq!(projected[0].body_event_count, 250); + assert_eq!(projected[0].modified_files[0].additions, 250); +} + +#[test] +fn requested_pending_turn_uses_successor_boundary_and_newest_is_bounded() { + let conn = setup(); + let first = vec![ + chunk( + "user-pending", + "raw", + imported_history::FUNCTION_USER_MESSAGE, + 0, + json!({}), + json!({"message":{"content":"start work"}}), + ), + chunk( + "lifecycle-start", + imported_history::ACTION_TYPE_TASK_START, + imported_history::ACTION_TYPE_TASK_START, + 1, + json!({}), + json!({}), + ), + chunk( + "edit-pending", + "tool_call", + "edit_file", + 2, + json!({"file_path":"src/pending.rs"}), + json!({"linesAdded":1}), + ), + ]; + let second = vec![chunk( + "user-next", + "raw", + imported_history::FUNCTION_USER_MESSAGE, + 3, + json!({}), + json!({"message":{"content":"next request"}}), + )]; + insert_turn(&conn, 0, &first); + insert_turn(&conn, 1, &second); + + let requested_ids = vec!["user-pending".to_string()]; + let requested = select_projection_turns( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + Some(&requested_ids), + ) + .expect("select requested turn"); + let projected = project_indexed_turns( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + &requested, + ) + .expect("project requested turn"); + assert_eq!(projected.len(), 1); + assert_eq!(projected[0].status, "interrupted"); + assert_eq!( + projected[0].ended_at.as_deref(), + Some(second[0].created_at.as_str()) + ); + assert_eq!(projected[0].event_count, 2); + assert_eq!(projected[0].body_event_count, 1); + + let newest = select_projection_turns( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + None, + ) + .expect("select newest turn"); + assert_eq!(newest.len(), 1); + assert_eq!(newest[0].header.turn_id, "user-next"); +} + +#[test] +fn virtual_turn_index_selectors_return_compact_previews_without_body_windows() { + let conn = setup(); + let first = vec![chunk( + "provider-user-0", + "raw", + imported_history::FUNCTION_USER_MESSAGE, + 0, + json!({}), + json!({"message":{"content":"first compact prompt"}}), + )]; + let second = vec![chunk( + "provider-user-1", + "raw", + imported_history::FUNCTION_USER_MESSAGE, + 1, + json!({}), + json!({"message":{"content":"second compact prompt"}}), + )]; + insert_turn(&conn, 0, &first); + insert_turn(&conn, 1, &second); + + let selector = format!("{TURN_INDEX_SELECTOR_PREFIX}0"); + let requested = vec![selector.clone()]; + let turns = select_projection_turns( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + Some(&requested), + ) + .expect("select virtual catalog row by turn index"); + let projected = project_indexed_turns( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + &turns, + ) + .expect("project compact catalog preview"); + + assert_eq!(projected.len(), 1); + assert_eq!(projected[0].turn_id, selector); + assert_eq!(projected[0].user_preview, "first compact prompt"); + assert_eq!(projected[0].event_count, 1); +} + +#[test] +fn projection_rejects_a_concurrent_generation_replacement() { + let conn = setup(); + insert_state(&conn, GENERATION); + load_projection_state_for_generation( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + "testing metadata projection", + ) + .expect("matching generation"); + + // Simulate another connection atomically publishing a replacement + // between header selection and body projection. + insert_state(&conn, "generation-2"); + let error = load_projection_state_for_generation( + &conn, + ImportedHistorySourceId::CodexApp, + SOURCE_SESSION_ID, + GENERATION, + "testing metadata projection", + ) + .expect_err("mixed generations must be rejected"); + + assert!(error.contains("expected generation-1, found generation-2")); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/mod.rs new file mode 100644 index 000000000..e7fa165bf --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/mod.rs @@ -0,0 +1,656 @@ +//! Storage-agnostic bounded replay for imported CLI histories. +//! +//! The public API exposes source-neutral generations, revisions and sequence +//! cursors. Driver positions (JSONL byte offsets, SQLite keys, whole-JSON +//! generations) stay private and are persisted in ORGII-owned compact index +//! tables. No API in this module falls back to a provider's full-history +//! loader. + +mod cache_policy; +#[cfg(test)] +mod conformance_tests; +mod drivers; +mod index; +mod metadata_projection; +mod model; +mod payload_artifact; +#[cfg(test)] +mod perf_tests; +pub mod registry; + +use drivers::jsonl::{self as jsonl_driver, codex as codex_jsonl, qoder_sidecar}; +use drivers::sqlite as sqlite_driver; +use drivers::structured as structured_driver; +use drivers::whole_json as whole_json_driver; + +use std::path::{Path, PathBuf}; + +use core_types::activity::ActivityChunk; +use rusqlite::{Connection, Transaction}; +use serde::{Deserialize, Serialize}; + +pub use cache_policy::{ + prune_cache, ReplayCacheEviction, ReplayCacheEvictionReason, ReplayCachePolicy, + ReplayCachePruneReport, DEFAULT_REPLAY_CACHE_MAX_BYTES, DEFAULT_REPLAY_CACHE_PROTECT_RECENT, + DEFAULT_REPLAY_CACHE_TARGET_BYTES, DEFAULT_REPLAY_CACHE_TTL, +}; +pub use model::*; +pub use registry::{ + ImportedHistorySourceId, ImportedReplayDescriptor, ReplayAdapterSupport, ReplayStorageFamily, +}; + +/// Register one already-resolved external transcript without scanning its +/// body. Lifecycle hooks can learn about a child transcript before the next +/// catalog refresh; replay still needs the canonical source path in order to +/// build its bounded compact index. Existing catalog metadata is deliberately +/// preserved -- this only establishes the source/session/path binding. +pub fn bind_source_path( + conn: &Connection, + source: ImportedHistorySourceId, + source_session_id: &str, + session_id: &str, + source_path: &Path, +) -> Result<(), String> { + source.validate_session_id(session_id)?; + let expected_source_session_id = source.source_session_id(session_id)?; + if expected_source_session_id != source_session_id { + return Err(format!( + "Replay source identity mismatch: session {session_id} resolves to {expected_source_session_id}, not {source_session_id}" + )); + } + if !source_path.is_file() { + return Err(format!( + "Replay source path is not a file: {}", + source_path.display() + )); + } + let source_path = source_path + .canonicalize() + .unwrap_or_else(|_| source_path.to_path_buf()); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source,source_session_id,session_id,source_path,source_record_key, + parser_version,listable,updated_at + ) VALUES (?1,?2,?3,?4,?2,?5,0,?6) + ON CONFLICT(source,source_session_id) DO UPDATE SET + session_id=excluded.session_id, + source_path=excluded.source_path, + source_record_key=excluded.source_record_key, + parser_version=excluded.parser_version, + updated_at=excluded.updated_at", + rusqlite::params![ + source.as_str(), + source_session_id, + session_id, + source_path.to_string_lossy(), + i64::from(source.descriptor().parser_version), + chrono::Utc::now().to_rfc3339(), + ], + ) + .map_err(|err| format!("bind imported replay source path: {err}"))?; + Ok(()) +} + +/// Physical sources whose mutations can invalidate one replay session. +/// +/// This is a backend lifecycle API, not part of the renderer wire protocol; +/// storage-specific sidecar locations never cross IPC. +pub fn watch_paths( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, +) -> Result, String> { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let resolved = index::resolve_source(conn, source, session_id)?; + let mut paths = vec![resolved.path.clone()]; + if source == ImportedHistorySourceId::Qoder { + paths.extend(qoder_sidecar::watch_paths( + &resolved.source_session_id, + &resolved.path, + )); + } + for path in &mut paths { + *path = path.canonicalize().unwrap_or_else(|_| path.clone()); + } + paths.sort(); + paths.dedup(); + Ok(paths) +} + +pub fn open_window( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let sync = index::sync_index(conn, source, session_id)?; + let mut window = index::read_recent_window(conn, source, session_id, limits.bounded())?; + merge_sync_stats(&mut window.stats, sync.stats); + Ok(window) +} + +/// Read the newest bounded window from an already-published compact index. +/// +/// This is an explicitly stale-tolerant read for secondary consumers that do +/// not own source synchronization. Visible UI opens must call [`open_window`] +/// on a short-lived foreground connection so a provider append is reflected +/// immediately. A cache miss remains explicit. +pub fn open_cached_window( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + limits: ReplayLimits, +) -> Result, String> { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let resolved = index::resolve_source(conn, source, session_id)?; + if index::load_state(conn, source, &resolved.source_session_id)?.is_none() { + return Ok(None); + } + index::read_recent_window(conn, source, session_id, limits.bounded()).map(Some) +} + +pub fn poll_delta( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + cursor: &ReplayCursor, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + validate_cursor_identity(source, session_id, cursor)?; + ensure_supported(source)?; + let sync = index::sync_index(conn, source, session_id)?; + index::read_delta(conn, source, session_id, cursor, limits.bounded(), sync) +} + +pub fn read_window( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + before_sequence: Option, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let sync = index::sync_index(conn, source, session_id)?; + let mut window = + index::read_window_before(conn, source, session_id, before_sequence, limits.bounded())?; + merge_sync_stats(&mut window.stats, sync.stats); + Ok(window) +} + +/// Read a bounded older window from the last atomically published generation. +/// +/// Cursor IDE and Windsurf cold indexes intentionally omit older turn bodies, +/// so their foreground pager must retain the synchronized hydration path. +pub fn read_cached_window( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + before_sequence: Option, + limits: ReplayLimits, +) -> Result, String> { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + return Ok(None); + } + let resolved = index::resolve_source(conn, source, session_id)?; + if index::load_state(conn, source, &resolved.source_session_id)?.is_none() { + return Ok(None); + } + index::read_window_before(conn, source, session_id, before_sequence, limits.bounded()).map(Some) +} + +/// Read one compact-index turn by its stable header id. This is the bounded +/// replacement for Cursor IDE's legacy lazy `cursorIdeTurnWindow` hydration. +pub fn read_turn_window( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + turn_id: &str, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + index::sync_index(conn, source, session_id)?; + index::read_window_for_turn(conn, source, session_id, turn_id, limits.bounded()) +} + +/// Read one already-materialized turn without synchronizing the provider. +pub fn read_cached_turn_window( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + turn_id: &str, + limits: ReplayLimits, +) -> Result, String> { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + return Ok(None); + } + let resolved = index::resolve_source(conn, source, session_id)?; + if index::load_state(conn, source, &resolved.source_session_id)?.is_none() { + return Ok(None); + } + index::read_window_for_turn(conn, source, session_id, turn_id, limits.bounded()).map(Some) +} + +pub fn read_turn_window_at_index( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + turn_index: i64, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + index::sync_index(conn, source, session_id)?; + index::read_window_for_turn_index(conn, source, session_id, turn_index, limits.bounded()) +} + +/// Read one already-materialized turn index without synchronizing the provider. +pub fn read_cached_turn_window_at_index( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + turn_index: i64, + limits: ReplayLimits, +) -> Result, String> { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + if matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) { + return Ok(None); + } + let resolved = index::resolve_source(conn, source, session_id)?; + if index::load_state(conn, source, &resolved.source_session_id)?.is_none() { + return Ok(None); + } + index::read_window_for_turn_index(conn, source, session_id, turn_index, limits.bounded()) + .map(Some) +} + +/// Project compact metadata for visible imported-history turns without +/// materializing an `ActivityChunk` transcript or applying the 200-event +/// renderer window limit. +pub fn project_turn_metadata( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + requested_turn_ids: Option<&[String]>, +) -> Result, String> { + ensure_supported(source)?; + metadata_projection::project_turn_metadata(conn, source, session_id, requested_turn_ids) +} + +/// Project compact metadata without synchronizing the provider or taking a +/// write transaction. Most adapters read only ORGII's published index; +/// Cursor/Windsurf additionally perform bounded, exact user-bubble lookups +/// against their read-only KV store. `None` means no compact index exists yet. +pub fn project_cached_turn_metadata( + conn: &Connection, + source: ImportedHistorySourceId, + session_id: &str, + requested_turn_ids: Option<&[String]>, +) -> Result>, String> { + ensure_supported(source)?; + metadata_projection::project_cached_turn_metadata(conn, source, session_id, requested_turn_ids) +} + +pub fn scan_window_after( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + after_sequence: i64, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let sync = index::sync_index(conn, source, session_id)?; + index::read_scan_after( + conn, + source, + session_id, + after_sequence, + limits.bounded(), + sync, + ) +} + +/// Continue a backend-only scan against one already-synchronized immutable +/// replay cursor. This is used by consumers that must atomically replace a +/// derived read model: they first complete a normal generation/revision-checked +/// scan, then replay that exact compact snapshot inside their own SQLite +/// transaction. It deliberately does not touch the external source or start a +/// nested replay-index transaction. +pub fn scan_window_after_generation( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + revision: u64, + after_sequence: i64, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let scan = index::read_scan_after( + conn, + source, + session_id, + after_sequence, + limits.bounded(), + index::ReplaySyncResult::default(), + )?; + if scan.cursor.generation != generation || scan.cursor.revision != revision { + return Err(format!( + "Replay cursor changed while publishing a derived snapshot: expected {generation}@{revision}, found {}@{}", + scan.cursor.generation, scan.cursor.revision + )); + } + Ok(scan) +} + +const MAX_PINNED_SCAN_RESTARTS: usize = 2; + +/// Materialize every compact turn through bounded pages, then return one +/// stable generation/revision that strict snapshot readers can replay. +/// +/// This first pass deliberately retains no chunks. It is required for lazy +/// SQLite/KV sources, whose older turns legitimately advance the replay +/// revision when materialized. Once all turns are present, a fresh source sync +/// verifies that the transcript did not change during preparation. A changing +/// source restarts at most twice; it can never keep a caller in an endless +/// chase of a live transcript. +pub fn prepare_pinned_scan( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + limits: ReplayLimits, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let limits = limits.bounded(); + + retry_pinned_scan_preparation(session_id, || { + prepare_pinned_scan_attempt(conn, source, session_id, limits) + }) +} + +fn retry_pinned_scan_preparation( + session_id: &str, + mut attempt: impl FnMut() -> Result, String>, +) -> Result { + for restart in 0..=MAX_PINNED_SCAN_RESTARTS { + if let Some(cursor) = attempt()? { + return Ok(cursor); + } + if restart == MAX_PINNED_SCAN_RESTARTS { + break; + } + } + Err(format!( + "Imported replay source {session_id} kept changing while preparing a pinned scan after {MAX_PINNED_SCAN_RESTARTS} restarts" + )) +} + +fn prepare_pinned_scan_attempt( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + limits: ReplayLimits, +) -> Result, String> { + let mut previous_sequence = -1_i64; + let mut scan = scan_window_after(conn, source, session_id, previous_sequence, limits)?; + loop { + if scan.has_more && scan.cursor.through_sequence <= previous_sequence { + return Err(format!( + "Bounded replay preparation made no progress for {session_id} after sequence {previous_sequence}" + )); + } + previous_sequence = scan.cursor.through_sequence; + if !scan.has_more { + break; + } + let Some(next) = + continue_pinned_scan_preparation(conn, source, session_id, &scan.cursor, limits)? + else { + return Ok(None); + }; + scan = next; + } + + let prepared = scan.cursor; + let verification = + scan_window_after(conn, source, session_id, prepared.through_sequence, limits)?; + if verification.cursor.generation != prepared.generation + || verification.cursor.revision != prepared.revision + || verification.has_more + || !verification.chunks.is_empty() + { + return Ok(None); + } + Ok(Some(verification.cursor)) +} + +/// Continue only the discard-only preparation pass without observing the +/// external source again. Revision growth here can only come from lazy turn +/// materialization on this connection. A concurrent index change requests a +/// bounded restart instead of weakening the final strict snapshot. +fn continue_pinned_scan_preparation( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + cursor: &ReplayCursor, + limits: ReplayLimits, +) -> Result, String> { + validate_cursor_identity(source, session_id, cursor)?; + let resolved = index::resolve_source(conn, source, session_id)?; + let state = index::load_state(conn, source, &resolved.source_session_id)? + .ok_or_else(|| "Replay index state disappeared before continuation".to_string())?; + if state.generation != cursor.generation || state.revision != cursor.revision { + return Ok(None); + } + let scan = index::read_scan_after( + conn, + source, + session_id, + cursor.through_sequence, + limits.bounded(), + index::ReplaySyncResult::default(), + )?; + if scan.cursor.generation != cursor.generation || scan.cursor.revision < cursor.revision { + return Ok(None); + } + Ok(Some(scan)) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub fn read_payload_range( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: Option, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + let max_bytes = max_bytes + .unwrap_or(DEFAULT_PAYLOAD_RANGE_BYTES) + .clamp(1, HARD_MAX_PAYLOAD_RANGE_BYTES); + index::read_payload_range( + conn, source, session_id, generation, event_id, field_path, offset, max_bytes, + ) +} + +/// Ensure one deferred payload has an immutable content-addressed artifact. +/// +/// The caller owns `tx` so publishing a derived Shell manifest can occur in +/// the same transaction as the content hash. Direct provider ranges are read +/// in bounded pages and streamed into SQLite incremental BLOB I/O; no complete +/// payload is assembled in Rust. +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub fn materialize_payload_artifact( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, +) -> Result { + source.validate_session_id(session_id)?; + ensure_supported(source)?; + index::materialize_payload_artifact(tx, source, session_id, generation, event_id, field_path) +} + +/// Store a payload produced by an ORGII-owned replay adapter (managed CLI or +/// collaboration snapshot) in the same content-addressed artifact store. +/// The caller keeps manifest publication in this transaction. +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub fn store_scoped_payload_artifact_streamed( + tx: &Transaction<'_>, + source_id: &str, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + total_bytes: u64, + produce: F, +) -> Result +where + F: FnOnce(&mut dyn std::io::Write) -> Result<(), String>, +{ + if source_id.is_empty() || source_session_id.is_empty() || generation.is_empty() { + return Err("Replay payload artifact scope must be non-empty".to_string()); + } + let content_hash = payload_artifact::store_streamed_for_scope( + tx, + source_id, + source_session_id, + generation, + event_id, + field_path, + total_bytes, + produce, + )?; + Ok(ReplayPayloadArtifactLocator { + source_id: source_id.to_string(), + source_session_id: source_session_id.to_string(), + generation: generation.to_string(), + content_hash, + total_bytes, + }) +} + +/// Reuse a body only inside an immutable ORGII-owned managed/snapshot epoch. +/// Vendor adapters intentionally do not call this API because their rows may +/// update in place while the surrounding replay generation remains stable. +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub fn find_scoped_payload_artifact( + tx: &Transaction<'_>, + source_id: &str, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + expected_bytes: u64, +) -> Result, String> { + if source_id.is_empty() || source_session_id.is_empty() || generation.is_empty() { + return Err("Replay payload artifact scope must be non-empty".to_string()); + } + let Some(content_hash) = payload_artifact::find_for_immutable_scope( + tx, + source_id, + source_session_id, + generation, + event_id, + field_path, + expected_bytes, + )? + else { + return Ok(None); + }; + Ok(Some(ReplayPayloadArtifactLocator { + source_id: source_id.to_string(), + source_session_id: source_session_id.to_string(), + generation: generation.to_string(), + content_hash, + total_bytes: expected_bytes, + })) +} + +/// Return whether a JSON object field exists only in a compact replay row. +/// +/// Compatibility serializers that walk a preview instead of replacing a +/// root payload must omit these fields. They are useful to ORGII's bounded UI +/// and metadata projection, but were not present in the provider transcript. +pub fn is_compact_only_replay_field(key: &str) -> bool { + matches!(key, "_replayTruncated" | "_preview") + || key == crate::development_artifact::REPLAY_GIT_ARTIFACTS_FIELD +} + +fn ensure_supported(source: ImportedHistorySourceId) -> Result<(), String> { + let descriptor = source.descriptor(); + if descriptor.support != ReplayAdapterSupport::Incremental { + return Err(format!( + "Bounded replay adapter for {} ({:?}) is not implemented; refusing full-history fallback", + descriptor.source_id, descriptor.storage_family + )); + } + Ok(()) +} + +fn merge_sync_stats(target: &mut ReplayStats, sync: ReplayStats) { + target.parsed_bytes = target.parsed_bytes.saturating_add(sync.parsed_bytes); + target.parsed_rows = target.parsed_rows.saturating_add(sync.parsed_rows); + target.normalized_events = target + .normalized_events + .saturating_add(sync.normalized_events); + target.upserted_events = target.upserted_events.saturating_add(sync.upserted_events); + target.removed_events = target.removed_events.saturating_add(sync.removed_events); + target.not_ready |= sync.not_ready; +} + +fn validate_cursor_identity( + source: ImportedHistorySourceId, + session_id: &str, + cursor: &ReplayCursor, +) -> Result<(), String> { + if cursor.source_id != source.as_str() || cursor.session_id != session_id { + return Err("Replay cursor belongs to another source/session".to_string()); + } + Ok(()) +} + +#[cfg(test)] +#[path = "tests/facade.rs"] +mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/model.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/model.rs new file mode 100644 index 000000000..5daea888a --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/model.rs @@ -0,0 +1,348 @@ +use super::*; + +pub const DEFAULT_MAX_TURNS: usize = 1; +pub const HARD_MAX_TURNS: usize = 10; +pub const DEFAULT_MAX_EVENTS: usize = 200; +pub const HARD_MAX_EVENTS: usize = 200; +pub const DEFAULT_MAX_IPC_BYTES: usize = 4 * 1024 * 1024; +pub const HARD_MAX_IPC_BYTES: usize = 4 * 1024 * 1024; +pub const NORMAL_PAYLOAD_PREVIEW_BYTES: usize = 8 * 1024; +pub const SHELL_PAYLOAD_PREVIEW_BYTES: usize = 32 * 1024; +pub const DEFAULT_PAYLOAD_RANGE_BYTES: usize = 64 * 1024; +pub const HARD_MAX_PAYLOAD_RANGE_BYTES: usize = 256 * 1024; +pub const INVALIDATED_EVENT_NAME: &str = "external-replay://invalidated"; + +#[derive(Debug, Clone, Copy, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReplayLimits { + #[serde(default = "default_max_turns")] + pub max_turns: usize, + #[serde(default = "default_max_events")] + pub max_events: usize, + #[serde(default = "default_max_ipc_bytes")] + pub max_ipc_bytes: usize, +} + +const fn default_max_turns() -> usize { + DEFAULT_MAX_TURNS +} +const fn default_max_events() -> usize { + DEFAULT_MAX_EVENTS +} +const fn default_max_ipc_bytes() -> usize { + DEFAULT_MAX_IPC_BYTES +} + +impl Default for ReplayLimits { + fn default() -> Self { + Self { + max_turns: DEFAULT_MAX_TURNS, + max_events: DEFAULT_MAX_EVENTS, + max_ipc_bytes: DEFAULT_MAX_IPC_BYTES, + } + } +} + +impl ReplayLimits { + pub fn bounded(self) -> Self { + Self { + max_turns: self.max_turns.clamp(1, HARD_MAX_TURNS), + max_events: self.max_events.clamp(1, HARD_MAX_EVENTS), + max_ipc_bytes: self.max_ipc_bytes.clamp(1, HARD_MAX_IPC_BYTES), + } + } + + /// Reserve the remaining exact-turn budget after retaining its first + /// rendering anchor. A selected turn may then use this budget for its + /// newest tail without exceeding either the event or compact-byte cap. + pub fn after_exact_turn_anchor(self, anchor_bytes: usize) -> Option { + let bounded = self.bounded(); + (bounded.max_events > 1).then(|| Self { + max_turns: 1, + max_events: bounded.max_events - 1, + max_ipc_bytes: bounded.max_ipc_bytes.saturating_sub(anchor_bytes).max(1), + }) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct ReplayCursor { + pub source_id: String, + pub session_id: String, + pub generation: String, + pub revision: u64, + pub through_sequence: i64, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct ReplayTurnHeader { + pub turn_id: String, + pub turn_index: i64, + pub start_sequence: i64, + pub end_sequence: Option, + pub started_at: String, + pub ended_at: Option, + pub event_count: u64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct ReplayStats { + pub parsed_bytes: u64, + pub parsed_rows: u64, + pub normalized_events: u64, + pub upserted_events: u64, + pub removed_events: u64, + pub ipc_bytes: u64, + pub not_ready: bool, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReplayPayloadDescriptor { + pub field_path: String, + pub kind: ReplayPayloadKind, + /// Encoding of bytes returned by payload range reads. New adapters must + /// set this explicitly; the legacy value exists only so replay rows + /// written before this field was introduced remain readable. + #[serde(default)] + pub encoding: ReplayPayloadEncoding, + /// Bounded semantic body selected before a root JSON value is compacted. + /// Markdown/CSV consumers use this instead of hydrating the whole root. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub body_projection: Option, + pub spans: Vec, + pub total_bytes: u64, + /// Ordinal of the normalized payload-bearing item within one JSONL line. + /// This is persisted only as a source locator; public cursors remain + /// storage-neutral. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source_ordinal: Option, + /// Stable provider row/KV key used by SQLite adapters for range reads. + /// It is an internal locator and is never exposed in [`ReplayCursor`]. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source_key: Option, +} + +impl ReplayPayloadDescriptor { + /// Resolve rows written before `encoding` became explicit without + /// preserving path-shape inference in new production call sites. + pub fn resolved_encoding(&self) -> ReplayPayloadEncoding { + match self.encoding { + ReplayPayloadEncoding::LegacyPathInferred => { + if self.field_path.contains('.') { + ReplayPayloadEncoding::Utf8Text + } else { + ReplayPayloadEncoding::JsonValue + } + } + encoding => encoding, + } + } +} + +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum ReplayPayloadEncoding { + /// Compatibility-only value for cached descriptors that predate this + /// field. Newly constructed descriptors must never use it. + #[default] + LegacyPathInferred, + /// Payload ranges form one complete serialized JSON value. + JsonValue, + /// Payload ranges form decoded UTF-8 string content. + Utf8Text, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct ReplayPayloadBodyProjection { + pub field_path: String, + pub text: String, + pub truncated: bool, +} + +/// Select and bound the same semantic body used by transcript renderers +/// before a canonical root JSON payload is replaced by a compact preview. +/// `fallback_json` should be the already-serialized root when the root has no +/// well-known text field, avoiding a second session-sized serialization. +pub fn replay_payload_body_projection( + root: &str, + value: &serde_json::Value, + fallback_json: Option<&str>, + max_bytes: usize, + tail: bool, +) -> Option { + let (field_path, text) = replay_body_candidate(root, value) + .or_else(|| fallback_json.map(|text| (root.to_string(), text)))?; + let text = text.trim(); + if text.is_empty() { + return None; + } + let (text, truncated) = bounded_body_preview(text, max_bytes, tail); + Some(ReplayPayloadBodyProjection { + field_path, + text: text.to_string(), + truncated, + }) +} + +fn replay_body_candidate<'a>( + root: &str, + value: &'a serde_json::Value, +) -> Option<(String, &'a str)> { + match value { + serde_json::Value::String(text) => Some((root.to_string(), text)), + serde_json::Value::Object(map) => { + if let Some(text) = map + .get("message") + .and_then(|message| message.get("content")) + .and_then(serde_json::Value::as_str) + { + if !text.trim().is_empty() { + return Some((format!("{root}.message.content"), text)); + } + } + for key in [ + "content", + "text", + "observation", + "cmd", + "command", + "body", + "summary", + "prompt", + "description", + ] { + let Some(text) = map.get(key).and_then(serde_json::Value::as_str) else { + continue; + }; + if !text.trim().is_empty() { + return Some((format!("{root}.{key}"), text)); + } + } + None + } + _ => None, + } +} + +fn bounded_body_preview(text: &str, max_bytes: usize, tail: bool) -> (&str, bool) { + if text.len() <= max_bytes { + return (text, false); + } + if tail { + let mut start = text.len().saturating_sub(max_bytes); + while start < text.len() && !text.is_char_boundary(start) { + start += 1; + } + (&text[start..], true) + } else { + let mut end = text.len().min(max_bytes); + while end > 0 && !text.is_char_boundary(end) { + end -= 1; + } + (&text[..end], true) + } +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum ReplayPayloadKind { + UserMessage, + AgentMessage, + AssistantContent, + Reasoning, + ToolOutput, + ToolArguments, + ToolDiff, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReplaySourceSpan { + pub start: u64, + pub end: u64, +} + +#[derive(Debug, Clone)] +pub struct ReplayIndexedChunk { + pub sequence: i64, + pub turn_index: i64, + pub chunk: ActivityChunk, + pub payloads: Vec, +} + +#[derive(Debug, Clone)] +pub struct ReplayChunkWindow { + pub cursor: ReplayCursor, + pub chunks: Vec, + /// Sequence before which an older-page request should continue. + /// + /// Selected large turns keep their first user event as a rendering anchor + /// plus a bounded newest tail. The continuation boundary is therefore the + /// start of that tail, not necessarily the minimum sequence in `chunks`. + pub window_start_sequence: Option, + pub turn_headers: Vec, + pub total_turn_count: u64, + pub total_event_count: u64, + pub has_older: bool, + pub stats: ReplayStats, +} + +#[derive(Debug, Clone)] +pub struct ReplayChunkDelta { + pub cursor: ReplayCursor, + pub chunks: Vec, + pub removed_event_ids: Vec, + pub reset_required: bool, + pub stats: ReplayStats, +} + +/// Source-neutral forward scan used by backend-only streaming consumers. +/// +/// Unlike `ReplayChunkWindow`, this does not apply turn pagination: callers +/// advance strictly by sequence and keep each batch bounded by `ReplayLimits`. +/// The type is intentionally not part of the renderer wire protocol. +#[derive(Debug, Clone)] +pub struct ReplayChunkScan { + pub cursor: ReplayCursor, + pub chunks: Vec, + pub has_more: bool, + pub stats: ReplayStats, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReplayPayloadRange { + pub event_id: String, + pub field_path: String, + pub offset: u64, + pub next_offset: u64, + pub eof: bool, + pub total_bytes: u64, + pub text: String, +} + +/// Immutable locator for one content-addressed payload body in ORGII's replay +/// cache. This type is backend-only: renderer wire types continue to expose +/// only source-neutral event/field payload references. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ReplayPayloadArtifactLocator { + pub source_id: String, + pub source_session_id: String, + pub generation: String, + pub content_hash: String, + pub total_bytes: u64, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReplayInvalidated { + pub session_id: String, + pub source_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub generation: Option, +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/payload_artifact.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/payload_artifact.rs new file mode 100644 index 000000000..cd3011dcf --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/payload_artifact.rs @@ -0,0 +1,492 @@ +//! Generation-scoped, rebuildable payload artifacts. +//! +//! Some provider locators (JSONL records, Cursor blobs and Warp protobuf task +//! rows) cannot serve a byte range without decoding the entire source record. +//! Doing that once per 256 KiB page turns a large Shell output into quadratic +//! work. These helpers materialize such payloads once while their source row +//! is already decoded, then let the common range reader use SQLite `SUBSTR`. +//! +//! The incremental writer reserves a `zeroblob` and fills it through SQLite's +//! blob API. Cross-record payloads therefore retain at most one decoded +//! source record in Rust, never a transcript- or payload-sized concatenation. + +use std::io::Write; + +use rusqlite::blob::ZeroBlob; +use rusqlite::{params, DatabaseName, OptionalExtension, Transaction}; +use sha2::{Digest, Sha256}; + +use super::ImportedHistorySourceId; + +const ARTIFACT_TABLE: &str = "imported_replay_payload_artifacts"; + +struct HashingWriter<'blob, 'conn> { + blob: &'blob mut rusqlite::blob::Blob<'conn>, + hash: Sha256, + written: u64, +} + +impl Write for HashingWriter<'_, '_> { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + let written = self.blob.write(bytes)?; + Digest::update(&mut self.hash, &bytes[..written]); + self.written = self.written.saturating_add(written as u64); + Ok(written) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.blob.flush() + } +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn store_text( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + text: &str, +) -> Result { + store_bytes( + tx, + source, + source_session_id, + generation, + event_id, + field_path, + text.as_bytes(), + ) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn store_bytes( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + payload: &[u8], +) -> Result { + let content_hash = sha256_hex(payload); + tx.execute( + "INSERT INTO imported_replay_payload_artifacts( + source,source_session_id,generation,content_hash,payload + ) VALUES (?1,?2,?3,?4,?5) + ON CONFLICT(source,source_session_id,generation,content_hash) DO NOTHING", + params![ + source.as_str(), + source_session_id, + generation, + content_hash, + payload + ], + ) + .map_err(|error| format!("store replay payload artifact: {error}"))?; + reference( + tx, + source, + source_session_id, + generation, + event_id, + field_path, + &content_hash, + )?; + Ok(content_hash) +} + +/// Stream exactly `total_bytes` into one content-addressed artifact. +/// +/// The producer may decode one JSONL line/blob at a time and call +/// `write_all`; the complete payload is never assembled in Rust. The +/// temporary key is generation/event scoped and is atomically replaced by +/// the final content hash inside the caller's replay-index transaction. +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn store_streamed( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + total_bytes: u64, + produce: F, +) -> Result +where + F: FnOnce(&mut dyn Write) -> Result<(), String>, +{ + store_streamed_for_scope( + tx, + source.as_str(), + source_session_id, + generation, + event_id, + field_path, + total_bytes, + produce, + ) +} + +/// Storage-neutral counterpart used by ORGII-owned managed/snapshot replay +/// adapters. Vendor adapters should continue to call [`store_streamed`]. +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn store_streamed_for_scope( + tx: &Transaction<'_>, + source_id: &str, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + total_bytes: u64, + produce: F, +) -> Result +where + F: FnOnce(&mut dyn Write) -> Result<(), String>, +{ + let blob_len = i32::try_from(total_bytes).map_err(|_| { + format!( + "Replay payload artifact is too large for incremental SQLite BLOB I/O: {total_bytes} bytes" + ) + })?; + let temporary_hash = temporary_hash(event_id, field_path); + tx.execute( + "DELETE FROM imported_replay_payload_artifacts + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND content_hash=?4", + params![source_id, source_session_id, generation, temporary_hash], + ) + .map_err(|error| format!("clear interrupted replay payload artifact: {error}"))?; + tx.execute( + "INSERT INTO imported_replay_payload_artifacts( + source,source_session_id,generation,content_hash,payload + ) VALUES (?1,?2,?3,?4,?5)", + params![ + source_id, + source_session_id, + generation, + temporary_hash, + ZeroBlob(blob_len) + ], + ) + .map_err(|error| format!("reserve replay payload artifact: {error}"))?; + let row_id = tx + .query_row( + "SELECT rowid FROM imported_replay_payload_artifacts + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND content_hash=?4", + params![source_id, source_session_id, generation, temporary_hash], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("locate replay payload artifact BLOB: {error}"))?; + + let (content_hash, written) = { + let mut blob = tx + .blob_open(DatabaseName::Main, ARTIFACT_TABLE, "payload", row_id, false) + .map_err(|error| format!("open replay payload artifact BLOB: {error}"))?; + let (content_hash, written) = { + let mut writer = HashingWriter { + blob: &mut blob, + hash: Sha256::new(), + written: 0, + }; + produce(&mut writer)?; + writer + .flush() + .map_err(|error| format!("flush replay payload artifact: {error}"))?; + (digest_hex(writer.hash.clone().finalize()), writer.written) + }; + blob.close() + .map_err(|error| format!("close replay payload artifact BLOB: {error}"))?; + (content_hash, written) + }; + if written != total_bytes { + return Err(format!( + "Replay payload artifact length changed while decoding: expected {total_bytes}, wrote {written}" + )); + } + + let duplicate_row_id = tx + .query_row( + "SELECT rowid FROM imported_replay_payload_artifacts + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND content_hash=?4", + params![source_id, source_session_id, generation, content_hash], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|error| format!("deduplicate replay payload artifact: {error}"))?; + if duplicate_row_id.is_some_and(|existing| existing != row_id) { + tx.execute( + "DELETE FROM imported_replay_payload_artifacts WHERE rowid=?1", + [row_id], + ) + .map_err(|error| format!("remove duplicate replay payload artifact: {error}"))?; + } else { + tx.execute( + "UPDATE imported_replay_payload_artifacts SET content_hash=?1 WHERE rowid=?2", + params![content_hash, row_id], + ) + .map_err(|error| format!("publish replay payload artifact hash: {error}"))?; + } + reference_for_scope( + tx, + source_id, + source_session_id, + generation, + event_id, + field_path, + &content_hash, + )?; + Ok(content_hash) +} + +/// Resolve an already-published artifact for an immutable ORGII-owned scope. +/// +/// This deliberately is not used by vendor replay drivers: some provider +/// stores can mutate a row without changing generation, so their normal +/// materialization path must re-check source identity. Managed CLI and +/// collaboration snapshot generations, by contrast, are immutable epochs. +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn find_for_immutable_scope( + tx: &Transaction<'_>, + source_id: &str, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + expected_bytes: u64, +) -> Result, String> { + let row = tx + .query_row( + "SELECT ref.content_hash,LENGTH(artifact.payload) + FROM imported_replay_payload_artifact_refs AS ref + JOIN imported_replay_payload_artifacts AS artifact + ON artifact.source=ref.source + AND artifact.source_session_id=ref.source_session_id + AND artifact.generation=ref.generation + AND artifact.content_hash=ref.content_hash + WHERE ref.source=?1 AND ref.source_session_id=?2 AND ref.generation=?3 + AND ref.event_id=?4 AND ref.field_path=?5", + params![ + source_id, + source_session_id, + generation, + event_id, + field_path + ], + |row| Ok((row.get::<_, String>(0)?, row.get::<_, i64>(1)?)), + ) + .optional() + .map_err(|error| format!("find immutable replay payload artifact: {error}"))?; + let Some((content_hash, stored_bytes)) = row else { + return Ok(None); + }; + if u64::try_from(stored_bytes).ok() != Some(expected_bytes) { + return Ok(None); + } + Ok(Some(content_hash)) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +pub(super) fn reference( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + content_hash: &str, +) -> Result<(), String> { + reference_for_scope( + tx, + source.as_str(), + source_session_id, + generation, + event_id, + field_path, + content_hash, + ) +} + +#[allow( + clippy::too_many_arguments, + reason = "Replay adapter boundaries keep cursor, generation, and payload fields explicit" +)] +fn reference_for_scope( + tx: &Transaction<'_>, + source_id: &str, + source_session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + content_hash: &str, +) -> Result<(), String> { + tx.execute( + "INSERT INTO imported_replay_payload_artifact_refs( + source,source_session_id,generation,event_id,field_path,content_hash + ) VALUES (?1,?2,?3,?4,?5,?6) + ON CONFLICT(source,source_session_id,generation,event_id,field_path) DO UPDATE SET + content_hash=excluded.content_hash", + params![ + source_id, + source_session_id, + generation, + event_id, + field_path, + content_hash + ], + ) + .map(|_| ()) + .map_err(|error| format!("reference replay payload artifact: {error}")) +} + +pub(super) fn delete_event_refs( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, + event_id: &str, +) -> Result<(), String> { + tx.execute( + "DELETE FROM imported_replay_payload_artifact_refs + WHERE source=?1 AND source_session_id=?2 AND generation=?3 AND event_id=?4", + params![source.as_str(), source_session_id, generation, event_id], + ) + .map(|_| ()) + .map_err(|error| format!("delete replay payload artifact refs: {error}")) +} + +pub(super) fn delete_orphans( + tx: &Transaction<'_>, + source: ImportedHistorySourceId, + source_session_id: &str, + generation: &str, +) -> Result<(), String> { + tx.execute( + "DELETE FROM imported_replay_payload_artifacts AS artifact + WHERE artifact.source=?1 AND artifact.source_session_id=?2 AND artifact.generation=?3 + AND artifact.content_hash NOT IN ( + SELECT ref.content_hash FROM imported_replay_payload_artifact_refs AS ref + WHERE ref.source=?1 + AND ref.source_session_id=?2 + AND ref.generation=?3 + ) + AND artifact.content_hash NOT IN ( + SELECT shell.content_hash FROM imported_replay_shell_segments AS shell + WHERE shell.source=?1 + AND shell.source_session_id=?2 + AND shell.generation=?3 + )", + params![source.as_str(), source_session_id, generation], + ) + .map(|_| ()) + .map_err(|error| format!("delete orphan replay payload artifacts: {error}")) +} + +fn temporary_hash(event_id: &str, field_path: &str) -> String { + let mut hash = Sha256::new(); + hash.update((event_id.len() as u64).to_le_bytes()); + hash.update(event_id.as_bytes()); + hash.update((field_path.len() as u64).to_le_bytes()); + hash.update(field_path.as_bytes()); + format!("pending-{}", digest_hex(hash.finalize())) +} + +fn sha256_hex(bytes: &[u8]) -> String { + digest_hex(Sha256::digest(bytes)) +} + +fn digest_hex(digest: impl AsRef<[u8]>) -> String { + digest + .as_ref() + .iter() + .map(|byte| format!("{byte:02x}")) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::store::sqlite::SqliteRecordStore; + + #[test] + fn streamed_artifact_is_exact_and_content_deduplicated() { + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("replay cache schema"); + let tx = conn.transaction().expect("artifact transaction"); + let source = ImportedHistorySourceId::CodexApp; + let expected = "你🙂payload".repeat(10_000); + let hash = store_streamed( + &tx, + source, + "session", + "generation", + "event-a", + "result.output", + expected.len() as u64, + |writer| { + for bytes in expected.as_bytes().chunks(997) { + writer.write_all(bytes).map_err(|error| error.to_string())?; + } + Ok(()) + }, + ) + .expect("stream artifact"); + for index in 1..50 { + let duplicate = store_text( + &tx, + source, + "session", + "generation", + &format!("event-{index}"), + "result.output", + &expected, + ) + .expect("deduplicated artifact"); + assert_eq!(hash, duplicate); + } + assert_eq!(hash.len(), 64); + let artifact_count = tx + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifacts", + [], + |row| row.get::<_, i64>(0), + ) + .expect("artifact count"); + assert_eq!(artifact_count, 1); + let reference_count = tx + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifact_refs", + [], + |row| row.get::<_, i64>(0), + ) + .expect("artifact reference count"); + assert_eq!(reference_count, 50); + } + + #[test] + fn canonical_hash_is_standard_sha256() { + assert_eq!( + sha256_hex(b"abc"), + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad" + ); + } +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/perf_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/perf_tests.rs new file mode 100644 index 000000000..1d902246a --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/perf_tests.rs @@ -0,0 +1,426 @@ +//! Explicit #443 memory acceptance harnesses. +//! +//! These stay ignored in the normal unit suite because they write a 300 MiB +//! deterministic fixture and sample OS peak RSS. Run the named test in an +//! otherwise idle process before merging replay/index changes. + +use std::fs::{File, OpenOptions}; +use std::io::{BufWriter, Write}; +use std::path::{Path, PathBuf}; +#[cfg(unix)] +use std::process::Command; + +use rusqlite::params; + +use super::{ + open_window, poll_delta, ImportedHistorySourceId, ReplayLimits, HARD_MAX_EVENTS, + HARD_MAX_IPC_BYTES, HARD_MAX_TURNS, +}; +use crate::store::sqlite::SqliteRecordStore; + +const MIB: usize = 1024 * 1024; +const THIRTY_MIB: u64 = 30 * MIB as u64; +const THREE_HUNDRED_MIB: u64 = 300 * MIB as u64; +#[cfg(unix)] +const RSS_CHILD_ENV: &str = "ORGII_ISSUE_443_RSS_CHILD"; +#[cfg(unix)] +const RSS_FIXTURE_PATH_ENV: &str = "ORGII_ISSUE_443_RSS_FIXTURE_PATH"; +#[cfg(unix)] +const REAL_CODEX_JSONL_ENV: &str = "ORGII_ISSUE_443_REAL_CODEX_JSONL"; + +struct TempFixture { + root: PathBuf, + path: PathBuf, +} + +impl TempFixture { + fn new() -> Self { + let root = std::env::temp_dir().join(format!( + "orgii-issue-443-rss-{}-{}", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + std::fs::create_dir(&root).expect("create isolated replay fixture directory"); + let path = root.join("session.jsonl"); + Self { root, path } + } +} + +impl Drop for TempFixture { + fn drop(&mut self) { + if let Err(error) = std::fs::remove_dir_all(&self.root) { + if error.kind() != std::io::ErrorKind::NotFound { + eprintln!( + "failed to clean #443 replay fixture {}: {error}", + self.root.display() + ); + } + } + } +} + +fn assistant_line() -> Vec { + let body = "r".repeat(32 * 1024); + let mut line = serde_json::json!({ + "timestamp": "2026-07-22T00:00:01Z", + "type": "event_msg", + "payload": { "type": "agent_message", "message": body }, + }) + .to_string() + .into_bytes(); + line.push(b'\n'); + line +} + +fn extend_fixture(path: &Path, target_bytes: u64) { + let file = OpenOptions::new() + .create(true) + .append(true) + .open(path) + .expect("open deterministic replay fixture"); + let mut writer = BufWriter::with_capacity(1024 * 1024, file); + let line = assistant_line(); + let mut size = std::fs::metadata(path).map_or(0, |metadata| metadata.len()); + if size == 0 { + let user = serde_json::json!({ + "timestamp": "2026-07-22T00:00:00Z", + "type": "event_msg", + "payload": { "type": "user_message", "message": "RSS fixture" }, + }) + .to_string(); + writer.write_all(user.as_bytes()).expect("write user row"); + writer.write_all(b"\n").expect("finish user row"); + size = (user.len() + 1) as u64; + } + while size < target_bytes { + writer.write_all(&line).expect("extend replay fixture"); + size = size.saturating_add(line.len() as u64); + } + writer.flush().expect("flush replay fixture"); +} + +#[cfg(unix)] +fn peak_rss_bytes() -> usize { + let mut usage = std::mem::MaybeUninit::::zeroed(); + let result = unsafe { libc::getrusage(libc::RUSAGE_SELF, usage.as_mut_ptr()) }; + assert_eq!(result, 0, "getrusage failed"); + let peak = unsafe { usage.assume_init() }.ru_maxrss as usize; + #[cfg(target_os = "macos")] + { + peak + } + #[cfg(not(target_os = "macos"))] + { + peak.saturating_mul(1024) + } +} + +#[cfg(unix)] +fn sqlite_memory_bytes(reset_highwater: bool) -> (usize, usize) { + let mut current = 0_i64; + let mut highwater = 0_i64; + let status = unsafe { + rusqlite::ffi::sqlite3_status64( + rusqlite::ffi::SQLITE_STATUS_MEMORY_USED, + &mut current, + &mut highwater, + i32::from(reset_highwater), + ) + }; + assert_eq!(status, rusqlite::ffi::SQLITE_OK, "sqlite3_status64 failed"); + (current.max(0) as usize, highwater.max(0) as usize) +} + +#[cfg(unix)] +fn disk_cache_diagnostics(conn: &rusqlite::Connection, cache_path: &Path) { + let mut cache_used = 0; + let mut ignored_highwater = 0; + let status = unsafe { + rusqlite::ffi::sqlite3_db_status( + conn.handle(), + rusqlite::ffi::SQLITE_DBSTATUS_CACHE_USED, + &mut cache_used, + &mut ignored_highwater, + 0, + ) + }; + assert_eq!(status, rusqlite::ffi::SQLITE_OK, "sqlite3_db_status failed"); + + let (event_count, compact_text_bytes): (u64, u64) = conn + .query_row( + "SELECT COUNT(*), + COALESCE(SUM( + length(args_preview_json) + + length(result_preview_json) + + length(payloads_json) + ), 0) + FROM imported_replay_events", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("measure compact replay rows"); + let page_count: u64 = conn + .query_row("PRAGMA page_count", [], |row| row.get(0)) + .expect("read replay cache page count"); + let page_size: u64 = conn + .query_row("PRAGMA page_size", [], |row| row.get(0)) + .expect("read replay cache page size"); + let main_file_bytes = std::fs::metadata(cache_path).map_or(0, |metadata| metadata.len()); + let mut wal_path = cache_path.as_os_str().to_os_string(); + wal_path.push("-wal"); + let wal_file_bytes = + std::fs::metadata(PathBuf::from(wal_path)).map_or(0, |metadata| metadata.len()); + + eprintln!( + "#443 disk cache: events={event_count}, compact text={:.1} MiB, SQLite page cache={:.1} MiB, logical pages={:.1} MiB, files main={:.1} MiB/WAL={:.1} MiB", + compact_text_bytes as f64 / MIB as f64, + cache_used as f64 / MIB as f64, + page_count.saturating_mul(page_size) as f64 / MIB as f64, + main_file_bytes as f64 / MIB as f64, + wal_file_bytes as f64 / MIB as f64, + ); +} + +fn replay_fixture(path: &Path, cache_path: &Path) -> (rusqlite::Connection, String) { + let conn = rusqlite::Connection::open(cache_path).expect("open disk-backed replay cache"); + // Mirror `database::db::configure_connection`, which is applied to the + // production sessions DB before replay cache tables are used. Measuring + // an in-memory database here would count every compact-index page as + // resident application memory and would not model production. + conn.execute_batch( + "PRAGMA journal_mode = WAL; + PRAGMA synchronous = NORMAL; + PRAGMA cache_size = -64000; + PRAGMA temp_store = MEMORY; + PRAGMA busy_timeout = 15000; + PRAGMA wal_autocheckpoint = 2000;", + ) + .expect("configure production-like replay cache connection"); + SqliteRecordStore::init_tables(&conn).expect("initialize replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("initialize source cache"); + let session_id = "codexapp-issue-443-rss".to_string(); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES ('codex_app', 'issue-443-rss', ?1, ?2)", + params![session_id, path.to_string_lossy()], + ) + .expect("register replay source"); + (conn, session_id) +} + +#[cfg(unix)] +fn run_isolated_rss_child() { + // The parent owns cleanup so even a crashing/aborting child cannot leave a + // 300 MiB fixture behind after the parent observes its exit. + let fixture = TempFixture::new(); + let current_exe = std::env::current_exe().expect("resolve the Rust test executable"); + let status = Command::new(current_exe) + .arg("jsonl_cold_index_and_growth_have_bounded_peak_rss") + .arg("--ignored") + .arg("--nocapture") + .arg("--test-threads=1") + .env(RSS_CHILD_ENV, "1") + .env(RSS_FIXTURE_PATH_ENV, &fixture.path) + .status() + .expect("spawn isolated #443 RSS test child"); + assert!( + status.success(), + "isolated #443 RSS test child exited with {status}" + ); +} + +#[cfg(unix)] +fn run_rss_workload() { + let fixture_path = PathBuf::from( + std::env::var_os(RSS_FIXTURE_PATH_ENV) + .expect("isolated #443 RSS child requires a parent-owned fixture path"), + ); + let cache_path = fixture_path + .parent() + .expect("fixture path must have an isolated parent directory") + .join("replay-cache.sqlite"); + extend_fixture(&fixture_path, THIRTY_MIB); + let (mut conn, session_id) = replay_fixture(&fixture_path, &cache_path); + let limits = ReplayLimits { + max_turns: HARD_MAX_TURNS, + max_events: HARD_MAX_EVENTS, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }; + + let baseline_peak = peak_rss_bytes(); + let (cold_sqlite_baseline, _) = sqlite_memory_bytes(true); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + limits, + ) + .expect("cold-index 30 MiB fixture"); + let cold_peak = peak_rss_bytes(); + let (cold_sqlite_current, cold_sqlite_highwater) = sqlite_memory_bytes(false); + let cold_delta = cold_peak.saturating_sub(baseline_peak); + eprintln!( + "#443 RSS: baseline={:.1} MiB, 30 MiB cold peak={:.1} MiB, delta={:.1} MiB", + baseline_peak as f64 / MIB as f64, + cold_peak as f64 / MIB as f64, + cold_delta as f64 / MIB as f64 + ); + eprintln!( + "#443 SQLite heap (cold): baseline={:.1} MiB, current={:.1} MiB, highwater={:.1} MiB", + cold_sqlite_baseline as f64 / MIB as f64, + cold_sqlite_current as f64 / MIB as f64, + cold_sqlite_highwater as f64 / MIB as f64, + ); + assert!( + cold_delta <= 128 * MIB, + "30 MiB cold index grew peak RSS by {cold_delta} bytes" + ); + + extend_fixture(&fixture_path, THREE_HUNDRED_MIB); + let before_growth_peak = peak_rss_bytes(); + let (growth_sqlite_baseline, _) = sqlite_memory_bytes(true); + let delta = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor, + limits, + ) + .expect("incrementally index fixture growth to 300 MiB"); + assert!(delta.stats.parsed_bytes >= 260 * MIB as u64); + let grown_peak = peak_rss_bytes(); + let (growth_sqlite_current, growth_sqlite_highwater) = sqlite_memory_bytes(false); + let growth_delta = grown_peak.saturating_sub(before_growth_peak); + eprintln!( + "#443 RSS: before growth={:.1} MiB, 300 MiB peak={:.1} MiB, delta={:.1} MiB, parsed={:.1} MiB", + before_growth_peak as f64 / MIB as f64, + grown_peak as f64 / MIB as f64, + growth_delta as f64 / MIB as f64, + delta.stats.parsed_bytes as f64 / MIB as f64 + ); + eprintln!( + "#443 SQLite heap (growth): baseline={:.1} MiB, current={:.1} MiB, highwater={:.1} MiB", + growth_sqlite_baseline as f64 / MIB as f64, + growth_sqlite_current as f64 / MIB as f64, + growth_sqlite_highwater as f64 / MIB as f64, + ); + disk_cache_diagnostics(&conn, &cache_path); + assert!( + growth_delta <= 64 * MIB, + "30 -> 300 MiB indexing grew peak RSS by {growth_delta} bytes" + ); +} + +#[cfg(unix)] +#[test] +#[ignore = "#443 serial OS RSS stress; writes a deterministic 300 MiB JSONL fixture"] +fn jsonl_cold_index_and_growth_have_bounded_peak_rss() { + if std::env::var_os(RSS_CHILD_ENV).is_some() { + run_rss_workload(); + } else { + run_isolated_rss_child(); + } +} + +#[cfg(unix)] +#[test] +#[ignore = "#443 real Codex JSONL acceptance; requires ORGII_ISSUE_443_REAL_CODEX_JSONL"] +fn real_codex_jsonl_open_poll_and_reopen_stay_bounded() { + let source_path = PathBuf::from( + std::env::var_os(REAL_CODEX_JSONL_ENV) + .expect("set ORGII_ISSUE_443_REAL_CODEX_JSONL to a read-only transcript"), + ); + let source_bytes = std::fs::metadata(&source_path) + .expect("stat real Codex JSONL") + .len(); + assert!( + source_bytes >= 30 * MIB as u64, + "real acceptance source is too small" + ); + + let cache = TempFixture::new(); + let cache_path = cache.root.join("real-codex-replay-cache.sqlite"); + let (mut conn, session_id) = replay_fixture(&source_path, &cache_path); + let limits = ReplayLimits { + max_turns: HARD_MAX_TURNS, + max_events: HARD_MAX_EVENTS, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }; + + let baseline_peak = peak_rss_bytes(); + let first_started = std::time::Instant::now(); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + limits, + ) + .expect("open real Codex JSONL through bounded replay"); + let first_elapsed = first_started.elapsed(); + let first_peak = peak_rss_bytes(); + let first_growth = first_peak.saturating_sub(baseline_peak); + assert!(opened.chunks.len() <= HARD_MAX_EVENTS); + assert!(opened.turn_headers.len() <= HARD_MAX_TURNS); + assert!(opened.stats.parsed_bytes <= source_bytes.saturating_add(MIB as u64)); + assert!( + first_growth <= 128 * MIB, + "real Codex cold open grew peak RSS by {first_growth} bytes" + ); + + let unchanged = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor, + limits, + ) + .expect("poll unchanged real Codex JSONL"); + assert!(!unchanged.reset_required); + assert!(unchanged.chunks.is_empty()); + assert!(unchanged.removed_event_ids.is_empty()); + assert_eq!(unchanged.stats.parsed_bytes, 0); + assert_eq!(unchanged.stats.parsed_rows, 0); + + let reopen_started = std::time::Instant::now(); + let reopened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + limits, + ) + .expect("reopen real Codex JSONL from compact index"); + let reopen_elapsed = reopen_started.elapsed(); + let reopened_peak = peak_rss_bytes(); + assert_eq!(reopened.stats.parsed_bytes, 0); + assert_eq!(reopened.stats.parsed_rows, 0); + assert_eq!(reopened.cursor.generation, opened.cursor.generation); + assert_eq!(reopened.cursor.revision, opened.cursor.revision); + assert_eq!(reopened.chunks.len(), opened.chunks.len()); + + eprintln!( + "#443 real Codex JSONL: source={:.1} MiB, rows={}, total_events={}, window_events={}, turns={}, first_open={first_elapsed:?}, reopen={reopen_elapsed:?}, baseline_peak={:.1} MiB, first_peak={:.1} MiB, reopened_peak={:.1} MiB, first_growth={:.1} MiB", + source_bytes as f64 / MIB as f64, + opened.stats.parsed_rows, + opened.total_event_count, + opened.chunks.len(), + opened.turn_headers.len(), + baseline_peak as f64 / MIB as f64, + first_peak as f64 / MIB as f64, + reopened_peak as f64 / MIB as f64, + first_growth as f64 / MIB as f64, + ); +} + +#[test] +fn deterministic_generator_reaches_requested_size_without_one_large_allocation() { + let fixture = TempFixture::new(); + let _ = File::create(&fixture.path).expect("create small generator fixture"); + extend_fixture(&fixture.path, MIB as u64); + assert!( + std::fs::metadata(&fixture.path) + .expect("fixture metadata") + .len() + >= MIB as u64 + ); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/registry.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/registry.rs new file mode 100644 index 000000000..021e5a09e --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/registry.rs @@ -0,0 +1,346 @@ +//! Exhaustive imported-history replay source registry. +//! +//! A source may share a storage driver with another source, but it must still +//! have an explicit registry entry. In particular there is no "try the old +//! full loader" catch-all: adding a sixteenth source requires choosing a +//! driver and implementing its replay adapter deliberately. + +use serde::{Deserialize, Serialize}; + +use crate::sources::imported_history::metadata::*; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ImportedHistorySourceId { + ClaudeCode, + CodexApp, + CursorIde, + CursorCli, + OpenCode, + Windsurf, + WorkBuddy, + Trae, + Cline, + Warp, + ZCode, + Qoder, + MimoCode, + Omp, + QoderCli, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ReplayStorageFamily { + JsonLines, + SqliteWal, + SqliteKeyValue, + SqliteManifestBlob, + SqliteTaskBlob, + WholeJson, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ImportedReplayDescriptor { + pub source: ImportedHistorySourceId, + pub source_id: &'static str, + pub session_prefix: &'static str, + pub storage_family: ReplayStorageFamily, + pub parser_version: u32, + pub support: ReplayAdapterSupport, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReplayAdapterSupport { + Incremental, + Pending, +} + +impl ImportedHistorySourceId { + pub const ALL: [Self; 15] = [ + Self::ClaudeCode, + Self::CodexApp, + Self::CursorIde, + Self::CursorCli, + Self::OpenCode, + Self::Windsurf, + Self::WorkBuddy, + Self::Trae, + Self::Cline, + Self::Warp, + Self::ZCode, + Self::Qoder, + Self::MimoCode, + Self::Omp, + Self::QoderCli, + ]; + + pub fn parse(value: &str) -> Result { + Self::ALL + .into_iter() + .find(|source| source.as_str() == value) + .ok_or_else(|| format!("Unknown imported replay source: {value}")) + } + + pub const fn as_str(self) -> &'static str { + match self { + Self::ClaudeCode => SOURCE_CLAUDE_CODE, + Self::CodexApp => SOURCE_CODEX_APP, + Self::CursorIde => SOURCE_CURSOR_IDE, + Self::CursorCli => SOURCE_CURSOR_CLI, + Self::OpenCode => SOURCE_OPENCODE, + Self::Windsurf => SOURCE_WINDSURF, + Self::WorkBuddy => SOURCE_WORKBUDDY, + Self::Trae => SOURCE_TRAE, + Self::Cline => SOURCE_CLINE, + Self::Warp => SOURCE_WARP, + Self::ZCode => SOURCE_ZCODE, + Self::Qoder => SOURCE_QODER, + Self::MimoCode => SOURCE_MIMO_CODE, + Self::Omp => SOURCE_OMP, + Self::QoderCli => SOURCE_QODER_CLI, + } + } + + pub const fn descriptor(self) -> ImportedReplayDescriptor { + use crate::sources; + match self { + Self::ClaudeCode => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_CLAUDE_CODE, + session_prefix: sources::claude_code::SESSION_PREFIX, + storage_family: ReplayStorageFamily::JsonLines, + // v2 materializes non-rangeable JSONL payloads once per + // generation instead of decoding the source line per page. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::CodexApp => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_CODEX_APP, + session_prefix: sources::codex::SESSION_PREFIX, + storage_family: ReplayStorageFamily::JsonLines, + // v4 rebuilds the compact catalog projection so historical + // tool-call names can no longer remain cached as titles. + parser_version: 4, + support: ReplayAdapterSupport::Incremental, + }, + Self::CursorIde => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_CURSOR_IDE, + session_prefix: sources::cursor_ide::CURSORIDE_SESSION_PREFIX, + storage_family: ReplayStorageFamily::SqliteKeyValue, + // v2 derives turn headers and event sequences from the same + // canonical order after filtering stale/duplicate KV entries. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::CursorCli => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_CURSOR_CLI, + session_prefix: sources::cursor_cli::SESSION_PREFIX, + storage_family: ReplayStorageFamily::SqliteManifestBlob, + // v2 persists decoded manifest-blob payload artifacts. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::OpenCode => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_OPENCODE, + session_prefix: sources::opencode::history::OPENCODE_SESSION_PREFIX, + storage_family: ReplayStorageFamily::SqliteWal, + parser_version: 1, + support: ReplayAdapterSupport::Incremental, + }, + Self::Windsurf => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_WINDSURF, + session_prefix: sources::windsurf::history::WINDSURF_SESSION_PREFIX, + storage_family: ReplayStorageFamily::SqliteKeyValue, + // v2 shares Cursor IDE's canonical KV ordering contract. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::WorkBuddy => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_WORKBUDDY, + session_prefix: sources::workbuddy::WORKBUDDY_SESSION_PREFIX, + storage_family: ReplayStorageFamily::JsonLines, + // v2 materializes non-rangeable JSONL payloads once. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::Trae => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_TRAE, + session_prefix: sources::trae::history::TRAE_SESSION_PREFIX, + storage_family: ReplayStorageFamily::JsonLines, + // v2 materializes non-rangeable JSONL payloads once. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::Cline => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_CLINE, + session_prefix: sources::cline::history::CLINE_SESSION_PREFIX, + storage_family: ReplayStorageFamily::WholeJson, + // v2 upgrades rebuildable artifact integrity keys to SHA-256. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::Warp => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_WARP, + session_prefix: sources::warp::history::WARP_SESSION_PREFIX, + storage_family: ReplayStorageFamily::SqliteTaskBlob, + // v2 persists decoded task-BLOB payload artifacts. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::ZCode => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_ZCODE, + session_prefix: sources::zcode::history::ZCODE_SESSION_PREFIX, + storage_family: ReplayStorageFamily::SqliteWal, + parser_version: 1, + support: ReplayAdapterSupport::Incremental, + }, + Self::Qoder => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_QODER, + session_prefix: sources::qoder::history::QODER_SESSION_PREFIX, + storage_family: ReplayStorageFamily::JsonLines, + // v4 rejects sequence overflow and rebuilds the sidecar cursor + // with pre-allocation record limits and SHA-256 identities. + parser_version: 4, + support: ReplayAdapterSupport::Incremental, + }, + Self::MimoCode => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_MIMO_CODE, + session_prefix: sources::mimo_code::history::MIMO_CODE_SESSION_PREFIX, + storage_family: ReplayStorageFamily::SqliteWal, + parser_version: 1, + support: ReplayAdapterSupport::Incremental, + }, + Self::Omp => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_OMP, + session_prefix: sources::omp::history::OMP_SESSION_PREFIX, + storage_family: ReplayStorageFamily::JsonLines, + // v2 materializes non-rangeable JSONL payloads once. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + Self::QoderCli => ImportedReplayDescriptor { + source: self, + source_id: SOURCE_QODER_CLI, + session_prefix: sources::qoder_cli::history::QODER_CLI_SESSION_PREFIX, + storage_family: ReplayStorageFamily::JsonLines, + // v2 materializes non-rangeable JSONL payloads once. + parser_version: 2, + support: ReplayAdapterSupport::Incremental, + }, + } + } + + /// Derive a source from its canonical imported session-id prefix. + /// Prefixes are deliberately centralized here so a caller cannot route a + /// Codex session through another provider's adapter. + pub fn from_session_id(session_id: &str) -> Option { + Self::ALL + .into_iter() + .find(|source| session_id.starts_with(source.descriptor().session_prefix)) + } + + pub fn validate_session_id(self, session_id: &str) -> Result<(), String> { + let derived = Self::from_session_id(session_id) + .ok_or_else(|| format!("Session id {session_id} has no imported-history prefix"))?; + if derived != self { + return Err(format!( + "Replay source/session mismatch: source={} session={session_id} belongs to {}", + self.as_str(), + derived.as_str() + )); + } + Ok(()) + } + + pub fn source_session_id(self, session_id: &str) -> Result<&str, String> { + self.validate_session_id(session_id)?; + Ok(&session_id[self.descriptor().session_prefix.len()..]) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn registry_is_exhaustive_and_unique() { + assert_eq!( + ImportedHistorySourceId::ALL.len(), + IMPORTED_HISTORY_SOURCES.len() + ); + let mut ids = std::collections::BTreeSet::new(); + let mut prefixes = std::collections::BTreeSet::new(); + for source in ImportedHistorySourceId::ALL { + let descriptor = source.descriptor(); + assert!(ids.insert(descriptor.source_id)); + assert!(prefixes.insert(descriptor.session_prefix)); + assert_eq!(ImportedHistorySourceId::parse(source.as_str()), Ok(source)); + } + assert_eq!( + ids, + IMPORTED_HISTORY_SOURCES.into_iter().collect(), + "every transcript-bearing imported source must declare an explicit replay adapter" + ); + } + + #[test] + fn source_and_session_prefix_must_agree() { + let codex = ImportedHistorySourceId::CodexApp; + assert!(codex.validate_session_id("codexapp-abc").is_ok()); + assert!(codex.validate_session_id("claudecodeapp-abc").is_err()); + } + + #[test] + fn codex_catalog_title_fix_forces_existing_replay_indexes_to_rebuild() { + assert_eq!( + ImportedHistorySourceId::CodexApp + .descriptor() + .parser_version, + 4 + ); + } + + #[test] + fn completed_jsonl_adapters_are_advertised_incremental() { + let incremental = ImportedHistorySourceId::ALL + .into_iter() + .filter(|source| { + source.descriptor().storage_family == ReplayStorageFamily::JsonLines + && source.descriptor().support == ReplayAdapterSupport::Incremental + }) + .collect::>(); + assert_eq!( + incremental, + vec![ + ImportedHistorySourceId::ClaudeCode, + ImportedHistorySourceId::CodexApp, + ImportedHistorySourceId::WorkBuddy, + ImportedHistorySourceId::Trae, + ImportedHistorySourceId::Qoder, + ImportedHistorySourceId::Omp, + ImportedHistorySourceId::QoderCli, + ] + ); + } + + #[test] + fn all_fifteen_sources_have_explicit_incremental_adapters() { + assert!(ImportedHistorySourceId::ALL + .into_iter() + .all(|source| source.descriptor().support == ReplayAdapterSupport::Incremental)); + } +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/cache_policy.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/cache_policy.rs new file mode 100644 index 000000000..6a7e1616e --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/cache_policy.rs @@ -0,0 +1,882 @@ +use std::path::Path; + +use rusqlite::Connection; +use serde_json::json; + +use super::*; +use crate::sources::imported_history::replay::{ + open_window, ImportedHistorySourceId, ReplayLimits, +}; +use crate::store::sqlite::SqliteRecordStore; + +fn cache() -> Connection { + let conn = Connection::open_in_memory().expect("replay cache"); + SqliteRecordStore::init_tables(&conn).expect("core schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("replay schema"); + conn +} + +fn cache_at(path: &Path) -> Connection { + let conn = Connection::open(path).expect("file-backed replay cache"); + SqliteRecordStore::init_tables(&conn).expect("core schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("replay schema"); + conn +} + +fn iso(ms: i64) -> String { + DateTime::from_timestamp_millis(ms) + .expect("test timestamp") + .to_rfc3339() +} + +fn bind_catalog( + conn: &Connection, + source: &str, + source_session_id: &str, + session_id: &str, + path: &str, +) { + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path,source_record_key, + source_mtime_ms,source_size_bytes,source_fingerprint,parser_version, + name,created_at_ms,updated_at_ms,model,input_tokens,output_tokens, + cache_read_tokens,cache_write_tokens,repo_path,branch,files_changed, + lines_added,lines_removed,touched_files_json,listable, + source_metadata_json,parent_session_id,updated_at + ) VALUES(?1,?2,?3,?4,?2,123,456,'provider-fingerprint',1, + 'replay title',111,222,'model',10,20,3,4,'/repo','branch',1, + 2,3,'[\"src/lib.rs\"]',1,'{\"continuationGroupKey\":\"g\"}', + 'parent',?5)", + params![ + source, + source_session_id, + session_id, + path, + Utc::now().to_rfc3339() + ], + ) + .expect("catalog binding"); +} + +fn seed_entry( + conn: &mut Connection, + source: &str, + source_session_id: &str, + updated_at_ms: i64, + payload_bytes: usize, + provider_size: i64, +) { + let session_id = format!("{source}-{source_session_id}"); + bind_catalog( + conn, + source, + source_session_id, + &session_id, + "/provider/truth", + ); + let generation = format!("generation-{source_session_id}"); + conn.execute( + "INSERT INTO imported_replay_state( + source,source_session_id,generation,revision,parser_version, + source_identity,driver_cursor_json,indexed_size_bytes,indexed_mtime_ns, + total_events,total_turns,valid,updated_at + ) VALUES(?1,?2,?3,1,1,'identity','{\"cursor\":1}',?4,10,1,1,1,?5)", + params![ + source, + source_session_id, + generation, + provider_size, + iso(updated_at_ms) + ], + ) + .expect("replay state"); + conn.execute( + "INSERT INTO imported_replay_turns( + source,source_session_id,generation,turn_index,turn_id, + start_sequence,end_sequence,started_at,ended_at,event_count + ) VALUES(?1,?2,?3,0,'turn',0,0,'start','end',1)", + params![source, source_session_id, generation], + ) + .expect("replay turn"); + conn.execute( + "INSERT INTO imported_replay_events( + source,source_session_id,generation,sequence,event_id,turn_index, + action_type,function_name,created_at,args_preview_json,result_preview_json, + args_size_bytes,result_size_bytes,source_start,source_end,payloads_json, + content_hash,event_revision + ) VALUES(?1,?2,?3,0,'event',0,'tool_call','Shell','now', + '{\"command\":\"echo\"}','{\"output\":\"preview\"}',10,20,0,10, + '[{\"fieldPath\":\"result.output\"}]','hash',1)", + params![source, source_session_id, generation], + ) + .expect("replay event"); + conn.execute( + "INSERT INTO imported_replay_source_rows( + source,source_session_id,generation,source_key,content_hash, + event_id,sequence,source_order,seen_revision + ) VALUES(?1,?2,?3,'source-key','hash','event',0,0,1)", + params![source, source_session_id, generation], + ) + .expect("replay source row"); + conn.execute( + "INSERT INTO imported_replay_structured_rows( + source,source_session_id,generation,source_key,content_hash,seen_revision + ) VALUES(?1,?2,?3,'structured-key','hash',1)", + params![source, source_session_id, generation], + ) + .expect("structured row"); + conn.execute( + "INSERT INTO imported_replay_structured_events( + source,source_session_id,generation,source_key,local_key,event_id,sequence + ) VALUES(?1,?2,?3,'structured-key','local','event',0)", + params![source, source_session_id, generation], + ) + .expect("structured event"); + conn.execute( + "INSERT INTO imported_replay_changes( + source,source_session_id,generation,change_revision,event_id,change_kind,sequence + ) VALUES(?1,?2,?3,1,'event','upsert',0)", + params![source, source_session_id, generation], + ) + .expect("replay change"); + conn.execute( + "INSERT INTO imported_replay_payload_artifacts( + source,source_session_id,generation,content_hash,payload + ) VALUES(?1,?2,?3,'hash',?4)", + params![ + source, + source_session_id, + generation, + vec![b'x'; payload_bytes] + ], + ) + .expect("payload artifact"); + conn.execute( + "INSERT INTO imported_replay_payload_artifact_refs( + source,source_session_id,generation,event_id,field_path,content_hash + ) VALUES(?1,?2,?3,'event','result.output','hash')", + params![source, source_session_id, generation], + ) + .expect("payload artifact ref"); + conn.execute( + "INSERT INTO imported_replay_shell_manifests( + session_id,logical_call_id,call_id,identity_hash,total_bytes, + last_sequence,terminal_preview,completed_at,accessed_at + ) VALUES(?1,'logical-shell','shell-call','identity',?2,0, + 'preview','now',?3)", + params![session_id, provider_size, iso(updated_at_ms)], + ) + .expect("replay Shell manifest"); + conn.execute( + "INSERT INTO imported_replay_shell_segments( + session_id,call_id,ordinal,stream,source,source_session_id, + generation,content_hash,output_byte_start,total_bytes, + first_sequence,frame_count + ) VALUES(?1,'shell-call',0,'stdout',?2,?3,?4,'hash',0,?5,0,1)", + params![ + session_id, + source, + source_session_id, + generation, + provider_size + ], + ) + .expect("replay Shell segment"); + conn.execute( + "INSERT INTO imported_replay_rejected_snapshots( + source,source_session_id,parser_version,source_identity, + source_size_bytes,source_mtime_ns,sample_fingerprint, + rejection_kind,rejected_at + ) VALUES(?1,?2,1,'identity',99,100,'sample','test',?3)", + params![source, source_session_id, iso(updated_at_ms)], + ) + .expect("rejected watermark"); + + // Replay overlays only its compact projection onto the mixed-ownership + // catalog row. Prune must later restore this adapter-owned baseline. + let cursor = json!({ + "catalog": { + "model": "replay-model", + "inputTokens": 100, + "outputTokens": 20, + "tokensObserved": true, + "continuationGroupKey": "replay-group", + "continuationObserved": true + } + }) + .to_string(); + let tx = conn.transaction().expect("catalog projection transaction"); + crate::sources::imported_history::catalog::publish_from_replay_tx( + &tx, + ImportedHistorySourceId::parse(source).expect("registered replay source"), + source_session_id, + &generation, + 1, + true, + 10_000_000, + &cursor, + ) + .expect("publish replay catalog projection"); + tx.commit().expect("commit replay catalog projection"); +} + +fn seed_shell_only_scope( + conn: &Connection, + source_session_id: &str, + accessed_at_ms: i64, + payload_bytes: usize, +) { + let source = "managed_readerless"; + let session_id = format!("managed-{source_session_id}"); + let call_id = format!("call-{source_session_id}"); + let generation = format!("generation-{source_session_id}"); + let content_hash = format!("hash-{source_session_id}"); + conn.execute( + "INSERT INTO imported_replay_payload_artifacts( + source,source_session_id,generation,content_hash,payload + ) VALUES(?1,?2,?3,?4,?5)", + params![ + source, + source_session_id, + generation, + content_hash, + vec![b'x'; payload_bytes] + ], + ) + .expect("readerless payload artifact"); + conn.execute( + "INSERT INTO imported_replay_shell_manifests( + session_id,logical_call_id,call_id,identity_hash,total_bytes, + last_sequence,terminal_preview,completed_at,accessed_at + ) VALUES(?1,?2,?3,'identity',?4,1,'preview', + '2000-01-01T00:00:00Z',?5)", + params![ + session_id, + format!("logical-{source_session_id}"), + call_id, + payload_bytes as i64, + iso(accessed_at_ms) + ], + ) + .expect("readerless Shell manifest"); + conn.execute( + "INSERT INTO imported_replay_shell_segments( + session_id,call_id,ordinal,stream,source,source_session_id, + generation,content_hash,output_byte_start,total_bytes, + first_sequence,frame_count + ) VALUES(?1,?2,0,'stdout',?3,?4,?5,?6,0,?7,1,1)", + params![ + session_id, + call_id, + source, + source_session_id, + generation, + content_hash, + payload_bytes as i64 + ], + ) + .expect("readerless Shell segment"); +} + +fn count(conn: &Connection, table: &str, source_session_id: &str) -> i64 { + conn.query_row( + &format!("SELECT COUNT(*) FROM {table} WHERE source_session_id=?1"), + [source_session_id], + |row| row.get(0), + ) + .expect("count replay rows") +} + +fn no_protection_policy(max_bytes: u64, target_bytes: u64) -> ReplayCachePolicy { + ReplayCachePolicy { + max_bytes, + target_bytes, + ttl: Duration::from_secs(365 * 24 * 60 * 60), + protect_recent: Duration::ZERO, + } +} + +fn candidate_entry(source_session_id: &str, last_accessed_ms: i64, bytes: u64) -> CacheEntry { + CacheEntry { + source: "cline".to_string(), + source_session_id: source_session_id.to_string(), + last_accessed_at: iso(last_accessed_ms), + last_accessed_ms, + approx_bytes: bytes, + protected: false, + } +} + +#[test] +fn cache_table_registry_is_exhaustive() { + let conn = cache(); + let mut statement = conn + .prepare( + "SELECT name FROM sqlite_master + WHERE type='table' AND name LIKE 'imported_replay_%' + ORDER BY name", + ) + .expect("prepare replay cache table registry"); + let mut actual = statement + .query_map([], |row| row.get::<_, String>(0)) + .expect("query replay cache table registry") + .collect::, _>>() + .expect("read replay cache table registry"); + let mut expected = REPLAY_CACHE_TABLES + .iter() + .map(|table| table.name.to_string()) + .collect::>(); + actual.sort(); + expected.sort(); + assert_eq!(actual, expected, "new replay tables need prune coverage"); + + let mut accounted = REPLAY_CACHE_BYTE_AGGREGATIONS + .iter() + .map(|aggregation| aggregation.table.to_string()) + .collect::>(); + accounted.sort(); + assert_eq!( + actual, accounted, + "new replay tables need set-based byte accounting" + ); +} + +#[test] +fn maintenance_selection_is_bounded_by_scope_count_and_approximate_bytes() { + let now = 1_800_000_000_000_i64; + let ten_small = (0..10) + .map(|index| candidate_entry(&format!("scope-{index}"), now + index, 1024)) + .collect::>(); + let selected = select_eviction_candidates(&ten_small, no_protection_policy(0, 0), 0, now); + assert_eq!(selected.len(), MAX_EVICTION_SCOPES_PER_RUN); + + let seventy_mib = 70 * 1024 * 1024; + let two_large = vec![ + candidate_entry("oldest", now, seventy_mib), + candidate_entry("newer", now + 1, seventy_mib), + ]; + let selected = select_eviction_candidates(&two_large, no_protection_policy(0, 0), 0, now); + assert_eq!(selected.len(), 1); + assert_eq!(selected[0].0, 0); + + let oversized = vec![ + candidate_entry("oversized", now, MAX_EVICTION_BYTES_PER_RUN + 1), + candidate_entry("later", now + 1, 1), + ]; + let selected = select_eviction_candidates(&oversized, no_protection_policy(0, 0), 0, now); + assert_eq!(selected.len(), 1, "oversized oldest scope runs alone"); + assert_eq!(selected[0].0, 0); +} + +#[test] +fn byte_accounting_runs_one_query_per_table_independent_of_scope_count() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_entry(&mut conn, "cline", "one", now, 32, 1); + CACHE_BYTE_AGGREGATION_QUERY_COUNT.with(|count| count.set(0)); + let one_scope = load_cache_entries(&conn).expect("measure one replay scope"); + assert_eq!(one_scope.len(), 1); + let one_scope_queries = CACHE_BYTE_AGGREGATION_QUERY_COUNT.with(|count| count.get()); + + for index in 2..=12 { + seed_entry( + &mut conn, + "cline", + &format!("scope-{index}"), + now + index, + 32, + 1, + ); + } + CACHE_BYTE_AGGREGATION_QUERY_COUNT.with(|count| count.set(0)); + let many_scopes = load_cache_entries(&conn).expect("measure many replay scopes"); + let many_scope_queries = CACHE_BYTE_AGGREGATION_QUERY_COUNT.with(|count| count.get()); + + assert_eq!(many_scopes.len(), 12); + assert_eq!(one_scope_queries, REPLAY_CACHE_BYTE_AGGREGATIONS.len()); + assert_eq!(many_scope_queries, one_scope_queries); +} + +#[test] +fn default_policy_matches_the_global_cache_contract() { + assert_eq!( + ReplayCachePolicy::default(), + ReplayCachePolicy { + max_bytes: 512 * 1024 * 1024, + target_bytes: 384 * 1024 * 1024, + ttl: Duration::from_secs(7 * 24 * 60 * 60), + protect_recent: Duration::from_secs(3 * 60), + } + ); +} + +#[test] +fn ttl_prune_deletes_every_rebuildable_table_but_keeps_provider_binding() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_entry( + &mut conn, + "cline", + "expired", + now - duration_millis(DEFAULT_REPLAY_CACHE_TTL) - 1, + 4096, + 10 * 1024 * 1024 * 1024, + ); + seed_entry(&mut conn, "cline", "fresh", now - 60_000, 64, 1); + + let report = + prune_cache_at(&mut conn, ReplayCachePolicy::default(), now).expect("TTL replay prune"); + assert_eq!(report.ttl_evictions, 1); + assert_eq!(report.budget_evictions, 0); + assert_eq!(report.evictions[0].source_session_id, "expired"); + assert!( + report.evictions[0].approx_bytes < 1024 * 1024, + "provider indexed_size_bytes must not count as cache bytes" + ); + for table in [ + "imported_replay_state", + "imported_replay_turns", + "imported_replay_events", + "imported_replay_source_rows", + "imported_replay_structured_rows", + "imported_replay_structured_events", + "imported_replay_changes", + "imported_replay_payload_artifact_refs", + "imported_replay_payload_artifacts", + "imported_replay_rejected_snapshots", + "imported_replay_catalog_derivations", + "imported_replay_shell_segments", + ] { + assert_eq!(count(&conn, table, "expired"), 0, "{table}"); + assert_eq!(count(&conn, table, "fresh"), 1, "{table}"); + } + let manifest_count = |session_id: &str| { + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_manifests + WHERE session_id=?1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .expect("count replay Shell manifests") + }; + assert_eq!(manifest_count("cline-expired"), 0); + assert_eq!(manifest_count("cline-fresh"), 1); + let catalog = conn + .query_row( + "SELECT source_path,name,model,files_changed,touched_files_json, + source_metadata_json,parent_session_id + FROM imported_history_session_cache + WHERE source='cline' AND source_session_id='expired'", + [], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, String>(2)?, + row.get::<_, i64>(3)?, + row.get::<_, String>(4)?, + row.get::<_, String>(5)?, + row.get::<_, String>(6)?, + )) + }, + ) + .expect("pruned catalog binding"); + assert_eq!(catalog.0, "/provider/truth"); + assert_eq!(catalog.1, "replay title"); + assert_eq!(catalog.2, "model"); + assert_eq!(catalog.3, 1); + assert_eq!(catalog.4, r#"["src/lib.rs"]"#); + assert_eq!(catalog.5, r#"{"continuationGroupKey":"g"}"#); + assert_eq!(catalog.6, "parent"); +} + +#[test] +fn ttl_prunes_rejected_snapshot_without_valid_generation() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + conn.execute( + "INSERT INTO imported_replay_rejected_snapshots( + source,source_session_id,parser_version,source_identity, + source_size_bytes,source_mtime_ns,sample_fingerprint, + rejection_kind,rejected_at + ) VALUES('cline','never-valid',1,'identity',99,100,'sample','json',?1)", + [iso(now - duration_millis(DEFAULT_REPLAY_CACHE_TTL) - 1)], + ) + .expect("standalone rejected snapshot"); + + let report = prune_cache_at(&mut conn, ReplayCachePolicy::default(), now) + .expect("prune standalone rejected snapshot"); + + assert_eq!(report.ttl_evictions, 1); + assert_eq!(report.evictions[0].source_session_id, "never-valid"); + assert_eq!( + count(&conn, "imported_replay_rejected_snapshots", "never-valid"), + 0 + ); +} + +#[test] +fn shell_only_managed_scopes_are_counted_and_evicted_by_orgii_access_time() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_shell_only_scope( + &conn, + "expired", + now - duration_millis(DEFAULT_REPLAY_CACHE_TTL) - 1, + 8192, + ); + seed_shell_only_scope(&conn, "fresh", now - 60_000, 4096); + assert_eq!(count(&conn, "imported_replay_state", "expired"), 0); + assert_eq!(count(&conn, "imported_replay_state", "fresh"), 0); + + let report = prune_cache_at(&mut conn, ReplayCachePolicy::default(), now) + .expect("prune readerless Shell cache"); + + assert!(report.before_bytes >= 8192 + 4096); + assert_eq!(report.ttl_evictions, 1); + assert_eq!(report.protected_entries, 1); + assert_eq!(report.evictions[0].source_id, "managed_readerless"); + assert_eq!(report.evictions[0].source_session_id, "expired"); + assert_eq!( + count(&conn, "imported_replay_payload_artifacts", "expired"), + 0 + ); + assert_eq!( + count(&conn, "imported_replay_payload_artifacts", "fresh"), + 1 + ); + assert_eq!(count(&conn, "imported_replay_shell_segments", "expired"), 0); + assert_eq!(count(&conn, "imported_replay_shell_segments", "fresh"), 1); + let manifest_count = |session_id: &str| { + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_manifests + WHERE session_id=?1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .expect("count readerless Shell manifest") + }; + assert_eq!(manifest_count("managed-expired"), 0); + assert_eq!(manifest_count("managed-fresh"), 1); +} + +#[test] +fn selected_shell_scope_touched_before_writer_lock_is_not_evicted() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_shell_only_scope( + &conn, + "became-active", + now - duration_millis(DEFAULT_REPLAY_CACHE_TTL) - 1, + 4096, + ); + let touched_at = iso(now - 60_000); + + let report = prune_cache_at_with_hook(&mut conn, ReplayCachePolicy::default(), now, |conn| { + conn.execute( + "UPDATE imported_replay_shell_manifests SET accessed_at=?1 + WHERE session_id='managed-became-active'", + [&touched_at], + ) + .map(|_| ()) + .map_err(|err| format!("simulate manifest delivery touch: {err}")) + }) + .expect("revalidate touched Shell scope"); + + assert_eq!(report.evicted_entries, 0); + assert_eq!(report.ttl_evictions, 0); + assert_eq!( + count(&conn, "imported_replay_payload_artifacts", "became-active"), + 1 + ); + assert_eq!( + count(&conn, "imported_replay_shell_segments", "became-active"), + 1 + ); +} + +#[test] +fn shell_only_managed_scopes_participate_in_byte_lru() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_shell_only_scope(&conn, "older", now - 30_000, 8192); + seed_shell_only_scope(&conn, "newer", now - 10_000, 4096); + let entries = load_cache_entries(&conn).expect("measure readerless Shell scopes"); + let total = entries.iter().map(|entry| entry.approx_bytes).sum::(); + let older_bytes = entries + .iter() + .find(|entry| entry.source_session_id == "older") + .expect("older readerless scope") + .approx_bytes; + + let report = prune_cache_at( + &mut conn, + no_protection_policy(total - 1, total.saturating_sub(older_bytes)), + now, + ) + .expect("byte-LRU readerless Shell cache"); + + assert_eq!(report.ttl_evictions, 0); + assert_eq!(report.budget_evictions, 1); + assert_eq!(report.evictions[0].source_session_id, "older"); + assert_eq!( + count(&conn, "imported_replay_payload_artifacts", "older"), + 0 + ); + assert_eq!( + count(&conn, "imported_replay_payload_artifacts", "newer"), + 1 + ); +} + +#[test] +fn manifest_accounting_deduplicates_segment_scope_and_manifest_key() { + let now = 1_800_000_000_000_i64; + let conn = cache(); + seed_shell_only_scope(&conn, "dedup", now, 64); + let before = load_cache_entries(&conn) + .expect("measure one Shell segment") + .into_iter() + .find(|entry| entry.source_session_id == "dedup") + .expect("readerless scope") + .approx_bytes; + + conn.execute( + "INSERT INTO imported_replay_shell_segments( + session_id,call_id,ordinal,stream,source,source_session_id, + generation,content_hash,output_byte_start,total_bytes, + first_sequence,frame_count + ) VALUES('managed-dedup','call-dedup',1,'stdout','managed_readerless', + 'dedup','generation-dedup','hash-dedup',64,64,2,1)", + [], + ) + .expect("second segment for the same Shell manifest"); + let after = load_cache_entries(&conn) + .expect("measure two Shell segments") + .into_iter() + .find(|entry| entry.source_session_id == "dedup") + .expect("readerless scope") + .approx_bytes; + + let second_segment_bytes = APPROX_ROW_OVERHEAD_BYTES as u64 + + 48 + + "managed-dedup".len() as u64 + + "call-dedup".len() as u64 + + "stdout".len() as u64 + + "managed_readerless".len() as u64 + + "dedup".len() as u64 + + "generation-dedup".len() as u64 + + "hash-dedup".len() as u64; + assert_eq!( + after - before, + second_segment_bytes, + "adding a locator must not count the shared manifest a second time" + ); +} + +#[test] +fn byte_budget_uses_oldest_unprotected_entry_until_target() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_entry(&mut conn, "cline", "oldest", now - 30_000, 16_384, 1); + seed_entry(&mut conn, "cline", "middle", now - 20_000, 8192, 1); + seed_entry(&mut conn, "cline", "newest", now - 10_000, 4096, 1); + let entries = load_cache_entries(&conn).expect("measure seeded replay entries"); + let total = entries.iter().map(|entry| entry.approx_bytes).sum::(); + let oldest_bytes = entries + .iter() + .find(|entry| entry.source_session_id == "oldest") + .expect("oldest entry") + .approx_bytes; + let policy = no_protection_policy(total - 1, total.saturating_sub(oldest_bytes)); + + let report = prune_cache_at(&mut conn, policy, now).expect("byte LRU prune"); + assert_eq!(report.ttl_evictions, 0); + assert_eq!(report.budget_evictions, 1); + assert_eq!(report.evictions[0].source_session_id, "oldest"); + assert!(report.after_bytes <= policy.target_bytes); + assert_eq!(count(&conn, "imported_replay_state", "middle"), 1); + assert_eq!(count(&conn, "imported_replay_state", "newest"), 1); +} + +#[test] +fn recent_entries_are_protected_even_when_cache_stays_over_budget() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_entry(&mut conn, "cline", "old", now - 10 * 60_000, 4096, 1); + seed_entry(&mut conn, "cline", "active", now - 60_000, 4096, 1); + let policy = ReplayCachePolicy { + max_bytes: 0, + target_bytes: 0, + ttl: Duration::ZERO, + protect_recent: DEFAULT_REPLAY_CACHE_PROTECT_RECENT, + }; + + let report = prune_cache_at(&mut conn, policy, now).expect("protected replay prune"); + assert_eq!(report.evicted_entries, 1); + assert_eq!(report.protected_entries, 1); + assert!(report.over_budget); + assert_eq!(count(&conn, "imported_replay_state", "old"), 0); + assert_eq!(count(&conn, "imported_replay_state", "active"), 1); +} + +#[test] +fn writer_lock_is_released_between_scope_transactions() { + let now = 1_800_000_000_000_i64; + let path = temp_path("writer-lock").with_extension("sqlite"); + let mut conn = cache_at(&path); + conn.execute_batch( + "CREATE TABLE cache_lock_probe(value INTEGER NOT NULL); + INSERT INTO cache_lock_probe(value) VALUES(0);", + ) + .expect("writer lock probe"); + seed_entry(&mut conn, "cline", "first", now - 20_000, 1024, 1); + seed_entry(&mut conn, "cline", "second", now - 10_000, 1024, 1); + + let peer = Connection::open(&path).expect("peer cache writer"); + peer.busy_timeout(Duration::from_millis(50)) + .expect("short peer writer timeout"); + let report = prune_cache_at_with_hooks( + &mut conn, + no_protection_policy(0, 0), + now, + |_| Ok(()), + |_| { + peer.execute("UPDATE cache_lock_probe SET value=value+1", []) + .map(|_| ()) + .map_err(|err| format!("peer writer remained blocked: {err}")) + }, + ) + .expect("scope-local replay prune transactions"); + + assert_eq!(report.evicted_entries, 2); + let peer_writes: i64 = peer + .query_row("SELECT value FROM cache_lock_probe", [], |row| row.get(0)) + .expect("peer writer count"); + assert_eq!(peer_writes, 2); + drop(peer); + drop(conn); + let _ = std::fs::remove_file(path); +} + +#[test] +fn failed_later_scope_keeps_prior_commit_and_rolls_back_failing_scope() { + let now = 1_800_000_000_000_i64; + let mut conn = cache(); + seed_entry(&mut conn, "cline", "a-first", now - 20_000, 1024, 1); + seed_entry(&mut conn, "cline", "z-broken", now - 10_000, 1024, 1); + conn.execute( + "UPDATE imported_replay_catalog_derivations SET baseline_json='{' + WHERE source='cline' AND source_session_id='z-broken'", + [], + ) + .expect("corrupt second derivation guard"); + + let error = prune_cache_at(&mut conn, no_protection_policy(0, 0), now) + .expect_err("malformed guard must abort prune"); + + assert!(error.contains("decode replay catalog prune baseline")); + assert_eq!(count(&conn, "imported_replay_state", "a-first"), 0); + assert_eq!( + count(&conn, "imported_replay_payload_artifacts", "a-first"), + 0 + ); + assert_eq!(count(&conn, "imported_replay_state", "z-broken"), 1); + assert_eq!( + count(&conn, "imported_replay_payload_artifacts", "z-broken"), + 1 + ); + let manifest_count: i64 = conn + .query_row( + "SELECT COUNT(*) FROM imported_replay_shell_manifests", + [], + |row| row.get(0), + ) + .expect("count partially committed Shell manifests"); + assert_eq!(manifest_count, 1); +} + +#[test] +fn pruning_one_database_identity_cannot_touch_another() { + let now = 1_800_000_000_000_i64; + let mut first = cache(); + let mut second = cache(); + seed_entry(&mut first, "cline", "shared-id", now - 10_000, 1024, 1); + seed_entry(&mut second, "cline", "shared-id", now - 10_000, 1024, 1); + + prune_cache_at(&mut first, no_protection_policy(0, 0), now) + .expect("prune first database identity"); + assert_eq!(count(&first, "imported_replay_state", "shared-id"), 0); + assert_eq!(count(&second, "imported_replay_state", "shared-id"), 1); +} + +fn temp_path(name: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!( + "orgii-replay-prune-{name}-{}-{}.json", + std::process::id(), + Utc::now().timestamp_nanos_opt().unwrap_or_default() + )) +} + +#[test] +fn pruned_entry_rebuilds_from_untouched_provider_truth() { + let path = temp_path("rebuild"); + let transcript = json!({ + "messages":[ + {"role":"user","content":[{"type":"text","text":"hello"}]}, + {"role":"assistant","content":[{"type":"text","text":"world"}]} + ] + }); + std::fs::write(&path, transcript.to_string()).expect("provider Cline transcript"); + let mut conn = cache(); + let session_id = "clineapp-rebuild"; + bind_catalog( + &conn, + "cline", + "rebuild", + session_id, + &path.to_string_lossy(), + ); + let first = open_window( + &mut conn, + ImportedHistorySourceId::Cline, + session_id, + ReplayLimits::default(), + ) + .expect("build initial replay cache"); + assert_eq!(first.chunks.len(), 2); + conn.execute( + "UPDATE imported_replay_state SET updated_at='2000-01-01T00:00:00Z' + WHERE source='cline' AND source_session_id='rebuild'", + [], + ) + .expect("age replay cache entry"); + + let report = prune_cache_at( + &mut conn, + ReplayCachePolicy { + max_bytes: 0, + target_bytes: 0, + ttl: Duration::ZERO, + protect_recent: Duration::ZERO, + }, + 1_800_000_000_000, + ) + .expect("prune replay entry"); + assert_eq!(report.evicted_entries, 1); + assert!(Path::new(&path).is_file(), "provider truth is untouched"); + assert_eq!(count(&conn, "imported_replay_state", "rebuild"), 0); + + let rebuilt = open_window( + &mut conn, + ImportedHistorySourceId::Cline, + session_id, + ReplayLimits::default(), + ) + .expect("rebuild replay from provider truth"); + assert_eq!(rebuilt.chunks.len(), 2); + assert_eq!(rebuilt.total_event_count, first.total_event_count); + assert_eq!(count(&conn, "imported_replay_state", "rebuild"), 1); + let _ = std::fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/codex_jsonl.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/codex_jsonl.rs new file mode 100644 index 000000000..c5b9f7ae1 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/codex_jsonl.rs @@ -0,0 +1,188 @@ +use super::*; +use crate::store::sqlite::SqliteRecordStore; + +#[test] +fn previews_respect_utf8_boundaries() { + let text = "你".repeat(10_000); + let (head, head_truncated) = head_preview(&text, 8 * 1024); + let (tail, tail_truncated) = tail_preview(&text, 8 * 1024); + assert!(head_truncated && tail_truncated); + assert!(head.is_char_boundary(head.len())); + assert!(tail.is_char_boundary(tail.len())); +} + +#[test] +fn pending_cursor_never_contains_complete_large_output() { + let mut background = PendingBackgroundGroup { + calls: Vec::new(), + spans: Vec::new(), + output_preview: String::new(), + output_bytes: 0, + git_artifacts: Vec::new(), + }; + append_background_output( + &mut background, + &"x".repeat(1024 * 1024), + ReplaySourceSpan { start: 0, end: 1 }, + ); + assert!(background.output_preview.len() <= SHELL_PAYLOAD_PREVIEW_BYTES); + assert_eq!(background.output_bytes, 1024 * 1024); +} + +#[test] +fn codex_fallback_payload_range_reports_adjusted_utf8_offsets() { + let line = serde_json::json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"event_msg", + "payload":{"type":"agent_message","message":"a你b"} + }) + .to_string(); + let path = std::env::temp_dir().join(format!( + "orgii-codex-utf8-fallback-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{line}\n")).expect("Codex UTF-8 fixture"); + let descriptor = ReplayPayloadDescriptor { + field_path: "result.content".to_string(), + kind: ReplayPayloadKind::AgentMessage, + encoding: ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: vec![ReplaySourceSpan { + start: 0, + end: line.len() as u64 + 1, + }], + total_bytes: "a你b".len() as u64, + source_ordinal: None, + source_key: None, + }; + let range = read_payload( + &path, + &serde_json::to_string(&vec![descriptor]).expect("Codex payload locator"), + "event", + "result.content", + 2, + 1, + ) + .expect("Codex UTF-8 fallback range"); + assert_eq!(range.offset, 4); + assert_eq!(range.next_offset, 5); + assert_eq!(range.text, "b"); + assert!(range.eof); + let _ = fs::remove_file(path); +} + +#[test] +fn ten_mib_tool_args_stay_out_of_index_and_cursor_but_reconstruct_exactly() { + let command = format!("printf start {} end", "中🙂x".repeat(1_250_000)); + let arguments = serde_json::json!({ + "command": command, + "workdir": "/tmp/project", + "path": "src/lib.rs" + }); + let line: CodexJsonlLine = serde_json::from_value(serde_json::json!({ + "timestamp": "2026-07-22T00:00:00Z", + "type": "response_item", + "payload": { + "type": "function_call", + "name": "shell_command", + "arguments": serde_json::to_string(&arguments).unwrap(), + "call_id": "call-large-args" + } + })) + .expect("Codex tool line"); + let (_, mut calls) = + pending_tool_calls_from_payload(&line.payload, "").expect("normalized Codex call"); + assert_eq!(calls.len(), 1); + let full_args = serde_json::to_string(&calls[0].args).expect("full normalized args"); + assert!(full_args.len() > 10 * 1024 * 1024); + let (descriptor, encoded) = + compact_codex_tool_args(&mut calls[0], ReplaySourceSpan { start: 0, end: 1 }, 0) + .expect("large args descriptor"); + let assigned = AssignedToolCall { + call: calls.remove(0), + sequence: 0, + turn_index: 0, + args_payload: Some(descriptor.clone()), + }; + let compact_args_bytes = serde_json::to_vec(&assigned.call.args).unwrap().len(); + assert!( + compact_args_bytes < 80 * 1024, + "compact args unexpectedly use {compact_args_bytes} bytes" + ); + assert!(serde_json::to_vec(&assigned).unwrap().len() < 96 * 1024); + assert_eq!(descriptor.total_bytes, full_args.len() as u64); + assert_eq!(encoded, full_args); + assert_eq!( + payload_text(&descriptor, &line).as_deref(), + Some(full_args.as_str()) + ); +} + +#[test] +fn cross_record_output_is_streamed_into_one_artifact_without_concatenation() { + let path = std::env::temp_dir().join(format!( + "orgii-codex-cross-record-artifact-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let lines = ["first-你", "second-🙂"] + .into_iter() + .map(|output| { + serde_json::json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"response_item", + "payload":{ + "type":"function_call_output", + "call_id":"background-call", + "output":output + } + }) + .to_string() + }) + .collect::>(); + let mut source = String::new(); + let mut spans = Vec::new(); + for line in &lines { + let start = source.len() as u64; + source.push_str(line); + source.push('\n'); + spans.push(ReplaySourceSpan { + start, + end: source.len() as u64, + }); + } + fs::write(&path, source).expect("cross-record source"); + + let mut conn = rusqlite::Connection::open_in_memory().expect("artifact DB"); + SqliteRecordStore::init_tables(&conn).expect("base schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("artifact schema"); + let tx = conn.transaction().expect("artifact transaction"); + let expected = "first-你second-🙂"; + stream_codex_output_artifact( + &tx, + "session", + "generation", + "event", + &path, + &spans, + expected.len() as u64, + ) + .expect("stream cross-record artifact"); + let payload = tx + .query_row( + "SELECT artifact.payload + FROM imported_replay_payload_artifact_refs AS ref + JOIN imported_replay_payload_artifacts AS artifact + ON artifact.source=ref.source + AND artifact.source_session_id=ref.source_session_id + AND artifact.generation=ref.generation + AND artifact.content_hash=ref.content_hash + WHERE ref.event_id='event' AND ref.field_path='result.output'", + [], + |row| row.get::<_, Vec>(0), + ) + .expect("read streamed artifact"); + assert_eq!(payload, expected.as_bytes()); + let _ = fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/facade.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/facade.rs new file mode 100644 index 000000000..ccebd6507 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/facade.rs @@ -0,0 +1,1136 @@ +use super::*; +use crate::store::sqlite::SqliteRecordStore; + +fn codex_fixture() -> (rusqlite::Connection, std::path::PathBuf, String) { + let conn = rusqlite::Connection::open_in_memory().expect("in-memory replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + let path = std::env::temp_dir().join(format!( + "orgii-codex-replay-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let session_id = "codexapp-replay-fixture".to_string(); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES ('codex_app', 'replay-fixture', ?1, ?2)", + rusqlite::params![session_id, path.to_string_lossy()], + ) + .expect("cache fixture source"); + (conn, path, session_id) +} + +fn jsonl(payload: serde_json::Value) -> String { + serde_json::json!({ + "timestamp": "2026-07-22T00:00:00Z", + "type": "event_msg", + "payload": payload, + }) + .to_string() +} + +#[test] +fn limits_are_always_hard_bounded() { + let limits = ReplayLimits { + max_turns: usize::MAX, + max_events: usize::MAX, + max_ipc_bytes: usize::MAX, + } + .bounded(); + assert_eq!(limits.max_turns, HARD_MAX_TURNS); + assert_eq!(limits.max_events, HARD_MAX_EVENTS); + assert_eq!(limits.max_ipc_bytes, HARD_MAX_IPC_BYTES); +} + +#[test] +fn compact_only_fields_are_explicit_for_compatibility_serializers() { + for key in [ + "_replayTruncated", + "_preview", + crate::development_artifact::REPLAY_GIT_ARTIFACTS_FIELD, + ] { + assert!(is_compact_only_replay_field(key), "{key}"); + } + assert!(!is_compact_only_replay_field("output")); + assert!(!is_compact_only_replay_field("path")); +} + +#[test] +fn payload_encoding_is_explicit_on_new_wire_rows_and_inferred_only_for_legacy_rows() { + let legacy_root: ReplayPayloadDescriptor = serde_json::from_value(serde_json::json!({ + "fieldPath":"args", + "kind":"tool_arguments", + "spans":[], + "totalBytes":12 + })) + .expect("legacy root descriptor"); + let legacy_nested: ReplayPayloadDescriptor = serde_json::from_value(serde_json::json!({ + "fieldPath":"result.content.0.text", + "kind":"assistant_content", + "spans":[], + "totalBytes":12 + })) + .expect("legacy nested descriptor"); + assert_eq!( + legacy_root.resolved_encoding(), + ReplayPayloadEncoding::JsonValue + ); + assert_eq!( + legacy_nested.resolved_encoding(), + ReplayPayloadEncoding::Utf8Text + ); + + let descriptor = ReplayPayloadDescriptor { + field_path: "args".to_string(), + kind: ReplayPayloadKind::ToolArguments, + encoding: ReplayPayloadEncoding::JsonValue, + body_projection: Some(ReplayPayloadBodyProjection { + field_path: "args.command".to_string(), + text: "cargo test".to_string(), + truncated: false, + }), + spans: Vec::new(), + total_bytes: 12, + source_ordinal: None, + source_key: None, + }; + let wire = serde_json::to_value(descriptor).expect("descriptor wire JSON"); + assert_eq!(wire["encoding"], "json_value"); + assert_eq!(wire["bodyProjection"]["fieldPath"], "args.command"); + assert_eq!(wire["bodyProjection"]["text"], "cargo test"); +} + +#[test] +fn root_body_projection_uses_semantic_priority_and_utf8_byte_limit() { + let value = serde_json::json!({ + "description":"lower-priority", + "command":format!("BEGIN{}END", "你".repeat(20)) + }); + let projection = replay_payload_body_projection("args", &value, None, 20, false) + .expect("semantic body projection"); + assert_eq!(projection.field_path, "args.command"); + assert!(projection.text.starts_with("BEGIN")); + assert!(projection.text.len() <= 20); + assert!(projection.truncated); + assert!(std::str::from_utf8(projection.text.as_bytes()).is_ok()); +} + +#[test] +fn lifecycle_source_binding_preserves_existing_catalog_metadata() { + let conn = rusqlite::Connection::open_in_memory().expect("in-memory replay DB"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + let source_session_id = "rollout-2026-07-22T00-00-00-fixture"; + let session_id = format!("codexapp-{source_session_id}"); + let source_path = std::env::temp_dir().join(format!( + "orgii-replay-binding-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + std::fs::write(&source_path, b"{}\n").expect("write replay source"); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path,name,input_tokens,listable + ) VALUES('codex_app',?1,?2,'/stale/path','curated title',42,1)", + rusqlite::params![source_session_id, session_id], + ) + .expect("insert catalog metadata"); + + bind_source_path( + &conn, + ImportedHistorySourceId::CodexApp, + source_session_id, + &session_id, + &source_path, + ) + .expect("bind lifecycle source"); + + let (bound_path, title, input_tokens, listable): (String, String, i64, i64) = conn + .query_row( + "SELECT source_path,name,input_tokens,listable + FROM imported_history_session_cache + WHERE source='codex_app' AND source_session_id=?1", + [source_session_id], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), + ) + .expect("read rebound catalog row"); + assert_eq!( + std::path::PathBuf::from(bound_path), + source_path.canonicalize().expect("canonical source") + ); + assert_eq!(title, "curated title"); + assert_eq!(input_tokens, 42); + assert_eq!(listable, 1); + + let mismatch = bind_source_path( + &conn, + ImportedHistorySourceId::CodexApp, + "different-source-key", + &session_id, + &source_path, + ) + .expect_err("source identity mismatch"); + assert!(mismatch.contains("Replay source identity mismatch")); + let _ = std::fs::remove_file(source_path); +} + +#[test] +fn codex_tool_names_do_not_mutate_catalog_title_across_open_poll_and_reopen() { + use std::io::Write; + + let (mut conn, path, session_id) = codex_fixture(); + conn.execute( + "UPDATE imported_history_session_cache + SET source_record_key='replay-fixture', name='Session index title' + WHERE source='codex_app' AND source_session_id='replay-fixture'", + [], + ) + .expect("set authoritative fixture title"); + std::fs::write( + &path, + [ + serde_json::json!({ + "timestamp": "2026-07-22T00:00:00Z", + "type": "session_meta", + "payload": { + "id": "replay-fixture", + "title": "Transcript title" + } + }) + .to_string(), + jsonl(serde_json::json!({ + "type": "user_message", + "message": "First genuine request" + })), + jsonl(serde_json::json!({ + "type": "function_call", + "name": "exec", + "arguments": "{\"command\":\"true\"}", + "call_id": "call-exec" + })), + ] + .join("\n") + + "\n", + ) + .expect("write Codex title fixture"); + + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open Codex title fixture"); + let read_title = |conn: &rusqlite::Connection| -> String { + conn.query_row( + "SELECT name FROM imported_history_session_cache + WHERE source='codex_app' AND source_session_id='replay-fixture'", + [], + |row| row.get(0), + ) + .expect("read Codex catalog title") + }; + assert_eq!(read_title(&conn), "Session index title"); + + let mut file = std::fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append Codex title fixture"); + writeln!( + file, + "{}", + jsonl(serde_json::json!({ + "type": "custom_tool_call", + "name": "update_plan", + "input": "{}", + "call_id": "call-plan" + })) + ) + .expect("append tool call"); + file.flush().expect("flush tool call"); + drop(file); + + let polled = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("poll Codex tool delta"); + assert_eq!(read_title(&conn), "Session index title"); + + let reopened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("reopen Codex title fixture"); + assert_eq!(reopened.cursor.generation, polled.cursor.generation); + assert_eq!(read_title(&conn), "Session index title"); + let _ = std::fs::remove_file(path); +} + +#[test] +fn every_registered_adapter_is_incremental() { + for source in ImportedHistorySourceId::ALL { + ensure_supported(source).expect("all 15 imported sources have bounded replay"); + } +} + +#[test] +fn codex_append_is_incremental_and_partial_tail_is_retried() { + use std::io::Write; + + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + format!( + "{}\n", + jsonl(serde_json::json!({"type":"user_message","message":"hello"})) + ), + ) + .expect("initial JSONL"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open bounded replay"); + assert_eq!(opened.chunks.len(), 1); + let cursor = opened.cursor; + + let partial = jsonl(serde_json::json!({ + "type":"agent_message", + "message":"arrives only after newline" + })); + let mut file = std::fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append fixture"); + file.write_all(partial.as_bytes()).expect("partial record"); + file.flush().expect("flush partial"); + let partial_delta = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &cursor, + ReplayLimits::default(), + ) + .expect("poll partial tail"); + assert!(partial_delta.chunks.is_empty()); + assert_eq!(partial_delta.stats.parsed_bytes, 0); + assert_eq!(partial_delta.cursor.revision, cursor.revision); + + file.write_all(b"\n").expect("finish record"); + file.flush().expect("flush newline"); + drop(file); + let completed_delta = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &cursor, + ReplayLimits::default(), + ) + .expect("poll completed record"); + assert_eq!(completed_delta.chunks.len(), 1); + assert!(completed_delta.stats.parsed_bytes > 0); + assert_eq!( + completed_delta.chunks[0].chunk.function, + super::super::FUNCTION_ASSISTANT + ); + let mut unchanged_cursor = completed_delta.cursor; + for poll in 0..20 { + let unchanged = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &unchanged_cursor, + ReplayLimits::default(), + ) + .unwrap_or_else(|error| panic!("unchanged poll {poll} failed: {error}")); + assert!(unchanged.chunks.is_empty(), "unchanged poll {poll}"); + assert_eq!( + unchanged.stats, + ReplayStats::default(), + "unchanged poll {poll} must parse, normalize, upsert, and send nothing" + ); + unchanged_cursor = unchanged.cursor; + } + let _ = std::fs::remove_file(path); +} + +#[test] +fn unchanged_integrity_sample_refreshes_the_sixty_second_watermark() { + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + format!( + "{}\n", + jsonl(serde_json::json!({"type":"user_message","message":"hello"})) + ), + ) + .expect("initial JSONL"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open bounded replay"); + + conn.execute( + "UPDATE imported_replay_state SET updated_at='1970-01-01T00:00:00Z' + WHERE source='codex_app' AND source_session_id='replay-fixture'", + [], + ) + .expect("expire integrity watermark"); + index::take_file_sample_count(); + let before_touch = chrono::Utc::now().timestamp_millis(); + let sampled = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("integrity poll"); + assert_eq!(sampled.stats, ReplayStats::default()); + assert_eq!(index::take_file_sample_count(), 1); + let touched = index::load_state(&conn, ImportedHistorySourceId::CodexApp, "replay-fixture") + .expect("load touched replay state") + .expect("touched replay state"); + assert!(touched.state_updated_at_ms >= before_touch); + + let fast = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &sampled.cursor, + ReplayLimits::default(), + ) + .expect("metadata-only poll"); + assert_eq!(fast.stats, ReplayStats::default()); + assert_eq!( + index::take_file_sample_count(), + 0, + "the refreshed watermark must prevent another full integrity sample" + ); + let _ = std::fs::remove_file(path); +} + +#[test] +fn codex_same_inode_growing_rewrite_resets_generation() { + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + format!( + "{}\n", + jsonl(serde_json::json!({"type":"user_message","message":"old"})) + ), + ) + .expect("initial JSONL"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open initial generation"); + + // `fs::write` truncates and rewrites the existing inode. Make the + // replacement longer than the previous complete-line cursor so size + // metadata alone would look exactly like an append. + std::fs::write( + &path, + format!( + "{}\n", + jsonl(serde_json::json!({ + "type":"user_message", + "message":"replacement-is-deliberately-longer-than-old" + })) + ), + ) + .expect("same-inode growing rewrite"); + let delta = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("poll rewritten source"); + assert!(delta.reset_required); + assert_ne!(delta.cursor.generation, opened.cursor.generation); + let replacement = serde_json::to_string( + &delta + .chunks + .iter() + .map(|chunk| &chunk.chunk) + .collect::>(), + ) + .expect("replacement chunks"); + assert!(replacement.contains("replacement-is-deliberately-longer-than-old")); + assert!(!replacement.contains("\"old\"")); + let _ = std::fs::remove_file(path); +} + +#[test] +fn codex_delta_honors_max_turns_across_many_new_turns() { + use std::io::Write; + + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + format!( + "{}\n", + jsonl(serde_json::json!({"type":"user_message","message":"seed"})) + ), + ) + .expect("seed JSONL"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open seed"); + let mut file = std::fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append turns"); + for turn in 0..11 { + writeln!( + file, + "{}", + jsonl(serde_json::json!({ + "type":"user_message", + "message":format!("turn-{turn}") + })) + ) + .expect("append turn"); + } + drop(file); + let limits = ReplayLimits { + max_turns: 10, + max_events: 200, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }; + let first = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor, + limits, + ) + .expect("first bounded delta"); + assert_eq!( + first + .chunks + .iter() + .map(|chunk| chunk.turn_index) + .collect::>() + .len(), + 10 + ); + let second = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &first.cursor, + limits, + ) + .expect("remaining turn delta"); + assert_eq!( + second + .chunks + .iter() + .map(|chunk| chunk.turn_index) + .collect::>() + .len(), + 1 + ); + let _ = std::fs::remove_file(path); +} + +#[test] +fn codex_large_turn_reads_newest_slice_then_continues_without_gaps() { + use std::io::Write; + + let (mut conn, path, session_id) = codex_fixture(); + let mut file = std::fs::File::create(&path).expect("create large-turn fixture"); + writeln!( + file, + "{}", + jsonl(serde_json::json!({"type":"user_message","message":"large turn"})) + ) + .expect("write user event"); + for event_index in 0..450 { + writeln!( + file, + "{}", + jsonl(serde_json::json!({ + "type":"agent_message", + "message":format!("assistant-{event_index}") + })) + ) + .expect("write assistant event"); + } + drop(file); + + let limits = ReplayLimits { + max_turns: 1, + max_events: 200, + max_ipc_bytes: HARD_MAX_IPC_BYTES, + }; + let newest = read_turn_window_at_index( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + 0, + limits, + ) + .expect("read newest turn slice"); + assert_eq!(newest.chunks.len(), 200); + assert_eq!(newest.chunks.first().map(|chunk| chunk.sequence), Some(0)); + assert_eq!(newest.window_start_sequence, Some(252)); + assert_eq!( + newest.chunks.get(1).map(|chunk| chunk.sequence), + Some(252), + "selected large turns retain the user anchor plus the newest bounded tail" + ); + assert_eq!(newest.chunks.last().map(|chunk| chunk.sequence), Some(450)); + assert!(newest.has_older); + + let middle = read_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + newest.window_start_sequence, + limits, + ) + .expect("read middle turn slice"); + let oldest = read_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + middle.chunks.first().map(|chunk| chunk.sequence), + limits, + ) + .expect("read oldest turn slice"); + + assert_eq!(middle.chunks.len(), 200); + assert_eq!(oldest.chunks.len(), 52); + for window in [&newest, &middle, &oldest] { + assert!(window.chunks.len() <= limits.max_events); + assert!(window.stats.ipc_bytes <= limits.max_ipc_bytes as u64); + } + let sequences = oldest + .chunks + .iter() + .chain(middle.chunks.iter()) + .chain(newest.chunks.iter().skip(1)) + .map(|chunk| chunk.sequence) + .collect::>(); + assert_eq!(sequences, (0..=450).collect::>()); + assert_eq!( + sequences + .iter() + .copied() + .collect::>() + .len(), + sequences.len() + ); + let _ = std::fs::remove_file(path); +} + +#[test] +fn codex_exact_small_turn_keeps_the_anchor_as_its_continuation_boundary() { + use std::io::Write; + + let (mut conn, path, session_id) = codex_fixture(); + let mut file = std::fs::File::create(&path).expect("create small-turn fixture"); + for payload in [ + serde_json::json!({"type":"user_message","message":"small turn"}), + serde_json::json!({"type":"agent_message","message":"assistant-1"}), + serde_json::json!({"type":"agent_message","message":"assistant-2"}), + ] { + writeln!(file, "{}", jsonl(payload)).expect("write small-turn event"); + } + drop(file); + + let window = read_turn_window_at_index( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + 0, + ReplayLimits::default(), + ) + .expect("read complete exact turn"); + assert_eq!( + window + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(), + vec![0, 1, 2] + ); + assert_eq!(window.window_start_sequence, Some(0)); + assert!(!window.has_older); + let _ = std::fs::remove_file(path); +} + +#[test] +fn codex_large_body_is_source_backed_and_range_bounded() { + let (mut conn, path, session_id) = codex_fixture(); + let large = format!("BEGIN-{}-END", "你".repeat(10_000)); + std::fs::write( + &path, + format!( + "{}\n{}\n", + jsonl(serde_json::json!({"type":"user_message","message":"hello"})), + jsonl(serde_json::json!({"type":"agent_message","message":large.clone()})), + ), + ) + .expect("large JSONL"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open bounded replay"); + let assistant = opened + .chunks + .iter() + .find(|chunk| chunk.chunk.function == super::super::FUNCTION_ASSISTANT) + .expect("assistant event"); + assert!(serde_json::to_vec(&assistant.chunk).unwrap().len() < 16 * 1024); + assert_eq!(assistant.payloads.len(), 1); + + let mut reconstructed = String::new(); + let mut offset = 0_u64; + loop { + let range = read_payload_range( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor.generation, + &assistant.chunk.chunk_id, + "result.content", + offset, + Some(1024), + ) + .expect("read source-backed payload range"); + assert!(range.text.len() <= 1024); + reconstructed.push_str(&range.text); + offset = range.next_offset; + if range.eof { + break; + } + } + assert_eq!(reconstructed, large); + let _ = std::fs::remove_file(path); +} + +#[test] +fn cached_codex_secondary_reads_do_not_mutate_source_sync() { + use std::io::Write; + + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + [ + jsonl(serde_json::json!({"type":"user_message","message":"first"})), + jsonl(serde_json::json!({"type":"agent_message","message":"first answer"})), + jsonl(serde_json::json!({"type":"user_message","message":"second"})), + jsonl(serde_json::json!({"type":"agent_message","message":"second answer"})), + ] + .join("\n") + + "\n", + ) + .expect("initial cached JSONL"); + + assert!(open_cached_window( + &conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("check cold replay cache") + .is_none()); + + let indexed = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("index cached replay fixture"); + let indexed_cursor = indexed.cursor.clone(); + let changes_before_cached_reads = conn.total_changes(); + + let mut file = std::fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append cached replay fixture"); + writeln!( + file, + "{}", + jsonl(serde_json::json!({"type":"user_message","message":"third"})) + ) + .expect("append third user turn"); + writeln!( + file, + "{}", + jsonl(serde_json::json!({"type":"agent_message","message":"third answer"})) + ) + .expect("append third assistant turn"); + file.flush().expect("flush appended replay fixture"); + drop(file); + + let cached = open_cached_window( + &conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("read last published replay generation") + .expect("published replay cache"); + assert_eq!(cached.cursor, indexed_cursor); + assert_eq!(cached.stats.parsed_bytes, 0); + assert_eq!(cached.stats.parsed_rows, 0); + + let first_turn = read_cached_turn_window_at_index( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + 0, + ReplayLimits::default(), + ) + .expect("read cached earlier turn") + .expect("cached earlier turn"); + assert_eq!(first_turn.turn_headers[0].turn_index, 0); + let cached_metadata = project_cached_turn_metadata( + &conn, + ImportedHistorySourceId::CodexApp, + &session_id, + Some(&[first_turn.turn_headers[0].turn_id.clone()]), + ) + .expect("project cached earlier-turn metadata") + .expect("cached metadata projection"); + assert_eq!(cached_metadata.len(), 1); + assert_eq!( + cached_metadata[0].turn_id, + first_turn.turn_headers[0].turn_id + ); + assert_eq!(conn.total_changes(), changes_before_cached_reads); + + let delta = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &indexed_cursor, + ReplayLimits::default(), + ) + .expect("publish appended replay delta"); + assert!(delta.stats.parsed_rows > 0); + assert!(delta.cursor.revision > indexed_cursor.revision); + let _ = std::fs::remove_file(path); +} + +#[test] +fn authoritative_codex_open_syncs_an_append_before_returning() { + use std::io::Write; + + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + [ + jsonl(serde_json::json!({"type":"user_message","message":"first"})), + jsonl(serde_json::json!({"type":"agent_message","message":"first answer"})), + ] + .join("\n") + + "\n", + ) + .expect("initial authoritative-open JSONL"); + let first = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("initial authoritative open"); + assert_eq!(first.total_turn_count, 1); + + let mut file = std::fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append authoritative-open fixture"); + writeln!( + file, + "{}", + jsonl(serde_json::json!({"type":"user_message","message":"second"})) + ) + .expect("append second user turn"); + writeln!( + file, + "{}", + jsonl(serde_json::json!({"type":"agent_message","message":"second answer"})) + ) + .expect("append second assistant turn"); + file.flush().expect("flush authoritative-open append"); + drop(file); + + let reopened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("authoritative reopen after append"); + + assert_eq!(reopened.total_turn_count, 2); + assert!(reopened.cursor.revision > first.cursor.revision); + assert!(reopened.stats.parsed_rows > 0); + let _ = std::fs::remove_file(path); +} + +#[test] +fn compact_ipc_budget_counts_payload_descriptors_and_fails_closed() { + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + format!( + "{}\n", + jsonl(serde_json::json!({ + "type":"agent_message", + "message":"x".repeat(20_000) + })) + ), + ) + .expect("descriptor fixture"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open descriptor fixture"); + assert_eq!(opened.chunks.len(), 1); + let chunk_only = serde_json::to_vec(&opened.chunks[0].chunk) + .expect("serialize chunk") + .len(); + let descriptors = serde_json::to_vec(&opened.chunks[0].payloads) + .expect("serialize descriptors") + .len(); + assert!(descriptors > 0); + + let error = read_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + None, + ReplayLimits { + max_turns: 1, + max_events: 200, + max_ipc_bytes: chunk_only, + }, + ) + .expect_err("descriptor bytes must not bypass the compact limit"); + assert!(error.contains("compact window budget")); + let _ = std::fs::remove_file(path); +} + +#[test] +fn codex_ten_mib_shell_payload_is_materialized_once_and_generation_scoped() { + fn hash_text(text: &str) -> u64 { + text.as_bytes() + .iter() + .fold(0xcbf29ce484222325_u64, |hash, byte| { + (hash ^ u64::from(*byte)).wrapping_mul(0x100000001b3) + }) + } + + fn transcript(output: &str) -> String { + format!( + "{}\n{}\n{}\n", + jsonl(serde_json::json!({"type":"user_message","message":"run it"})), + jsonl(serde_json::json!({ + "type":"function_call", + "name":"shell_command", + "arguments":"{\"command\":\"printf payload\"}", + "call_id":"call-large-output" + })), + jsonl(serde_json::json!({ + "type":"function_call_output", + "call_id":"call-large-output", + "output":output + })), + ) + } + + let (mut conn, path, session_id) = codex_fixture(); + let old_output = format!("OLD:{}:END", "A".repeat(10 * 1024 * 1024)); + std::fs::write(&path, transcript(&old_output)).expect("10 MiB Codex transcript"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("index 10 MiB Codex transcript"); + assert_eq!(opened.stats.parsed_rows, 3); + let shell = opened + .chunks + .iter() + .find(|chunk| chunk.chunk.function == super::super::FUNCTION_RUN_COMMAND_LINE) + .expect("Codex Shell event"); + let artifact_count = conn + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifacts + WHERE source='codex_app' AND generation=?1", + [&opened.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("Codex artifact count"); + assert_eq!(artifact_count, 1); + + codex_jsonl::reset_payload_fallback_decodes(); + let mut reconstructed = String::with_capacity(old_output.len()); + let mut offset = 0_u64; + loop { + let range = read_payload_range( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor.generation, + &shell.chunk.chunk_id, + "result.output", + offset, + Some(HARD_MAX_PAYLOAD_RANGE_BYTES), + ) + .expect("read Codex Shell artifact page"); + assert!(range.text.len() <= HARD_MAX_PAYLOAD_RANGE_BYTES); + assert!(range.next_offset > offset || range.eof); + reconstructed.push_str(&range.text); + offset = range.next_offset; + if range.eof { + break; + } + } + assert_eq!(hash_text(&reconstructed), hash_text(&old_output)); + assert_eq!(reconstructed.len(), old_output.len()); + assert_eq!(codex_jsonl::payload_fallback_decodes(), 0); + + let new_output = format!("NEW:{}:END", "B".repeat(10 * 1024 * 1024)); + assert_eq!(new_output.len(), old_output.len()); + std::fs::write(&path, transcript(&new_output)).expect("same-size replacement"); + let replaced = poll_delta( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("replace Codex generation"); + assert!(replaced.reset_required); + assert_ne!(replaced.cursor.generation, opened.cursor.generation); + let replacement_shell = replaced + .chunks + .iter() + .find(|chunk| chunk.chunk.function == super::super::FUNCTION_RUN_COMMAND_LINE) + .expect("replacement Shell event"); + let replacement = read_payload_range( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &replaced.cursor.generation, + &replacement_shell.chunk.chunk_id, + "result.output", + 0, + Some(64), + ) + .expect("replacement artifact page"); + assert!(replacement.text.starts_with("NEW:BBBB")); + assert_eq!(codex_jsonl::payload_fallback_decodes(), 0); + let generations = conn + .query_row( + "SELECT COUNT(DISTINCT generation) FROM imported_replay_payload_artifacts + WHERE source='codex_app'", + [], + |row| row.get::<_, i64>(0), + ) + .expect("live artifact generations"); + assert_eq!(generations, 1, "replaced generation artifacts are retired"); + let _ = std::fs::remove_file(path); +} + +#[test] +fn pinned_generation_scan_rejects_a_mixed_snapshot() { + let (mut conn, path, session_id) = codex_fixture(); + std::fs::write( + &path, + format!( + "{}\n", + jsonl(serde_json::json!({"type":"user_message","message":"pinned"})) + ), + ) + .expect("pinned scan fixture"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + ReplayLimits::default(), + ) + .expect("open pinned generation"); + let error = scan_window_after_generation( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + "another-generation", + opened.cursor.revision, + -1, + ReplayLimits::default(), + ) + .expect_err("derived snapshots must not mix generations"); + assert!(error.contains("expected another-generation")); + let error = scan_window_after_generation( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor.generation, + opened.cursor.revision.saturating_add(1), + -1, + ReplayLimits::default(), + ) + .expect_err("derived snapshots must not mix revisions"); + assert!(error.contains(&format!("@{}", opened.cursor.revision.saturating_add(1)))); + let pinned = scan_window_after_generation( + &mut conn, + ImportedHistorySourceId::CodexApp, + &session_id, + &opened.cursor.generation, + opened.cursor.revision, + -1, + ReplayLimits::default(), + ) + .expect("scan pinned generation"); + assert_eq!(pinned.cursor.generation, opened.cursor.generation); + let _ = std::fs::remove_file(path); +} + +#[test] +fn pinned_scan_preparation_restarts_only_twice() { + let attempts = std::cell::Cell::new(0usize); + let error = retry_pinned_scan_preparation("codexapp-changing", || { + attempts.set(attempts.get().saturating_add(1)); + Ok(None) + }) + .expect_err("continuously changing source must stop"); + assert_eq!(attempts.get(), MAX_PINNED_SCAN_RESTARTS + 1); + assert!(error.contains("kept changing")); + assert!(error.contains("after 2 restarts")); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/index.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/index.rs new file mode 100644 index 000000000..8ee149699 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/index.rs @@ -0,0 +1,264 @@ +use super::*; +use super::{source_identity::*, sync::*}; + +#[test] +fn every_registered_source_has_one_matching_exhaustive_storage_driver() { + use super::super::ReplayStorageFamily; + + for source in ImportedHistorySourceId::ALL { + let family = source.descriptor().storage_family; + let compatible = match replay_driver_kind(source) { + ReplayDriverKind::CodexJsonl | ReplayDriverKind::SharedJsonl => { + family == ReplayStorageFamily::JsonLines + } + ReplayDriverKind::Sqlite => matches!( + family, + ReplayStorageFamily::SqliteWal | ReplayStorageFamily::SqliteKeyValue + ), + ReplayDriverKind::StructuredSqlite => matches!( + family, + ReplayStorageFamily::SqliteManifestBlob | ReplayStorageFamily::SqliteTaskBlob + ), + ReplayDriverKind::WholeJson => family == ReplayStorageFamily::WholeJson, + }; + assert!( + compatible, + "{} declares {family:?} but routes through {:?}", + source.as_str(), + replay_driver_kind(source) + ); + } +} + +fn replay_schema() -> Connection { + use crate::store::sqlite::SqliteRecordStore; + + let conn = Connection::open_in_memory().expect("replay schema"); + SqliteRecordStore::init_tables(&conn).expect("initialize replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn) + .expect("initialize replay source-cache schema"); + conn +} + +#[test] +fn source_resolution_treats_percent_and_underscore_as_literal_suffix_bytes() { + let conn = replay_schema(); + let source = ImportedHistorySourceId::CodexApp; + let requested_key = "thread_%"; + let display_session_id = format!("{}{}", source.descriptor().session_prefix, requested_key); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path,updated_at_ms + ) VALUES(?1,'rollout-thread-xy','wrong','/tmp/wrong.jsonl',2)", + [source.as_str()], + ) + .expect("seed wildcard-shaped near match"); + + let error = resolve_source(&conn, source, &display_session_id) + .expect_err("LIKE wildcard bytes must not match a different source session"); + assert!(error.contains("not indexed yet")); + + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path,updated_at_ms + ) VALUES(?1,?2,'exact','/tmp/exact.jsonl',1)", + params![source.as_str(), format!("rollout-{requested_key}")], + ) + .expect("seed literal suffix match"); + let resolved = + resolve_source(&conn, source, &display_session_id).expect("resolve literal suffix"); + assert_eq!( + resolved.source_session_id, + format!("rollout-{requested_key}") + ); + assert_eq!(resolved.path, PathBuf::from("/tmp/exact.jsonl")); +} + +#[test] +fn replay_write_transaction_reserves_writer_before_streaming_reads() { + let path = std::env::temp_dir().join(format!( + "orgii-replay-immediate-{}-{}.sqlite", + std::process::id(), + Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let mut replay_conn = Connection::open(&path).expect("open replay writer"); + replay_conn + .execute_batch("PRAGMA journal_mode=WAL; CREATE TABLE probe(value INTEGER);") + .expect("initialize writer probe"); + replay_conn + .busy_timeout(std::time::Duration::from_secs(1)) + .expect("configure replay busy timeout"); + let tx = begin_replay_write_transaction(&mut replay_conn, "test replay") + .expect("reserve replay writer"); + tx.query_row("SELECT COUNT(*) FROM probe", [], |row| row.get::<_, i64>(0)) + .expect("streaming read before first replay write"); + + let peer = Connection::open(&path).expect("open peer writer"); + peer.busy_timeout(std::time::Duration::from_millis(25)) + .expect("configure peer timeout"); + let error = peer + .execute("INSERT INTO probe(value) VALUES (1)", []) + .expect_err("IMMEDIATE replay transaction must already own the writer slot"); + assert!( + matches!( + error, + rusqlite::Error::SqliteFailure(ref details, _) + if matches!( + details.code, + rusqlite::ErrorCode::DatabaseBusy + | rusqlite::ErrorCode::DatabaseLocked + ) + ), + "unexpected peer error: {error}" + ); + + tx.execute("INSERT INTO probe(value) VALUES (2)", []) + .expect("first replay write cannot hit a stale snapshot"); + tx.commit().expect("commit replay write"); + peer.busy_timeout(std::time::Duration::from_secs(1)) + .expect("extend peer timeout"); + peer.execute("INSERT INTO probe(value) VALUES (3)", []) + .expect("writer slot released after replay commit"); + drop(peer); + drop(replay_conn); + let _ = std::fs::remove_file(path.with_extension("sqlite-shm")); + let _ = std::fs::remove_file(path.with_extension("sqlite-wal")); + let _ = std::fs::remove_file(path); +} + +#[test] +fn cursor_cli_lineage_rejection_watermark_matches_only_the_same_snapshot() { + let mut conn = replay_schema(); + let source = ImportedHistorySourceId::CursorCli; + let parser_version = source.descriptor().parser_version; + let snapshot = SourcePhysicalSnapshot { + identity: "/tmp/cursor-state.vscdb:1:2".to_string(), + size_bytes: 42_000, + mtime_ns: 1_700_000_000_000_000_000, + sample_fingerprint: "db-and-wal-sample-a".to_string(), + }; + conn.execute( + "INSERT INTO imported_replay_state( + source,source_session_id,generation,revision,parser_version, + source_identity,driver_cursor_json,indexed_size_bytes, + indexed_mtime_ns,total_events,total_turns,valid,updated_at + ) VALUES('cursor_cli','cursor-1','valid-generation',7,?1, + 'old-identity','{}',10,20,3,1,1,?2)", + params![i64::from(parser_version), Utc::now().to_rfc3339()], + ) + .expect("seed last valid Cursor CLI generation"); + + record_rejected_snapshot( + &mut conn, + source, + "cursor-1", + parser_version, + &snapshot, + RejectedSnapshotKind::CursorCliLineageChanged, + ) + .expect("record Cursor CLI lineage rejection"); + + for _ in 0..20 { + assert!(rejected_snapshot_matches( + &conn, + source, + "cursor-1", + parser_version, + &snapshot, + RejectedSnapshotKind::CursorCliLineageChanged, + ) + .expect("match unchanged Cursor CLI rejected snapshot")); + } + let valid = load_state(&conn, source, "cursor-1") + .expect("load last valid Cursor CLI state") + .expect("last valid Cursor CLI state remains visible"); + assert_eq!(valid.generation, "valid-generation"); + assert_eq!(valid.revision, 7); + + let mut changed = snapshot.clone(); + changed.sample_fingerprint = "db-and-wal-sample-b".to_string(); + assert!(!rejected_snapshot_matches( + &conn, + source, + "cursor-1", + parser_version, + &changed, + RejectedSnapshotKind::CursorCliLineageChanged, + ) + .expect("changed Cursor CLI snapshot is retryable")); + + let tx = conn + .transaction() + .expect("start successful Cursor CLI publish transaction"); + clear_rejected_snapshot(&tx, source, "cursor-1") + .expect("clear successful Cursor CLI rejection"); + tx.commit() + .expect("commit successful Cursor CLI rejection clear"); + assert!(!rejected_snapshot_matches( + &conn, + source, + "cursor-1", + parser_version, + &snapshot, + RejectedSnapshotKind::CursorCliLineageChanged, + ) + .expect("cleared Cursor CLI snapshot is retryable")); +} + +#[test] +fn fingerprint_reads_a_bounded_sample() { + let dir = std::env::temp_dir(); + let path = dir.join(format!( + "orgii-replay-fingerprint-{}.jsonl", + std::process::id() + )); + std::fs::write(&path, vec![b'x'; 32 * 1024]).expect("fixture"); + let first = sampled_file_fingerprint(&path, 32 * 1024).expect("fingerprint"); + let mut bytes = std::fs::read(&path).expect("read fixture"); + bytes[16 * 1024] = b'y'; + std::fs::write(&path, bytes).expect("rewrite fixture"); + let second = sampled_file_fingerprint(&path, 32 * 1024).expect("fingerprint"); + let _ = std::fs::remove_file(path); + assert_ne!(first, second); +} + +#[test] +fn sqlite_logical_snapshot_ignores_shm_but_tracks_wal() { + let path = std::env::temp_dir().join(format!( + "orgii-replay-sqlite-snapshot-{}-{}.db", + std::process::id(), + Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let wal = PathBuf::from(format!("{}-wal", path.to_string_lossy())); + let shm = PathBuf::from(format!("{}-shm", path.to_string_lossy())); + std::fs::write(&path, b"database").expect("main fixture"); + std::fs::write(&wal, b"committed-wal").expect("WAL fixture"); + std::fs::write(&shm, b"reader-lock-a").expect("SHM fixture"); + + let before = source_snapshot(&path, ImportedHistorySourceId::OpenCode) + .expect("snapshot before SHM churn"); + let before_fingerprint = + sqlite_physical_fingerprint(&path).expect("fingerprint before SHM churn"); + std::fs::write(&shm, b"reader-lock-b-with-different-size").expect("simulate SHM lock churn"); + let after_shm = source_snapshot(&path, ImportedHistorySourceId::OpenCode) + .expect("snapshot after SHM churn"); + let after_shm_fingerprint = + sqlite_physical_fingerprint(&path).expect("fingerprint after SHM churn"); + assert_eq!(before.identity, after_shm.identity); + assert_eq!(before.size_bytes, after_shm.size_bytes); + assert_eq!(before.mtime_ns, after_shm.mtime_ns); + assert_eq!(before_fingerprint, after_shm_fingerprint); + + std::fs::write(&wal, b"committed-wal-with-new-logical-row") + .expect("simulate committed WAL change"); + let after_wal = source_snapshot(&path, ImportedHistorySourceId::OpenCode) + .expect("snapshot after WAL change"); + let after_wal_fingerprint = + sqlite_physical_fingerprint(&path).expect("fingerprint after WAL change"); + assert_ne!(after_shm.size_bytes, after_wal.size_bytes); + assert_ne!(after_shm_fingerprint, after_wal_fingerprint); + + let _ = std::fs::remove_file(shm); + let _ = std::fs::remove_file(wal); + let _ = std::fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/jsonl_driver.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/jsonl_driver.rs new file mode 100644 index 000000000..52e9f5347 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/jsonl_driver.rs @@ -0,0 +1,990 @@ +use super::*; +use std::io::Write; + +use crate::store::sqlite::SqliteRecordStore; + +#[test] +fn qoder_wrapped_user_text_is_preserved_without_internal_context() { + let events = normalize_line( + ImportedHistorySourceId::Qoder, + &json!({ + "role":"user", "message":{"content":[{"type":"text","text":"xhello"}]} + }), + ); + assert!(matches!(&events[0].kind, NormalizedKind::UserText(text) if text == "hello")); +} + +#[test] +fn anthropic_tool_use_and_result_normalize_without_full_transcript() { + let call = normalize_line( + ImportedHistorySourceId::Omp, + &json!({ + "type":"assistant", "message":{"role":"assistant","content":[{"type":"tool_use","id":"c1","name":"bash","input":{"command":"pwd"}}]} + }), + ); + let result = normalize_line( + ImportedHistorySourceId::Omp, + &json!({ + "type":"user", "message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"c1","content":"/repo"}]} + }), + ); + assert!(matches!(&call[0].kind, NormalizedKind::ToolUse(tool) if tool.call_id == "c1")); + assert!( + matches!(&result[0].kind, NormalizedKind::ToolResult { output, .. } if output == "/repo") + ); +} + +#[test] +fn trae_line_stays_one_turn() { + let events = normalize_line( + ImportedHistorySourceId::Trae, + &json!({ + "intent":"fix it", "outcome":"done", "actions":["edit"], "learned":[] + }), + ); + assert_eq!(events.len(), 2); + assert!(events[0].starts_turn); + assert!(!events[1].starts_turn); +} + +fn fixture_lines(source: ImportedHistorySourceId) -> (String, String) { + match source { + ImportedHistorySourceId::ClaudeCode => ( + json!({"type":"user","timestamp":"2026-07-22T00:00:00Z","message":{"role":"user","content":"hello"}}).to_string(), + json!({"type":"assistant","timestamp":"2026-07-22T00:00:01Z","message":{"role":"assistant","content":[{"type":"text","text":"world"}]}}).to_string(), + ), + ImportedHistorySourceId::WorkBuddy => ( + json!({"type":"message","timestamp":"2026-07-22T00:00:00Z","role":"user","content":"hello"}).to_string(), + json!({"type":"message","timestamp":"2026-07-22T00:00:01Z","role":"assistant","content":"world"}).to_string(), + ), + ImportedHistorySourceId::Trae => ( + json!({"intent":"hello","outcome":"ok","actions":[],"learned":[],"message_summary_time":"2026-07-22 08:00:00"}).to_string(), + json!({"intent":"again","outcome":"done","actions":[],"learned":[],"message_summary_time":"2026-07-22 08:00:01"}).to_string(), + ), + ImportedHistorySourceId::Qoder => ( + json!({"role":"user","message":{"content":[{"type":"text","text":"hello"}]}}).to_string(), + json!({"role":"assistant","message":{"content":[{"type":"text","text":"world"}]}}).to_string(), + ), + ImportedHistorySourceId::Omp | ImportedHistorySourceId::QoderCli => ( + json!({"type":"user","timestamp":1_753_152_000_000_i64,"message":{"role":"user","content":[{"type":"text","text":"hello"}]}}).to_string(), + json!({"type":"assistant","timestamp":1_753_152_001_000_i64,"message":{"role":"assistant","content":[{"type":"text","text":"world"}]}}).to_string(), + ), + _ => unreachable!("JSONL conformance source"), + } +} + +fn state_from(outcome: &JsonlSyncOutcome, source: ImportedHistorySourceId) -> ReplayIndexState { + ReplayIndexState { + generation: "generation-1".to_string(), + revision: 1, + parser_version: 1, + source_identity: source.as_str().to_string(), + driver_cursor_json: outcome.driver_cursor_json.clone(), + indexed_size_bytes: outcome.indexed_size_bytes, + indexed_mtime_ns: 0, + total_events: outcome.total_events, + total_turns: outcome.total_turns, + state_updated_at_ms: 0, + } +} + +#[test] +fn oversized_jsonl_record_is_rejected_without_publishing_a_cursor_or_event() { + let source = ImportedHistorySourceId::ClaudeCode; + let path = std::env::temp_dir().join(format!( + "orgii-jsonl-oversize-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + let mut file = fs::File::create(&path).expect("oversize fixture"); + file.write_all(br#"{"type":"user","message":{"role":"user","content":""#) + .expect("write oversize prefix"); + let block = vec![b'x'; 64 * 1024]; + let blocks = MAX_JSONL_RECORD_BYTES / block.len() + 1; + for _ in 0..blocks { + file.write_all(&block).expect("write oversize body"); + } + file.write_all(b"\"}}\n").expect("write oversize suffix"); + file.flush().expect("flush oversize fixture"); + + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + let tx = conn.transaction().expect("oversize transaction"); + let error = sync( + &tx, + source, + "claudecodeapp-oversize", + "oversize", + &path, + "generation-oversize", + 1, + None, + "sample", + ) + .expect_err("oversize record must fail before parsing"); + assert!(error.contains("32 MiB safety limit"), "{error}"); + tx.rollback().expect("rollback oversize sync"); + let events = conn + .query_row("SELECT COUNT(*) FROM imported_replay_events", [], |row| { + row.get::<_, i64>(0) + }) + .expect("count rejected events"); + assert_eq!(events, 0); + let _ = fs::remove_file(path); +} + +#[test] +fn qoder_primary_sequence_overflow_fails_instead_of_saturating() { + let source = ImportedHistorySourceId::Qoder; + let (line, _) = fixture_lines(source); + let path = std::env::temp_dir().join(format!( + "orgii-qoder-sequence-overflow-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{line}\n")).expect("overflow fixture"); + let cursor = JsonlCursor { + next_sequence: i64::MAX - QODER_PRIMARY_SEQUENCE_STEP + 1, + ..JsonlCursor::default() + }; + let previous = ReplayIndexState { + generation: "generation-overflow".to_string(), + revision: 1, + parser_version: source.descriptor().parser_version, + source_identity: "fixture".to_string(), + driver_cursor_json: serde_json::to_string(&cursor).expect("overflow cursor"), + indexed_size_bytes: 0, + indexed_mtime_ns: 0, + total_events: 0, + total_turns: 0, + state_updated_at_ms: 0, + }; + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + let tx = conn.transaction().expect("overflow transaction"); + let error = sync( + &tx, + source, + "qoderapp-overflow", + "overflow", + &path, + "generation-overflow", + 2, + Some(&previous), + "sample", + ) + .expect_err("sequence overflow must fail closed"); + assert!(error.contains("sequence space exhausted"), "{error}"); + tx.rollback().expect("rollback overflow sync"); + let events = conn + .query_row("SELECT COUNT(*) FROM imported_replay_events", [], |row| { + row.get::<_, i64>(0) + }) + .expect("count rolled-back overflow events"); + assert_eq!(events, 0); + let _ = fs::remove_file(path); +} + +#[test] +fn fallback_payload_range_reports_adjusted_utf8_offsets_and_rejects_missing_spans() { + let source = ImportedHistorySourceId::ClaudeCode; + let assistant = json!({ + "type":"assistant", + "message":{"role":"assistant","content":[{"type":"text","text":"a你b"}]} + }) + .to_string(); + let user = json!({ + "type":"user", + "message":{"role":"user","content":"not assistant payload"} + }) + .to_string(); + let source_text = format!("{assistant}\n{user}\n"); + let first_end = assistant.len() as u64 + 1; + let path = std::env::temp_dir().join(format!( + "orgii-jsonl-utf8-fallback-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, source_text).expect("UTF-8 fallback fixture"); + + let descriptor = ReplayPayloadDescriptor { + field_path: "result.content".to_string(), + kind: ReplayPayloadKind::AssistantContent, + encoding: ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: vec![ReplaySourceSpan { + start: 0, + end: first_end, + }], + total_bytes: "a你b".len() as u64, + source_ordinal: Some(0), + source_key: None, + }; + let range = read_payload( + source, + &path, + &serde_json::to_string(&vec![descriptor.clone()]).expect("payload locator"), + "event", + "result.content", + 2, + 1, + ) + .expect("UTF-8 fallback range"); + assert_eq!(range.offset, 4); + assert_eq!(range.next_offset, 5); + assert_eq!(range.text, "b"); + assert!(range.eof); + + let missing = ReplayPayloadDescriptor { + spans: vec![ + descriptor.spans[0], + ReplaySourceSpan { + start: first_end, + end: (assistant.len() + user.len() + 2) as u64, + }, + ], + total_bytes: 10, + ..descriptor + }; + let error = read_payload( + source, + &path, + &serde_json::to_string(&vec![missing]).expect("missing-span locator"), + "event", + "result.content", + 5, + 2, + ) + .expect_err("a missing decoded span must not shift subsequent offsets"); + assert!(error.contains("no longer yields"), "{error}"); + let _ = fs::remove_file(path); +} + +#[test] +fn every_jsonl_adapter_obeys_incremental_and_partial_tail_contract() { + let sources = [ + ImportedHistorySourceId::ClaudeCode, + ImportedHistorySourceId::WorkBuddy, + ImportedHistorySourceId::Trae, + ImportedHistorySourceId::Qoder, + ImportedHistorySourceId::Omp, + ImportedHistorySourceId::QoderCli, + ]; + for source in sources { + let (first, second) = fixture_lines(source); + let path = std::env::temp_dir().join(format!( + "orgii-{}-jsonl-replay-{}-{}.jsonl", + source.as_str(), + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{first}\n")).expect("write cold fixture"); + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache replay schema"); + + let cold = { + let tx = conn.transaction().expect("cold transaction"); + let outcome = sync( + &tx, + source, + &format!("{}fixture", source.descriptor().session_prefix), + "fixture", + &path, + "generation-1", + 1, + None, + "sample-1", + ) + .expect("cold sync"); + tx.commit().expect("commit cold sync"); + outcome + }; + assert!(cold.stats.parsed_rows >= 1, "{} cold", source.as_str()); + assert!(cold.total_events >= 1, "{} cold events", source.as_str()); + let cold_state = state_from(&cold, source); + + let unchanged = { + let tx = conn.transaction().expect("unchanged transaction"); + let outcome = sync( + &tx, + source, + &format!("{}fixture", source.descriptor().session_prefix), + "fixture", + &path, + "generation-1", + 2, + Some(&cold_state), + "sample-1", + ) + .expect("unchanged sync"); + tx.commit().expect("commit unchanged sync"); + outcome + }; + assert_eq!( + unchanged.stats.parsed_rows, + 0, + "{} unchanged", + source.as_str() + ); + assert_eq!( + unchanged.stats.parsed_bytes, + 0, + "{} unchanged bytes", + source.as_str() + ); + + let mut file = fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append fixture"); + file.write_all(second.as_bytes()).expect("write torn tail"); + file.flush().expect("flush torn tail"); + let partial = { + let tx = conn.transaction().expect("partial transaction"); + let outcome = sync( + &tx, + source, + &format!("{}fixture", source.descriptor().session_prefix), + "fixture", + &path, + "generation-1", + 2, + Some(&cold_state), + "sample-2", + ) + .expect("partial sync"); + tx.commit().expect("commit partial sync"); + outcome + }; + assert_eq!(partial.stats.parsed_rows, 0, "{} partial", source.as_str()); + assert_eq!(partial.indexed_size_bytes, cold.indexed_size_bytes); + + file.write_all(b"\n").expect("complete tail"); + file.flush().expect("flush complete tail"); + drop(file); + let completed = { + let tx = conn.transaction().expect("append transaction"); + let outcome = sync( + &tx, + source, + &format!("{}fixture", source.descriptor().session_prefix), + "fixture", + &path, + "generation-1", + 2, + Some(&cold_state), + "sample-2", + ) + .expect("append sync"); + tx.commit().expect("commit append sync"); + outcome + }; + assert_eq!(completed.stats.parsed_rows, 1, "{} append", source.as_str()); + assert!(completed.indexed_size_bytes > cold.indexed_size_bytes); + + fs::write( + &path, + format!("{}\n{}\n", second, "x".repeat(first.len() + 32)), + ) + .expect("replace fixture"); + assert!(!cursor_matches_source(&path, &completed.driver_cursor_json)); + let _ = fs::remove_file(path); + } +} + +#[test] +fn every_jsonl_adapter_conforms_through_public_open_poll_and_reset() { + let sources = [ + ImportedHistorySourceId::ClaudeCode, + ImportedHistorySourceId::WorkBuddy, + ImportedHistorySourceId::Trae, + ImportedHistorySourceId::Qoder, + ImportedHistorySourceId::Omp, + ImportedHistorySourceId::QoderCli, + ]; + for source in sources { + let (first, second) = fixture_lines(source); + let source_session_id = format!("public-{}", source.as_str()); + let session_id = format!( + "{}{}", + source.descriptor().session_prefix, + source_session_id + ); + let path = std::env::temp_dir().join(format!( + "orgii-public-{}-{}-{}.jsonl", + source.as_str(), + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{first}\n")).expect("write public fixture"); + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("base schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES (?1, ?2, ?3, ?4)", + params![ + source.as_str(), + source_session_id, + session_id, + path.to_string_lossy() + ], + ) + .expect("cache public source"); + + let opened = crate::sources::imported_history::replay::open_window( + &mut conn, + source, + &session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("public cold open"); + assert!(!opened.chunks.is_empty(), "{} cold window", source.as_str()); + assert!( + opened.stats.parsed_rows >= 1, + "{} cold telemetry", + source.as_str() + ); + let unchanged = crate::sources::imported_history::replay::poll_delta( + &mut conn, + source, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("public unchanged poll"); + assert_eq!( + unchanged.stats.parsed_rows, + 0, + "{} unchanged", + source.as_str() + ); + assert!(unchanged.chunks.is_empty()); + + let mut file = fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append public"); + file.write_all(second.as_bytes()) + .expect("write public partial"); + file.flush().expect("flush public partial"); + let partial = crate::sources::imported_history::replay::poll_delta( + &mut conn, + source, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("public partial poll"); + assert_eq!(partial.stats.parsed_rows, 0, "{} partial", source.as_str()); + assert_eq!(partial.cursor.revision, opened.cursor.revision); + + file.write_all(b"\n").expect("complete public tail"); + file.flush().expect("flush public complete"); + drop(file); + let completed = crate::sources::imported_history::replay::poll_delta( + &mut conn, + source, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("public append poll"); + assert!( + !completed.chunks.is_empty(), + "{} append delta", + source.as_str() + ); + assert_eq!(completed.stats.parsed_rows, 1); + + // Truncation publishes a new generation atomically and asks the + // caller to replace its bounded window. + fs::write(&path, format!("{first}\n")).expect("truncate public fixture"); + let reset = crate::sources::imported_history::replay::poll_delta( + &mut conn, + source, + &session_id, + &completed.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("public reset poll"); + assert!(reset.reset_required, "{} truncate reset", source.as_str()); + assert_ne!(reset.cursor.generation, completed.cursor.generation); + let _ = fs::remove_file(path); + } +} + +#[test] +fn shared_jsonl_open_window_and_range_are_end_to_end_bounded() { + let source = ImportedHistorySourceId::ClaudeCode; + let session_id = "claudecodeapp-range-fixture"; + let source_session_id = "range-fixture"; + let large = format!("BEGIN-{}-END", "你".repeat(10_000)); + let user = json!({ + "type":"user", "timestamp":"2026-07-22T00:00:00Z", + "message":{"role":"user","content":"hello"} + }); + let assistant = json!({ + "type":"assistant", "timestamp":"2026-07-22T00:00:01Z", + "message":{"role":"assistant","content":[{"type":"text","text":large}]} + }); + let path = std::env::temp_dir().join(format!( + "orgii-claude-range-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{user}\n{assistant}\n")).expect("range fixture"); + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("base schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES (?1, ?2, ?3, ?4)", + params![ + source.as_str(), + source_session_id, + session_id, + path.to_string_lossy() + ], + ) + .expect("cache source path"); + + let opened = crate::sources::imported_history::replay::open_window( + &mut conn, + source, + session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("open bounded Claude replay"); + let assistant = opened + .chunks + .iter() + .find(|chunk| chunk.chunk.function == imported_history::FUNCTION_ASSISTANT) + .expect("assistant preview"); + // The canonical assistant result intentionally mirrors the preview in + // `content` and `observation`; each field remains under the 8 KiB + // preview cap even though the serialized compatibility shape is ~16 KiB. + assert!(serde_json::to_vec(&assistant.chunk).unwrap().len() < 20 * 1024); + assert_eq!(assistant.payloads.len(), 1); + + let mut reconstructed = String::new(); + let mut offset = 0; + loop { + let range = crate::sources::imported_history::replay::read_payload_range( + &mut conn, + source, + session_id, + &opened.cursor.generation, + &assistant.chunk.chunk_id, + "result.content", + offset, + Some(1024), + ) + .expect("range read"); + assert!(range.text.len() <= 1024); + reconstructed.push_str(&range.text); + offset = range.next_offset; + if range.eof { + break; + } + } + assert_eq!(reconstructed, large); + let _ = fs::remove_file(path); +} + +#[test] +fn shared_jsonl_ten_mib_shell_ranges_never_reparse_the_source_row() { + fn update_hash(mut hash: u64, text: &str) -> u64 { + for byte in text.as_bytes() { + hash ^= u64::from(*byte); + hash = hash.wrapping_mul(0x100000001b3); + } + hash + } + + let source = ImportedHistorySourceId::ClaudeCode; + let session_id = "claudecodeapp-large-shell-artifact"; + let source_session_id = "large-shell-artifact"; + let output = format!("BEGIN:{}:END", "x".repeat(10 * 1024 * 1024)); + let user = json!({ + "type":"user", "timestamp":"2026-07-22T00:00:00Z", + "message":{"role":"user","content":"run a large command"} + }); + let call = json!({ + "type":"assistant", "timestamp":"2026-07-22T00:00:01Z", + "message":{"role":"assistant","content":[{ + "type":"tool_use", "id":"shell-large", "name":"Bash", + "input":{"command":"printf payload"} + }]} + }); + let result = json!({ + "type":"user", "timestamp":"2026-07-22T00:00:02Z", + "message":{"role":"user","content":[{ + "type":"tool_result", "tool_use_id":"shell-large", "content":output.clone() + }]} + }); + let path = std::env::temp_dir().join(format!( + "orgii-claude-large-shell-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{user}\n{call}\n{result}\n")).expect("large Claude JSONL"); + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("base schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES (?1, ?2, ?3, ?4)", + params![ + source.as_str(), + source_session_id, + session_id, + path.to_string_lossy() + ], + ) + .expect("cache source path"); + + let opened = crate::sources::imported_history::replay::open_window( + &mut conn, + source, + session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("index large Claude Shell output"); + assert_eq!(opened.stats.parsed_rows, 3); + let shell = opened + .chunks + .iter() + .find(|chunk| chunk.chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE) + .expect("Claude Shell event"); + let artifact_count = conn + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifacts + WHERE source=?1 AND generation=?2", + params![source.as_str(), &opened.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("Claude artifact count"); + assert_eq!(artifact_count, 1); + + reset_payload_fallback_decodes(); + let mut actual_hash = 0xcbf29ce484222325_u64; + let mut actual_bytes = 0_usize; + let mut offset = 0_u64; + loop { + let range = crate::sources::imported_history::replay::read_payload_range( + &mut conn, + source, + session_id, + &opened.cursor.generation, + &shell.chunk.chunk_id, + "result.output", + offset, + Some(crate::sources::imported_history::replay::HARD_MAX_PAYLOAD_RANGE_BYTES), + ) + .expect("read Claude artifact page"); + assert!( + range.text.len() + <= crate::sources::imported_history::replay::HARD_MAX_PAYLOAD_RANGE_BYTES + ); + assert!(range.next_offset > offset || range.eof); + actual_hash = update_hash(actual_hash, &range.text); + actual_bytes = actual_bytes.saturating_add(range.text.len()); + offset = range.next_offset; + if range.eof { + break; + } + } + assert_eq!(actual_bytes, output.len()); + assert_eq!(actual_hash, update_hash(0xcbf29ce484222325, &output)); + assert_eq!(payload_fallback_decodes(), 0); + let _ = fs::remove_file(path); +} + +#[test] +fn large_real_driver_rows_keep_edit_and_git_metadata_projection() { + let source = ImportedHistorySourceId::ClaudeCode; + let session_id = "claudecodeapp-metadata-fixture"; + let source_session_id = "metadata-fixture"; + let user = json!({ + "type":"user", "timestamp":"2026-07-22T00:00:00Z", + "message":{"role":"user","content":"edit and commit"} + }); + let edit_call = json!({ + "type":"assistant", "timestamp":"2026-07-22T00:00:01Z", + "message":{"role":"assistant","content":[{ + "type":"tool_use", "id":"edit-1", "name":"Edit", + "input":{ + "file_path":"src/large.rs", + "old_string":"old\n".repeat(3_000), + "new_string":"new\n".repeat(3_000) + } + }]} + }); + let edit_result = json!({ + "type":"user", "timestamp":"2026-07-22T00:00:02Z", + "message":{"role":"user","content":[{ + "type":"tool_result", "tool_use_id":"edit-1", "content":"updated" + }]}, + "toolUseResult":{ + "filePath":"src/large.rs", + "structuredPatch":[{ + "oldStart":1,"oldLines":1,"newStart":1,"newLines":2, + "lines":["-old","+new","+another"] + }] + } + }); + let git_command = format!("git commit -m metadata # {}", "x".repeat(40 * 1024)); + let git_call = json!({ + "type":"assistant", "timestamp":"2026-07-22T00:00:03Z", + "message":{"role":"assistant","content":[{ + "type":"tool_use", "id":"git-1", "name":"Bash", + "input":{"command":git_command} + }]} + }); + let git_output = format!( + "[feature abc1234] metadata\n{}\nhttps://github.com/acme/repo/pull/42\n{}", + "middle".repeat(8 * 1024), + "tail".repeat(12 * 1024) + ); + let git_result = json!({ + "type":"user", "timestamp":"2026-07-22T00:00:04Z", + "message":{"role":"user","content":[{ + "type":"tool_result", "tool_use_id":"git-1", "content":git_output + }]} + }); + let path = std::env::temp_dir().join(format!( + "orgii-claude-metadata-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write( + &path, + format!("{user}\n{edit_call}\n{edit_result}\n{git_call}\n{git_result}\n"), + ) + .expect("metadata fixture"); + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("base schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES (?1, ?2, ?3, ?4)", + params![ + source.as_str(), + source_session_id, + session_id, + path.to_string_lossy() + ], + ) + .expect("cache metadata source"); + + let opened = crate::sources::imported_history::replay::open_window( + &mut conn, + source, + session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("index metadata fixture"); + let turn_id = opened.turn_headers[0].turn_id.clone(); + let projected = crate::sources::imported_history::replay::project_turn_metadata( + &mut conn, + source, + session_id, + Some(std::slice::from_ref(&turn_id)), + ) + .expect("project compact driver rows"); + + assert_eq!(projected.len(), 1); + assert_eq!(projected[0].modified_files.len(), 1); + assert_eq!(projected[0].modified_files[0].path, "src/large.rs"); + assert_eq!(projected[0].modified_files[0].additions, 2); + assert_eq!(projected[0].modified_files[0].deletions, 1); + assert!(projected[0] + .git_artifacts + .iter() + .any(|artifact| artifact.sha.as_deref() == Some("abc1234"))); + assert!(projected[0] + .git_artifacts + .iter() + .any(|artifact| artifact.pr_number == Some(42))); + + let (edit_args, git_args, git_result): (String, String, String) = conn + .query_row( + "SELECT + MAX(CASE WHEN function_name LIKE 'edit%' THEN args_preview_json END), + MAX(CASE WHEN function_name='run_command_line' THEN args_preview_json END), + MAX(CASE WHEN function_name='run_command_line' THEN result_preview_json END) + FROM imported_replay_events", + [], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .expect("compact metadata rows"); + assert!(edit_args.len() < 20 * 1024); + assert!(edit_args.contains("src/large.rs")); + assert!(git_args.len() < 100 * 1024); + assert!(git_result.contains("_replayGitArtifacts")); + let _ = fs::remove_file(path); +} + +#[test] +fn cross_line_tool_result_updates_the_stable_call_without_cursor_payload() { + let source = ImportedHistorySourceId::Omp; + let call = json!({ + "type":"assistant", "timestamp":"2026-07-22T00:00:00Z", + "message":{"role":"assistant","content":[{ + "type":"tool_use","id":"call-1","name":"bash","input":{"command":"pwd"} + }]} + }); + let result = json!({ + "type":"user", "timestamp":"2026-07-22T00:00:01Z", + "message":{"role":"user","content":[{ + "type":"tool_result","tool_use_id":"call-1","content":"/repo" + }]} + }); + let path = std::env::temp_dir().join(format!( + "orgii-omp-tool-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{call}\n")).expect("tool call fixture"); + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("base schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + let cold = { + let tx = conn.transaction().expect("cold transaction"); + let outcome = sync( + &tx, + source, + "ompapp-tool-fixture", + "tool-fixture", + &path, + "generation-1", + 1, + None, + "sample-1", + ) + .expect("index tool call"); + tx.commit().expect("commit tool call"); + outcome + }; + let event_before: (String, String) = conn + .query_row( + "SELECT event_id, result_preview_json FROM imported_replay_events", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("pending tool event"); + assert!(!event_before.1.contains("/repo")); + + let mut file = fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("append result"); + writeln!(file, "{result}").expect("write tool result"); + drop(file); + let state = state_from(&cold, source); + let completed = { + let tx = conn.transaction().expect("result transaction"); + let outcome = sync( + &tx, + source, + "ompapp-tool-fixture", + "tool-fixture", + &path, + "generation-1", + 2, + Some(&state), + "sample-2", + ) + .expect("index tool result"); + tx.commit().expect("commit tool result"); + outcome + }; + let event_after: (String, String) = conn + .query_row( + "SELECT event_id, result_preview_json FROM imported_replay_events", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("completed tool event"); + assert_eq!(event_after.0, event_before.0); + assert!(event_after.1.contains("/repo")); + assert_eq!(completed.total_events, 1); + assert!(!completed.driver_cursor_json.contains("/repo")); + let _ = fs::remove_file(path); +} + +#[test] +fn ten_mib_single_line_keeps_only_preview_locator_and_compact_cursor() { + let source = ImportedHistorySourceId::ClaudeCode; + let session_id = "claudecodeapp-ten-mib-fixture"; + let source_session_id = "ten-mib-fixture"; + let body = "x".repeat(10 * 1024 * 1024); + let line = json!({ + "type":"assistant", "timestamp":"2026-07-22T00:00:00Z", + "message":{"role":"assistant","content":[{"type":"text","text":body}]} + }); + let path = std::env::temp_dir().join(format!( + "orgii-claude-ten-mib-{}-{}.jsonl", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )); + fs::write(&path, format!("{line}\n")).expect("10 MiB fixture"); + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("base schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES (?1, ?2, ?3, ?4)", + params![ + source.as_str(), + source_session_id, + session_id, + path.to_string_lossy() + ], + ) + .expect("cache 10 MiB source"); + + let opened = crate::sources::imported_history::replay::open_window( + &mut conn, + source, + session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("open 10 MiB bounded replay"); + assert_eq!(opened.chunks.len(), 1); + assert!(opened.stats.parsed_bytes >= 10 * 1024 * 1024); + assert!(serde_json::to_vec(&opened.chunks[0].chunk).unwrap().len() < 20 * 1024); + assert_eq!(opened.chunks[0].payloads[0].total_bytes, 10 * 1024 * 1024); + let cursor_json: String = conn + .query_row( + "SELECT driver_cursor_json FROM imported_replay_state + WHERE source=?1 AND source_session_id=?2", + params![source.as_str(), source_session_id], + |row| row.get(0), + ) + .expect("compact cursor"); + assert!(cursor_json.len() < 64 * 1024); + assert!(!cursor_json.contains(&"x".repeat(1024))); + + let unchanged = crate::sources::imported_history::replay::poll_delta( + &mut conn, + source, + session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("unchanged 10 MiB poll"); + assert_eq!(unchanged.stats.parsed_bytes, 0); + assert_eq!(unchanged.stats.parsed_rows, 0); + assert!(unchanged.chunks.is_empty()); + let _ = fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/qoder_sidecar.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/qoder_sidecar.rs new file mode 100644 index 000000000..93ff693b4 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/qoder_sidecar.rs @@ -0,0 +1,609 @@ +use super::*; +use std::io::Write; + +use crate::store::sqlite::SqliteRecordStore; + +fn unique_path(label: &str) -> PathBuf { + std::env::temp_dir().join(format!( + "orgii-qoder-sidecar-{label}-{}-{}", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )) +} + +fn fixture_log() -> String { + [ + r#"2026-07-16 19:42:04.351 [info] [ChatSessionService] ACP progress: task-aaa111.session.execution, rid=u, type=current_model_update"#, + r#"2026-07-16 19:42:09.000 [info] [ChatSessionService] ACP progress: task-aaa111.session.execution, rid=u, type=tool_call, toolCallId=call-a"#, + r#"2026-07-16 19:42:09.200 [info] [SubAgentService] Registered SubAgent: {"parentToolCallId":"call-a","parentSessionId":"task-aaa111.session.execution","agentType":"GeneralPurpose","rawInputDescription":"inspect","prompt":"inspect memory"}"#, + r#"2026-07-16 19:42:09.500 [info] ToolInvoke : run_in_terminal"#, + r#"{"command":"vm_stat","cwd":"/workspace/a"}"#, + r#"2026-07-16 19:42:09.600 [info] [ChatSessionService] ACP progress: task-bbb222.session.execution, rid=u, type=current_model_update"#, + // No path signal while both windows overlap: must stay unowned. + r#"2026-07-16 19:42:10.000 [info] [ToolInvokeHandlerContribution] Tool invoke request: rid-x, grep_search, {"query":"ambiguous"}"#, + r#"2026-07-16 19:42:11.000 [info] [ChatSessionService] ACP progress: task-aaa111.session.execution, rid=u, type=chat_finish"#, + r#"2026-07-16 19:42:12.000 [info] [ChatSessionService] ACP progress: task-bbb222.session.execution, rid=u, type=chat_finish"#, + ] + .join("\n") + + "\n" +} + +fn prepare_replay_db( + conn: &mut rusqlite::Connection, + source_session_id: &str, + display_session_id: &str, +) { + SqliteRecordStore::init_tables(conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(conn).expect("source cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,repo_path + ) VALUES(?1,?2,?3,?4)", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + display_session_id, + "/workspace/a" + ], + ) + .expect("cache row"); + let tx = conn.transaction().expect("primary tx"); + tx.execute( + "INSERT INTO imported_replay_turns( + source,source_session_id,generation,turn_index,turn_id, + start_sequence,end_sequence,started_at,event_count + ) VALUES(?1,?2,'g',0,'qoder-turn-0',0,?3,'2026-07-16T00:00:00Z',0)", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + QODER_PRIMARY_SEQUENCE_STEP - 1 + ], + ) + .expect("turn"); + let mut stats = ReplayStats::default(); + let user = imported_history::user_message_chunk( + display_session_id, + "qoder", + 0, + "2026-07-16T00:00:00Z", + "check memory", + ); + upsert_chunk( + &tx, + ImportedHistorySourceId::Qoder, + source_session_id, + "g", + 1, + 0, + 0, + &user, + &[], + ReplaySourceSpan { start: 0, end: 1 }, + &mut stats, + ) + .expect("user"); + let assistant = imported_history::assistant_message_chunk( + display_session_id, + "qoder", + QODER_PRIMARY_SEQUENCE_STEP as usize, + "2026-07-16T00:00:01Z", + "done", + ); + upsert_chunk( + &tx, + ImportedHistorySourceId::Qoder, + source_session_id, + "g", + 1, + 0, + QODER_PRIMARY_SEQUENCE_STEP, + &assistant, + &[], + ReplaySourceSpan { start: 2, end: 3 }, + &mut stats, + ) + .expect("assistant"); + tx.commit().expect("primary commit"); +} + +fn probed_file(path: &Path) -> ProbedFile { + let metadata = fs::metadata(path).expect("fixture metadata"); + let canonical = fs::canonicalize(path).expect("fixture canonical path"); + ProbedFile { + identity: file_identity(&canonical, &metadata), + size_bytes: metadata.len(), + mtime_ns: metadata_mtime_ns(&metadata), + path: canonical, + } +} + +#[test] +fn reserved_sequence_is_stable_and_inside_primary_gap() { + let mut conn = rusqlite::Connection::open_in_memory().expect("DB"); + conn.execute_batch( + "CREATE TABLE imported_replay_events( + source TEXT,source_session_id TEXT,generation TEXT,sequence INTEGER,event_id TEXT + );", + ) + .expect("schema"); + let tx = conn.transaction().expect("tx"); + let first = sidecar_sequence( + &tx, + "p/task", + "g", + QODER_PRIMARY_SEQUENCE_STEP, + 1_784_691_200_000, + "qoder-log-a", + ) + .expect("sequence"); + let second = sidecar_sequence( + &tx, + "p/task", + "g", + QODER_PRIMARY_SEQUENCE_STEP, + 1_784_691_200_001, + "qoder-log-b", + ) + .expect("sequence"); + assert!(first > QODER_PRIMARY_SEQUENCE_STEP); + assert!(first < QODER_PRIMARY_SEQUENCE_STEP * 2); + assert!(second > first); +} + +#[test] +fn reserved_sequence_refuses_base_overflow_without_reusing_i64_max() { + let mut conn = rusqlite::Connection::open_in_memory().expect("DB"); + prepare_replay_db(&mut conn, "p/task", "qoderapp-p/task"); + let tx = conn.transaction().expect("tx"); + let error = sidecar_sequence( + &tx, + "p/task", + "g", + i64::MAX - 10, + 1_784_691_200_000, + "qoder-log-overflow", + ) + .expect_err("overflowing sidecar sequence must fail closed"); + assert!(error.contains("sequence space exhausted"), "{error}"); +} + +#[test] +fn torn_tail_is_not_acknowledged() { + let mut reader = BufReader::new(&b"complete\npartial"[..]); + let (_, bytes) = read_complete_line(&mut reader) + .expect("line") + .expect("complete"); + assert_eq!(bytes, 9); + assert!(read_complete_line(&mut reader).expect("tail").is_none()); +} + +#[test] +fn compact_sidecar_matches_legacy_attribution_and_order() { + let path = unique_path("differential.log"); + fs::write(&path, fixture_log()).expect("fixture"); + let source_session_id = "project-a/task-aaa"; + let display_session_id = "qoderapp-project-a/task-aaa"; + let mut conn = rusqlite::Connection::open_in_memory().expect("DB"); + prepare_replay_db(&mut conn, source_session_id, display_session_id); + let tx = conn.transaction().expect("sidecar tx"); + ensure_raw_table(&tx).expect("raw table"); + let mut stats = ReplayStats::default(); + let file = probed_file(&path); + let (offset, inserted) = + ingest_file(&tx, source_session_id, "g", &file, 0, &mut stats).expect("ingest"); + assert!(inserted); + assert_eq!(offset, file.size_bytes); + assert!(fold_sidecar_events( + &tx, + display_session_id, + source_session_id, + "g", + 2, + &mut stats, + ) + .expect("fold")); + tx.commit().expect("commit"); + + let mut statement = conn + .prepare( + "SELECT function_name,args_preview_json,result_preview_json,sequence + FROM imported_replay_events + WHERE event_id LIKE 'qoder-log-%' ORDER BY sequence", + ) + .expect("query"); + let indexed = statement + .query_map([], |row| { + Ok(( + row.get::<_, String>(0)?, + serde_json::from_str::(&row.get::<_, String>(1)?).unwrap(), + serde_json::from_str::(&row.get::<_, String>(2)?).unwrap(), + row.get::<_, i64>(3)?, + )) + }) + .expect("rows") + .collect::, _>>() + .expect("indexed events"); + assert_eq!(indexed.len(), 2); + assert_eq!(indexed[0].0, "subagent"); + assert_eq!(indexed[0].1["agentType"], "GeneralPurpose"); + assert_eq!(indexed[0].2["call_id"], "call-a"); + assert_eq!(indexed[1].0, imported_history::FUNCTION_RUN_COMMAND_LINE); + assert_eq!(indexed[1].1["cmd"], "vm_stat"); + assert!(indexed + .iter() + .all(|event| { event.3 > 0 && event.3 < QODER_PRIMARY_SEQUENCE_STEP })); + + let base = vec![ + imported_history::user_message_chunk(display_session_id, "qoder", 0, "", "check memory"), + imported_history::assistant_message_chunk(display_session_id, "qoder", 1, "", "done"), + ]; + let legacy = log_enrichment::enrich_chunks_from_log_fixture( + display_session_id, + "task-aaa", + "project-a", + Some("/workspace/a"), + base, + &fixture_log(), + ); + let legacy_tools = legacy + .iter() + .filter(|chunk| chunk.action_type == imported_history::ACTION_TYPE_TOOL_CALL) + .collect::>(); + assert_eq!(legacy_tools.len(), indexed.len()); + for (legacy, current) in legacy_tools.iter().zip(indexed.iter()) { + assert_eq!(legacy.function, current.0); + assert_eq!(legacy.args, current.1); + assert_eq!(legacy.result["call_id"], current.2["call_id"]); + assert_eq!(legacy.result["raw_tool_name"], current.2["raw_tool_name"]); + assert_eq!(legacy.result["recovered_from"], current.2["recovered_from"]); + } + drop(statement); + let _ = fs::remove_file(path); +} + +#[test] +fn append_reads_only_new_complete_bytes_and_upserts_only_new_tool() { + let path = unique_path("append.log"); + fs::write(&path, fixture_log()).expect("fixture"); + let source_session_id = "project-a/task-aaa"; + let display_session_id = "qoderapp-project-a/task-aaa"; + let mut conn = rusqlite::Connection::open_in_memory().expect("DB"); + prepare_replay_db(&mut conn, source_session_id, display_session_id); + let first_offset; + { + let tx = conn.transaction().expect("initial tx"); + ensure_raw_table(&tx).expect("raw table"); + let mut stats = ReplayStats::default(); + let file = probed_file(&path); + first_offset = ingest_file(&tx, source_session_id, "g", &file, 0, &mut stats) + .expect("initial ingest") + .0; + fold_sidecar_events( + &tx, + display_session_id, + source_session_id, + "g", + 2, + &mut stats, + ) + .expect("initial fold"); + tx.commit().expect("initial commit"); + } + let append = [ + r#"2026-07-16 19:42:13.000 [info] [ChatSessionService] ACP progress: task-aaa111.session.execution, rid=u, type=tool_call, toolCallId=call-new"#, + r#"2026-07-16 19:42:13.100 [info] [ToolInvokeHandlerContribution] Tool invoke request: rid-new, read_file, {"file_path":"/workspace/a/src/lib.rs"}"#, + ] + .join("\n") + + "\n"; + fs::OpenOptions::new() + .append(true) + .open(&path) + .expect("open append") + .write_all(append.as_bytes()) + .expect("append"); + let tx = conn.transaction().expect("append tx"); + let mut stats = ReplayStats::default(); + let file = probed_file(&path); + let (next_offset, inserted) = + ingest_file(&tx, source_session_id, "g", &file, first_offset, &mut stats) + .expect("append ingest"); + assert!(inserted); + assert_eq!(stats.parsed_bytes, append.len() as u64); + assert_eq!(next_offset, first_offset + append.len() as u64); + fold_sidecar_events( + &tx, + display_session_id, + source_session_id, + "g", + 3, + &mut stats, + ) + .expect("append fold"); + assert_eq!(stats.upserted_events, 1); + tx.commit().expect("append commit"); + + let tx = conn.transaction().expect("unchanged tx"); + let mut unchanged = ReplayStats::default(); + let file = probed_file(&path); + let (same_offset, inserted) = ingest_file( + &tx, + source_session_id, + "g", + &file, + next_offset, + &mut unchanged, + ) + .expect("unchanged ingest"); + assert_eq!(same_offset, next_offset); + assert!(!inserted); + assert_eq!(unchanged.parsed_bytes, 0); + assert_eq!(unchanged.parsed_rows, 0); + tx.rollback().expect("rollback unchanged"); + let _ = fs::remove_file(path); +} + +#[test] +fn rotation_or_truncation_breaks_lineage_but_new_log_does_not() { + let original_path = unique_path("lineage-original.log"); + fs::write(&original_path, fixture_log()).expect("original"); + let original = probed_file(&original_path); + let cursor = QoderSidecarCursor { + version: SIDECAR_CURSOR_VERSION, + signature: "old".to_string(), + edit_signature: String::new(), + files: vec![SidecarFileCursor { + path: original.path.to_string_lossy().into_owned(), + identity: original.identity.clone(), + byte_offset: original.size_bytes, + boundary_fingerprint: boundary_fingerprint(&original.path, original.size_bytes) + .expect("boundary"), + }], + }; + let cursor_json = json!({ "qoder_sidecar": cursor }).to_string(); + let new_path = unique_path("lineage-new.log"); + fs::write(&new_path, "new launch\n").expect("new log"); + let with_new = SidecarProbe { + files: vec![original.clone(), probed_file(&new_path)], + signature: "new".to_string(), + edit_signature: String::new(), + }; + assert!(cursor_lineage_matches(&cursor_json, &with_new)); + + fs::write(&original_path, "short\n").expect("truncate"); + let truncated = SidecarProbe { + files: vec![probed_file(&original_path), probed_file(&new_path)], + signature: "truncated".to_string(), + edit_signature: String::new(), + }; + assert!(!cursor_lineage_matches(&cursor_json, &truncated)); + + let missing = SidecarProbe { + files: vec![probed_file(&new_path)], + signature: "missing".to_string(), + edit_signature: String::new(), + }; + assert!(!cursor_lineage_matches(&cursor_json, &missing)); + let _ = fs::remove_file(original_path); + let _ = fs::remove_file(new_path); +} + +#[test] +fn backend_watch_paths_include_transcript_and_project_spill_root_without_duplicates() { + let root = unique_path("watch-root"); + let transcript = root + .join("project-a") + .join("conversation-history") + .join("task-aaa") + .join("task-aaa.jsonl"); + let spill_root = root.join("project-a").join("agent-tools"); + fs::create_dir_all(transcript.parent().expect("transcript parent")).expect("history dirs"); + fs::create_dir_all(&spill_root).expect("spill root"); + fs::write(&transcript, "{}\n").expect("transcript"); + let conn = rusqlite::Connection::open_in_memory().expect("DB"); + SqliteRecordStore::init_tables(&conn).expect("schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,?2,?3,?4)", + params![ + ImportedHistorySourceId::Qoder.as_str(), + "project-a/task-aaa", + "qoderapp-project-a/task-aaa", + transcript.to_string_lossy() + ], + ) + .expect("cache row"); + let paths = crate::sources::imported_history::replay::watch_paths( + &conn, + ImportedHistorySourceId::Qoder, + "qoderapp-project-a/task-aaa", + ) + .expect("watch paths"); + assert!(paths.contains(&fs::canonicalize(&transcript).expect("canonical transcript"))); + assert!(paths.contains(&fs::canonicalize(&spill_root).expect("canonical spill"))); + let mut unique = paths.clone(); + unique.sort(); + unique.dedup(); + assert_eq!(paths.len(), unique.len()); + let _ = fs::remove_dir_all(root); +} + +#[test] +fn unchanged_transcript_sidecar_append_is_delta_and_rotation_is_reset() { + let transcript_path = unique_path("e2e.jsonl"); + let log_path = unique_path("e2e.log"); + fs::write( + &transcript_path, + concat!( + "{\"role\":\"user\",\"message\":{\"content\":[{\"type\":\"text\",\"text\":\"check memory\"}]}}\n", + "{\"role\":\"assistant\",\"message\":{\"content\":[{\"type\":\"text\",\"text\":\"done\"}]}}\n" + ), + ) + .expect("transcript"); + fs::write(&log_path, fixture_log()).expect("log"); + let source_session_id = "project-a/task-aaa"; + let display_session_id = "qoderapp-project-a/task-aaa"; + let mut conn = rusqlite::Connection::open_in_memory().expect("DB"); + SqliteRecordStore::init_tables(&conn).expect("schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("cache schema"); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path,repo_path + ) VALUES(?1,?2,?3,?4,?5)", + params![ + ImportedHistorySourceId::Qoder.as_str(), + source_session_id, + display_session_id, + transcript_path.to_string_lossy(), + "/workspace/a" + ], + ) + .expect("cache row"); + log_enrichment::with_qoder_log_paths_for_test(vec![log_path.clone()], || { + let opened = crate::sources::imported_history::replay::open_window( + &mut conn, + ImportedHistorySourceId::Qoder, + display_session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("initial open"); + let functions = opened + .chunks + .iter() + .map(|chunk| chunk.chunk.function.as_str()) + .collect::>(); + assert_eq!( + functions, + vec![ + imported_history::FUNCTION_USER_MESSAGE, + "subagent", + imported_history::FUNCTION_RUN_COMMAND_LINE, + imported_history::FUNCTION_ASSISTANT, + ] + ); + + let append = [ + r#"2026-07-16 19:42:13.000 [info] [ChatSessionService] ACP progress: task-aaa111.session.execution, rid=u, type=tool_call, toolCallId=call-new"#, + r#"2026-07-16 19:42:13.100 [info] [ToolInvokeHandlerContribution] Tool invoke request: rid-new, read_file, {"file_path":"/workspace/a/src/lib.rs"}"#, + ] + .join("\n") + + "\n"; + fs::OpenOptions::new() + .append(true) + .open(&log_path) + .expect("open append") + .write_all(append.as_bytes()) + .expect("append"); + let delta = crate::sources::imported_history::replay::poll_delta( + &mut conn, + ImportedHistorySourceId::Qoder, + display_session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("append delta"); + assert!(!delta.reset_required); + assert_eq!(delta.stats.parsed_bytes, append.len() as u64); + assert_eq!(delta.stats.upserted_events, 1); + assert_eq!(delta.chunks.len(), 1); + assert_eq!( + delta.chunks[0].chunk.function, + imported_history::FUNCTION_READ_FILE + ); + + let unchanged = crate::sources::imported_history::replay::poll_delta( + &mut conn, + ImportedHistorySourceId::Qoder, + display_session_id, + &delta.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("unchanged poll"); + assert!(!unchanged.reset_required); + assert!(unchanged.chunks.is_empty()); + assert_eq!(unchanged.stats.parsed_bytes, 0); + assert_eq!(unchanged.stats.parsed_rows, 0); + assert_eq!(unchanged.stats.normalized_events, 0); + assert_eq!(unchanged.stats.upserted_events, 0); + + let ambiguous = + "2026-07-16 19:42:14.000 [info] [ChatSessionService] ACP progress: task-aaa222.session.execution, rid=u, type=current_model_update\n"; + fs::OpenOptions::new() + .append(true) + .open(&log_path) + .expect("open ambiguous append") + .write_all(ambiguous.as_bytes()) + .expect("append ambiguous task"); + let conservative = crate::sources::imported_history::replay::poll_delta( + &mut conn, + ImportedHistorySourceId::Qoder, + display_session_id, + &unchanged.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("ambiguous task delta"); + assert!(!conservative.reset_required); + assert!(conservative.chunks.is_empty()); + assert_eq!(conservative.stats.removed_events, 3); + assert_eq!(conservative.removed_event_ids.len(), 3); + + fs::write( + &log_path, + concat!( + "2026-07-16 20:00:00.000 [info] [ChatSessionService] ACP progress: task-aaa111.session.execution, rid=u, type=current_model_update\n", + "2026-07-16 20:00:00.100 [info] [ToolInvokeHandlerContribution] Tool invoke request: rid-r, read_file, {\"file_path\":\"/workspace/a/rotated.rs\"}\n" + ), + ) + .expect("rotate"); + let reset = crate::sources::imported_history::replay::poll_delta( + &mut conn, + ImportedHistorySourceId::Qoder, + display_session_id, + &conservative.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("rotation reset"); + assert!(reset.reset_required); + assert_ne!(reset.cursor.generation, conservative.cursor.generation); + }); + let _ = fs::remove_file(transcript_path); + let _ = fs::remove_file(log_path); +} + +#[test] +#[ignore = "30 MiB deterministic sidecar streaming stress"] +fn large_log_streams_without_materializing_irrelevant_rows() { + let path = unique_path("large.log"); + let mut file = fs::File::create(&path).expect("large fixture"); + let irrelevant = "2026-07-16 19:42:00.000 [info] heartbeat heartbeat heartbeat\n"; + let block = irrelevant.repeat(1024); + let mut written = 0_usize; + while written < 30 * 1024 * 1024 { + file.write_all(block.as_bytes()).expect("write large log"); + written = written.saturating_add(block.len()); + } + file.write_all(fixture_log().as_bytes()) + .expect("write events"); + drop(file); + let mut conn = rusqlite::Connection::open_in_memory().expect("DB"); + prepare_replay_db( + &mut conn, + "project-a/task-aaa", + "qoderapp-project-a/task-aaa", + ); + let tx = conn.transaction().expect("tx"); + ensure_raw_table(&tx).expect("table"); + let mut stats = ReplayStats::default(); + let probed = probed_file(&path); + ingest_file(&tx, "project-a/task-aaa", "g", &probed, 0, &mut stats).expect("stream large"); + let rows: i64 = tx + .query_row(&format!("SELECT COUNT(*) FROM {RAW_TABLE}"), [], |row| { + row.get(0) + }) + .expect("raw count"); + assert_eq!(stats.parsed_bytes, probed.size_bytes); + assert_eq!(rows, 8); + tx.rollback().expect("rollback"); + let _ = fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/sqlite_driver.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/sqlite_driver.rs new file mode 100644 index 000000000..78dad757e --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/sqlite_driver.rs @@ -0,0 +1,1167 @@ +use super::*; +use crate::projectors::turn_metadata::TurnMetadataAccumulator; +use crate::sources::imported_history::replay::{ + materialize_payload_artifact, open_window, poll_delta, prepare_pinned_scan, read_payload_range, + scan_window_after, scan_window_after_generation, ReplayLimits, +}; +use crate::store::sqlite::SqliteRecordStore; + +fn hash_text(text: &str) -> u64 { + text.as_bytes() + .iter() + .fold(0xcbf29ce484222325_u64, |hash, byte| { + (hash ^ u64::from(*byte)).wrapping_mul(0x100000001b3) + }) +} + +fn read_full_payload( + cache: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, +) -> String { + let mut restored = String::new(); + let mut offset = 0_u64; + loop { + let range = read_payload_range( + cache, + source, + session_id, + generation, + event_id, + field_path, + offset, + Some(256 * 1024), + ) + .expect("SQLite replay payload range"); + assert_eq!(range.offset, offset); + assert!(range.next_offset > offset || range.eof); + restored.push_str(&range.text); + offset = range.next_offset; + if range.eof { + assert_eq!(offset, range.total_bytes); + break; + } + } + restored +} + +fn temp_db(label: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!( + "orgii-replay-{label}-{}-{}.sqlite", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )) +} + +fn cache_conn( + source: ImportedHistorySourceId, + source_session_id: &str, + path: &Path, +) -> (Connection, String) { + let conn = Connection::open_in_memory().expect("cache DB"); + SqliteRecordStore::init_tables(&conn).expect("replay tables"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache"); + let session_id = format!( + "{}{}", + source.descriptor().session_prefix, + source_session_id + ); + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,?2,?3,?4)", + params![ + source.as_str(), + source_session_id, + session_id, + path.to_string_lossy() + ], + ) + .expect("cache source path"); + (conn, session_id) +} + +fn create_part_db(path: &Path, session_id: &str) -> Connection { + let conn = Connection::open(path).expect("source DB"); + conn.execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE session( + id TEXT PRIMARY KEY,time_created INTEGER,time_updated INTEGER + ); + CREATE TABLE message( + id TEXT PRIMARY KEY,session_id TEXT,time_created INTEGER,data TEXT + ); + CREATE TABLE part( + id TEXT PRIMARY KEY,message_id TEXT,session_id TEXT, + time_created INTEGER,data TEXT + );", + ) + .expect("part schema"); + conn.execute("INSERT INTO session VALUES(?1,1,1)", [session_id]) + .expect("session row"); + conn +} + +fn insert_part(conn: &Connection, session_id: &str, ordinal: usize, role: &str, part: Value) { + let message_id = format!("message-{ordinal:06}"); + let part_id = format!("part-{ordinal:06}"); + let timestamp = ordinal as i64 + 1; + conn.execute( + "INSERT INTO message(id,session_id,time_created,data) VALUES(?1,?2,?3,?4)", + params![ + message_id, + session_id, + timestamp, + serde_json::json!({"role":role}).to_string() + ], + ) + .expect("message row"); + conn.execute( + "INSERT INTO part(id,message_id,session_id,time_created,data) + VALUES(?1,?2,?3,?4,?5)", + params![part_id, message_id, session_id, timestamp, part.to_string()], + ) + .expect("part row"); +} + +fn create_kv_db(path: &Path, composer_id: &str) -> Connection { + let conn = Connection::open(path).expect("KV source DB"); + conn.execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE cursorDiskKV(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("KV schema"); + write_kv_transcript(&conn, composer_id, &[1, 2]); + conn +} + +fn write_kv_transcript(conn: &Connection, composer_id: &str, bubble_types: &[i64]) { + let headers = bubble_types + .iter() + .enumerate() + .map(|(index, bubble_type)| { + serde_json::json!({"bubbleId":format!("b{index}"),"type":bubble_type}) + }) + .collect::>(); + let composer = serde_json::json!({ + "composerId":composer_id, + "createdAt":1, + "lastUpdatedAt":bubble_types.len() as i64, + "fullConversationHeadersOnly":headers, + }); + conn.execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2) + ON CONFLICT(key) DO UPDATE SET value=excluded.value", + params![format!("composerData:{composer_id}"), composer.to_string()], + ) + .expect("composer row"); + for (index, bubble_type) in bubble_types.iter().copied().enumerate() { + let bubble = serde_json::json!({ + "bubbleId":format!("b{index}"), + "type":bubble_type, + "createdAt":format!("2026-07-22T00:00:{index:02}Z"), + "text":if bubble_type == 1 { format!("user {index}") } else { format!("assistant {index}") }, + }); + conn.execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2) + ON CONFLICT(key) DO UPDATE SET value=excluded.value", + params![ + format!("bubbleId:{composer_id}:b{index}"), + bubble.to_string() + ], + ) + .expect("bubble row"); + } +} + +fn write_dirty_kv_transcript(conn: &Connection, composer_id: &str) { + let composer = serde_json::json!({ + "composerId":composer_id, + "createdAt":1, + "lastUpdatedAt":99, + "fullConversationHeadersOnly":[ + {"bubbleId":"","type":1}, + {"bubbleId":"missing","type":2}, + {"bubbleId":"b0","type":1}, + {"bubbleId":"b0","type":2}, + {"bubbleId":"b1","type":2}, + {"bubbleId":"b2","type":1}, + {"bubbleId":"b3","type":2} + ], + }); + conn.execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2) + ON CONFLICT(key) DO UPDATE SET value=excluded.value", + params![format!("composerData:{composer_id}"), composer.to_string()], + ) + .expect("dirty composer row"); + + for (bubble_id, bubble_type, created_at) in [ + ("b0", 1_i64, "2026-07-22T00:00:00Z"), + ("b1", 2_i64, "2026-07-22T00:00:01Z"), + ("b2", 1_i64, "2026-07-22T00:00:02Z"), + ("b3", 2_i64, "2026-07-22T00:00:03Z"), + ("b4", 1_i64, "2026-07-22T00:00:04Z"), + ("b5", 2_i64, "2026-07-22T00:00:05Z"), + ("undefined", 2_i64, "2026-07-22T00:00:06Z"), + ] { + let bubble = serde_json::json!({ + "bubbleId":bubble_id, + "type":bubble_type, + "createdAt":created_at, + "text":if bubble_type == 1 { + format!("user {bubble_id}") + } else { + format!("assistant {bubble_id}") + }, + }); + conn.execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2) + ON CONFLICT(key) DO UPDATE SET value=excluded.value", + params![ + format!("bubbleId:{composer_id}:{bubble_id}"), + bubble.to_string() + ], + ) + .expect("dirty bubble row"); + } +} + +#[test] +fn preview_is_utf8_safe_and_bounded() { + let preview = head_preview(&"你".repeat(10_000), NORMAL_PAYLOAD_PREVIEW_BYTES); + assert!(preview.is_char_boundary(preview.len())); + assert!(preview.len() < NORMAL_PAYLOAD_PREVIEW_BYTES + 64); +} + +#[test] +fn compact_sqlite_rows_keep_line_stats_and_full_output_git_summary() { + let mut edit = ActivityChunk::new("opencodeapp-s1", "tool_call", "edit_file"); + edit.args = serde_json::json!({ + "file_path":"src/large.rs", + "content":"line\n".repeat(4_000) + }); + edit.result = serde_json::json!({ + "linesAdded":7, + "linesRemoved":3, + "output":"x".repeat(16 * 1024) + }); + compact_chunk(&mut edit, "part-edit"); + let mut edit_metadata = TurnMetadataAccumulator::new(); + edit_metadata.add_event_values_at( + Some(&edit.function), + &edit.args, + &edit.result, + "2026-07-22T00:00:00Z", + ); + assert_eq!(edit_metadata.modified_files()[0].path, "src/large.rs"); + assert_eq!(edit_metadata.modified_files()[0].additions, 7); + assert_eq!(edit_metadata.modified_files()[0].deletions, 3); + + let mut shell = ActivityChunk::new( + "opencodeapp-s1", + "tool_call", + imported_history::FUNCTION_RUN_COMMAND_LINE, + ); + shell.args = serde_json::json!({"command":"git commit -m metadata"}); + shell.result = serde_json::json!({ + "output":format!( + "[feature abc1234] metadata\n{}\nhttps://github.com/acme/repo/pull/42", + "middle".repeat(10 * 1024) + ) + }); + compact_chunk(&mut shell, "part-shell"); + let mut shell_metadata = TurnMetadataAccumulator::new(); + shell_metadata.add_event_values_at( + Some(&shell.function), + &shell.args, + &shell.result, + "2026-07-22T00:00:01Z", + ); + assert!(shell_metadata + .git_artifacts() + .iter() + .any(|artifact| artifact.sha.as_deref() == Some("abc1234"))); + assert!(shell_metadata + .git_artifacts() + .iter() + .any(|artifact| artifact.pr_number == Some(42))); +} + +#[test] +fn stable_id_ignores_rowid_and_uses_provider_key() { + let first = stable_event_id(ImportedHistorySourceId::OpenCode, "part-1"); + let second = stable_event_id(ImportedHistorySourceId::OpenCode, "part-1"); + assert_eq!(first, second); +} + +#[test] +fn all_five_sqlite_sources_open_bounded_and_poll_unchanged_without_parsing() { + for source in [ + ImportedHistorySourceId::OpenCode, + ImportedHistorySourceId::MimoCode, + ImportedHistorySourceId::ZCode, + ] { + let path = temp_db(source.as_str()); + let writer = create_part_db(&path, "s1"); + insert_part( + &writer, + "s1", + 0, + "user", + serde_json::json!({"type":"text","text":"hello"}), + ); + insert_part( + &writer, + "s1", + 1, + "assistant", + serde_json::json!({"type":"text","text":"world"}), + ); + let (mut cache, session_id) = cache_conn(source, "s1", &path); + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("open part replay"); + assert_eq!(opened.chunks.len(), 2, "{}", source.as_str()); + let unchanged = poll_delta( + &mut cache, + source, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("unchanged part poll"); + assert_eq!(unchanged.stats, ReplayStats::default()); + drop(writer); + let _ = std::fs::remove_file(path); + } + + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let path = temp_db(source.as_str()); + let writer = create_kv_db(&path, "c1"); + let (mut cache, session_id) = cache_conn(source, "c1", &path); + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("open KV replay"); + assert_eq!(opened.chunks.len(), 2, "{}", source.as_str()); + let unchanged = poll_delta( + &mut cache, + source, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("unchanged KV poll"); + assert_eq!(unchanged.stats, ReplayStats::default()); + drop(writer); + let _ = std::fs::remove_file(path); + } +} + +#[test] +fn opencode_append_update_delete_and_checkpoint_are_incremental() { + let path = temp_db("opencode-delta"); + let writer = create_part_db(&path, "s1"); + insert_part( + &writer, + "s1", + 0, + "user", + serde_json::json!({"type":"text","text":"hello"}), + ); + let (mut cache, session_id) = cache_conn(ImportedHistorySourceId::OpenCode, "s1", &path); + let opened = open_window( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + ReplayLimits::default(), + ) + .expect("open replay"); + + let tx = writer.unchecked_transaction().expect("append transaction"); + for ordinal in 1..=1_000 { + insert_part( + &tx, + "s1", + ordinal, + "assistant", + serde_json::json!({"type":"text","text":format!("answer {ordinal}")}), + ); + } + tx.execute("UPDATE session SET time_updated=2 WHERE id='s1'", []) + .unwrap(); + tx.commit().unwrap(); + let mut appended = poll_delta( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("append delta"); + assert!(appended.stats.parsed_rows <= 1_010); + assert_eq!(appended.stats.parsed_rows, 1_000); + assert_eq!(appended.stats.upserted_events, 1_000); + let mut append_ids = std::collections::HashSet::new(); + loop { + for chunk in &appended.chunks { + assert!( + append_ids.insert(chunk.chunk.chunk_id.clone()), + "append delta repeated an event" + ); + } + if append_ids.len() == 1_000 { + break; + } + assert!(!appended.chunks.is_empty(), "append continuation stalled"); + appended = poll_delta( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &appended.cursor, + ReplayLimits::default(), + ) + .expect("continued append delta"); + assert_eq!(appended.stats.parsed_rows, 0); + assert_eq!(appended.stats.upserted_events, 0); + } + assert_eq!(append_ids.len(), 1_000); + + writer + .execute_batch("PRAGMA wal_checkpoint(PASSIVE);") + .unwrap(); + let checkpoint = poll_delta( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &appended.cursor, + ReplayLimits::default(), + ) + .expect("checkpoint poll"); + assert_eq!(checkpoint.stats, ReplayStats::default()); + + writer + .execute( + "UPDATE part SET data=?1 WHERE id='part-000000'", + [serde_json::json!({"type":"text","text":"edited"}).to_string()], + ) + .unwrap(); + writer + .execute("UPDATE session SET time_updated=3 WHERE id='s1'", []) + .unwrap(); + let updated = poll_delta( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &checkpoint.cursor, + ReplayLimits::default(), + ) + .expect("update delta"); + assert_eq!(updated.stats.parsed_rows, 1); + assert_eq!(updated.chunks.len(), 1); + + writer + .execute("DELETE FROM part WHERE id='part-000000'", []) + .unwrap(); + writer + .execute("UPDATE session SET time_updated=4 WHERE id='s1'", []) + .unwrap(); + let deleted = poll_delta( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &updated.cursor, + ReplayLimits::default(), + ) + .expect("delete delta"); + assert_eq!(deleted.removed_event_ids.len(), 1); + assert_eq!(deleted.stats.removed_events, 1); + + writer + .execute_batch("PRAGMA wal_checkpoint(TRUNCATE); VACUUM;") + .unwrap(); + let vacuumed = poll_delta( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &deleted.cursor, + ReplayLimits::default(), + ) + .expect("VACUUM reset"); + assert!(vacuumed.reset_required); + assert_ne!(vacuumed.cursor.generation, deleted.cursor.generation); + drop(writer); + let _ = std::fs::remove_file(path); +} + +#[test] +fn cursor_and_windsurf_kv_updates_and_deletes_are_deltas() { + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let path = temp_db(&format!("{}-delta", source.as_str())); + let writer = create_kv_db(&path, "c1"); + let (mut cache, session_id) = cache_conn(source, "c1", &path); + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("open KV delta fixture"); + + let updated_bubble = serde_json::json!({ + "bubbleId":"b1","type":2,"createdAt":"2026-07-22T00:00:01Z", + "text":"assistant edited" + }); + writer + .execute( + "UPDATE cursorDiskKV SET value=?1 WHERE key='bubbleId:c1:b1'", + [updated_bubble.to_string()], + ) + .unwrap(); + let composer = serde_json::json!({ + "composerId":"c1","createdAt":1,"lastUpdatedAt":99, + "fullConversationHeadersOnly":[ + {"bubbleId":"b0","type":1},{"bubbleId":"b1","type":2} + ] + }); + writer + .execute( + "UPDATE cursorDiskKV SET value=?1 WHERE key='composerData:c1'", + [composer.to_string()], + ) + .unwrap(); + let updated = poll_delta( + &mut cache, + source, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("KV update delta"); + assert_eq!(updated.stats.parsed_rows, 1, "{}", source.as_str()); + assert_eq!(updated.chunks.len(), 1, "{}", source.as_str()); + + writer + .execute("DELETE FROM cursorDiskKV WHERE key='bubbleId:c1:b0'", []) + .unwrap(); + let composer = serde_json::json!({ + "composerId":"c1","createdAt":1,"lastUpdatedAt":100, + "fullConversationHeadersOnly":[{"bubbleId":"b1","type":2}] + }); + writer + .execute( + "UPDATE cursorDiskKV SET value=?1 WHERE key='composerData:c1'", + [composer.to_string()], + ) + .unwrap(); + let deleted = poll_delta( + &mut cache, + source, + &session_id, + &updated.cursor, + ReplayLimits::default(), + ) + .expect("KV delete delta"); + assert_eq!(deleted.removed_event_ids.len(), 1, "{}", source.as_str()); + drop(writer); + let _ = std::fs::remove_file(path); + } +} + +#[test] +fn kv_cold_open_reads_only_latest_turn_and_older_turn_hydrates_by_index() { + let path = temp_db("cursor-lazy-turn"); + let writer = create_kv_db(&path, "c1"); + write_kv_transcript(&writer, "c1", &[1, 2, 1, 2]); + let (mut cache, session_id) = cache_conn(ImportedHistorySourceId::CursorIde, "c1", &path); + let opened = open_window( + &mut cache, + ImportedHistorySourceId::CursorIde, + &session_id, + ReplayLimits::default(), + ) + .expect("cold bounded KV replay"); + assert_eq!(opened.stats.parsed_rows, 2); + assert_eq!(opened.chunks.len(), 2); + assert_eq!(opened.total_turn_count, 2); + assert_eq!(opened.total_event_count, 4); + + let older = crate::sources::imported_history::replay::read_turn_window_at_index( + &mut cache, + ImportedHistorySourceId::CursorIde, + &session_id, + 0, + ReplayLimits::default(), + ) + .expect("hydrate older KV turn"); + assert_eq!(older.chunks.len(), 2); + assert_eq!(older.turn_headers[0].turn_index, 0); + drop(writer); + let _ = std::fs::remove_file(path); +} + +#[test] +fn kv_visible_turn_metadata_reads_bounded_user_previews_without_hydrating_bodies() { + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let path = temp_db(&format!("{}-compact-turn-metadata", source.as_str())); + let writer = create_kv_db(&path, "c1"); + write_kv_transcript(&writer, "c1", &[1, 2, 1, 2]); + let large_user_text = format!( + "compact target prompt {}", + "user-payload ".repeat(128 * 1024) + ); + let large_user_bubble = serde_json::json!({ + "bubbleId":"b0", + "type":1, + "createdAt":"2026-07-22T00:00:00Z", + "text":large_user_text, + }); + let large_assistant_bubble = serde_json::json!({ + "bubbleId":"b1", + "type":2, + "createdAt":"2026-07-22T00:00:01Z", + "text":"assistant-body ".repeat(128 * 1024), + }); + writer + .execute( + "UPDATE cursorDiskKV SET value=?1 WHERE key='bubbleId:c1:b0'", + [large_user_bubble.to_string()], + ) + .expect("large user bubble"); + writer + .execute( + "UPDATE cursorDiskKV SET value=?1 WHERE key='bubbleId:c1:b1'", + [large_assistant_bubble.to_string()], + ) + .expect("large assistant bubble"); + + let (mut cache, session_id) = cache_conn(source, "c1", &path); + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("cold bounded KV replay"); + assert_eq!(opened.stats.parsed_rows, 2, "{}", source.as_str()); + assert_eq!( + cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_events + WHERE source=?1 AND source_session_id='c1' + AND generation=?2 AND turn_index=0", + params![source.as_str(), opened.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("count old indexed events"), + 0, + "{}", + source.as_str() + ); + let changes_before = cache.total_changes(); + let selector = "__external_replay_turn_index__:0".to_string(); + let projected = crate::sources::imported_history::replay::project_cached_turn_metadata( + &cache, + source, + &session_id, + Some(std::slice::from_ref(&selector)), + ) + .expect("read cached KV turn metadata") + .expect("published compact KV index"); + + assert_eq!(projected.len(), 1, "{}", source.as_str()); + assert_eq!(projected[0].turn_id, selector, "{}", source.as_str()); + assert!( + projected[0] + .user_preview + .starts_with("compact target prompt"), + "{}", + source.as_str() + ); + assert!( + projected[0].user_preview.chars().count() <= 160, + "{}", + source.as_str() + ); + assert_eq!(projected[0].started_at, "2026-07-22T00:00:00Z"); + assert_eq!(projected[0].event_count, 2); + assert_eq!(projected[0].body_event_count, 1); + assert_eq!( + cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_events + WHERE source=?1 AND source_session_id='c1' + AND generation=?2 AND turn_index=0", + params![source.as_str(), opened.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("recount old indexed events"), + 0, + "{}", + source.as_str() + ); + assert_eq!( + cache.total_changes(), + changes_before, + "metadata read must not write the compact index for {}", + source.as_str() + ); + + drop(cache); + drop(writer); + let _ = std::fs::remove_file(path); + } +} + +#[test] +fn kv_cold_forward_scan_hydrates_turns_in_order_without_gaps() { + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let path = temp_db(&format!("{}-forward-scan", source.as_str())); + let writer = create_kv_db(&path, "c1"); + write_kv_transcript(&writer, "c1", &[1, 2, 1, 2, 1, 2]); + let (mut cache, session_id) = cache_conn(source, "c1", &path); + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("cold bounded KV replay"); + assert_eq!(opened.stats.parsed_rows, 2, "{}", source.as_str()); + + let limits = ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: 4 * 1024 * 1024, + }; + let mut after_sequence = -1; + let mut sequences = Vec::new(); + for _ in 0..10 { + let scan = scan_window_after(&mut cache, source, &session_id, after_sequence, limits) + .expect("bounded forward KV scan"); + sequences.extend(scan.chunks.iter().map(|chunk| chunk.sequence)); + assert!(scan.cursor.through_sequence > after_sequence || !scan.has_more); + after_sequence = scan.cursor.through_sequence; + if !scan.has_more { + break; + } + } + assert_eq!(sequences, vec![0, 1, 2, 3, 4, 5], "{}", source.as_str()); + drop(writer); + let _ = std::fs::remove_file(path); + } +} + +#[test] +fn cursor_and_windsurf_share_canonical_dirty_kv_ordering() { + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let path = temp_db(&format!("{}-dirty-order", source.as_str())); + let writer = create_kv_db(&path, "c1"); + write_dirty_kv_transcript(&writer, "c1"); + let (mut cache, session_id) = cache_conn(source, "c1", &path); + + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("open dirty KV fixture"); + assert_eq!(opened.total_event_count, 6, "{}", source.as_str()); + assert_eq!(opened.total_turn_count, 3, "{}", source.as_str()); + assert_eq!( + opened + .chunks + .iter() + .map(|chunk| (chunk.sequence, chunk.chunk.chunk_id.clone())) + .collect::>(), + vec![ + (4, stable_event_id(source, "bubbleId:c1:b4")), + (5, stable_event_id(source, "bubbleId:c1:b5")), + ], + "{}", + source.as_str() + ); + + let limits = ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: 4 * 1024 * 1024, + }; + let mut after_sequence = -1; + let mut actual = Vec::new(); + for _ in 0..10 { + let scan = scan_window_after(&mut cache, source, &session_id, after_sequence, limits) + .expect("scan canonical dirty KV fixture"); + actual.extend( + scan.chunks + .iter() + .map(|chunk| (chunk.sequence, chunk.chunk.chunk_id.clone())), + ); + after_sequence = scan.cursor.through_sequence; + if !scan.has_more { + break; + } + } + assert_eq!( + actual, + ["b0", "b1", "b2", "b3", "b4", "b5"] + .into_iter() + .enumerate() + .map(|(sequence, bubble_id)| { + ( + sequence as i64, + stable_event_id(source, &format!("bubbleId:c1:{bubble_id}")), + ) + }) + .collect::>(), + "{}", + source.as_str() + ); + + let headers = { + let mut statement = cache + .prepare( + "SELECT turn_index,turn_id,start_sequence,end_sequence,event_count + FROM imported_replay_turns + WHERE source=?1 AND source_session_id='c1' + ORDER BY turn_index", + ) + .expect("prepare compact KV turn query"); + statement + .query_map([source.as_str()], |row| { + Ok(( + row.get::<_, i64>(0)?, + row.get::<_, String>(1)?, + row.get::<_, i64>(2)?, + row.get::<_, i64>(3)?, + row.get::<_, i64>(4)?, + )) + }) + .expect("query compact KV turns") + .collect::, _>>() + .expect("collect compact KV turns") + }; + assert_eq!( + headers, + vec![ + (0, "b0".to_string(), 0, 1, 2), + (1, "b2".to_string(), 2, 3, 2), + (2, "b4".to_string(), 4, 5, 2), + ], + "{}", + source.as_str() + ); + + drop(writer); + let _ = std::fs::remove_file(path); + } +} + +#[test] +fn kv_prepare_then_strict_scan_crosses_three_lazy_turns() { + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let path = temp_db(&format!("{}-cursor-continuation", source.as_str())); + let writer = create_kv_db(&path, "c1"); + write_kv_transcript(&writer, "c1", &[1, 2, 1, 2, 1, 2]); + let (mut cache, session_id) = cache_conn(source, "c1", &path); + open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("cold bounded KV replay"); + + let limits = ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: 4 * 1024 * 1024, + }; + let prepared = prepare_pinned_scan(&mut cache, source, &session_id, limits) + .expect("prepare stable lazy KV scan"); + let pinned_generation = prepared.generation.clone(); + let pinned_revision = prepared.revision; + let mut after_sequence = -1; + let mut sequences = Vec::new(); + for _ in 0..10 { + let scan = scan_window_after_generation( + &mut cache, + source, + &session_id, + &pinned_generation, + pinned_revision, + after_sequence, + limits, + ) + .expect("strict scan across prepared KV turns"); + sequences.extend(scan.chunks.iter().map(|chunk| chunk.sequence)); + assert_eq!(scan.cursor.generation, pinned_generation); + assert_eq!(scan.cursor.revision, pinned_revision); + assert!(scan.cursor.through_sequence > after_sequence || !scan.has_more); + after_sequence = scan.cursor.through_sequence; + if !scan.has_more { + break; + } + } + assert_eq!(sequences, vec![0, 1, 2, 3, 4, 5], "{}", source.as_str()); + drop(writer); + let _ = std::fs::remove_file(path); + } +} + +#[test] +fn kv_reorder_rebuilds_stable_events_without_sequence_collisions() { + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let path = temp_db(&format!("{}-reorder", source.as_str())); + let writer = create_kv_db(&path, "c1"); + write_kv_transcript(&writer, "c1", &[1, 2, 2]); + let (mut cache, session_id) = cache_conn(source, "c1", &path); + let opened = open_window(&mut cache, source, &session_id, ReplayLimits::default()) + .expect("open KV reorder fixture"); + + let reordered = serde_json::json!({ + "composerId":"c1","createdAt":1,"lastUpdatedAt":99, + "fullConversationHeadersOnly":[ + {"bubbleId":"b0","type":1}, + {"bubbleId":"b2","type":2}, + {"bubbleId":"b1","type":2} + ] + }); + writer + .execute( + "UPDATE cursorDiskKV SET value=?1 WHERE key='composerData:c1'", + [reordered.to_string()], + ) + .unwrap(); + let delta = poll_delta( + &mut cache, + source, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("reordered KV delta"); + assert!(!delta.reset_required, "{}", source.as_str()); + assert_eq!(delta.removed_event_ids.len(), 0, "{}", source.as_str()); + + let reordered_window = crate::sources::imported_history::replay::read_turn_window_at_index( + &mut cache, + source, + &session_id, + 0, + ReplayLimits::default(), + ) + .expect("read reordered KV turn"); + let actual = reordered_window + .chunks + .iter() + .map(|chunk| (chunk.sequence, chunk.chunk.chunk_id.clone())) + .collect::>(); + let expected = vec![ + (0, stable_event_id(source, "bubbleId:c1:b0")), + (1, stable_event_id(source, "bubbleId:c1:b2")), + (2, stable_event_id(source, "bubbleId:c1:b1")), + ]; + assert_eq!(actual, expected, "{}", source.as_str()); + drop(writer); + let _ = std::fs::remove_file(path); + } +} + +#[test] +fn unknown_sqlite_schema_is_explicit_and_never_falls_back() { + let path = temp_db("unknown-schema"); + let source_conn = Connection::open(&path).unwrap(); + source_conn + .execute_batch("CREATE TABLE unrelated(id INTEGER PRIMARY KEY,value TEXT);") + .unwrap(); + let (mut cache, session_id) = cache_conn(ImportedHistorySourceId::OpenCode, "s1", &path); + let error = open_window( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + ReplayLimits::default(), + ) + .unwrap_err(); + assert!(error.contains("Unsupported opencode replay schema")); + assert!(error.contains("will not fall back")); + drop(source_conn); + let _ = std::fs::remove_file(path); +} + +#[test] +fn sqlite_wal_path_plus_ten_mib_content_uses_exact_root_args_artifact() { + let path = temp_db("opencode-root-args"); + let writer = create_part_db(&path, "s1"); + let original_args = serde_json::json!({ + "path":"src/huge.txt", + "content":format!("BEGIN{}END", "你".repeat((10 * 1024 * 1024) / 3)), + }); + let expected_json = serde_json::to_string(&original_args).expect("expected args JSON"); + let source_part = serde_json::json!({ + "type":"tool", + "tool":"custom_tool", + "callID":"call-root-args", + "state":{ + "status":"completed", + "input":original_args, + "output":"ok" + } + }); + insert_part(&writer, "s1", 0, "assistant", source_part); + let (mut cache, session_id) = cache_conn(ImportedHistorySourceId::OpenCode, "s1", &path); + let opened = open_window( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + ReplayLimits::default(), + ) + .expect("open SQLite root args fixture"); + let indexed = opened.chunks.first().expect("custom tool event"); + assert_eq!(indexed.chunk.args["path"], "src/huge.txt"); + assert_eq!(indexed.chunk.args["_replayTruncated"], true); + assert!(indexed.chunk.args.get("content").is_none()); + assert_eq!(indexed.payloads.len(), 1); + assert_eq!(indexed.payloads[0].field_path, "args"); + assert!(indexed.payloads[0].source_key.is_none()); + + let generation = opened.cursor.generation.clone(); + let event_id = indexed.chunk.chunk_id.clone(); + let restored = read_full_payload( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &generation, + &event_id, + "args", + ); + assert_eq!(restored.len(), expected_json.len()); + assert_eq!(hash_text(&restored), hash_text(&expected_json)); + assert_eq!( + serde_json::from_str::(&restored).expect("restored args JSON"), + serde_json::from_str::(&expected_json).expect("baseline args JSON") + ); + assert!(!restored.contains("_replayTruncated")); + assert!(!restored.contains("[payload truncated]")); + let artifact_count = cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifact_refs + WHERE source='opencode' AND generation=?1 AND event_id=?2 AND field_path='args'", + params![generation, event_id], + |row| row.get::<_, i64>(0), + ) + .expect("root args artifact ref"); + assert_eq!(artifact_count, 1); + drop(writer); + let _ = std::fs::remove_file(path); +} + +#[test] +fn same_length_sqlite_row_update_replaces_materialized_payload_hash() { + let path = temp_db("opencode-same-length-shell-update"); + let writer = create_part_db(&path, "s1"); + let first_output = format!("BEGIN{}END", "A".repeat(96 * 1024)); + let second_output = format!("BEGIN{}END", "B".repeat(96 * 1024)); + assert_eq!(first_output.len(), second_output.len()); + let part = |output: &str| { + serde_json::json!({ + "type":"tool", + "tool":"bash", + "callID":"call-same-length", + "state":{ + "status":"completed", + "input":{"command":"emit same length"}, + "output":output + } + }) + }; + insert_part(&writer, "s1", 0, "assistant", part(&first_output)); + let (mut cache, session_id) = cache_conn(ImportedHistorySourceId::OpenCode, "s1", &path); + let opened = open_window( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + ReplayLimits::default(), + ) + .expect("open same-length SQLite Shell fixture"); + let indexed = opened.chunks.first().expect("SQLite Shell event"); + let event_id = indexed.chunk.chunk_id.clone(); + assert!(indexed + .payloads + .iter() + .any(|payload| payload.field_path == "result.output")); + let first_hash = { + let tx = cache.transaction().expect("first payload artifact"); + let locator = materialize_payload_artifact( + &tx, + ImportedHistorySourceId::OpenCode, + &session_id, + &opened.cursor.generation, + &event_id, + "result.output", + ) + .expect("materialize first SQLite Shell payload"); + tx.commit().expect("commit first payload artifact"); + locator.content_hash + }; + + writer + .execute( + "UPDATE part SET data=?1 WHERE id='part-000000'", + [part(&second_output).to_string()], + ) + .expect("same-length SQLite row update"); + writer + .execute("UPDATE session SET time_updated=2 WHERE id='s1'", []) + .expect("advance SQLite session clock"); + let updated = poll_delta( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &opened.cursor, + ReplayLimits::default(), + ) + .expect("poll same-length SQLite update"); + assert!(!updated.reset_required); + assert_eq!(updated.cursor.generation, opened.cursor.generation); + assert_eq!(updated.stats.parsed_rows, 1); + + let second_hash = { + let tx = cache.transaction().expect("updated payload artifact"); + let locator = materialize_payload_artifact( + &tx, + ImportedHistorySourceId::OpenCode, + &session_id, + &updated.cursor.generation, + &event_id, + "result.output", + ) + .expect("materialize changed SQLite Shell payload"); + tx.commit().expect("commit changed payload artifact"); + locator.content_hash + }; + assert_ne!(first_hash, second_hash, "content, not length, is identity"); + let restored = read_full_payload( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &updated.cursor.generation, + &event_id, + "result.output", + ); + assert_eq!(restored, second_output); + assert_eq!( + cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifacts + WHERE source='opencode' AND generation=?1", + [&updated.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("live same-length artifact count"), + 1 + ); + drop(writer); + let _ = std::fs::remove_file(path); +} + +#[path = "sqlite_payload_driver.rs"] +mod payload_tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/sqlite_payload_driver.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/sqlite_payload_driver.rs new file mode 100644 index 000000000..9b9605788 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/sqlite_payload_driver.rs @@ -0,0 +1,130 @@ +use super::*; + +#[test] +fn sqlite_git_projection_metadata_does_not_leak_from_exact_result() { + let path = temp_db("opencode-git-root-result"); + let writer = create_part_db(&path, "s1"); + let output = format!( + "[feature abc1234] exact\n{}\nhttps://github.com/acme/repo/pull/42", + "middle".repeat(8 * 1024) + ); + let source_part = serde_json::json!({ + "type":"tool", + "tool":"bash", + "callID":"call-git", + "state":{ + "status":"completed", + "input":{"command":"git commit -m exact"}, + "output":output + } + }); + let expected = crate::sources::opencode::history::replay_chunk_from_part_json( + "opencodeapp-s1", + "opencode", + 0, + "part-000000".to_string(), + "message-000000".to_string(), + "assistant".to_string(), + &source_part.to_string(), + 1, + ) + .expect("normalize old full result") + .expect("old full tool event"); + insert_part(&writer, "s1", 0, "assistant", source_part); + let (mut cache, session_id) = cache_conn(ImportedHistorySourceId::OpenCode, "s1", &path); + let opened = open_window( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + ReplayLimits::default(), + ) + .expect("open SQLite Git fixture"); + let indexed = opened.chunks.first().expect("Git shell event"); + assert!(indexed.chunk.result.get("_replayGitArtifacts").is_some()); + assert_eq!( + indexed + .payloads + .iter() + .map(|payload| payload.field_path.as_str()) + .collect::>(), + vec!["result"] + ); + let restored = read_full_payload( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &opened.cursor.generation, + &indexed.chunk.chunk_id, + "result", + ); + let restored: Value = serde_json::from_str(&restored).expect("restored exact result"); + assert_eq!(restored, expected.result); + assert!(restored.get("_replayGitArtifacts").is_none()); + drop(writer); + let _ = std::fs::remove_file(path); +} + +#[test] +fn ten_megabyte_command_keeps_semantic_preview_and_round_trips_by_range() { + let path = temp_db("opencode-large-args"); + let writer = create_part_db(&path, "s1"); + let command = format!("BEGIN{}END", "你".repeat((10 * 1024 * 1024) / 3)); + insert_part( + &writer, + "s1", + 0, + "assistant", + serde_json::json!({ + "type":"tool", + "tool":"bash", + "callID":"call-1", + "state":{"status":"completed","input":{"command":command},"output":"ok"} + }), + ); + let (mut cache, session_id) = cache_conn(ImportedHistorySourceId::OpenCode, "s1", &path); + let opened = open_window( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + ReplayLimits::default(), + ) + .expect("open large args"); + let event = opened.chunks.first().expect("tool event"); + let preview = event + .chunk + .args + .get("command") + .and_then(Value::as_str) + .expect("semantic command preview"); + assert!(preview.len() < SHELL_PAYLOAD_PREVIEW_BYTES + 64); + let payload = event + .payloads + .iter() + .find(|payload| payload.field_path == "args") + .expect("root args payload"); + assert_eq!(payload.encoding, ReplayPayloadEncoding::JsonValue); + let projection = payload + .body_projection + .as_ref() + .expect("bounded root body projection"); + assert_eq!(projection.field_path, "args.cmd"); + assert!(projection.truncated); + assert!(projection.text.len() <= SHELL_PAYLOAD_PREVIEW_BYTES); + + let reconstructed = read_full_payload( + &mut cache, + ImportedHistorySourceId::OpenCode, + &session_id, + &opened.cursor.generation, + &event.chunk.chunk_id, + "args", + ); + let reconstructed: Value = + serde_json::from_str(&reconstructed).expect("complete normalized args JSON"); + assert_eq!(reconstructed["command"], command); + assert_eq!(reconstructed["cmd"], command); + assert_eq!(reconstructed["payload"]["command"], command); + assert!(reconstructed.get("_replayTruncated").is_none()); + drop(writer); + let _ = std::fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/structured_driver.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/structured_driver.rs new file mode 100644 index 000000000..31a12cf58 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/structured_driver.rs @@ -0,0 +1,973 @@ +use prost_reflect::prost::Message as _; + +use super::*; +use crate::projectors::turn_metadata::TurnMetadataAccumulator; + +fn temp_db(name: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!( + "orgii-structured-replay-{name}-{}-{}.db", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )) +} + +fn update_test_hash(mut hash: u64, text: &str) -> u64 { + for byte in text.as_bytes() { + hash ^= u64::from(*byte); + hash = hash.wrapping_mul(0x100000001b3); + } + hash +} + +fn read_full_payload( + cache: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, +) -> String { + let mut restored = String::new(); + let mut offset = 0_u64; + loop { + let range = crate::sources::imported_history::replay::read_payload_range( + cache, + source, + session_id, + generation, + event_id, + field_path, + offset, + Some(crate::sources::imported_history::replay::HARD_MAX_PAYLOAD_RANGE_BYTES), + ) + .expect("structured replay payload range"); + assert_eq!(range.offset, offset); + assert!(range.next_offset > offset || range.eof); + restored.push_str(&range.text); + offset = range.next_offset; + if range.eof { + assert_eq!(offset, range.total_bytes); + break; + } + } + restored +} + +fn cache_for( + source: ImportedHistorySourceId, + source_session_id: &str, + source_path: &Path, +) -> (Connection, String) { + use crate::store::sqlite::SqliteRecordStore; + + let cache = Connection::open_in_memory().expect("replay cache"); + SqliteRecordStore::init_tables(&cache).expect("replay tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source cache tables"); + let session_id = format!( + "{}{}", + source.descriptor().session_prefix, + source_session_id + ); + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,?2,?3,?4)", + params![ + source.as_str(), + source_session_id, + session_id, + source_path.to_string_lossy() + ], + ) + .expect("cache source binding"); + (cache, session_id) +} + +fn cursor_manifest(ids: &[String]) -> Vec { + let mut manifest = Vec::new(); + for id in ids { + manifest.extend_from_slice(&[0x0a, 32]); + manifest.extend_from_slice(&hex_decode(id).expect("blob id")); + } + manifest +} + +fn put_cursor_blob(conn: &Connection, byte: u8, data: &[u8]) -> String { + let id = hex_encode(&[byte; 32]); + conn.execute( + "INSERT OR REPLACE INTO blobs(id,data) VALUES(?1,?2)", + params![id, data], + ) + .expect("insert Cursor blob"); + id +} + +fn publish_cursor_root(conn: &Connection, root_byte: u8, ids: &[String]) { + let root_id = put_cursor_blob(conn, root_byte, &cursor_manifest(ids)); + let meta = json!({ + "agentId":"cursor-1", + "latestRootBlobId":root_id, + "createdAt":1_700_000_000_000_i64, + }) + .to_string(); + conn.execute( + "INSERT INTO meta(key,value) VALUES('0',?1) + ON CONFLICT(key) DO UPDATE SET value=excluded.value", + [hex_encode(meta.as_bytes())], + ) + .expect("publish Cursor root"); +} + +fn encode_warp_fixture(value: Value) -> Vec { + let pool = match &*WARP_DESCRIPTOR_POOL { + Ok(pool) => pool, + Err(error) => panic!("Warp descriptor: {error}"), + }; + let descriptor = pool + .get_message_by_name(WARP_TASK_PROTO_NAME) + .expect("Warp task descriptor"); + let encoded = value.to_string(); + let mut deserializer = serde_json::Deserializer::from_str(&encoded); + DynamicMessage::deserialize(descriptor, &mut deserializer) + .expect("Warp task JSON") + .encode_to_vec() +} + +fn metadata_from_chunks(chunks: &[ActivityChunk]) -> TurnMetadataAccumulator { + let mut metadata = TurnMetadataAccumulator::new(); + for chunk in chunks { + metadata.add_event_values_at( + Some(&chunk.function), + &chunk.args, + &chunk.result, + &chunk.created_at, + ); + } + metadata +} + +fn assert_projected_metadata_matches( + cache: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + expected: &TurnMetadataAccumulator, +) { + let projected = crate::sources::imported_history::replay::project_turn_metadata( + cache, source, session_id, None, + ) + .expect("project compact replay metadata"); + assert_eq!(projected.len(), 1); + assert_eq!(projected[0].modified_files, expected.modified_files()); + assert_eq!( + serde_json::to_value(&projected[0].resource_interactions).unwrap(), + serde_json::to_value(expected.resource_interactions()).unwrap() + ); + let mut actual_artifacts = projected[0] + .git_artifacts + .iter() + .map(|artifact| serde_json::to_string(artifact).unwrap()) + .collect::>(); + let mut expected_artifacts = expected + .git_artifacts() + .iter() + .map(|artifact| serde_json::to_string(artifact).unwrap()) + .collect::>(); + actual_artifacts.sort(); + expected_artifacts.sort(); + assert_eq!(actual_artifacts, expected_artifacts); +} + +#[test] +fn cursor_manifest_prefix_hash_detects_reorder() { + fn manifest(ids: &[[u8; 32]]) -> Vec { + let mut out = Vec::new(); + for id in ids { + out.extend_from_slice(&[0x0a, 32]); + out.extend_from_slice(id); + } + out + } + let first = manifest(&[[1; 32], [2; 32]]); + let appended = manifest(&[[1; 32], [2; 32], [3; 32]]); + let reordered = manifest(&[[2; 32], [1; 32], [3; 32]]); + let (_, expected) = manifest_prefix_hash(&first, 2).expect("prefix"); + assert_eq!(manifest_prefix_hash(&appended, 2).unwrap().1, expected); + assert_ne!(manifest_prefix_hash(&reordered, 2).unwrap().1, expected); +} + +#[test] +fn range_reader_preserves_utf8_boundaries() { + let text = "你".repeat(100); + let range = range_from_text("event", "result.output", &text, 1, 17).expect("range"); + assert!(range.text.is_char_boundary(range.text.len())); + assert!(range.next_offset > range.offset); + + let one_byte = range_from_text("event", "result.output", &text, 0, 1).expect("small range"); + assert_eq!(one_byte.text, "你"); + assert_eq!(one_byte.next_offset, 3); +} + +#[test] +fn structured_compaction_keeps_edit_scalars_and_full_git_summary() { + let mut edit = ActivityChunk::new( + "structured", + "tool_call", + imported_history::FUNCTION_EDIT_FILE, + ); + edit.args = json!({ + "file_path":"src/large.rs", + "action":"replace", + "operation":"update", + "linesAdded":17, + "linesRemoved":9, + "content":"line\n".repeat(4_000), + }); + edit.result = json!({"output":"updated"}); + compact_chunk(&mut edit, "edit-locator"); + assert_eq!(edit.args["linesAdded"], 17); + assert_eq!(edit.args["linesRemoved"], 9); + assert_eq!(edit.args["operation"], "update"); + assert_eq!(edit.result["linesAdded"], 17); + assert_eq!(edit.result["linesRemoved"], 9); + + let mut shell = ActivityChunk::new( + "structured", + "tool_call", + imported_history::FUNCTION_RUN_COMMAND_LINE, + ); + shell.args = json!({"command":"git commit -m metadata"}); + shell.result = json!({ + "output":format!( + "[feature abc1234] metadata\n{}\nhttps://github.com/acme/repo/pull/42", + "middle".repeat(14 * 1024) + ) + }); + assert!(shell.result["output"].as_str().unwrap().len() > 80 * 1024); + compact_chunk(&mut shell, "shell-locator"); + let metadata = metadata_from_chunks(&[edit, shell]); + assert_eq!(metadata.modified_files()[0].additions, 17); + assert_eq!(metadata.modified_files()[0].deletions, 9); + assert!(metadata + .git_artifacts() + .iter() + .any(|artifact| artifact.sha.as_deref() == Some("abc1234"))); + assert!(metadata + .git_artifacts() + .iter() + .any(|artifact| artifact.pr_number == Some(42))); +} + +#[test] +fn structured_path_plus_ten_mib_content_round_trips_exact_root_args() { + let path = temp_db("cursor-root-args"); + let source = Connection::open(&path).expect("Cursor root-args source"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE blobs(id TEXT PRIMARY KEY,data BLOB); + CREATE TABLE meta(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("Cursor root-args schema"); + let user = put_cursor_blob( + &source, + 71, + br#"{"role":"user","content":"large args"}"#, + ); + let original_args = json!({ + "path":"src/structured-huge.txt", + "content":format!("BEGIN{}END", "你".repeat((10 * 1024 * 1024) / 3)), + }); + let expected_json = serde_json::to_string(&original_args).expect("baseline args JSON"); + let tool_call = put_cursor_blob( + &source, + 72, + json!({ + "role":"assistant", + "content":[{ + "type":"tool-call", + "toolCallId":"large-args-call", + "toolName":"custom_tool", + "args":original_args + }] + }) + .to_string() + .as_bytes(), + ); + let tool_result = put_cursor_blob( + &source, + 73, + br#"{"role":"tool","content":[{"type":"tool-result","toolCallId":"large-args-call","result":"ok"}]}"#, + ); + publish_cursor_root(&source, 74, &[user, tool_call, tool_result]); + drop(source); + + let (mut cache, session_id) = cache_for(ImportedHistorySourceId::CursorCli, "cursor-1", &path); + let legacy = crate::sources::cursor_cli::history::load_cursor_cli_history_for_session( + &cache, + &session_id, + ) + .expect("old full Cursor history baseline"); + let expected_args = legacy + .iter() + .find(|chunk| chunk.function == "custom_tool") + .expect("legacy custom tool") + .args + .clone(); + assert_eq!( + expected_args, + serde_json::from_str::(&expected_json).unwrap() + ); + + let opened = crate::sources::imported_history::replay::open_window( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("open structured root args fixture"); + let indexed = opened + .chunks + .iter() + .find(|event| event.chunk.function == "custom_tool") + .expect("bounded custom tool event"); + assert_eq!(indexed.chunk.args["path"], "src/structured-huge.txt"); + assert_eq!(indexed.chunk.args["_replayTruncated"], true); + assert!(indexed.chunk.args.get("content").is_none()); + assert_eq!(indexed.payloads.len(), 1); + assert_eq!(indexed.payloads[0].field_path, "args"); + + reset_payload_fallback_decodes(); + let restored = read_full_payload( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &opened.cursor.generation, + &indexed.chunk.chunk_id, + "args", + ); + assert_eq!(restored.len(), expected_json.len()); + assert_eq!( + update_test_hash(0xcbf29ce484222325, &restored), + update_test_hash(0xcbf29ce484222325, &expected_json) + ); + assert_eq!( + serde_json::from_str::(&restored).expect("restored structured args"), + expected_args + ); + assert!(!restored.contains("_replayTruncated")); + assert!(!restored.contains("[payload truncated]")); + assert_eq!(payload_fallback_decodes(), 0); + let _ = std::fs::remove_file(path); +} + +#[test] +fn cursor_cli_public_replay_is_bounded_incremental_and_resets_on_reorder() { + let path = temp_db("cursor"); + let source = Connection::open(&path).expect("Cursor source"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE blobs(id TEXT PRIMARY KEY,data BLOB); + CREATE TABLE meta(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("Cursor schema"); + let user = put_cursor_blob( + &source, + 1, + br#"{"role":"user","content":"hello"}"#, + ); + let large_text = "cursor-large-".repeat(900_000); + let assistant = put_cursor_blob( + &source, + 2, + json!({"role":"assistant","content":[{"type":"text","text":large_text}]}) + .to_string() + .as_bytes(), + ); + let tool_call = put_cursor_blob( + &source, + 4, + br#"{"role":"assistant","content":[{"type":"tool-call","toolCallId":"call-1","toolName":"shell","args":{"command":"pwd"}}]}"#, + ); + publish_cursor_root( + &source, + 20, + &[user.clone(), assistant.clone(), tool_call.clone()], + ); + drop(source); + + let (mut cache, session_id) = cache_for(ImportedHistorySourceId::CursorCli, "cursor-1", &path); + let opened = crate::sources::imported_history::replay::open_window( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("open Cursor bounded replay"); + assert_eq!(opened.chunks.len(), 3); + assert!(opened.stats.parsed_bytes > 0); + let assistant_event = opened + .chunks + .iter() + .find(|event| event.chunk.function == imported_history::FUNCTION_ASSISTANT) + .expect("assistant event"); + assert!( + assistant_event.chunk.result["content"] + .as_str() + .unwrap_or_default() + .len() + < NORMAL_PAYLOAD_PREVIEW_BYTES + 64 + ); + let artifact_count = cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifacts + WHERE source='cursor_cli' AND generation=?1", + [&opened.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("Cursor payload artifact count"); + assert_eq!(artifact_count, 1); + reset_payload_fallback_decodes(); + let mut cursor_payload_hash = 0xcbf29ce484222325_u64; + let mut cursor_payload_bytes = 0_usize; + let mut payload_offset = 0_u64; + loop { + let range = crate::sources::imported_history::replay::read_payload_range( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &opened.cursor.generation, + &assistant_event.chunk.chunk_id, + "result.content", + payload_offset, + Some(crate::sources::imported_history::replay::HARD_MAX_PAYLOAD_RANGE_BYTES), + ) + .expect("Cursor payload artifact page"); + assert!( + range.text.len() + <= crate::sources::imported_history::replay::HARD_MAX_PAYLOAD_RANGE_BYTES + ); + assert!(range.next_offset > payload_offset || range.eof); + cursor_payload_hash = update_test_hash(cursor_payload_hash, &range.text); + cursor_payload_bytes = cursor_payload_bytes.saturating_add(range.text.len()); + payload_offset = range.next_offset; + if range.eof { + break; + } + } + assert_eq!(cursor_payload_bytes, large_text.len()); + assert_eq!( + cursor_payload_hash, + update_test_hash(0xcbf29ce484222325, &large_text) + ); + assert_eq!(payload_fallback_decodes(), 0); + let pending_tool_id = opened + .chunks + .iter() + .find(|event| event.chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE) + .expect("pending Cursor tool") + .chunk + .chunk_id + .clone(); + + let unchanged = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("unchanged Cursor poll"); + assert_eq!(unchanged.stats.parsed_rows, 0); + assert_eq!(unchanged.stats.upserted_events, 0); + + let source = Connection::open(&path).expect("reopen Cursor source"); + let second_user = put_cursor_blob( + &source, + 3, + br#"{"role":"user","content":"second"}"#, + ); + publish_cursor_root( + &source, + 21, + &[ + user.clone(), + assistant.clone(), + tool_call.clone(), + second_user.clone(), + ], + ); + drop(source); + let appended = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Cursor append delta"); + assert!(!appended.reset_required); + assert_eq!(appended.chunks.len(), 1); + assert_eq!(appended.stats.parsed_rows, 1); + + let source = Connection::open(&path).expect("reopen Cursor result source"); + let tool_result = put_cursor_blob( + &source, + 5, + br#"{"role":"tool","content":[{"type":"tool-result","toolCallId":"call-1","result":"/repo"}]}"#, + ); + publish_cursor_root( + &source, + 22, + &[ + user.clone(), + assistant.clone(), + tool_call.clone(), + second_user.clone(), + tool_result.clone(), + ], + ); + drop(source); + let completed_tool = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &appended.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Cursor cross-root tool result delta"); + assert!(!completed_tool.reset_required); + assert_eq!(completed_tool.chunks.len(), 1); + assert_eq!(completed_tool.chunks[0].chunk.chunk_id, pending_tool_id); + assert!(completed_tool.chunks[0] + .chunk + .result + .to_string() + .contains("/repo")); + + let source = Connection::open(&path).expect("reopen Cursor source for fork"); + publish_cursor_root( + &source, + 23, + &[second_user, user, assistant, tool_call, tool_result], + ); + drop(source); + let reset = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &completed_tool.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Cursor reorder reset"); + assert!(reset.reset_required); + assert_ne!(reset.cursor.generation, completed_tool.cursor.generation); + let _ = std::fs::remove_file(path); +} + +#[test] +fn cursor_cli_compact_projection_matches_full_large_edit_and_git_output() { + let path = temp_db("cursor-metadata"); + let source = Connection::open(&path).expect("Cursor metadata source"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE blobs(id TEXT PRIMARY KEY,data BLOB); + CREATE TABLE meta(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("Cursor metadata schema"); + let user = put_cursor_blob( + &source, + 31, + br#"{"role":"user","content":"metadata"}"#, + ); + let edit_args = json!({ + "file_path":"src/cursor-large.rs", + "old_string":"old\nvalue", + "new_string":"new line\n".repeat(2_000), + "operation":"replace", + }); + assert!(edit_args.to_string().len() > NORMAL_PAYLOAD_PREVIEW_BYTES); + let edit_call = put_cursor_blob( + &source, + 32, + json!({ + "role":"assistant", + "content":[{ + "type":"tool-call","toolCallId":"edit-1", + "toolName":"search_replace","args":edit_args + }] + }) + .to_string() + .as_bytes(), + ); + let edit_result = put_cursor_blob( + &source, + 33, + br#"{"role":"tool","content":[{"type":"tool-result","toolCallId":"edit-1","result":"done"}]}"#, + ); + let shell_call = put_cursor_blob( + &source, + 34, + br#"{"role":"assistant","content":[{"type":"tool-call","toolCallId":"git-1","toolName":"shell","args":{"command":"git commit -m metadata"}}]}"#, + ); + let git_output = format!( + "[feature abc1234] metadata\n{}\nhttps://github.com/acme/cursor/pull/77", + "middle".repeat(14 * 1024) + ); + assert!(git_output.len() > 80 * 1024); + let shell_result = put_cursor_blob( + &source, + 35, + json!({ + "role":"tool", + "content":[{"type":"tool-result","toolCallId":"git-1","result":git_output}] + }) + .to_string() + .as_bytes(), + ); + publish_cursor_root( + &source, + 36, + &[user, edit_call, edit_result, shell_call, shell_result], + ); + drop(source); + + let (mut cache, session_id) = cache_for(ImportedHistorySourceId::CursorCli, "cursor-1", &path); + let legacy = crate::sources::cursor_cli::history::load_cursor_cli_history_for_session( + &cache, + &session_id, + ) + .expect("load full Cursor metadata baseline"); + let expected_shell_result = legacy + .iter() + .find(|chunk| chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE) + .expect("legacy Cursor shell") + .result + .clone(); + let expected = metadata_from_chunks(&legacy); + assert!(expected + .modified_files() + .iter() + .any(|file| file.path == "src/cursor-large.rs")); + assert!(expected + .git_artifacts() + .iter() + .any(|artifact| artifact.sha.as_deref() == Some("abc1234"))); + assert!(expected + .git_artifacts() + .iter() + .any(|artifact| artifact.pr_number == Some(77))); + assert_projected_metadata_matches( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &expected, + ); + let opened = crate::sources::imported_history::replay::open_window( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("open compact Cursor metadata replay"); + let compact_shell = opened + .chunks + .iter() + .find(|event| event.chunk.function == imported_history::FUNCTION_RUN_COMMAND_LINE) + .expect("compact Cursor shell"); + assert!(compact_shell + .chunk + .result + .get("_replayGitArtifacts") + .is_some()); + assert!(compact_shell + .payloads + .iter() + .any(|payload| payload.field_path == "result")); + assert!(!compact_shell + .payloads + .iter() + .any(|payload| payload.field_path.starts_with("result."))); + let restored_result = read_full_payload( + &mut cache, + ImportedHistorySourceId::CursorCli, + &session_id, + &opened.cursor.generation, + &compact_shell.chunk.chunk_id, + "result", + ); + let restored_result: Value = + serde_json::from_str(&restored_result).expect("exact Cursor shell result"); + assert_eq!(restored_result, expected_shell_result); + assert!(restored_result.get("_replayGitArtifacts").is_none()); + let _ = std::fs::remove_file(path); +} + +#[test] +fn warp_task_rows_reconcile_insert_delete_rowid_reuse_and_schema_reset() { + let path = temp_db("warp"); + let source = Connection::open(&path).expect("Warp source"); + source + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE agent_conversations( + id INTEGER PRIMARY KEY,conversation_id TEXT,conversation_data TEXT, + last_modified_at TEXT,summary TEXT + ); + CREATE TABLE agent_tasks( + id INTEGER PRIMARY KEY,conversation_id TEXT,task_id TEXT, + task BLOB,last_modified_at TEXT + );", + ) + .expect("Warp schema"); + source + .execute( + "INSERT INTO agent_conversations( + id,conversation_id,conversation_data,last_modified_at,summary + ) VALUES(1,'conversation-1','{}','2026-07-14 01:00:06','{}')", + [], + ) + .expect("Warp conversation"); + let mut fixture: Value = serde_json::from_str(include_str!("../../../fixtures/warp_task.json")) + .expect("Warp fixture JSON"); + fixture["messages"][2]["toolCall"]["runShellCommand"]["command"] = + Value::String("git commit -m metadata".to_string()); + let git_output = format!( + "[feature def5678] metadata\n{}\nhttps://github.com/acme/warp/pull/88", + "middle".repeat(14 * 1024) + ); + assert!(git_output.len() > 80 * 1024); + fixture["messages"][3]["toolCallResult"]["runShellCommand"]["commandFinished"]["output"] = + Value::String(git_output); + fixture["messages"][4]["toolCall"]["applyFileDiffs"]["diffs"][0]["replace"] = + Value::String("new line\n".repeat(2_000)); + assert!( + fixture["messages"][4]["toolCall"]["applyFileDiffs"] + .to_string() + .len() + > NORMAL_PAYLOAD_PREVIEW_BYTES + ); + let large_text = "warp-large-".repeat(900_000); + fixture["messages"][6]["agentOutput"]["text"] = Value::String(large_text.clone()); + let blob = encode_warp_fixture(fixture.clone()); + let legacy_chunks = normalize_warp_task("warpapp-conversation-1", &blob, 0) + .expect("normalize full Warp metadata baseline"); + let legacy_edit = legacy_chunks + .iter() + .find(|chunk| chunk.function == imported_history::FUNCTION_EDIT_FILE) + .expect("Warp edit chunk"); + assert_eq!(legacy_edit.args["file_path"], "src/importer.rs"); + let expected = metadata_from_chunks(&legacy_chunks); + assert!(expected + .modified_files() + .iter() + .any(|file| file.path == "src/importer.rs")); + assert!(expected + .git_artifacts() + .iter() + .any(|artifact| artifact.sha.as_deref() == Some("def5678"))); + assert!(expected + .git_artifacts() + .iter() + .any(|artifact| artifact.pr_number == Some(88))); + source + .execute( + "INSERT INTO agent_tasks( + id,conversation_id,task_id,task,last_modified_at + ) VALUES(1,'conversation-1','task-root',?1,'2026-07-14 01:00:06')", + [&blob], + ) + .expect("Warp task"); + drop(source); + + let (mut cache, session_id) = cache_for(ImportedHistorySourceId::Warp, "conversation-1", &path); + let opened = crate::sources::imported_history::replay::open_window( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("open Warp bounded replay"); + assert_eq!(opened.chunks.len(), 5); + assert_projected_metadata_matches( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &expected, + ); + let assistant_event = opened + .chunks + .iter() + .find(|event| event.chunk.function == imported_history::FUNCTION_ASSISTANT) + .expect("Warp assistant"); + let assistant_artifacts = cache + .query_row( + "SELECT COUNT(DISTINCT content_hash) FROM imported_replay_payload_artifact_refs + WHERE source='warp' AND generation=?1 AND event_id=?2", + params![&opened.cursor.generation, &assistant_event.chunk.chunk_id], + |row| row.get::<_, i64>(0), + ) + .expect("Warp payload artifact ref count"); + assert_eq!( + assistant_artifacts, 1, + "duplicate compatibility fields share one body" + ); + reset_payload_fallback_decodes(); + let mut warp_payload_hash = 0xcbf29ce484222325_u64; + let mut warp_payload_bytes = 0_usize; + let mut payload_offset = 0_u64; + loop { + let range = crate::sources::imported_history::replay::read_payload_range( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &opened.cursor.generation, + &assistant_event.chunk.chunk_id, + "result.content", + payload_offset, + Some(crate::sources::imported_history::replay::HARD_MAX_PAYLOAD_RANGE_BYTES), + ) + .expect("Warp payload artifact page"); + assert!( + range.text.len() + <= crate::sources::imported_history::replay::HARD_MAX_PAYLOAD_RANGE_BYTES + ); + assert!(range.next_offset > payload_offset || range.eof); + warp_payload_hash = update_test_hash(warp_payload_hash, &range.text); + warp_payload_bytes = warp_payload_bytes.saturating_add(range.text.len()); + payload_offset = range.next_offset; + if range.eof { + break; + } + } + assert_eq!(warp_payload_bytes, large_text.len()); + assert_eq!( + warp_payload_hash, + update_test_hash(0xcbf29ce484222325, &large_text) + ); + assert_eq!(payload_fallback_decodes(), 0); + + let unchanged = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("unchanged Warp poll"); + assert_eq!(unchanged.stats.parsed_rows, 0); + assert_eq!(unchanged.stats.upserted_events, 0); + + let source = Connection::open(&path).expect("reopen Warp source"); + source + .execute( + "UPDATE agent_tasks SET last_modified_at='2026-07-14 01:00:06.5' WHERE id=1", + [], + ) + .expect("update Warp row metadata"); + drop(source); + let metadata_only = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &unchanged.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Warp metadata-only poll"); + assert_eq!(metadata_only.stats.parsed_rows, 0); + assert_eq!(metadata_only.stats.upserted_events, 0); + assert_eq!(metadata_only.cursor.revision, unchanged.cursor.revision); + + let source = Connection::open(&path).expect("reopen Warp source"); + fixture["messages"][6]["agentOutput"]["text"] = Value::String(format!("{large_text}-changed")); + let updated_blob = encode_warp_fixture(fixture); + source + .execute( + "UPDATE agent_tasks SET task=?1,last_modified_at='2026-07-14 01:00:07' WHERE id=1", + [&updated_blob], + ) + .expect("update one Warp event inside task BLOB"); + drop(source); + let updated = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &metadata_only.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Warp task update delta"); + assert_eq!(updated.stats.parsed_rows, 1); + assert_eq!(updated.stats.upserted_events, 1); + assert_eq!(updated.chunks.len(), 1); + let updated_assistant = updated + .chunks + .iter() + .find(|event| event.chunk.function == imported_history::FUNCTION_ASSISTANT) + .expect("updated Warp assistant"); + let updated_tail = crate::sources::imported_history::replay::read_payload_range( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &updated.cursor.generation, + &updated_assistant.chunk.chunk_id, + "result.content", + large_text.len() as u64, + Some(32), + ) + .expect("updated Warp artifact tail"); + assert_eq!(updated_tail.text, "-changed"); + assert_eq!(payload_fallback_decodes(), 0); + + let source = Connection::open(&path).expect("reopen Warp source"); + source + .execute("DELETE FROM agent_tasks WHERE id=1", []) + .expect("delete Warp task"); + drop(source); + let deleted = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &updated.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Warp delete delta"); + assert_eq!(deleted.removed_event_ids.len(), 5); + + let source = Connection::open(&path).expect("reopen Warp rowid source"); + source + .execute( + "INSERT INTO agent_tasks( + id,conversation_id,task_id,task,last_modified_at + ) VALUES(1,'conversation-1','task-root',?1,'2026-07-14 01:00:07')", + [&blob], + ) + .expect("reuse Warp rowid"); + drop(source); + let reinserted = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &deleted.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Warp rowid reuse delta"); + assert_eq!(reinserted.chunks.len(), 5); + + let source = Connection::open(&path).expect("reopen Warp schema source"); + source + .execute("ALTER TABLE agent_tasks ADD COLUMN extra TEXT", []) + .expect("change Warp schema"); + drop(source); + let reset = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Warp, + &session_id, + &reinserted.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("Warp schema reset"); + assert!(reset.reset_required); + let _ = std::fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/whole_json_driver.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/whole_json_driver.rs new file mode 100644 index 000000000..45dfa6eb6 --- /dev/null +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/replay/tests/whole_json_driver.rs @@ -0,0 +1,353 @@ +use std::io::{BufWriter, Write}; + +use super::*; +use crate::projectors::turn_metadata::TurnMetadataAccumulator; + +fn replay_cache(path: &Path) -> (rusqlite::Connection, String) { + use crate::store::sqlite::SqliteRecordStore; + + let cache = rusqlite::Connection::open_in_memory().expect("replay cache"); + SqliteRecordStore::init_tables(&cache).expect("replay tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source cache tables"); + let session_id = "clineapp-cline-1".to_string(); + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES('cline','cline-1',?1,?2)", + params![session_id, path.to_string_lossy()], + ) + .expect("cache Cline source"); + (cache, session_id) +} + +fn temp_path(name: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!( + "orgii-cline-replay-{name}-{}-{}.json", + std::process::id(), + chrono::Utc::now().timestamp_nanos_opt().unwrap_or_default() + )) +} + +fn metadata_from_chunks(chunks: &[ActivityChunk]) -> TurnMetadataAccumulator { + let mut metadata = TurnMetadataAccumulator::new(); + for chunk in chunks { + metadata.add_event_values_at( + Some(&chunk.function), + &chunk.args, + &chunk.result, + &chunk.created_at, + ); + } + metadata +} + +#[test] +fn invalid_partial_rewrite_is_reported_not_ready() { + let path = temp_path("partial"); + std::fs::write(&path, br#"{"messages":[{"role":"user","content":["#).expect("partial fixture"); + assert!(visit_messages(&path, |_| Ok(())).is_err()); + std::fs::write( + &path, + br#"{"messages":[{"role":"user","content":[{"type":"text","text":"ok"}]}]}"#, + ) + .expect("complete fixture"); + visit_messages(&path, |_| Ok(())).expect("complete probe"); + let _ = std::fs::remove_file(path); +} + +#[test] +fn sink_failures_are_not_misclassified_as_invalid_source_snapshots() { + let path = temp_path("sink-error"); + std::fs::write( + &path, + br#"{"messages":[{"role":"user","content":[{"type":"text","text":"ok"}]}]}"#, + ) + .expect("valid Cline fixture"); + let error = visit_messages(&path, |_| Err("replay index write failed".to_string())) + .expect_err("sink failure must propagate"); + assert!(error.starts_with("process Cline replay source")); + assert!(error.contains("replay index write failed")); + assert!(!error.starts_with("parse Cline replay source")); + let _ = std::fs::remove_file(path); +} + +#[test] +fn thirty_mib_document_streams_one_message_at_a_time() { + let path = temp_path("30mib"); + let file = File::create(&path).expect("fixture file"); + let mut writer = BufWriter::new(file); + writer.write_all(br#"{"messages":["#).unwrap(); + let padding = "x".repeat(30 * 1024); + for index in 0..1024 { + if index > 0 { + writer.write_all(b",").unwrap(); + } + serde_json::to_writer( + &mut writer, + &json!({ + "role":"assistant", + "content":[{"type":"text","text":padding}], + "ts":1_700_000_000_000_i64 + index, + }), + ) + .unwrap(); + } + writer.write_all(b"]}").unwrap(); + writer.flush().unwrap(); + assert!(std::fs::metadata(&path).unwrap().len() >= 30 * 1024 * 1024); + let mut count = 0_u64; + visit_messages(&path, |_| { + count += 1; + Ok(()) + }) + .expect("stream 30 MiB fixture"); + assert_eq!(count, 1024); + let _ = std::fs::remove_file(path); +} + +#[test] +fn cline_compact_projection_matches_full_large_edit_and_git_output() { + let path = temp_path("metadata"); + let large_edit = "new line\n".repeat(2_000); + let git_output = format!( + "[feature cab1234] metadata\n{}\nhttps://github.com/acme/cline/pull/99", + "middle".repeat(14 * 1024) + ); + assert!( + large_edit.len() > crate::sources::imported_history::replay::NORMAL_PAYLOAD_PREVIEW_BYTES + ); + assert!(git_output.len() > 80 * 1024); + let transcript = json!({ + "messages":[ + {"role":"user","content":[{"type":"text","text":"metadata"}],"ts":1_700_000_000_000_i64}, + {"role":"assistant","content":[{ + "type":"tool_use","id":"edit-1","name":"editor", + "input":{"path":"src/cline-large.rs","old_text":"old\nvalue","new_text":large_edit} + }],"ts":1_700_000_000_001_i64}, + {"role":"user","content":[{ + "type":"tool_result","tool_use_id":"edit-1","content":"done" + }],"ts":1_700_000_000_002_i64}, + {"role":"assistant","content":[{ + "type":"tool_use","id":"git-1","name":"run_commands", + "input":{"commands":["git commit -m metadata"]} + }],"ts":1_700_000_000_003_i64}, + {"role":"user","content":[{ + "type":"tool_result","tool_use_id":"git-1", + "content":[{"result":git_output,"success":true}] + }],"ts":1_700_000_000_004_i64} + ] + }); + std::fs::write(&path, transcript.to_string()).expect("Cline metadata transcript"); + let (mut cache, session_id) = replay_cache(&path); + let legacy = + crate::sources::cline::history::load_cline_history_for_session(&cache, &session_id) + .expect("load full Cline metadata baseline"); + let expected = metadata_from_chunks(&legacy); + assert_eq!(expected.modified_files()[0].path, "src/cline-large.rs"); + assert_eq!(expected.modified_files()[0].additions, 2_000); + assert_eq!(expected.modified_files()[0].deletions, 2); + assert!(expected + .git_artifacts() + .iter() + .any(|artifact| artifact.sha.as_deref() == Some("cab1234"))); + // The legacy full loader caps one Cline tool result at 50k chars, so + // the tail PR URL is the exact metadata that the bounded adapter must + // improve on rather than reproduce losing. + assert!(!expected + .git_artifacts() + .iter() + .any(|artifact| artifact.pr_number == Some(99))); + + let projected = crate::sources::imported_history::replay::project_turn_metadata( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + None, + ) + .expect("project compact Cline metadata"); + assert_eq!(projected.len(), 1); + assert_eq!(projected[0].modified_files, expected.modified_files()); + assert_eq!( + serde_json::to_value(&projected[0].resource_interactions).unwrap(), + serde_json::to_value(expected.resource_interactions()).unwrap() + ); + assert!(projected[0] + .git_artifacts + .iter() + .any(|artifact| artifact.sha.as_deref() == Some("cab1234"))); + assert!(projected[0] + .git_artifacts + .iter() + .any(|artifact| artifact.pr_number == Some(99))); + let _ = std::fs::remove_file(path); +} + +#[test] +fn cline_public_replay_keeps_last_valid_generation_during_partial_rewrite() { + let path = temp_path("atomic"); + let large_text = "cline-large-".repeat(900_000); + let initial = json!({ + "messages":[ + {"role":"user","content":[{"type":"text","text":"hello"}],"ts":1_700_000_000_000_i64}, + {"role":"assistant","content":[{"type":"text","text":large_text}],"ts":1_700_000_000_001_i64} + ] + }); + std::fs::write(&path, initial.to_string()).expect("initial Cline transcript"); + let (mut cache, session_id) = replay_cache(&path); + let opened = crate::sources::imported_history::replay::open_window( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("open Cline bounded replay"); + assert_eq!(opened.chunks.len(), 2); + assert_eq!(take_sync_attempts(), 1, "initial document indexed once"); + let assistant = opened + .chunks + .iter() + .find(|event| event.chunk.function == imported_history::FUNCTION_ASSISTANT) + .expect("Cline assistant"); + let artifact_count = cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifacts + WHERE source='cline' AND generation=?1", + [&opened.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("count deduplicated Cline artifacts"); + let artifact_ref_count = cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifact_refs + WHERE source='cline' AND generation=?1", + [&opened.cursor.generation], + |row| row.get::<_, i64>(0), + ) + .expect("count Cline artifact refs"); + assert_eq!( + artifact_count, 1, + "identical compatibility fields share bytes" + ); + assert_eq!(artifact_ref_count, 2); + let first_range = crate::sources::imported_history::replay::read_payload_range( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + &opened.cursor.generation, + &assistant.chunk.chunk_id, + "result.content", + 0, + Some(2048), + ) + .expect("Cline payload artifact"); + assert_eq!(first_range.text, large_text[..2048]); + + std::fs::write(&path, br#"{"messages":[{"role":"assistant","content":["#) + .expect("partial Cline rewrite"); + let partial = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("serve previous Cline generation"); + assert!(partial.stats.not_ready); + assert!(!partial.reset_required); + assert_eq!(partial.cursor.generation, opened.cursor.generation); + assert_eq!(partial.stats.parsed_bytes, 0); + assert_eq!(partial.stats.parsed_rows, 0); + assert_eq!(partial.stats.upserted_events, 0); + assert_eq!(take_sync_attempts(), 1, "invalid snapshot parsed once"); + let rejected_rows = cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_rejected_snapshots + WHERE source='cline' AND source_session_id='cline-1' + AND rejection_kind='cline_invalid_document'", + [], + |row| row.get::<_, i64>(0), + ) + .expect("count rejected Cline snapshot"); + assert_eq!(rejected_rows, 1); + + for _ in 0..20 { + let unchanged_invalid = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("serve unchanged rejected Cline snapshot"); + assert!(unchanged_invalid.stats.not_ready); + assert!(!unchanged_invalid.reset_required); + assert_eq!( + unchanged_invalid.cursor.generation, + opened.cursor.generation + ); + assert_eq!(unchanged_invalid.stats.parsed_bytes, 0); + assert_eq!(unchanged_invalid.stats.parsed_rows, 0); + assert_eq!(unchanged_invalid.stats.upserted_events, 0); + } + assert_eq!( + take_sync_attempts(), + 0, + "unchanged rejected snapshot must not be reparsed" + ); + let old_range = crate::sources::imported_history::replay::read_payload_range( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + &opened.cursor.generation, + &assistant.chunk.chunk_id, + "result.content", + 2048, + Some(2048), + ) + .expect("old Cline artifact during invalid rewrite"); + assert_eq!(old_range.text, large_text[2048..4096]); + + let complete = json!({ + "messages":[ + {"role":"user","content":[{"type":"text","text":"hello"}]}, + {"role":"assistant","content":[{"type":"text","text":"done"}]}, + {"role":"user","content":[{"type":"text","text":"second"}]}, + {"role":"assistant","content":[{"type":"text","text":"second done"}]} + ] + }); + std::fs::write(&path, complete.to_string()).expect("complete Cline rewrite"); + let reset = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + &opened.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("publish complete Cline generation"); + assert!(reset.reset_required); + assert_ne!(reset.cursor.generation, opened.cursor.generation); + assert_eq!(reset.chunks.len(), 2); + assert_eq!(take_sync_attempts(), 1, "changed snapshot retried once"); + let rejected_rows = cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_rejected_snapshots + WHERE source='cline' AND source_session_id='cline-1'", + [], + |row| row.get::<_, i64>(0), + ) + .expect("count cleared Cline rejection watermark"); + assert_eq!(rejected_rows, 0, "successful publish clears watermark"); + + let unchanged = crate::sources::imported_history::replay::poll_delta( + &mut cache, + ImportedHistorySourceId::Cline, + &session_id, + &reset.cursor, + crate::sources::imported_history::replay::ReplayLimits::default(), + ) + .expect("unchanged Cline poll"); + assert_eq!(unchanged.stats.parsed_rows, 0); + assert_eq!(unchanged.stats.upserted_events, 0); + let _ = std::fs::remove_file(path); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/router.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/router.rs index f2d55aef0..60c280e6e 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/router.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/router.rs @@ -1,122 +1,66 @@ -use core_types::activity::ActivityChunk; +//! Source-neutral bounded replay routing for backend consumers. +//! +//! This module intentionally exposes pages from the compact replay index. It +//! must never grow a provider-specific full-history fallback: adding a source +//! without a replay adapter is an error in the exhaustive replay registry. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum ImportedHistoryLoader { - ClaudeCode, - Codex, - Cursor, - CursorCli, - OpenCode, - Windsurf, - WorkBuddy, - Trae, - Cline, - Warp, - ZCode, - Qoder, - MimoCode, - Omp, - QoderCli, -} +use rusqlite::Connection; -fn imported_history_loader(session_id: &str) -> Option { - if session_id.starts_with(super::super::claude_code::SESSION_PREFIX) { - Some(ImportedHistoryLoader::ClaudeCode) - } else if session_id.starts_with(super::super::codex::SESSION_PREFIX) { - Some(ImportedHistoryLoader::Codex) - } else if session_id.starts_with(super::super::cursor_ide::CURSORIDE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Cursor) - } else if session_id.starts_with(super::super::cursor_cli::SESSION_PREFIX) { - Some(ImportedHistoryLoader::CursorCli) - } else if session_id.starts_with(super::super::opencode::history::OPENCODE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::OpenCode) - } else if session_id.starts_with(super::super::windsurf::history::WINDSURF_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Windsurf) - } else if session_id.starts_with(super::super::workbuddy::WORKBUDDY_SESSION_PREFIX) { - Some(ImportedHistoryLoader::WorkBuddy) - } else if session_id.starts_with(super::super::trae::history::TRAE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Trae) - } else if session_id.starts_with(super::super::cline::history::CLINE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Cline) - } else if session_id.starts_with(super::super::warp::history::WARP_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Warp) - } else if session_id.starts_with(super::super::zcode::history::ZCODE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::ZCode) - } else if session_id.starts_with(super::super::qoder::history::QODER_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Qoder) - } else if session_id.starts_with(super::super::mimo_code::history::MIMO_CODE_SESSION_PREFIX) { - Some(ImportedHistoryLoader::MimoCode) - } else if session_id.starts_with(super::super::omp::history::OMP_SESSION_PREFIX) { - Some(ImportedHistoryLoader::Omp) - } else if session_id.starts_with(super::super::qoder_cli::history::QODER_CLI_SESSION_PREFIX) { - Some(ImportedHistoryLoader::QoderCli) - } else { - None - } +use super::replay::{self, ImportedHistorySourceId, ReplayChunkScan, ReplayCursor, ReplayLimits}; + +/// Resolve one of the fifteen built-in imported-history sources from its +/// canonical session-id prefix. Unknown ids are left for third-party loaders. +pub fn source_for_session(session_id: &str) -> Option { + ImportedHistorySourceId::from_session_id(session_id) } -/// Load one imported provider session through its existing canonical history -/// reader. `None` means the id is not owned by an imported-history provider; -/// `Some(empty)` is a known provider session whose source currently has no -/// readable chunks. +/// Read the next bounded compact-index page for a built-in imported session. /// -/// This is the single provider router for cross-provider projections such as -/// per-round Orgtrack metadata. It deliberately delegates parsing to the -/// established source modules instead of introducing another transcript -/// reader. -pub fn load_activity_chunks_for_session( - conn: &rusqlite::Connection, +/// `None` means the id is not owned by a built-in source. Passing no cursor +/// starts by boundedly materializing lazy turns and verifying a stable source, +/// then reads the first page from that strict generation/revision. A cursor +/// continues the same immutable compact snapshot; external changes are never +/// mixed into an in-progress consumer. +/// Each returned `Vec` is constrained by [`ReplayLimits`] and is never a full +/// transcript. +pub fn scan_activity_chunks_for_session( + conn: &mut Connection, session_id: &str, -) -> Result>, String> { - let chunks = match imported_history_loader(session_id) { - Some(ImportedHistoryLoader::ClaudeCode) => { - super::super::claude_code::history::load_claude_code_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Codex) => { - super::super::codex::app::load_codex_app_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Cursor) => { - super::super::cursor_ide::history::load_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::CursorCli) => { - super::super::cursor_cli::history::load_cursor_cli_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::OpenCode) => { - super::super::opencode::history::load_opencode_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::Windsurf) => { - super::super::windsurf::history::load_windsurf_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::WorkBuddy) => { - super::super::workbuddy::load_workbuddy_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Trae) => { - super::super::trae::history::load_trae_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Cline) => { - super::super::cline::history::load_cline_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Warp) => { - super::super::warp::history::load_warp_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::ZCode) => { - super::super::zcode::history::load_zcode_history_for_session(session_id)? - } - Some(ImportedHistoryLoader::Qoder) => { - super::super::qoder::history::load_qoder_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::MimoCode) => { - super::super::mimo_code::history::load_mimo_code_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::Omp) => { - super::super::omp::history::load_omp_history_for_session(conn, session_id)? - } - Some(ImportedHistoryLoader::QoderCli) => { - super::super::qoder_cli::history::load_qoder_cli_history_for_session(conn, session_id)? + cursor: Option<&ReplayCursor>, + limits: ReplayLimits, +) -> Result, String> { + let Some(source) = source_for_session(session_id) else { + return Ok(None); + }; + let scan = match cursor { + Some(cursor) => { + if cursor.source_id != source.as_str() || cursor.session_id != session_id { + return Err("Replay cursor belongs to another source/session".to_string()); + } + replay::scan_window_after_generation( + conn, + source, + session_id, + &cursor.generation, + cursor.revision, + cursor.through_sequence, + limits, + )? + } + None => { + let prepared = replay::prepare_pinned_scan(conn, source, session_id, limits)?; + replay::scan_window_after_generation( + conn, + source, + session_id, + &prepared.generation, + prepared.revision, + -1, + limits, + )? } - None => return Ok(None), }; - Ok(Some(chunks)) + Ok(Some(scan)) } #[cfg(test)] @@ -124,28 +68,70 @@ mod tests { use super::*; #[test] - fn routes_every_imported_provider_to_its_existing_history_loader() { + fn routes_every_builtin_to_the_exhaustive_replay_registry() { let cases = [ - ("claudecodeapp-id", ImportedHistoryLoader::ClaudeCode), - ("codexapp-id", ImportedHistoryLoader::Codex), - ("cursoride-id", ImportedHistoryLoader::Cursor), - ("cursorcliapp-id", ImportedHistoryLoader::CursorCli), - ("opencodeapp-id", ImportedHistoryLoader::OpenCode), - ("windsurfapp-id", ImportedHistoryLoader::Windsurf), - ("workbuddyapp-id", ImportedHistoryLoader::WorkBuddy), - ("traeapp-id", ImportedHistoryLoader::Trae), - ("clineapp-id", ImportedHistoryLoader::Cline), - ("warpapp-id", ImportedHistoryLoader::Warp), - ("zcodeapp-id", ImportedHistoryLoader::ZCode), - ("qoderapp-id", ImportedHistoryLoader::Qoder), - ("mimocodeapp-id", ImportedHistoryLoader::MimoCode), - ("ompapp-id", ImportedHistoryLoader::Omp), - ("qodercliapp-id", ImportedHistoryLoader::QoderCli), + "claudecodeapp-id", + "codexapp-id", + "cursoride-id", + "cursorcliapp-id", + "opencodeapp-id", + "windsurfapp-id", + "workbuddyapp-id", + "traeapp-id", + "clineapp-id", + "warpapp-id", + "zcodeapp-id", + "qoderapp-id", + "mimocodeapp-id", + "ompapp-id", + "qodercliapp-id", ]; - for (session_id, expected) in cases { - assert_eq!(imported_history_loader(session_id), Some(expected)); - } - assert_eq!(imported_history_loader("org2-native-id"), None); + assert_eq!(cases.len(), ImportedHistorySourceId::ALL.len()); + let mut routed = std::collections::HashSet::new(); + for session_id in cases { + let source = source_for_session(session_id).expect(session_id); + assert_eq!(source.validate_session_id(session_id), Ok(())); + assert!(routed.insert(source), "duplicate route for {session_id}"); + } + assert_eq!( + routed, + ImportedHistorySourceId::ALL.into_iter().collect(), + "every built-in replay adapter must be reachable from its canonical prefix" + ); + assert_eq!(source_for_session("org2-native-id"), None); + } + + #[test] + fn unknown_ids_do_not_touch_sqlite_or_fall_back() { + let mut conn = Connection::open_in_memory().expect("in-memory DB"); + assert!(scan_activity_chunks_for_session( + &mut conn, + "plugin-owned-id", + None, + ReplayLimits::default(), + ) + .expect("unknown route") + .is_none()); + } + + #[test] + fn continuation_rejects_a_cursor_from_another_session_before_reading() { + let mut conn = Connection::open_in_memory().expect("in-memory DB"); + let cursor = ReplayCursor { + source_id: ImportedHistorySourceId::CodexApp.as_str().to_string(), + session_id: "codexapp-other".to_string(), + generation: "generation".to_string(), + revision: 1, + through_sequence: 10, + }; + let error = scan_activity_chunks_for_session( + &mut conn, + "codexapp-target", + Some(&cursor), + ReplayLimits::default(), + ) + .expect_err("mismatched cursor"); + assert!(error.contains("another source/session")); } } diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/scan_snapshot_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/scan_snapshot_tests.rs index 4893d9ec9..8bf8a6245 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/scan_snapshot_tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/scan_snapshot_tests.rs @@ -25,12 +25,7 @@ fn temp_tree(tag: &str) -> PathBuf { dir } -type WalkResult = ( - Vec, - HashMap, - usize, - usize, -); +type WalkResult = (Vec, HashMap, usize, usize); fn walk(previous: &HashMap, root: &Path) -> WalkResult { let mut walker = SnapshotDirWalker::new(previous, "jsonl", "Test"); @@ -56,7 +51,10 @@ fn snapshot_rows_roundtrip_and_replace() { ); write_dir_snapshots_from_conn(&conn, "claude_code", &snapshots).expect("write"); - assert_eq!(read_dir_snapshots_from_conn(&conn, "claude_code"), snapshots); + assert_eq!( + read_dir_snapshots_from_conn(&conn, "claude_code"), + snapshots + ); assert!(read_dir_snapshots_from_conn(&conn, "codex_app").is_empty()); let mut replacement = HashMap::new(); diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark.rs index 0a9899ce5..0daebb70e 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark.rs @@ -191,9 +191,8 @@ impl WatermarkedTranscriptReader { } if resume_state_json.is_none() { hasher = PrefixHasher::default(); - file.seek(SeekFrom::Start(0)).map_err(|err| { - format!("Failed to rewind {error_label} history: {err}") - })?; + file.seek(SeekFrom::Start(0)) + .map_err(|err| format!("Failed to rewind {error_label} history: {err}"))?; } } @@ -237,12 +236,7 @@ impl WatermarkedTranscriptReader { let read = self .reader .read_until(b'\n', &mut self.buf) - .map_err(|err| { - format!( - "Failed to read {} history line: {err}", - self.error_label - ) - })?; + .map_err(|err| format!("Failed to read {} history line: {err}", self.error_label))?; if read == 0 { return Ok(None); } @@ -259,12 +253,7 @@ impl WatermarkedTranscriptReader { } } let text = std::str::from_utf8(&self.buf[..end]) - .map_err(|err| { - format!( - "Failed to read {} history line: {err}", - self.error_label - ) - })? + .map_err(|err| format!("Failed to read {} history line: {err}", self.error_label))? .to_string(); Ok(Some(TranscriptLine { text, terminated })) } diff --git a/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark_tests.rs index cb2e2fd07..ea61d8dba 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark_tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/imported_history/watermark_tests.rs @@ -49,8 +49,8 @@ fn resume_reads_only_the_appended_suffix() { let path = temp_transcript("resume", "alpha\nbeta\n"); let (mtime, size) = stat(&path); - let mut full = WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size) - .expect("open full"); + let mut full = + WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size).expect("open full"); assert!(full.resume_state_json().is_none()); assert_eq!( read_all(&mut full), @@ -65,9 +65,15 @@ fn resume_reads_only_the_appended_suffix() { .and_then(|mut file| std::io::Write::write_all(&mut file, b"gamma\n")) .expect("append"); let (mtime_after, size_after) = stat(&path); - let mut resumed = - WatermarkedTranscriptReader::open(&path, "Test", Some(&watermark), 1, mtime_after, size_after) - .expect("open resumed"); + let mut resumed = WatermarkedTranscriptReader::open( + &path, + "Test", + Some(&watermark), + 1, + mtime_after, + size_after, + ) + .expect("open resumed"); assert_eq!(resumed.resume_state_json(), Some("state-1")); assert_eq!(read_all(&mut resumed), vec![("gamma".to_string(), true)]); let next = resumed.into_watermark(1, mtime_after, size_after, "state-2".to_string()); @@ -81,8 +87,8 @@ fn unterminated_tail_is_returned_but_never_watermarked() { let path = temp_transcript("tail", "alpha\npart"); let (mtime, size) = stat(&path); - let mut reader = WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size) - .expect("open full"); + let mut reader = + WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size).expect("open full"); assert_eq!( read_all(&mut reader), vec![("alpha".to_string(), true), ("part".to_string(), false)] @@ -92,9 +98,15 @@ fn unterminated_tail_is_returned_but_never_watermarked() { fs::write(&path, "alpha\npartial-done\n").expect("complete tail"); let (mtime_after, size_after) = stat(&path); - let mut resumed = - WatermarkedTranscriptReader::open(&path, "Test", Some(&watermark), 1, mtime_after, size_after) - .expect("open resumed"); + let mut resumed = WatermarkedTranscriptReader::open( + &path, + "Test", + Some(&watermark), + 1, + mtime_after, + size_after, + ) + .expect("open resumed"); assert_eq!(resumed.resume_state_json(), Some("state-1")); assert_eq!( read_all(&mut resumed), @@ -108,16 +120,22 @@ fn unterminated_tail_is_returned_but_never_watermarked() { fn prefix_mutation_forces_a_full_reparse() { let path = temp_transcript("mutated", "aa\nbb\n"); let (mtime, size) = stat(&path); - let mut reader = WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size) - .expect("open full"); + let mut reader = + WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size).expect("open full"); read_all(&mut reader); let watermark = reader.into_watermark(1, mtime, size, "state-1".to_string()); fs::write(&path, "xx\nbb\ncc\n").expect("rewrite prefix"); let (mtime_after, size_after) = stat(&path); - let mut reopened = - WatermarkedTranscriptReader::open(&path, "Test", Some(&watermark), 1, mtime_after, size_after) - .expect("open reopened"); + let mut reopened = WatermarkedTranscriptReader::open( + &path, + "Test", + Some(&watermark), + 1, + mtime_after, + size_after, + ) + .expect("open reopened"); assert!(reopened.resume_state_json().is_none()); assert_eq!( read_all(&mut reopened), @@ -135,8 +153,8 @@ fn prefix_mutation_forces_a_full_reparse() { fn size_regression_and_parser_version_change_force_a_full_reparse() { let path = temp_transcript("invalidate", "aa\nbb\n"); let (mtime, size) = stat(&path); - let mut reader = WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size) - .expect("open full"); + let mut reader = + WatermarkedTranscriptReader::open(&path, "Test", None, 1, mtime, size).expect("open full"); read_all(&mut reader); let watermark = reader.into_watermark(1, mtime, size, "state-1".to_string()); diff --git a/src-tauri/crates/orgtrack-core/src/sources/mimo_code/history.rs b/src-tauri/crates/orgtrack-core/src/sources/mimo_code/history.rs index 34473c561..dad9c6fbd 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/mimo_code/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/mimo_code/history.rs @@ -21,7 +21,9 @@ use crate::sources::imported_history::{ paths as imported_paths, ImportedHistoryRecentPath, ImportedHistorySessionPage, ImportedHistorySessionRow, }; -use crate::sources::opencode::history::load_opencode_compatible_history_from_conn; +use crate::sources::opencode::history::{ + load_opencode_compatible_history_from_conn, load_opencode_compatible_impact_from_conn, +}; pub const MIMO_CODE_SESSION_PREFIX: &str = "mimocodeapp-"; const MIMO_CODE_PROVIDER_SLUG: &str = "mimo_code"; @@ -92,7 +94,18 @@ pub fn load_mimo_code_history_for_session( ) } +pub(crate) fn refresh_catalog(cache_conn: &mut Connection) -> Result<(), String> { + sync_mimo_code_history_cache_inner(cache_conn, false) +} + fn sync_mimo_code_history_cache(cache_conn: &mut Connection) -> Result<(), String> { + sync_mimo_code_history_cache_inner(cache_conn, true) +} + +fn sync_mimo_code_history_cache_inner( + cache_conn: &mut Connection, + include_legacy_impact: bool, +) -> Result<(), String> { let mut metas = Vec::new(); for db_path in mimo_code_history_candidate_paths() { if !db_path.is_file() { @@ -123,15 +136,22 @@ fn sync_mimo_code_history_cache(cache_conn: &mut Connection) -> Result<(), Strin .into_iter() .filter(|meta| changed_ids.contains(&meta.source_session_id)) { - let session_id = format!("{MIMO_CODE_SESSION_PREFIX}{}", meta.source_session_id); - let conn = open_mimo_code_db_at(Path::new(&meta.source_path))?; - let chunks = load_opencode_compatible_history_from_conn( - &conn, - &session_id, + if include_legacy_impact { + let session_id = format!("{MIMO_CODE_SESSION_PREFIX}{}", meta.source_session_id); + let conn = open_mimo_code_db_at(Path::new(&meta.source_path))?; + meta.impact = load_opencode_compatible_impact_from_conn( + &conn, + &session_id, + &meta.source_session_id, + MIMO_CODE_PROVIDER_SLUG, + )?; + } else if let Some(cached) = imported_cache::query_cached_session_from_conn( + cache_conn, + SOURCE_MIMO_CODE, &meta.source_session_id, - MIMO_CODE_PROVIDER_SLUG, - )?; - meta.impact = imported_history::impact_from_edit_chunks(&chunks); + )? { + meta.impact = cached.impact; + } inputs.push(meta_to_cache_input(meta, &container_parent_ids)); } diff --git a/src-tauri/crates/orgtrack-core/src/sources/omp/history.rs b/src-tauri/crates/orgtrack-core/src/sources/omp/history.rs index 440e0859f..463214c54 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/omp/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/omp/history.rs @@ -23,6 +23,10 @@ fn config() -> AnthropicJsonlSource { } } +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + anthropic_jsonl::refresh_catalog(&config(), conn) +} + pub fn list_omp_history_sessions_paginated( conn: &mut Connection, limit: usize, diff --git a/src-tauri/crates/orgtrack-core/src/sources/opencode/history/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/opencode/history/mod.rs index ddc8217cd..464cbbae3 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/opencode/history/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/opencode/history/mod.rs @@ -149,6 +149,10 @@ pub fn load_opencode_history_for_session(session_id: &str) -> Result Result<(), String> { + sync_opencode_history_cache_inner(cache_conn, false) +} + /// Parse the message/part schema shared by OpenCode-compatible stores. /// /// Mimo Code persists the same normalized part records in its own SQLite @@ -170,6 +174,102 @@ pub(crate) fn load_opencode_compatible_history_from_conn( Ok(chunks) } +/// Fold edit impact directly from SQLite rows without constructing a +/// session-sized part or `ActivityChunk` vector. Only edit-capable tool rows +/// are copied out of SQLite; assistant/reasoning/shell output never enters the +/// catalog refresh path. +pub(crate) fn load_opencode_compatible_impact_from_conn( + conn: &Connection, + session_id: &str, + source_session_id: &str, + provider_slug: &str, +) -> Result { + let mut stmt = conn + .prepare( + "SELECT p.id, p.message_id, json_extract(m.data, '$.role'), p.data, p.time_created + FROM part p + JOIN message m ON m.id = p.message_id + WHERE p.session_id = ?1 + AND json_extract(p.data, '$.type') = 'tool' + AND lower(COALESCE(json_extract(p.data, '$.tool'), '')) + IN ('write', 'edit', 'patch', 'apply_patch') + ORDER BY p.time_created ASC, p.id ASC", + ) + .map_err(|err| format!("Failed to prepare OpenCode compact impact query: {err}"))?; + let mut rows = stmt + .query([source_session_id]) + .map_err(|err| format!("Failed to query OpenCode compact impact rows: {err}"))?; + let mut touched = std::collections::BTreeSet::new(); + let mut impact = ImportedHistoryImpactStats::default(); + let mut sequence = 0usize; + while let Some(row) = rows + .next() + .map_err(|err| format!("Failed to read OpenCode compact impact row: {err}"))? + { + let Some(raw_data) = row + .get::<_, Option>(3) + .map_err(|err| format!("Failed to read OpenCode compact tool payload: {err}"))? + else { + continue; + }; + let chunk = replay_chunk_from_part_json( + session_id, + provider_slug, + sequence, + row.get::<_, Option>(0) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + row.get::<_, Option>(1) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + row.get::<_, Option>(2) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + &raw_data, + row.get::<_, Option>(4) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + )?; + sequence = sequence.saturating_add(1); + let Some(chunk) = chunk else { continue }; + let one = imported_history::impact_from_edit_chunks(std::slice::from_ref(&chunk)); + impact.lines_added = impact.lines_added.saturating_add(one.lines_added); + impact.lines_removed = impact.lines_removed.saturating_add(one.lines_removed); + touched.extend(one.touched_files); + } + impact.touched_files = touched.into_iter().collect(); + impact.files_changed = impact.touched_files.len() as i64; + Ok(impact) +} + +/// Normalize one OpenCode-compatible `part` row for bounded replay. +/// +/// Unlike [`load_opencode_compatible_history_from_conn`], this entry point +/// never collects a session-sized part vector. Replay drivers call it only +/// after the row's compact content hash changed. +#[allow(clippy::too_many_arguments)] +pub(crate) fn replay_chunk_from_part_json( + session_id: &str, + provider_slug: &str, + sequence: usize, + part_id: String, + message_id: String, + role: String, + raw_data: &str, + time_created: i64, +) -> Result, String> { + let part = serde_json::from_str::(raw_data) + .map_err(|err| format!("Failed to parse {provider_slug} replay part {part_id}: {err}"))?; + let row = OpenCodePartRow { + part_id, + message_id, + role, + part, + time_created, + }; + Ok(part_row_to_chunk(session_id, provider_slug, sequence, &row)) +} + #[cfg(test)] fn load_opencode_history_from_conn( conn: &Connection, diff --git a/src-tauri/crates/orgtrack-core/src/sources/opencode/history/sync.rs b/src-tauri/crates/orgtrack-core/src/sources/opencode/history/sync.rs index 41e612a4d..4c02db226 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/opencode/history/sync.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/opencode/history/sync.rs @@ -3,6 +3,13 @@ use super::*; pub(super) fn sync_opencode_history_cache(cache_conn: &mut Connection) -> Result<(), String> { + sync_opencode_history_cache_inner(cache_conn, true) +} + +pub(super) fn sync_opencode_history_cache_inner( + cache_conn: &mut Connection, + include_legacy_impact: bool, +) -> Result<(), String> { let Some((conn, db_path)) = open_opencode_db()? else { imported_cache::sync_source_cache_from_conn( cache_conn, @@ -49,14 +56,7 @@ pub(super) fn sync_opencode_history_cache(cache_conn: &mut Connection) -> Result .into_iter() .filter(|meta| changed_ids.contains(&meta.source_session_id)) { - let session_id = format!("{OPENCODE_SESSION_PREFIX}{}", meta.source_session_id); - let chunks = load_opencode_compatible_history_from_conn( - &conn, - &session_id, - &meta.source_session_id, - OPENCODE_PROVIDER_SLUG, - )?; - meta.impact = imported_history::impact_from_edit_chunks(&chunks); + populate_opencode_impact(&conn, cache_conn, &mut meta, include_legacy_impact)?; inputs.push(session_meta_to_cache_input( meta, &container_parent_ids, @@ -66,6 +66,33 @@ pub(super) fn sync_opencode_history_cache(cache_conn: &mut Connection) -> Result imported_cache::sync_source_cache_from_conn(cache_conn, SOURCE_OPENCODE, live_ids, inputs) } +pub(super) fn populate_opencode_impact( + source_conn: &Connection, + cache_conn: &Connection, + meta: &mut OpenCodeSessionMeta, + include_legacy_impact: bool, +) -> Result<(), String> { + if include_legacy_impact { + let session_id = format!("{OPENCODE_SESSION_PREFIX}{}", meta.source_session_id); + meta.impact = load_opencode_compatible_impact_from_conn( + source_conn, + &session_id, + &meta.source_session_id, + OPENCODE_PROVIDER_SLUG, + )?; + } else if let Some(cached) = imported_cache::query_cached_session_from_conn( + cache_conn, + SOURCE_OPENCODE, + &meta.source_session_id, + )? { + // Catalog refresh preserves the compact projection already published + // for this session. It never replays historical tool rows merely + // because the shared DB/WAL changed. + meta.impact = cached.impact; + } + Ok(()) +} + fn opencode_meta_signature(meta: &OpenCodeSessionMeta) -> ImportedHistoryRecordSignature { ImportedHistoryRecordSignature { source_session_id: meta.source_session_id.clone(), diff --git a/src-tauri/crates/orgtrack-core/src/sources/opencode/history_tests.rs b/src-tauri/crates/orgtrack-core/src/sources/opencode/history_tests.rs index c067661a3..afe5cf8bc 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/opencode/history_tests.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/opencode/history_tests.rs @@ -672,3 +672,45 @@ fn ignores_mutual_parent_cycle() { assert!(inputs.iter().all(|input| input.listable)); assert!(inputs.iter().all(|input| input.parent_session_id.is_none())); } + +#[test] +fn catalog_refresh_does_not_query_thirty_mib_part_bodies() { + use crate::store::sqlite::SqliteRecordStore; + + let source = Connection::open_in_memory().expect("source DB"); + source + .execute_batch( + "CREATE TABLE part (data BLOB); + INSERT INTO part(data) VALUES (zeroblob(31457280));", + ) + .expect("create large body fixture"); + // Deliberately omit the message/session schema required by the legacy + // impact loader. A catalog-only projection succeeds only if it never + // touches the source's historical part rows. + let cache = Connection::open_in_memory().expect("cache DB"); + SqliteRecordStore::init_tables(&cache).expect("cache schema"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source-cache schema"); + let mut meta = OpenCodeSessionMeta { + source_session_id: "large-session".to_string(), + source_path: "/tmp/opencode.db".to_string(), + source_record_key: "large-session".to_string(), + source_mtime_ms: 1, + source_size_bytes: 30 * 1024 * 1024, + source_fingerprint: "large".to_string(), + title: "Large session".to_string(), + directory: "/work/opencode".to_string(), + model: Some("gpt-5".to_string()), + input_tokens: 0, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + time_created: 1, + time_updated: 2, + parent_id: None, + impact: ImportedHistoryImpactStats::default(), + }; + + populate_opencode_impact(&source, &cache, &mut meta, false) + .expect("catalog projection must not query part bodies"); + assert_eq!(meta.impact.files_changed, 0); +} diff --git a/src-tauri/crates/orgtrack-core/src/sources/qoder/history.rs b/src-tauri/crates/orgtrack-core/src/sources/qoder/history.rs index 4a0ca0e34..c78c3df07 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/qoder/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/qoder/history.rs @@ -13,6 +13,7 @@ use std::collections::HashMap; use std::fs; +use std::io::{BufRead, BufReader, Read}; use std::path::{Path, PathBuf}; use core_types::activity::ActivityChunk; @@ -30,6 +31,8 @@ use crate::sources::imported_history::{ ImportedHistorySessionRow, ImportedToolCall, }; +const CATALOG_PREFIX_BYTES: u64 = 1024 * 1024; + pub const QODER_SESSION_PREFIX: &str = "qoderapp-"; const QODER_PROVIDER_SLUG: &str = "qoder"; // Version 2 derives per-session file impact from the chat-editing snapshot @@ -148,6 +151,10 @@ pub fn load_qoder_history_for_session( )) } +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + sync_qoder_history_cache(conn) +} + fn sync_qoder_history_cache(conn: &mut Connection) -> Result<(), String> { let discovered = discover_qoder_history_records()?; let signatures = discovered @@ -158,10 +165,20 @@ fn sync_qoder_history_cache(conn: &mut Connection) -> Result<(), String> { imported_cache::changed_records_from_conn(conn, SOURCE_QODER, &discovered, |record| { record.signature() })?; - let inputs = changed - .into_iter() - .map(|record| session_meta_to_cache_input(parse_qoder_session_meta(record))) - .collect(); + let mut inputs = Vec::new(); + for discovered in changed { + if imported_cache::advance_cached_catalog_record_from_conn( + conn, + SOURCE_QODER, + &discovered.record, + None, + )? { + continue; + } + inputs.push(session_meta_to_cache_input(parse_qoder_session_meta( + discovered, + ))); + } imported_cache::sync_source_cache_from_conn( conn, SOURCE_QODER, @@ -293,7 +310,6 @@ fn quest_task_fingerprint(task: &QoderQuestTask) -> String { fn parse_qoder_session_meta(discovered: &QoderDiscoveredRecord) -> QoderHistoryMeta { let record = &discovered.record; let snapshot = discovered.snapshot.as_ref(); - let transcript = read_qoder_transcript(&record.source_path).unwrap_or_default(); // The signature mtime is nanoseconds (see `file_metadata_signature`); // scale it down where a real epoch-ms value is needed. @@ -315,7 +331,7 @@ fn parse_qoder_session_meta(discovered: &QoderDiscoveredRecord) -> QoderHistoryM .find(|value| !value.is_empty()) .map(str::to_string) }) - .or_else(|| first_user_text(&transcript)) + .or_else(|| first_user_text_from_path(&record.source_path)) .map(|value| imported_history::truncate_name(&value, 200)) .unwrap_or_else(|| record.source_session_id.clone()); @@ -325,9 +341,9 @@ fn parse_qoder_session_meta(discovered: &QoderDiscoveredRecord) -> QoderHistoryM .map(str::to_string); let session_id = format!("{QODER_SESSION_PREFIX}{}", record.source_session_id); - // Edits never appear in the transcript, so the +/- stats come from the - // chat-editing snapshot store; the transcript-derived impact is kept as a - // fallback in case a future Qoder version starts persisting tool blocks. + // Edits live in Qoder's compact snapshot store. Catalog refresh must not + // deserialize the whole JSONL merely to search for a hypothetical + // fallback edit block; the bounded replay adapter owns transcript bodies. let task_dir_name = record .source_session_id .split_once('/') @@ -337,11 +353,7 @@ fn parse_qoder_session_meta(discovered: &QoderDiscoveredRecord) -> QoderHistoryM task_dir_name, snapshot.map(|task| task.id.as_str()), ); - let impact = if edit_impact.files_changed > 0 { - edit_impact - } else { - imported_history::impact_from_edit_chunks(&transcript_to_chunks(&session_id, &transcript)) - }; + let impact = edit_impact; QoderHistoryMeta { source_session_id: record.source_session_id.clone(), @@ -597,6 +609,20 @@ fn first_user_text(transcript: &[QoderTranscriptLine]) -> Option { None } +fn first_user_text_from_path(path: &Path) -> Option { + let file = fs::File::open(path).ok()?; + for line in BufReader::new(file.take(CATALOG_PREFIX_BYTES)).lines() { + let Ok(line) = line else { continue }; + let Ok(parsed) = serde_json::from_str::(line.trim()) else { + continue; + }; + if let Some(text) = first_user_text(std::slice::from_ref(&parsed)) { + return Some(text); + } + } + None +} + /// Flatten a `tool_result.content` value (string, array of blocks, or object) /// into readable text, capped so a huge output can't bloat the payload. fn value_to_text(value: Option<&Value>) -> String { diff --git a/src-tauri/crates/orgtrack-core/src/sources/qoder/log_enrichment.rs b/src-tauri/crates/orgtrack-core/src/sources/qoder/log_enrichment.rs index f8419d8ae..c2899855c 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/qoder/log_enrichment.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/qoder/log_enrichment.rs @@ -28,6 +28,7 @@ use std::collections::HashMap; use std::fs; +use std::io::{BufRead, BufReader, Read}; use std::path::{Path, PathBuf}; use chrono::{Local, NaiveDateTime, TimeZone}; @@ -40,6 +41,17 @@ use crate::sources::imported_history::{ use super::history::MAX_TOOL_OUTPUT_CHARS; +#[cfg(test)] +thread_local! { + static QODER_LOG_PATH_OVERRIDE: std::cell::RefCell>> = const { + std::cell::RefCell::new(None) + }; +} + +type SnapshotContents = (String, String); +type EditSnapshots = HashMap; +type EditSnapshotLoader<'a> = dyn Fn(&str) -> EditSnapshots + 'a; + const ACP_PROGRESS_MARKER: &str = "[ChatSessionService] ACP progress: "; const SUBAGENT_MARKER: &str = "[SubAgentService] Registered SubAgent: "; const TOOL_INVOKE_MARKER: &str = "[ToolInvokeHandlerContribution] Tool invoke request: "; @@ -48,13 +60,13 @@ const FILE_CHANGE_MARKER: &str = "[FileChangeTracking] "; const SESSION_ID_SUFFIX: &str = ".session.execution"; /// Pad around a session's first/last ACP event when window-attributing /// invoke lines with no content signal. -const WINDOW_PAD_MS: i64 = 2_000; +pub(crate) const WINDOW_PAD_MS: i64 = 2_000; /// An invoke usually trails its ACP `tool_call` event by well under a second; /// pair them within this window to recover the call id. -const CALL_ID_PAIR_MS: i64 = 2_500; +pub(crate) const CALL_ID_PAIR_MS: i64 = 2_500; #[derive(Debug, Clone)] -enum LogEvent { +pub(crate) enum LogEvent { /// Any ACP progress line; `tool_call_id` set only for `type=tool_call`. Acp { ts_ms: i64, @@ -88,14 +100,12 @@ enum LogEvent { /// Which session an invoke's args point at, judged purely by its paths. #[derive(Debug, PartialEq)] -enum ContentSignal { +pub(crate) enum ContentSignal { Ours, Theirs, Silent, } -type EditSnapshotMap = HashMap; - /// Enrich a session's text-only chunks with the tool trajectory recovered /// from Qoder's launch logs. `task_dir_name`/`project_dir_name` are the /// conversation-history composite id halves; `workspace_path` is the @@ -109,8 +119,8 @@ pub(super) fn enrich_with_agent_log( ) -> Vec { let mut events = Vec::new(); for log_path in qoder_launch_log_paths() { - if let Ok(content) = fs::read_to_string(&log_path) { - parse_launch_log(&content, &mut events); + if let Ok(file) = fs::File::open(&log_path) { + parse_launch_log_reader(BufReader::new(file), &mut events); } } enrich_chunks_with_events( @@ -131,7 +141,7 @@ fn enrich_chunks_with_events( workspace_path: Option<&str>, chunks: Vec, events: &[LogEvent], - edit_snapshots: &dyn Fn(&str) -> EditSnapshotMap, + edit_snapshots: &EditSnapshotLoader<'_>, ) -> Vec { // Resolve the truncated dir name to the full task id seen in the logs. // Two distinct matches would mean we cannot tell the sessions apart — @@ -359,7 +369,7 @@ fn enrich_chunks_with_events( /// Judge which session an invoke belongs to from the paths in its args. /// `file_path` under a project cache dir or `cwd` inside the workspace are /// decisive; paths that name a *different* project cache dir disown it. -fn invoke_content_signal( +pub(crate) fn invoke_content_signal( args: &Value, project_dir_name: &str, workspace_path: Option<&str>, @@ -399,7 +409,7 @@ fn paired_call_id(our_acp_calls: &[(i64, &str)], ts_ms: i64) -> Option { /// Map Qoder tool names onto the canonical functions the replay UI has typed /// cards for; unknown names pass through as generic cards. -fn canonical_tool_name(name: &str) -> String { +pub(crate) fn canonical_tool_name(name: &str) -> String { match name { "read_file" => imported_history::FUNCTION_READ_FILE.to_string(), "run_in_terminal" => imported_history::FUNCTION_RUN_COMMAND_LINE.to_string(), @@ -412,7 +422,7 @@ fn canonical_tool_name(name: &str) -> String { } /// Reshape args into the keys the frontend extractors read. -fn normalized_args(name: &str, args: &Value) -> Value { +pub(crate) fn normalized_args(name: &str, args: &Value) -> Value { if name == "run_in_terminal" { if let Some(command) = args.get("command") { let mut merged = args.clone(); @@ -444,19 +454,19 @@ fn normalized_args(name: &str, args: &Value) -> Value { /// When a `read_file` targets an `agent-tools` spill file, its content is the /// missing tool OUTPUT — attach it (capped). Other paths are live workspace /// files that may have changed since; leave those empty. -fn spill_file_output(args: &Value) -> String { +pub(crate) fn spill_file_output(args: &Value) -> String { let Some(path) = args.get("file_path").and_then(Value::as_str) else { return String::new(); }; if !path.contains("/agent-tools/") { return String::new(); } - let Ok(content) = fs::read_to_string(Path::new(path)) else { + let Some((content, truncated)) = read_text_prefix(Path::new(path), MAX_TOOL_OUTPUT_CHARS) + else { return String::new(); }; - if content.chars().count() > MAX_TOOL_OUTPUT_CHARS { - let truncated: String = content.chars().take(MAX_TOOL_OUTPUT_CHARS).collect(); - format!("{truncated}\n… (truncated)") + if truncated { + format!("{content}\n… (truncated)") } else { content.trim_end().to_string() } @@ -464,127 +474,159 @@ fn spill_file_output(args: &Value) -> String { /// Parse one launch log (agent.log or an exthost output log — the markers are /// disjoint, so one parser handles both). +#[cfg(test)] fn parse_launch_log(content: &str, events: &mut Vec) { - let mut lines = content.lines().peekable(); + parse_launch_log_reader(std::io::Cursor::new(content.as_bytes()), events); +} + +fn parse_launch_log_reader(reader: impl BufRead, events: &mut Vec) { + let mut lines = reader.lines().peekable(); while let Some(line) = lines.next() { - let Some(ts_ms) = parse_line_timestamp_ms(line) else { - continue; + let Ok(line) = line else { + break; }; - if let Some(rest) = substring_after(line, ACP_PROGRESS_MARKER) { - // `, rid=, type=[, toolCallId=]` - let mut session_task_id = String::new(); - let mut event_type = ""; - let mut tool_call_id = ""; - for (index, part) in rest.split(", ").enumerate() { - if index == 0 { - session_task_id = part.trim().trim_end_matches(SESSION_ID_SUFFIX).to_string(); - } else if let Some(value) = part.trim().strip_prefix("type=") { - event_type = value; - } else if let Some(value) = part.trim().strip_prefix("toolCallId=") { - tool_call_id = value; - } - } - if session_task_id.is_empty() { - continue; - } - let tool_call_id = (event_type == "tool_call" && !tool_call_id.is_empty()) - .then(|| tool_call_id.to_string()); - events.push(LogEvent::Acp { - ts_ms, - session_task_id, - tool_call_id, - }); - } else if let Some(rest) = substring_after(line, SUBAGENT_MARKER) { - let Ok(payload) = serde_json::from_str::(rest.trim()) else { - continue; - }; - let field = |key: &str| { - payload - .get(key) - .and_then(Value::as_str) - .unwrap_or_default() - .to_string() - }; - let session_task_id = field("parentSessionId") - .trim_end_matches(SESSION_ID_SUFFIX) - .to_string(); - let tool_call_id = field("parentToolCallId"); - if session_task_id.is_empty() || tool_call_id.is_empty() { - continue; - } - events.push(LogEvent::Subagent { - ts_ms, - session_task_id, - tool_call_id, - agent_type: field("agentType"), - description: field("rawInputDescription"), - prompt: field("prompt"), - }); - } else if let Some(rest) = substring_after(line, TOOL_INVOKE_MARKER) { - // agent.log: `, , {args json}` — args may contain - // ", " so split only the first two fields. - let Some((_rid, rest)) = rest.split_once(", ") else { - continue; - }; - let Some((name, args_raw)) = rest.split_once(", ") else { - continue; - }; - push_invoke(events, ts_ms, name, args_raw); - } else if let Some(rest) = substring_after(line, FILE_CHANGE_MARKER) { - // ` | source=agent | session=, request= | Agent ` - // (the marker also logs a pipe-less "Agent file tracked:" shape — - // the parts count filters that out). - let parts: Vec<&str> = rest.split(" | ").collect(); - if parts.len() < 4 || !parts.contains(&"source=agent") { - continue; - } - let path = parts[0].trim(); - let session_dir_name = parts - .iter() - .flat_map(|part| part.split(", ")) - .find_map(|field| field.trim().strip_prefix("session=")); - let operation = parts - .last() - .and_then(|part| part.trim().strip_prefix("Agent ")) - .map(str::trim) - .filter(|op| !op.is_empty()); - let (Some(session_dir_name), Some(operation)) = (session_dir_name, operation) else { - continue; - }; - if path.is_empty() { - continue; - } - events.push(LogEvent::FileEdit { - ts_ms, - session_dir_name: session_dir_name.to_string(), - path: path.to_string(), - operation: operation.to_string(), - }); - } else if let Some(name) = substring_after(line, EXTHOST_INVOKE_MARKER) { - // exthost log: ` [info] ToolInvoke : ` with the args - // JSON on the following line. - let Some(args_line) = lines.peek() else { - continue; - }; - if args_line.trim_start().starts_with('{') { - let args_raw = lines.next().unwrap_or_default(); - push_invoke(events, ts_ms, name, args_raw); - } + let next_line = lines + .peek() + .and_then(|line| line.as_ref().ok()) + .map(String::as_str); + let (event, consumed_following) = parse_launch_log_record(&line, next_line); + if consumed_following { + let _ = lines.next(); + } + if let Some(event) = event { + events.push(event); } } } -fn push_invoke(events: &mut Vec, ts_ms: i64, name: &str, args_raw: &str) { +/// Parse one complete Qoder log record. Exthost tool invokes span two lines; +/// callers must leave the first line unacknowledged when the second line is a +/// torn tail. The returned boolean tells a streaming caller whether the second +/// complete line belongs to this record. +pub(crate) fn parse_launch_log_record( + line: &str, + following_line: Option<&str>, +) -> (Option, bool) { + let Some(ts_ms) = parse_line_timestamp_ms(line) else { + return (None, false); + }; + let event = if let Some(rest) = substring_after(line, ACP_PROGRESS_MARKER) { + // `, rid=, type=[, toolCallId=]` + let mut session_task_id = String::new(); + let mut event_type = ""; + let mut tool_call_id = ""; + for (index, part) in rest.split(", ").enumerate() { + if index == 0 { + session_task_id = part.trim().trim_end_matches(SESSION_ID_SUFFIX).to_string(); + } else if let Some(value) = part.trim().strip_prefix("type=") { + event_type = value; + } else if let Some(value) = part.trim().strip_prefix("toolCallId=") { + tool_call_id = value; + } + } + if session_task_id.is_empty() { + return (None, false); + } + let tool_call_id = (event_type == "tool_call" && !tool_call_id.is_empty()) + .then(|| tool_call_id.to_string()); + Some(LogEvent::Acp { + ts_ms, + session_task_id, + tool_call_id, + }) + } else if let Some(rest) = substring_after(line, SUBAGENT_MARKER) { + let Ok(payload) = serde_json::from_str::(rest.trim()) else { + return (None, false); + }; + let field = |key: &str| { + payload + .get(key) + .and_then(Value::as_str) + .unwrap_or_default() + .to_string() + }; + let session_task_id = field("parentSessionId") + .trim_end_matches(SESSION_ID_SUFFIX) + .to_string(); + let tool_call_id = field("parentToolCallId"); + if session_task_id.is_empty() || tool_call_id.is_empty() { + return (None, false); + } + Some(LogEvent::Subagent { + ts_ms, + session_task_id, + tool_call_id, + agent_type: field("agentType"), + description: field("rawInputDescription"), + prompt: field("prompt"), + }) + } else if let Some(rest) = substring_after(line, TOOL_INVOKE_MARKER) { + // agent.log: `, , {args json}` — args may contain + // ", " so split only the first two fields. + let Some((_rid, rest)) = rest.split_once(", ") else { + return (None, false); + }; + let Some((name, args_raw)) = rest.split_once(", ") else { + return (None, false); + }; + parse_invoke(ts_ms, name, args_raw) + } else if let Some(rest) = substring_after(line, FILE_CHANGE_MARKER) { + // ` | source=agent | session=, request= | Agent ` + // (the marker also logs a pipe-less "Agent file tracked:" shape — + // the parts count filters that out). + let parts: Vec<&str> = rest.split(" | ").collect(); + if parts.len() < 4 || !parts.contains(&"source=agent") { + return (None, false); + } + let path = parts[0].trim(); + let session_dir_name = parts + .iter() + .flat_map(|part| part.split(", ")) + .find_map(|field| field.trim().strip_prefix("session=")); + let operation = parts + .last() + .and_then(|part| part.trim().strip_prefix("Agent ")) + .map(str::trim) + .filter(|op| !op.is_empty()); + let (Some(session_dir_name), Some(operation)) = (session_dir_name, operation) else { + return (None, false); + }; + if path.is_empty() { + return (None, false); + } + Some(LogEvent::FileEdit { + ts_ms, + session_dir_name: session_dir_name.to_string(), + path: path.to_string(), + operation: operation.to_string(), + }) + } else if let Some(name) = substring_after(line, EXTHOST_INVOKE_MARKER) { + // exthost log: ` [info] ToolInvoke : ` with the args + // JSON on the following line. + let Some(args_line) = following_line else { + return (None, false); + }; + if args_line.trim_start().starts_with('{') { + return (parse_invoke(ts_ms, name, args_line), true); + } + None + } else { + None + }; + (event, false) +} + +fn parse_invoke(ts_ms: i64, name: &str, args_raw: &str) -> Option { let name = name.trim(); if name.is_empty() { - return; + return None; } let args = serde_json::from_str(args_raw.trim()).unwrap_or(Value::Null); - events.push(LogEvent::ToolInvoke { + Some(LogEvent::ToolInvoke { ts_ms, name: name.to_string(), args, - }); + }) } /// Real before/after contents for the files a session edited, from VS Code's @@ -593,7 +635,7 @@ fn push_invoke(events: &mut Vec, ts_ms: i64, name: &str, args_raw: &st /// holds a `state.json` mapping each resource to `originalHash`/`currentHash`, /// with the content-addressed snapshot bodies under `contents/`. /// Returns `path → (old_content, new_content)`. -fn edit_snapshots_for_task(task_id: &str) -> HashMap { +pub(crate) fn edit_snapshots_for_task(task_id: &str) -> EditSnapshots { edit_snapshots(task_id, Some(task_id)) } @@ -654,7 +696,7 @@ fn numstat_between(old_content: &str, new_content: &str) -> (i64, i64) { /// Change-signature of the session's edit store (`state.json` mtime+size per /// workspace). Folded into the discovery fingerprint so edits that land after /// a sync re-parse the session even when the transcript itself is unchanged. -pub(super) fn edit_store_signature(task_dir_name: &str, full_task_id: Option<&str>) -> String { +pub(crate) fn edit_store_signature(task_dir_name: &str, full_task_id: Option<&str>) -> String { edit_store_paths(&qoder_workspace_storage_dirs(), task_dir_name, full_task_id) .iter() .filter_map(|dir| { @@ -775,14 +817,30 @@ fn read_snapshot_content(session_dir: &Path, hash: &str) -> String { if hash.is_empty() { return String::new(); } - let Ok(content) = fs::read_to_string(session_dir.join("contents").join(hash)) else { + let Some((content, _truncated)) = read_text_prefix( + &session_dir.join("contents").join(hash), + MAX_TOOL_OUTPUT_CHARS, + ) else { return String::new(); }; - if content.chars().count() > MAX_TOOL_OUTPUT_CHARS { - content.chars().take(MAX_TOOL_OUTPUT_CHARS).collect() - } else { - content - } + content +} + +/// Read at most enough bytes to recover `max_chars` UTF-8 characters. Qoder +/// spill/snapshot files can be arbitrarily large; enrichment needs only the +/// compatibility preview and must never allocate the complete body. +fn read_text_prefix(path: &Path, max_chars: usize) -> Option<(String, bool)> { + let file = fs::File::open(path).ok()?; + let total_bytes = file.metadata().ok()?.len(); + let max_bytes = max_chars.saturating_mul(4).saturating_add(1); + let known_bytes = usize::try_from(total_bytes).unwrap_or(usize::MAX); + let mut bytes = Vec::with_capacity(max_bytes.min(known_bytes)); + file.take(max_bytes as u64).read_to_end(&mut bytes).ok()?; + let decoded = String::from_utf8_lossy(&bytes); + let mut chars = decoded.chars(); + let content = chars.by_ref().take(max_chars).collect::(); + let truncated = chars.next().is_some() || total_bytes > bytes.len() as u64; + Some((content, truncated)) } /// `file:///a/b%20c.py` → `/a/b c.py`. @@ -831,19 +889,13 @@ fn substring_after<'a>(line: &'a str, marker: &str) -> Option<&'a str> { /// Every trajectory-bearing log across launch folders: /// `/Qoder/logs//questWindow/agent.log` and /// `/Qoder/logs//questWindow/exthost/output_logging_*/1-Qoder.log`. -fn qoder_launch_log_paths() -> Vec { - let mut roots = Vec::new(); - if let Some(data) = dirs::data_dir() { - roots.push(data); +pub(crate) fn qoder_launch_log_paths() -> Vec { + #[cfg(test)] + if let Some(paths) = QODER_LOG_PATH_OVERRIDE.with(|slot| slot.borrow().clone()) { + return paths; } - if let Some(config) = dirs::config_dir() { - roots.push(config); - } - roots.sort(); - roots.dedup(); - let mut logs = Vec::new(); - for root in roots { + for root in qoder_data_roots() { let logs_dir = root.join("Qoder").join("logs"); let Ok(entries) = fs::read_dir(&logs_dir) else { continue; @@ -873,9 +925,81 @@ fn qoder_launch_log_paths() -> Vec { } } } + logs.sort(); + logs.dedup(); logs } +/// Existing directories whose mutations can change Qoder replay enrichment. +/// These paths stay backend-only; the renderer sees only source-neutral replay +/// invalidations. +pub(crate) fn replay_sidecar_watch_paths(task_dir_name: &str) -> Vec { + let mut paths = qoder_data_roots() + .into_iter() + .map(|root| root.join("Qoder").join("logs")) + .filter(|path| path.is_dir()) + .collect::>(); + paths.extend(edit_store_paths( + &qoder_workspace_storage_dirs(), + task_dir_name, + None, + )); + paths.sort(); + paths.dedup(); + paths +} + +fn qoder_data_roots() -> Vec { + let mut roots = Vec::new(); + if let Some(data) = dirs::data_dir() { + roots.push(data); + } + if let Some(config) = dirs::config_dir() { + roots.push(config); + } + roots.sort(); + roots.dedup(); + roots +} + +#[cfg(test)] +pub(crate) fn with_qoder_log_paths_for_test(paths: Vec, run: impl FnOnce() -> T) -> T { + struct Reset; + impl Drop for Reset { + fn drop(&mut self) { + QODER_LOG_PATH_OVERRIDE.with(|slot| *slot.borrow_mut() = None); + } + } + + QODER_LOG_PATH_OVERRIDE.with(|slot| *slot.borrow_mut() = Some(paths)); + let reset = Reset; + let output = run(); + drop(reset); + output +} + +#[cfg(test)] +pub(crate) fn enrich_chunks_from_log_fixture( + session_id: &str, + task_dir_name: &str, + project_dir_name: &str, + workspace_path: Option<&str>, + chunks: Vec, + content: &str, +) -> Vec { + let mut events = Vec::new(); + parse_launch_log(content, &mut events); + enrich_chunks_with_events( + session_id, + task_dir_name, + project_dir_name, + workspace_path, + chunks, + &events, + &|_| HashMap::new(), + ) +} + #[cfg(test)] #[path = "log_enrichment_tests.rs"] mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/qoder/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/qoder/mod.rs index 89f77a982..e5aa2e2e4 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/qoder/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/qoder/mod.rs @@ -10,4 +10,4 @@ //! matched to a transcript by task-id prefix + workspace basename; the snapshot //! is enrichment only — discovery works from the JSONL store alone. pub mod history; -mod log_enrichment; +pub(crate) mod log_enrichment; diff --git a/src-tauri/crates/orgtrack-core/src/sources/qoder_cli/history.rs b/src-tauri/crates/orgtrack-core/src/sources/qoder_cli/history.rs index a5e066251..064af389c 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/qoder_cli/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/qoder_cli/history.rs @@ -25,6 +25,10 @@ fn config() -> AnthropicJsonlSource { } } +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + anthropic_jsonl::refresh_catalog(&config(), conn) +} + pub fn list_qoder_cli_history_sessions_paginated( conn: &mut Connection, limit: usize, diff --git a/src-tauri/crates/orgtrack-core/src/sources/registry.rs b/src-tauri/crates/orgtrack-core/src/sources/registry.rs index 44e757cab..d9a171794 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/registry.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/registry.rs @@ -1,21 +1,15 @@ //! Unified source registry and scan dispatch. //! -//! Every imported-history source ships a `list_*_history_sessions_paginated` -//! loader that fuses three steps behind one call: discover the provider's -//! sessions on disk, incrementally upsert them into the source cache tables of -//! the passed connection, and read back one page of normalized -//! [`ImportedHistorySessionPage`] rows. Those loaders live next to each -//! provider's parser, which is the right home for them — but the *set* of -//! providers, and the routing from a stable `source` id to the right loader, -//! was until now open-coded in every host (the desktop app's -//! `history_commands`, and any CLI). This module is the single place that owns -//! that mapping. +//! Every imported-history source owns a compact catalog refresher and a +//! storage-specific bounded replay adapter. Registry scans deliberately call +//! only the catalog refresher, then page ORGII's compact cache; they never call +//! the legacy transcript loaders that materialize complete histories. //! -//! The read side already has its router: -//! [`super::imported_history::load_activity_chunks_for_session`] takes a -//! `session_id` and returns the session's [`core_types::activity::ActivityChunk`] -//! stream regardless of which provider owns it. This module is the write/scan -//! twin — enumerate the providers, scan one (or all) of them into a +//! The read side already has its bounded router: +//! [`super::imported_history::router::scan_activity_chunks_for_session`] takes +//! a `session_id` and returns compact replay pages regardless of which provider +//! owns it. This module is the write/scan twin — enumerate the providers, scan +//! one (or all) of them into a //! connection, and let the analytics layer ([`crate::usage_dashboard`], //! [`crate::session_usage`]) read the result. Hosts that want a bare, //! app-independent store (tests, the `orgtrack` CLI) get the whole loading @@ -27,25 +21,17 @@ use rusqlite::Connection; -use super::imported_history::{metadata, ImportedHistorySessionPage, ImportedHistorySessionRow}; -use super::{ - claude_code, cline, codex, cursor_cli, cursor_ide, mimo_code, omp, opencode, qoder, qoder_cli, - trae, warp, windsurf, workbuddy, zcode, +use super::imported_history::{ + cache, catalog, metadata, replay::ImportedHistorySourceId, ImportedHistorySessionPage, }; -/// Signature every provider's paginated session loader shares. The `&mut -/// Connection` is the source cache store the scan writes through; `limit` / -/// `offset` page the returned rows (the full provider set is always synced to -/// the cache regardless of the page window). -type ScanFn = fn(&mut Connection, usize, usize) -> Result; - /// One registered provider: its stable `source` id (matches the /// `metadata::SOURCE_*` constants and the `source` column written to every /// cache table), a human label for CLI/UI listing, and its scan loader. pub struct RegisteredSource { pub id: &'static str, pub label: &'static str, - scan: ScanFn, + source: ImportedHistorySourceId, } impl RegisteredSource { @@ -57,7 +43,8 @@ impl RegisteredSource { limit: usize, offset: usize, ) -> Result { - (self.scan)(conn, limit, offset) + catalog::refresh_source(conn, self.source)?; + cache::query_imported_session_page_from_conn(conn, self.id, limit, offset) } } @@ -70,122 +57,80 @@ static REGISTERED: &[RegisteredSource] = &[ RegisteredSource { id: metadata::SOURCE_CLAUDE_CODE, label: "Claude Code", - scan: claude_code::history::list_claude_code_history_sessions_paginated, + source: ImportedHistorySourceId::ClaudeCode, }, RegisteredSource { id: metadata::SOURCE_CODEX_APP, label: "Codex", - scan: codex::app::list_codex_app_sessions_paginated, + source: ImportedHistorySourceId::CodexApp, }, RegisteredSource { id: metadata::SOURCE_CURSOR_CLI, label: "Cursor CLI", - scan: cursor_cli::history::list_cursor_cli_history_sessions_paginated, + source: ImportedHistorySourceId::CursorCli, }, RegisteredSource { id: metadata::SOURCE_CURSOR_IDE, label: "Cursor IDE", - scan: scan_cursor_ide, + source: ImportedHistorySourceId::CursorIde, }, RegisteredSource { id: metadata::SOURCE_OPENCODE, label: "OpenCode", - scan: opencode::history::list_opencode_history_sessions_paginated, + source: ImportedHistorySourceId::OpenCode, }, RegisteredSource { id: metadata::SOURCE_CLINE, label: "Cline", - scan: cline::history::list_cline_history_sessions_paginated, + source: ImportedHistorySourceId::Cline, }, RegisteredSource { id: metadata::SOURCE_WINDSURF, label: "Windsurf", - scan: windsurf::history::list_windsurf_history_sessions_paginated, + source: ImportedHistorySourceId::Windsurf, }, RegisteredSource { id: metadata::SOURCE_WARP, label: "Warp", - scan: warp::history::list_warp_history_sessions_paginated, + source: ImportedHistorySourceId::Warp, }, RegisteredSource { id: metadata::SOURCE_TRAE, label: "Trae", - scan: trae::history::list_trae_history_sessions_paginated, + source: ImportedHistorySourceId::Trae, }, RegisteredSource { id: metadata::SOURCE_ZCODE, label: "ZCode", - scan: zcode::history::list_zcode_history_sessions_paginated, + source: ImportedHistorySourceId::ZCode, }, RegisteredSource { id: metadata::SOURCE_QODER, label: "Qoder", - scan: qoder::history::list_qoder_history_sessions_paginated, + source: ImportedHistorySourceId::Qoder, }, RegisteredSource { id: metadata::SOURCE_QODER_CLI, label: "Qoder CLI", - scan: qoder_cli::history::list_qoder_cli_history_sessions_paginated, + source: ImportedHistorySourceId::QoderCli, }, RegisteredSource { id: metadata::SOURCE_MIMO_CODE, label: "Mimo Code", - scan: mimo_code::history::list_mimo_code_history_sessions_paginated, + source: ImportedHistorySourceId::MimoCode, }, RegisteredSource { id: metadata::SOURCE_OMP, label: "OMP", - scan: omp::history::list_omp_history_sessions_paginated, + source: ImportedHistorySourceId::Omp, }, RegisteredSource { id: metadata::SOURCE_WORKBUDDY, label: "WorkBuddy", - scan: workbuddy::list_workbuddy_history_sessions_paginated, + source: ImportedHistorySourceId::WorkBuddy, }, ]; -/// Cursor IDE's loader predates the shared row type and returns its own -/// `CursorIdeSessionRow` (identical apart from carrying no parent-session -/// linkage). Normalize it here so the registry exposes one uniform page type. -fn scan_cursor_ide( - conn: &mut Connection, - limit: usize, - offset: usize, -) -> Result { - let page = cursor_ide::history::list_cursor_ide_sessions_paginated(conn, limit, offset)?; - Ok(ImportedHistorySessionPage { - has_more: page.has_more, - sessions: page - .sessions - .into_iter() - .map(|row| ImportedHistorySessionRow { - session_id: row.session_id, - name: row.name, - status: row.status, - created_at: row.created_at, - updated_at: row.updated_at, - category: row.category, - read_only: row.read_only, - model: row.model, - total_tokens: row.total_tokens, - background: row.background, - is_active: row.is_active, - repo_path: row.repo_path, - repo_root_path: row.repo_root_path, - repo_remote_urls: row.repo_remote_urls, - storage_path: row.storage_path, - repo_name: row.repo_name, - branch: row.branch, - files_changed: row.files_changed, - lines_added: row.lines_added, - lines_removed: row.lines_removed, - touched_files: row.touched_files, - parent_session_id: None, - }) - .collect(), - }) -} - /// Every provider the registry can scan, in display order. pub fn registered_sources() -> &'static [RegisteredSource] { REGISTERED @@ -234,6 +179,14 @@ mod tests { assert!(!source.label.is_empty(), "empty label for {}", source.id); assert!(seen.insert(source.id), "duplicate source id {}", source.id); } + let authoritative = ImportedHistorySourceId::ALL + .into_iter() + .map(ImportedHistorySourceId::as_str) + .collect::>(); + assert_eq!( + seen, authoritative, + "the scan registry and bounded replay registry must contain exactly the same sources" + ); } #[test] diff --git a/src-tauri/crates/orgtrack-core/src/sources/trae/history.rs b/src-tauri/crates/orgtrack-core/src/sources/trae/history.rs index 1bc4f4fe3..77f5cd983 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/trae/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/trae/history.rs @@ -11,7 +11,7 @@ use std::collections::HashSet; use std::fs; -use std::io::{BufRead, BufReader}; +use std::io::{BufRead, BufReader, Read}; use std::path::{Path, PathBuf}; use core_types::activity::ActivityChunk; @@ -29,6 +29,8 @@ use crate::sources::imported_history::{ ImportedHistorySessionRow, }; +const CATALOG_PREFIX_BYTES: u64 = 1024 * 1024; + pub const TRAE_SESSION_PREFIX: &str = "traeapp-"; const TRAE_PROVIDER_SLUG: &str = "trae"; // v2: re-parse fixes — `message_summary_time` now read as local (not UTC) @@ -112,6 +114,43 @@ pub fn load_trae_history_for_session( load_trae_history_from_path(session_id, &path) } +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + refresh_trae_catalog(conn) +} + +fn refresh_trae_catalog(conn: &mut Connection) -> Result<(), String> { + let discovered = discover_trae_history_records()?; + let signatures = discovered + .iter() + .map(ImportedHistoryDiscoveredRecord::signature) + .collect::>(); + let changed = + imported_cache::changed_records_from_conn(conn, SOURCE_TRAE, &discovered, |record| { + record.signature() + })?; + let session_index = if changed.is_empty() { + index::TraeSessionIndex::new() + } else { + index::load_trae_session_index() + }; + let mut inputs = Vec::new(); + for record in changed { + if imported_cache::advance_cached_catalog_record_from_conn(conn, SOURCE_TRAE, record, None)? + { + continue; + } + if let Some(meta) = parse_trae_session_meta(record, &session_index)? { + inputs.push(session_meta_to_cache_input(meta)); + } + } + imported_cache::sync_source_cache_from_conn( + conn, + SOURCE_TRAE, + imported_cache::live_ids_from_signatures(&signatures), + inputs, + ) +} + fn sync_trae_history_cache(conn: &mut Connection) -> Result<(), String> { let discovered = discover_trae_history_records()?; let signatures = discovered @@ -226,7 +265,7 @@ fn parse_trae_session_meta( record.source_path.display() ) })?; - let reader = BufReader::new(file); + let reader = BufReader::new(file.take(CATALOG_PREFIX_BYTES)); let mut created_at_ms = 0; let mut updated_at_ms = 0; diff --git a/src-tauri/crates/orgtrack-core/src/sources/warp/history.rs b/src-tauri/crates/orgtrack-core/src/sources/warp/history.rs index 4392120f3..170b00601 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/warp/history.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/warp/history.rs @@ -111,6 +111,7 @@ impl WarpConversationRecord { struct WarpTaskAnalysis { chunks: Vec, initial_query: Option, + #[cfg(test)] root_description: Option, model: Option, created_at_ms: Option, @@ -154,6 +155,10 @@ pub fn load_warp_history_for_session(session_id: &str) -> Result Result<(), String> { + sync_warp_history_cache(cache_conn) +} + fn sync_warp_history_cache(cache_conn: &mut Connection) -> Result<(), String> { let Some((source_conn, db_path)) = open_warp_db()? else { imported_cache::sync_source_cache_from_conn( @@ -177,18 +182,7 @@ fn sync_warp_history_cache(cache_conn: &mut Connection) -> Result<(), String> { let mut inputs = Vec::with_capacity(changed.len()); for record in changed { - let fallback_ms = parse_warp_timestamp_ms(&record.last_modified_at).unwrap_or(0); - let task_blobs = load_task_blobs(&source_conn, &record.conversation_id)?; - let analysis = analyze_task_blobs( - &format!("{WARP_SESSION_PREFIX}{}", record.conversation_id), - &task_blobs, - fallback_ms, - ); - inputs.push(conversation_to_cache_input( - record.clone(), - analysis, - &db_path, - )); + inputs.push(conversation_to_catalog_input(record.clone(), &db_path)); } imported_cache::sync_source_cache_from_conn( @@ -199,6 +193,81 @@ fn sync_warp_history_cache(cache_conn: &mut Connection) -> Result<(), String> { ) } +/// Build sidebar metadata from Warp's compact conversation/summary rows. +/// `agent_tasks.task` can contain an entire encoded transcript, so catalog +/// refresh never copies or decodes those BLOBs; the task-blob replay adapter +/// owns incremental body indexing. +fn conversation_to_catalog_input( + record: WarpConversationRecord, + db_path: &Path, +) -> ImportedHistoryCacheInput { + let signature = record.signature(db_path); + let summary = record + .summary_json + .as_deref() + .and_then(|raw| serde_json::from_str::(raw).ok()) + .unwrap_or_default(); + let data = serde_json::from_str::(&record.conversation_data_json) + .unwrap_or_default(); + let usage = data.conversation_usage_metadata.clone().unwrap_or_default(); + let total_tokens = usage + .token_usage + .iter() + .map(|item| { + i64::from(item.warp_tokens) + + i64::from(item.byok_tokens) + + i64::from(item.custom_endpoint_tokens) + }) + .sum(); + let model = usage + .token_usage + .iter() + .rev() + .map(|item| item.model_id.trim()) + .find(|model| !model.is_empty()) + .map(str::to_string); + let title = non_empty(Some(&summary.title)) + .or_else(|| non_empty(Some(&summary.initial_query))) + .or_else(|| non_empty(data.agent_name.as_deref())) + .unwrap_or_else(|| "Warp conversation".to_string()); + let updated_at_ms = parse_warp_timestamp_ms(&record.last_modified_at).unwrap_or(0); + let parent_session_id = non_empty(data.parent_conversation_id.as_deref()) + .map(|parent| format!("{WARP_SESSION_PREFIX}{parent}")); + let listable = + !data.is_remote_child && !summary.is_unlisted_auto_code_diff && record.task_count > 0; + let source_metadata_json = serde_json::to_string(&json!({ + "initialQuery": non_empty(Some(&summary.initial_query)), + "taskCount": record.task_count, + })) + .ok(); + + ImportedHistoryCacheInput { + source: SOURCE_WARP, + source_session_id: record.conversation_id.clone(), + session_id: format!("{WARP_SESSION_PREFIX}{}", record.conversation_id), + source_path: signature.source_path, + source_record_key: record.conversation_id, + source_mtime_ms: signature.source_mtime_ms, + source_size_bytes: signature.source_size_bytes, + source_fingerprint: signature.source_fingerprint, + parser_version: signature.parser_version, + name: imported_history::truncate_name(&title, 200), + created_at_ms: updated_at_ms, + updated_at_ms, + model, + input_tokens: total_tokens, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + repo_path: non_empty(summary.initial_working_directory.as_deref()), + branch: None, + impact: ImportedHistoryImpactStats::default(), + listable, + source_metadata_json, + parent_session_id, + } +} + fn list_conversation_records(conn: &Connection) -> Result, String> { if !table_exists(conn, "agent_conversations")? || !table_exists(conn, "agent_tasks")? { return Ok(Vec::new()); @@ -243,6 +312,7 @@ fn list_conversation_records(conn: &Connection) -> Result Option { }) } +#[cfg(test)] fn is_root_task(task: &Value) -> bool { field(task, &["dependencies"]) .and_then(|dependencies| field_str(dependencies, &["parentTaskId", "parent_task_id"])) diff --git a/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/mod.rs index ae31439b6..d4eb14c54 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/mod.rs @@ -174,6 +174,32 @@ pub fn load_windsurf_history_for_session(session_id: &str) -> Result Result<(), String> { + sync_windsurf_history_cache_inner(cache_conn, false) +} + +/// Normalize a single Windsurf KV bubble without hydrating the composer. +/// Bounded replay passes rows here only after their stable-key hash changes. +pub(crate) fn replay_chunk_from_bubble_json( + conn: &Connection, + session_id: &str, + _sequence: usize, + bubble_id: &str, + header_type: i64, + raw_json: &str, +) -> Result, String> { + let raw = serde_json::from_str::(raw_json) + .map_err(|err| format!("Failed to parse Windsurf replay bubble {bubble_id}: {err}"))?; + let bubble = OrderedBubble { + bubble_id: bubble_id.to_string(), + bubble_type: header_type, + raw, + }; + Ok(bubbles_to_chunks(conn, session_id, &[bubble]) + .into_iter() + .next()) +} + #[cfg(test)] #[path = "../history_tests.rs"] mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/parts.rs b/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/parts.rs index cbbe33945..6b9fddaf6 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/parts.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/parts.rs @@ -225,7 +225,7 @@ fn assistant_tool_bubble_to_chunk( Some(chunk) } -fn normalize_windsurf_tool_call(raw_name: &str, args: Value) -> (String, Value) { +pub(super) fn normalize_windsurf_tool_call(raw_name: &str, args: Value) -> (String, Value) { match raw_name { "shell" | "run_command" | "terminal" | "terminal_command" => ( imported_history::FUNCTION_RUN_COMMAND_LINE.to_string(), diff --git a/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/sync.rs b/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/sync.rs index 3c44d4961..759809f58 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/sync.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/windsurf/history/sync.rs @@ -3,6 +3,13 @@ use super::*; pub(super) fn sync_windsurf_history_cache(cache_conn: &mut Connection) -> Result<(), String> { + sync_windsurf_history_cache_inner(cache_conn, true) +} + +pub(super) fn sync_windsurf_history_cache_inner( + cache_conn: &mut Connection, + include_legacy_impact: bool, +) -> Result<(), String> { let Some((conn, db_path)) = open_windsurf_db() else { imported_cache::sync_source_cache_from_conn( cache_conn, @@ -14,8 +21,24 @@ pub(super) fn sync_windsurf_history_cache(cache_conn: &mut Connection) -> Result }; let (source_mtime_ms, source_size_bytes) = imported_paths::file_metadata_signature(&db_path, "Windsurf")?; - let metas = - list_windsurf_composer_meta_from_conn(&conn, &db_path, source_mtime_ms, source_size_bytes)?; + let mut metas = list_windsurf_composer_meta_from_conn_inner( + &conn, + &db_path, + source_mtime_ms, + source_size_bytes, + include_legacy_impact, + )?; + if !include_legacy_impact { + for meta in &mut metas { + if let Some(cached) = imported_cache::query_cached_session_from_conn( + cache_conn, + SOURCE_WINDSURF, + &meta.source_session_id, + )? { + meta.impact = cached.impact; + } + } + } let live_ids = metas .iter() .map(|meta| meta.source_session_id.clone()) @@ -27,11 +50,28 @@ pub(super) fn sync_windsurf_history_cache(cache_conn: &mut Connection) -> Result imported_cache::sync_source_cache_from_conn(cache_conn, SOURCE_WINDSURF, live_ids, inputs) } +#[cfg(test)] pub(super) fn list_windsurf_composer_meta_from_conn( conn: &Connection, db_path: &Path, source_mtime_ms: i64, source_size_bytes: i64, +) -> Result, String> { + list_windsurf_composer_meta_from_conn_inner( + conn, + db_path, + source_mtime_ms, + source_size_bytes, + true, + ) +} + +fn list_windsurf_composer_meta_from_conn_inner( + conn: &Connection, + db_path: &Path, + source_mtime_ms: i64, + source_size_bytes: i64, + include_legacy_impact: bool, ) -> Result, String> { let mut stmt = conn .prepare("SELECT value FROM cursorDiskKV WHERE key LIKE 'composerData:%'") @@ -56,18 +96,12 @@ pub(super) fn list_windsurf_composer_meta_from_conn( if composer.composer_id.trim().is_empty() { continue; } - let bubbles = load_bubbles_by_id( - conn, - &composer.composer_id, - &composer.full_conversation_headers_only, - )?; - let chunks = bubbles_to_chunks( - conn, - &format!("{WINDSURF_SESSION_PREFIX}{}", composer.composer_id), - &bubbles, - ); - let listable = is_listable_composer(&composer, &chunks); - let impact = imported_history::impact_from_edit_chunks(&chunks); + let listable = is_listable_composer(&composer); + let impact = if include_legacy_impact { + composer_impact(conn, &composer)? + } else { + ImportedHistoryImpactStats::default() + }; let source_fingerprint = windsurf_source_fingerprint(&composer, &sidecar_signature); metas.push(WindsurfComposerMeta { source_session_id: composer.composer_id.clone(), @@ -108,14 +142,74 @@ fn windsurf_source_fingerprint(composer: &RawComposerData, sidecar_signature: &s .join("|") } -fn is_listable_composer(composer: &RawComposerData, chunks: &[ActivityChunk]) -> bool { +fn is_listable_composer(composer: &RawComposerData) -> bool { if composer.composer_id.trim().is_empty() || composer.name.trim().is_empty() { return false; } if composer.subagent_info.is_some() || composer.full_conversation_headers_only.is_empty() { return false; } - !chunks.is_empty() + true +} + +fn composer_impact( + conn: &Connection, + composer: &RawComposerData, +) -> Result { + let key_prefix = format!("bubbleId:{}:", composer.composer_id); + let mut stmt = conn + .prepare( + "SELECT + json_extract(value, '$.toolFormerData.name'), + json_extract(value, '$.toolFormerData.params') + FROM cursorDiskKV + WHERE substr(key, 1, length(?1)) = ?1 + AND json_valid(value) + AND lower(COALESCE(json_extract(value, '$.toolFormerData.name'), '')) + IN ('edit_file', 'edit_file_v2', 'write_file', 'apply_patch')", + ) + .map_err(|err| format!("Failed to prepare Windsurf compact impact query: {err}"))?; + let mut rows = stmt + .query([key_prefix]) + .map_err(|err| format!("Failed to query Windsurf compact edit rows: {err}"))?; + let mut touched_files = std::collections::BTreeSet::new(); + let mut impact = ImportedHistoryImpactStats::default(); + while let Some(row) = rows + .next() + .map_err(|err| format!("Failed to read Windsurf compact edit row: {err}"))? + { + let raw_name = row + .get::<_, Option>(0) + .map_err(|err| format!("Failed to read Windsurf compact tool name: {err}"))? + .unwrap_or_default(); + let raw_args = row + .get::<_, Option>(1) + .map_err(|err| format!("Failed to read Windsurf compact tool args: {err}"))? + .unwrap_or_default(); + let (canonical_name, args) = + normalize_windsurf_tool_call(&raw_name, imported_history::parse_inner_json(&raw_args)); + let call = ImportedToolCall { + call_id: String::new(), + raw_name, + canonical_name, + args, + created_at: String::new(), + }; + let chunk = imported_history::tool_call_chunk( + "windsurfapp-catalog", + WINDSURF_PROVIDER_SLUG, + 0, + &call, + "", + ); + let one = imported_history::impact_from_edit_chunks(&[chunk]); + impact.lines_added = impact.lines_added.saturating_add(one.lines_added); + impact.lines_removed = impact.lines_removed.saturating_add(one.lines_removed); + touched_files.extend(one.touched_files); + } + impact.touched_files = touched_files.into_iter().collect(); + impact.files_changed = impact.touched_files.len() as i64; + Ok(impact) } pub(super) fn composer_meta_to_cache_input( diff --git a/src-tauri/crates/orgtrack-core/src/sources/workbuddy/discovery.rs b/src-tauri/crates/orgtrack-core/src/sources/workbuddy/discovery.rs index 42a983863..c6810472e 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/workbuddy/discovery.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/workbuddy/discovery.rs @@ -28,6 +28,38 @@ pub(super) fn sync_workbuddy_history_cache(conn: &mut Connection) -> Result<(), ) } +pub(super) fn refresh_workbuddy_catalog(conn: &mut Connection) -> Result<(), String> { + let discovered = discover_workbuddy_history_records()?; + let signatures = discovered + .iter() + .map(ImportedHistoryDiscoveredRecord::signature) + .collect::>(); + let changed = + imported_cache::changed_records_from_conn(conn, SOURCE_WORKBUDDY, &discovered, |record| { + record.signature() + })?; + let mut inputs = Vec::new(); + for record in changed { + if imported_cache::advance_cached_catalog_record_from_conn( + conn, + SOURCE_WORKBUDDY, + record, + None, + )? { + continue; + } + if let Some(meta) = parse_workbuddy_session_meta(record)? { + inputs.push(session_meta_to_cache_input(meta)); + } + } + imported_cache::sync_source_cache_from_conn( + conn, + SOURCE_WORKBUDDY, + imported_cache::live_ids_from_signatures(&signatures), + inputs, + ) +} + pub(super) fn discover_workbuddy_history_records( ) -> Result, String> { let mut files = Vec::new(); @@ -101,7 +133,7 @@ pub(super) fn parse_workbuddy_session_meta( record.source_path.display() ) })?; - let reader = BufReader::new(file); + let reader = BufReader::new(file.take(CATALOG_PREFIX_BYTES)); let mut created_at_ms = 0; let mut updated_at_ms = 0; diff --git a/src-tauri/crates/orgtrack-core/src/sources/workbuddy/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/workbuddy/mod.rs index 78cf7c6ff..03523df25 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/workbuddy/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/workbuddy/mod.rs @@ -7,7 +7,7 @@ use std::collections::{BTreeSet, HashMap, HashSet}; use std::fs; -use std::io::{BufRead, BufReader}; +use std::io::{BufRead, BufReader, Read}; use std::path::{Path, PathBuf}; use core_types::activity::ActivityChunk; @@ -39,6 +39,7 @@ use transcript::*; pub const WORKBUDDY_SESSION_PREFIX: &str = "workbuddyapp-"; const WORKBUDDY_PROVIDER_SLUG: &str = "workbuddy"; +const CATALOG_PREFIX_BYTES: u64 = 1024 * 1024; // Version 2 imports `subagents/agent-*.jsonl` and links them to their parent session. const WORKBUDDY_METADATA_PARSER_VERSION: i64 = 2; @@ -194,6 +195,10 @@ pub fn load_workbuddy_history_for_session( load_workbuddy_history_from_path(session_id, &path) } +pub(crate) fn refresh_catalog(conn: &mut Connection) -> Result<(), String> { + refresh_workbuddy_catalog(conn) +} + #[cfg(test)] #[path = "tests.rs"] mod tests; diff --git a/src-tauri/crates/orgtrack-core/src/sources/zcode/history/mod.rs b/src-tauri/crates/orgtrack-core/src/sources/zcode/history/mod.rs index 0edfda6ba..5d927723c 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/zcode/history/mod.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/zcode/history/mod.rs @@ -143,6 +143,96 @@ pub fn load_zcode_history_for_session(session_id: &str) -> Result Result<(), String> { + sync_zcode_history_cache_inner(cache_conn, false) +} + +fn load_zcode_impact_from_conn( + conn: &Connection, + session_id: &str, + source_session_id: &str, +) -> Result { + let mut stmt = conn + .prepare( + "SELECT p.id, p.message_id, json_extract(m.data, '$.role'), p.data, p.time_created + FROM part p + JOIN message m ON m.id = p.message_id + WHERE p.session_id = ?1 + AND json_extract(p.data, '$.type') = 'tool' + AND lower(COALESCE(json_extract(p.data, '$.tool'), '')) + IN ('write', 'edit', 'patch', 'apply_patch') + ORDER BY p.time_created ASC, p.id ASC", + ) + .map_err(|err| format!("Failed to prepare ZCode compact impact query: {err}"))?; + let mut rows = stmt + .query([source_session_id]) + .map_err(|err| format!("Failed to query ZCode compact impact rows: {err}"))?; + let mut touched = std::collections::BTreeSet::new(); + let mut impact = ImportedHistoryImpactStats::default(); + let mut sequence = 0usize; + while let Some(row) = rows + .next() + .map_err(|err| format!("Failed to read ZCode compact impact row: {err}"))? + { + let Some(raw_data) = row + .get::<_, Option>(3) + .map_err(|err| format!("Failed to read ZCode compact tool payload: {err}"))? + else { + continue; + }; + let chunk = replay_chunk_from_part_json( + session_id, + sequence, + row.get::<_, Option>(0) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + row.get::<_, Option>(1) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + row.get::<_, Option>(2) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + &raw_data, + row.get::<_, Option>(4) + .map_err(|err| err.to_string())? + .unwrap_or_default(), + )?; + sequence = sequence.saturating_add(1); + let Some(chunk) = chunk else { continue }; + let one = imported_history::impact_from_edit_chunks(std::slice::from_ref(&chunk)); + impact.lines_added = impact.lines_added.saturating_add(one.lines_added); + impact.lines_removed = impact.lines_removed.saturating_add(one.lines_removed); + touched.extend(one.touched_files); + } + impact.touched_files = touched.into_iter().collect(); + impact.files_changed = impact.touched_files.len() as i64; + Ok(impact) +} + +/// Normalize one ZCode `part` row for the bounded SQLite replay driver. +/// The caller streams rows and invokes this only for new or changed hashes. +#[allow(clippy::too_many_arguments)] +pub(crate) fn replay_chunk_from_part_json( + session_id: &str, + sequence: usize, + part_id: String, + message_id: String, + role: String, + raw_data: &str, + time_created: i64, +) -> Result, String> { + let part = serde_json::from_str::(raw_data) + .map_err(|err| format!("Failed to parse ZCode replay part {part_id}: {err}"))?; + let row = ZCodePartRow { + part_id, + message_id, + role, + part, + time_created, + }; + Ok(part_row_to_chunk(session_id, sequence, &row)) +} + /// Candidate on-disk locations for ZCode's CLI history database. Exposed so the /// external-CLI detection layer can report the store path. pub fn zcode_history_candidate_paths() -> Vec { diff --git a/src-tauri/crates/orgtrack-core/src/sources/zcode/history/parts.rs b/src-tauri/crates/orgtrack-core/src/sources/zcode/history/parts.rs index 02ae135f2..e74d815d1 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/zcode/history/parts.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/zcode/history/parts.rs @@ -65,7 +65,7 @@ fn load_ordered_parts( Ok(parts) } -fn part_row_to_chunk( +pub(super) fn part_row_to_chunk( session_id: &str, sequence: usize, row: &ZCodePartRow, diff --git a/src-tauri/crates/orgtrack-core/src/sources/zcode/history/sync.rs b/src-tauri/crates/orgtrack-core/src/sources/zcode/history/sync.rs index aa75bc029..97004f64e 100644 --- a/src-tauri/crates/orgtrack-core/src/sources/zcode/history/sync.rs +++ b/src-tauri/crates/orgtrack-core/src/sources/zcode/history/sync.rs @@ -3,6 +3,13 @@ use super::*; pub(super) fn sync_zcode_history_cache(cache_conn: &mut Connection) -> Result<(), String> { + sync_zcode_history_cache_inner(cache_conn, true) +} + +pub(super) fn sync_zcode_history_cache_inner( + cache_conn: &mut Connection, + include_legacy_impact: bool, +) -> Result<(), String> { let Some((conn, db_path)) = open_zcode_db()? else { imported_cache::sync_source_cache_from_conn( cache_conn, @@ -34,9 +41,16 @@ pub(super) fn sync_zcode_history_cache(cache_conn: &mut Connection) -> Result<() .into_iter() .filter(|meta| changed_ids.contains(&meta.source_session_id)) { - let session_id = format!("{ZCODE_SESSION_PREFIX}{}", meta.source_session_id); - let chunks = load_zcode_history_from_conn(&conn, &session_id, &meta.source_session_id)?; - meta.impact = imported_history::impact_from_edit_chunks(&chunks); + if include_legacy_impact { + let session_id = format!("{ZCODE_SESSION_PREFIX}{}", meta.source_session_id); + meta.impact = load_zcode_impact_from_conn(&conn, &session_id, &meta.source_session_id)?; + } else if let Some(cached) = imported_cache::query_cached_session_from_conn( + cache_conn, + SOURCE_ZCODE, + &meta.source_session_id, + )? { + meta.impact = cached.impact; + } inputs.push(session_meta_to_cache_input(meta)); } imported_cache::sync_source_cache_from_conn(cache_conn, SOURCE_ZCODE, live_ids, inputs) diff --git a/src-tauri/crates/orgtrack-core/src/store/sqlite/schema.rs b/src-tauri/crates/orgtrack-core/src/store/sqlite/schema.rs index c49aeaae9..466b3f025 100644 --- a/src-tauri/crates/orgtrack-core/src/store/sqlite/schema.rs +++ b/src-tauri/crates/orgtrack-core/src/store/sqlite/schema.rs @@ -624,6 +624,240 @@ impl SqliteRecordStore<'_> { entries_json TEXT NOT NULL DEFAULT '{}', PRIMARY KEY (source, directory_path) );", + )?; + + // #443: compact, rebuildable replay index for external CLI histories. + // + // These tables intentionally live in ORGII's sessions.db rather than + // in any provider-owned JSONL/SQLite store. Event rows contain only + // bounded previews and locators. Payloads that cannot be truly range + // read (JSONL/manifest/protobuf) may have one generation-scoped, + // rebuildable artifact; the provider transcript remains the source + // of truth and directly addressable bodies are still read on demand. + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS imported_replay_state ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + revision INTEGER NOT NULL DEFAULT 0, + parser_version INTEGER NOT NULL, + source_identity TEXT NOT NULL, + driver_cursor_json TEXT NOT NULL DEFAULT '{}', + indexed_size_bytes INTEGER NOT NULL DEFAULT 0, + indexed_mtime_ns INTEGER NOT NULL DEFAULT 0, + total_events INTEGER NOT NULL DEFAULT 0, + total_turns INTEGER NOT NULL DEFAULT 0, + valid INTEGER NOT NULL DEFAULT 1, + updated_at TEXT NOT NULL, + PRIMARY KEY (source, source_session_id) + ); + CREATE INDEX IF NOT EXISTS idx_imported_replay_state_generation + ON imported_replay_state(source, source_session_id, generation); + + -- A failed whole-document/manifest read must not replace the last + -- valid generation. Keep its physical snapshot separately so an + -- unchanged, still-invalid source can be served from the previous + -- generation without repeating the expensive parse on every poll. + CREATE TABLE IF NOT EXISTS imported_replay_rejected_snapshots ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + parser_version INTEGER NOT NULL, + source_identity TEXT NOT NULL, + source_size_bytes INTEGER NOT NULL, + source_mtime_ns INTEGER NOT NULL, + sample_fingerprint TEXT NOT NULL, + rejection_kind TEXT NOT NULL, + rejected_at TEXT NOT NULL, + PRIMARY KEY (source, source_session_id) + ); + + -- The imported session catalog is shared with source discovery. + -- Preserve the adapter-owned row before replay overlays compact + -- metadata so cache eviction can remove only replay-derived values. + CREATE TABLE IF NOT EXISTS imported_replay_catalog_derivations ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + baseline_json TEXT NOT NULL, + applied_json TEXT NOT NULL, + updated_at TEXT NOT NULL, + PRIMARY KEY (source, source_session_id) + ); + + CREATE TABLE IF NOT EXISTS imported_replay_turns ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + turn_index INTEGER NOT NULL, + turn_id TEXT NOT NULL, + start_sequence INTEGER NOT NULL, + end_sequence INTEGER, + started_at TEXT NOT NULL, + ended_at TEXT, + event_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (source, source_session_id, generation, turn_index) + ); + CREATE UNIQUE INDEX IF NOT EXISTS idx_imported_replay_turn_id + ON imported_replay_turns(source, source_session_id, generation, turn_id); + + CREATE TABLE IF NOT EXISTS imported_replay_events ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + sequence INTEGER NOT NULL, + event_id TEXT NOT NULL, + turn_index INTEGER NOT NULL, + action_type TEXT NOT NULL, + function_name TEXT NOT NULL, + created_at TEXT NOT NULL, + args_preview_json TEXT NOT NULL DEFAULT '{}', + result_preview_json TEXT NOT NULL DEFAULT '{}', + args_size_bytes INTEGER NOT NULL DEFAULT 0, + result_size_bytes INTEGER NOT NULL DEFAULT 0, + thread_id TEXT, + process_id TEXT, + source_start INTEGER NOT NULL DEFAULT 0, + source_end INTEGER NOT NULL DEFAULT 0, + payloads_json TEXT NOT NULL DEFAULT '[]', + content_hash TEXT NOT NULL, + event_revision INTEGER NOT NULL DEFAULT 1, + PRIMARY KEY (source, source_session_id, generation, sequence), + UNIQUE (source, source_session_id, generation, event_id) + ); + CREATE INDEX IF NOT EXISTS idx_imported_replay_events_turn + ON imported_replay_events( + source, source_session_id, generation, turn_index, sequence + ); + + CREATE TABLE IF NOT EXISTS imported_replay_source_rows ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + source_key TEXT NOT NULL, + content_hash TEXT NOT NULL, + event_id TEXT, + sequence INTEGER, + source_order INTEGER NOT NULL, + seen_revision INTEGER NOT NULL, + PRIMARY KEY (source, source_session_id, generation, source_key) + ); + CREATE INDEX IF NOT EXISTS idx_imported_replay_source_rows_seen + ON imported_replay_source_rows( + source, source_session_id, generation, seen_revision + ); + + CREATE TABLE IF NOT EXISTS imported_replay_structured_rows ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + source_key TEXT NOT NULL, + content_hash TEXT NOT NULL, + seen_revision INTEGER NOT NULL, + PRIMARY KEY (source, source_session_id, generation, source_key) + ); + CREATE TABLE IF NOT EXISTS imported_replay_structured_events ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + source_key TEXT NOT NULL, + local_key TEXT NOT NULL, + event_id TEXT NOT NULL, + sequence INTEGER NOT NULL, + PRIMARY KEY ( + source, source_session_id, generation, source_key, local_key + ) + ); + CREATE TABLE IF NOT EXISTS imported_replay_payload_artifacts ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + content_hash TEXT NOT NULL, + payload BLOB NOT NULL, + PRIMARY KEY ( + source, source_session_id, generation, content_hash + ) + ); + CREATE TABLE IF NOT EXISTS imported_replay_payload_artifact_refs ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + event_id TEXT NOT NULL, + field_path TEXT NOT NULL, + content_hash TEXT NOT NULL, + PRIMARY KEY ( + source, source_session_id, generation, event_id, field_path + ) + ); + + -- External CLI Shell cards reuse the canonical replay payload + -- artifact instead of copying it into one `.slog` per call. The + -- manifest is deliberately small: ordered stream boundaries and + -- content hashes only; provider locators never cross IPC. + CREATE TABLE IF NOT EXISTS imported_replay_shell_manifests ( + session_id TEXT NOT NULL, + logical_call_id TEXT NOT NULL, + call_id TEXT NOT NULL, + identity_hash TEXT NOT NULL, + total_bytes INTEGER NOT NULL, + last_sequence INTEGER NOT NULL, + terminal_preview TEXT NOT NULL, + completed_at TEXT, + accessed_at TEXT NOT NULL, + PRIMARY KEY (session_id, call_id), + UNIQUE (session_id, logical_call_id) + ); + CREATE TABLE IF NOT EXISTS imported_replay_shell_segments ( + session_id TEXT NOT NULL, + call_id TEXT NOT NULL, + ordinal INTEGER NOT NULL, + stream TEXT NOT NULL CHECK(stream IN ('stdout','stderr')), + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + content_hash TEXT NOT NULL, + output_byte_start INTEGER NOT NULL, + total_bytes INTEGER NOT NULL, + first_sequence INTEGER NOT NULL, + frame_count INTEGER NOT NULL, + PRIMARY KEY (session_id, call_id, ordinal), + FOREIGN KEY (session_id, call_id) + REFERENCES imported_replay_shell_manifests(session_id, call_id) + ON DELETE CASCADE + ); + CREATE INDEX IF NOT EXISTS idx_imported_replay_shell_segment_artifact + ON imported_replay_shell_segments( + source, source_session_id, generation, content_hash + ); + + CREATE TABLE IF NOT EXISTS imported_replay_changes ( + source TEXT NOT NULL, + source_session_id TEXT NOT NULL, + generation TEXT NOT NULL, + change_revision INTEGER NOT NULL, + event_id TEXT NOT NULL, + change_kind TEXT NOT NULL, + sequence INTEGER, + PRIMARY KEY(source, source_session_id, generation, change_revision) + ); + CREATE INDEX IF NOT EXISTS idx_imported_replay_changes_event + ON imported_replay_changes( + source, source_session_id, generation, event_id + );", + )?; + ensure_column( + conn, + "imported_replay_events", + "event_revision", + "INTEGER NOT NULL DEFAULT 1", + )?; + // The external Shell manifest table was introduced on the issue-443 + // branch before cache eviction gained an ORGII-owned access clock. + // Keep developer databases made by that intermediate schema readable; + // an epoch default makes them cold until publish/read touches them. + ensure_column( + conn, + "imported_replay_shell_manifests", + "accessed_at", + "TEXT NOT NULL DEFAULT '1970-01-01T00:00:00.000Z'", ) } } diff --git a/src-tauri/crates/orgtrack-core/src/usage_dashboard.rs b/src-tauri/crates/orgtrack-core/src/usage_dashboard.rs index cdcc8037a..d4f7971b2 100644 --- a/src-tauri/crates/orgtrack-core/src/usage_dashboard.rs +++ b/src-tauri/crates/orgtrack-core/src/usage_dashboard.rs @@ -33,8 +33,8 @@ mod tests; use accumulator::UsageHeadlineAccumulator; pub use overview::{usage_overview, usage_rounds, usage_trends, UsageOverview}; -pub use rounds::UsageRoundRow; use rounds::visit_rounds; +pub use rounds::UsageRoundRow; use rusqlite::Connection; use serde::Serialize; diff --git a/src-tauri/crates/orgtrack-core/src/usage_dashboard/overview.rs b/src-tauri/crates/orgtrack-core/src/usage_dashboard/overview.rs index 29f963c45..e444c5cc2 100644 --- a/src-tauri/crates/orgtrack-core/src/usage_dashboard/overview.rs +++ b/src-tauri/crates/orgtrack-core/src/usage_dashboard/overview.rs @@ -11,7 +11,9 @@ use serde::Serialize; use super::accumulator::UsageHeadlineAccumulator; use super::rounds::{visit_rounds, UsageRoundRow}; -use super::{SessionSort, TrendBucket, UsageFilter, UsageRoundQuery, UsageSummary, UsageTrendPoint}; +use super::{ + SessionSort, TrendBucket, UsageFilter, UsageRoundQuery, UsageSummary, UsageTrendPoint, +}; const ROUND_PAGE_TABLE: &str = "usage_dashboard_round_page"; diff --git a/src-tauri/crates/orgtrack-core/src/usage_dashboard/tests.rs b/src-tauri/crates/orgtrack-core/src/usage_dashboard/tests.rs index 634f519ed..7ee9e968a 100644 --- a/src-tauri/crates/orgtrack-core/src/usage_dashboard/tests.rs +++ b/src-tauri/crates/orgtrack-core/src/usage_dashboard/tests.rs @@ -158,8 +158,8 @@ fn rounds_fallback_when_no_round_rows() { // No imported_history_round_usage rows: native sessions expand to their // per-turn rows, imported codex gets one synthesized fallback round. let conn = seeded_conn(); - let rows = usage_rounds(&conn, &UsageFilter::default(), SessionSort::Recent, 0, 100) - .expect("rounds"); + let rows = + usage_rounds(&conn, &UsageFilter::default(), SessionSort::Recent, 0, 100).expect("rounds"); // claude 2 native turns + org2 1 native turn + codex 1 fallback = 4. assert_eq!(rows.len(), 4); assert!(rows.iter().all(|r| r.session_id != "mirror-claude")); @@ -198,8 +198,8 @@ fn rounds_use_real_rows_and_session_filter() { ms("2026-07-18T02:10:00Z"), ); - let rows = usage_rounds(&conn, &UsageFilter::default(), SessionSort::Recent, 0, 100) - .expect("rounds"); + let rows = + usage_rounds(&conn, &UsageFilter::default(), SessionSort::Recent, 0, 100).expect("rounds"); // claude 2 + org2 1 + codex 2 real = 5. assert_eq!(rows.len(), 5); let codex: Vec<_> = rows @@ -599,8 +599,7 @@ fn sessions_table_paginates() { #[test] fn trends_use_per_turn_native_and_lumped_imported() { let conn = seeded_conn(); - let series = - usage_trends(&conn, &UsageFilter::default(), TrendBucket::Hour).expect("trends"); + let series = usage_trends(&conn, &UsageFilter::default(), TrendBucket::Hour).expect("trends"); // Distinct hour buckets: codex 02:00, claude 03:00, org2 04:00, claude 05:00. let keys: Vec = series.iter().map(|p| p.bucket_ms).collect(); assert_eq!( @@ -631,8 +630,7 @@ fn trends_use_per_turn_native_and_lumped_imported() { #[test] fn trends_day_bucket_collapses_hours() { let conn = seeded_conn(); - let series = - usage_trends(&conn, &UsageFilter::default(), TrendBucket::Day).expect("trends"); + let series = usage_trends(&conn, &UsageFilter::default(), TrendBucket::Day).expect("trends"); assert_eq!(series.len(), 1); assert_eq!(series[0].bucket_ms, ms("2026-07-18T00:00:00Z")); } diff --git a/src-tauri/crates/session-persistence/Cargo.toml b/src-tauri/crates/session-persistence/Cargo.toml index 96d4ae4d2..b8697dfb4 100644 --- a/src-tauri/crates/session-persistence/Cargo.toml +++ b/src-tauri/crates/session-persistence/Cargo.toml @@ -45,3 +45,6 @@ tracing = "0.1" # Error derive macros — `IntentError` in `turn_intents`. thiserror = { workspace = true } + +[target.'cfg(unix)'.dev-dependencies] +libc = "0.2" diff --git a/src-tauri/crates/session-persistence/src/crud.rs b/src-tauri/crates/session-persistence/src/crud.rs index fa88899b9..222b2ea82 100644 --- a/src-tauri/crates/session-persistence/src/crud.rs +++ b/src-tauri/crates/session-persistence/src/crud.rs @@ -140,8 +140,8 @@ fn refresh_session_metadata_from_events( conn: &Connection, session_id: &str, ) -> SqliteResult { - let (event_count, time_start, time_end): (i64, Option, Option) = - conn.query_row( + let (event_count, time_start, time_end): (i64, Option, Option) = conn + .query_row( "SELECT COUNT(*), MIN(created_at), MAX(created_at) FROM events WHERE session_id=?1", [session_id], @@ -920,11 +920,8 @@ mod tests { ], ) .expect("append first import page"); - save_events_deferred( - session_id, - &[cached_event(session_id, "event-3", t3)], - ) - .expect("append second import page"); + save_events_deferred(session_id, &[cached_event(session_id, "event-3", t3)]) + .expect("append second import page"); assert!( get_session_metadata(session_id) diff --git a/src-tauri/crates/session-persistence/src/lib.rs b/src-tauri/crates/session-persistence/src/lib.rs index 1f2e85f6d..abed829bd 100644 --- a/src-tauri/crates/session-persistence/src/lib.rs +++ b/src-tauri/crates/session-persistence/src/lib.rs @@ -56,10 +56,10 @@ pub use connection::get_connection; pub use schema::init_session_tables; pub use crud::{ - clear_old_sessions, delete_session, find_awaiting_user_events_by_function, get_all_sessions, - finalize_deferred_event_import, get_cache_stats, get_event, get_session_metadata, load_events, - load_session, save_events, save_events_deferred, save_session, search_all_sessions, - search_events, update_session_specs, + clear_old_sessions, delete_session, finalize_deferred_event_import, + find_awaiting_user_events_by_function, get_all_sessions, get_cache_stats, get_event, + get_session_metadata, load_events, load_session, save_events, save_events_deferred, + save_session, search_all_sessions, search_events, update_session_specs, }; pub use editing::{clear_session_history, delete_event, truncate_after_event, update_event}; diff --git a/src-tauri/crates/session-persistence/src/turn_index.rs b/src-tauri/crates/session-persistence/src/turn_index.rs index 984913ce9..ccb2a8861 100644 --- a/src-tauri/crates/session-persistence/src/turn_index.rs +++ b/src-tauri/crates/session-persistence/src/turn_index.rs @@ -3,7 +3,8 @@ use chrono::{DateTime, Utc}; use core_types::extracted::ExtractedGitArtifactData; use orgtrack_core::projectors::turn_metadata::{ - TurnMetadataAccumulator, TurnModifiedFile, TurnResourceInteraction, + metadata_projection_requirements, TurnMetadataAccumulator, TurnModifiedFile, + TurnResourceInteraction, }; use rusqlite::{params, Connection, OptionalExtension, Result as SqliteResult}; use serde::{Deserialize, Serialize}; @@ -29,7 +30,9 @@ const TURN_STATUS_FAILED: &str = "failed"; /// Orgtrack instead of interpreting ORG2 tool names in this host crate. /// v11: treat the normalized imported-history `user` function as the same /// turn boundary as the native `user_message` function. -const TURN_INDEX_VERSION: i64 = 11; +/// v12: stream event rows into turn summaries and skip payload columns that +/// cannot contribute projected resource/edit/Git metadata. +const TURN_INDEX_VERSION: i64 = 12; #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] @@ -103,18 +106,6 @@ struct UserMessageRow { images: Option, } -fn index_event_row(row: &rusqlite::Row<'_>) -> SqliteResult { - Ok(IndexEventRow { - id: row.get(0)?, - function_name: row.get(1)?, - args_json: row.get(2)?, - result_json: row.get(3)?, - content: row.get(4)?, - created_at: row.get(5)?, - order_sequence: row.get(6)?, - }) -} - fn event_state(conn: &Connection, session_id: &str) -> SqliteResult<(i64, Option)> { conn.query_row( "SELECT COUNT(*), MAX(COALESCE(history_sequence, rowid)) @@ -125,8 +116,8 @@ fn event_state(conn: &Connection, session_id: &str) -> SqliteResult<(i64, Option ) } -fn is_synthetic_user_input(row: &IndexEventRow) -> bool { - serde_json::from_str::(&row.result_json) +fn is_synthetic_user_input(result_json: &str) -> bool { + serde_json::from_str::(result_json) .ok() .and_then(|result| { result @@ -139,8 +130,8 @@ fn is_synthetic_user_input(row: &IndexEventRow) -> bool { /// Extract the canonical user-intent id from a user_message row's /// `result_json`. Returns `None` for legacy rows (no id was minted) and /// for malformed JSON. -fn turn_intent_id_for_row(row: &IndexEventRow) -> Option { - serde_json::from_str::(&row.result_json) +fn turn_intent_id_for_result(result_json: &str) -> Option { + serde_json::from_str::(result_json) .ok() .and_then(|result| { result @@ -155,7 +146,7 @@ fn is_user_message(row: &IndexEventRow) -> bool { matches!( row.function_name.as_deref(), Some(USER_MESSAGE_FUNCTION | IMPORTED_USER_MESSAGE_FUNCTION) - ) && !is_synthetic_user_input(row) + ) && !is_synthetic_user_input(&row.result_json) } /// Lookup of intent ids that the indexer must treat as not yielding a @@ -171,26 +162,20 @@ type StaleIntentIds = std::collections::HashSet; /// when no body events landed. type IntentStatusOverlay = std::collections::HashMap; -fn load_stale_intent_ids(session_id: &str) -> StaleIntentIds { - super::turn_intents::list_for_session(session_id) - .map(|rows| { - rows.into_iter() - .filter(|row| row.status.is_pre_durable_terminal()) - .map(|row| row.turn_intent_id) - .collect() - }) - .unwrap_or_default() -} - -fn load_intent_status_overlay(session_id: &str) -> IntentStatusOverlay { - super::turn_intents::list_for_session(session_id) - .map(|rows| { - rows.into_iter() - .filter(|row| !row.status.is_pre_durable_terminal()) - .map(|row| (row.turn_intent_id, row.status.as_str().to_string())) - .collect() - }) - .unwrap_or_default() +fn load_intent_overlays(session_id: &str) -> (StaleIntentIds, IntentStatusOverlay) { + let mut stale = StaleIntentIds::new(); + let mut statuses = IntentStatusOverlay::new(); + let Ok(rows) = super::turn_intents::list_for_session(session_id) else { + return (stale, statuses); + }; + for row in rows { + if row.status.is_pre_durable_terminal() { + stale.insert(row.turn_intent_id); + } else { + statuses.insert(row.turn_intent_id, row.status.as_str().to_string()); + } + } + (stale, statuses) } fn user_event_id_for_message(message_id: &str) -> String { @@ -216,28 +201,6 @@ fn user_content_dedup_key(content: &str) -> String { content[..end].to_string() } -fn load_user_messages(conn: &Connection, session_id: &str) -> SqliteResult> { - let mut stmt = conn.prepare_cached( - "SELECT id, content, sequence, created_at, images - FROM agent_messages - WHERE session_id = ?1 AND role = 'user' - ORDER BY sequence ASC, created_at ASC, id ASC", - )?; - - let rows = stmt - .query_map([session_id], |row| { - Ok(UserMessageRow { - id: row.get(0)?, - content: row.get(1)?, - sequence: row.get(2)?, - created_at: row.get(3)?, - images: row.get(4)?, - }) - })? - .collect::>>()?; - Ok(rows) -} - fn load_existing_user_event_keys( conn: &Connection, session_id: &str, @@ -262,16 +225,7 @@ fn load_existing_user_event_keys( })?; for row in rows { let (id, content, result_json) = row?; - let event_row = IndexEventRow { - id: id.clone(), - function_name: Some(USER_MESSAGE_FUNCTION.to_string()), - args_json: String::new(), - result_json, - content: content.clone(), - created_at: String::new(), - order_sequence: 0, - }; - if is_synthetic_user_input(&event_row) { + if is_synthetic_user_input(&result_json) { continue; } ids.insert(id); @@ -288,15 +242,30 @@ fn load_existing_user_event_keys( } fn backfill_missing_user_events(conn: &Connection, session_id: &str) -> SqliteResult { - let messages = load_user_messages(conn, session_id)?; - if messages.is_empty() { - return Ok(0); - } - let (existing_ids, mut existing_content_counts) = load_existing_user_event_keys(conn, session_id)?; + let mut messages_stmt = conn.prepare_cached( + "SELECT id, content, sequence, created_at, images + FROM agent_messages + WHERE session_id = ?1 AND role = 'user' + ORDER BY sequence ASC, created_at ASC, id ASC", + )?; + let mut insert_stmt = conn.prepare_cached( + "INSERT OR IGNORE INTO events + (id, session_id, event_type, function_name, thread_id, args_json, result_json, + content, created_at, meta_json, history_sequence) + VALUES (?1, ?2, 'raw', 'user_message', NULL, '{}', ?3, ?4, ?5, ?6, ?7)", + )?; + let mut messages = messages_stmt.query([session_id])?; let mut inserted = 0; - for message in messages { + while let Some(row) = messages.next()? { + let message = UserMessageRow { + id: row.get(0)?, + content: row.get(1)?, + sequence: row.get(2)?, + created_at: row.get(3)?, + images: row.get(4)?, + }; let event_id = user_event_id_for_message(&message.id); if existing_ids.contains(&event_id) { continue; @@ -349,21 +318,15 @@ fn backfill_missing_user_events(conn: &Connection, session_id: &str) -> SqliteRe }); let content = format!("user_message {}", message.content); - let affected = conn.execute( - "INSERT OR IGNORE INTO events - (id, session_id, event_type, function_name, thread_id, args_json, result_json, - content, created_at, meta_json, history_sequence) - VALUES (?1, ?2, 'raw', 'user_message', NULL, '{}', ?3, ?4, ?5, ?6, ?7)", - params![ - event_id, - session_id, - serde_json::to_string(&result).unwrap_or_else(|_| "{}".to_string()), - content, - message.created_at, - serde_json::to_string(&meta).unwrap_or_else(|_| "{}".to_string()), - message.sequence, - ], - )?; + let affected = insert_stmt.execute(params![ + event_id, + session_id, + serde_json::to_string(&result).unwrap_or_else(|_| "{}".to_string()), + content, + message.created_at, + serde_json::to_string(&meta).unwrap_or_else(|_| "{}".to_string()), + message.sequence, + ])?; inserted += affected; } @@ -389,31 +352,32 @@ fn max_timestamp(left: &str, right: &str) -> String { } } -struct TurnDraftBuilder<'a> { - stale_intent_ids: &'a StaleIntentIds, - drafts: Vec, +struct TurnDraftFolder<'a> { current: Option, + stale_intent_ids: &'a StaleIntentIds, } -impl<'a> TurnDraftBuilder<'a> { +impl<'a> TurnDraftFolder<'a> { fn new(stale_intent_ids: &'a StaleIntentIds) -> Self { Self { - stale_intent_ids, - drafts: Vec::new(), current: None, + stale_intent_ids, } } - fn push(&mut self, row: &IndexEventRow) { + /// Fold one row and return a completed materialized turn, if this row + /// closes one. At most the open turn is retained, so memory is bounded by + /// one turn's compact metadata rather than the session's event payloads. + fn push(&mut self, row: &IndexEventRow) -> Option { if is_user_message(row) { - let row_intent_id = turn_intent_id_for_row(row); + let row_intent_id = turn_intent_id_for_result(&row.result_json); // Lifecycle-pre-durable terminal: this intent will never yield // a durable round (Stale = invalidated). Drop the row entirely // so the indexer does not paint a phantom turn. if let Some(ref intent_id) = row_intent_id { if self.stale_intent_ids.contains(intent_id) { - return; + return None; } } @@ -427,15 +391,15 @@ impl<'a> TurnDraftBuilder<'a> { turn.user_event_ids.push(row.id.clone()); turn.event_count += 1; turn.ended_at = Some(max_timestamp(&turn.started_at, &row.created_at)); - return; + return None; } } - if let Some(mut completed) = self.current.take() { + let completed = self.current.take().and_then(|mut completed| { completed.end_sequence = Some(row.order_sequence); completed.next_turn_id = Some(row.id.clone()); - self.drafts.push(completed); - } + (completed.body_event_count > 0).then_some(completed) + }); self.current = Some(TurnDraft { turn_id: row.id.clone(), @@ -451,7 +415,7 @@ impl<'a> TurnDraftBuilder<'a> { turn_intent_id: row_intent_id, metadata_accumulator: TurnMetadataAccumulator::new(), }); - return; + return completed; } if let Some(ref mut turn) = self.current { @@ -465,61 +429,69 @@ impl<'a> TurnDraftBuilder<'a> { &row.created_at, ); } + None } - fn finish(mut self) -> Vec { - if let Some(turn) = self.current.take() { - self.drafts.push(turn); - } - materialized_turn_drafts(self.drafts) + /// The newest user-only turn remains visible as pending, matching the + /// previous `materialized_turn_drafts` last-item rule. + fn finish(self) -> Option { + self.current } } #[cfg(test)] fn build_turn_drafts(rows: &[IndexEventRow], stale_intent_ids: &StaleIntentIds) -> Vec { - let mut builder = TurnDraftBuilder::new(stale_intent_ids); + let mut folder = TurnDraftFolder::new(stale_intent_ids); + let mut drafts = Vec::new(); for row in rows { - builder.push(row); + if let Some(completed) = folder.push(row) { + drafts.push(completed); + } } - builder.finish() -} - -/// Aggregate directly from SQLite's cursor so a GiB-scale transcript is -/// never retained as a second in-memory vector during index construction. -fn stream_turn_drafts( - conn: &Connection, - session_id: &str, - stale_intent_ids: &StaleIntentIds, -) -> SqliteResult> { - let mut stmt = conn.prepare_cached( - "SELECT id, function_name, args_json, result_json, content, created_at, - history_sequence AS order_sequence - FROM events - WHERE session_id = ?1 - ORDER BY history_sequence ASC, created_at ASC, id ASC", - )?; - let rows = stmt.query_map([session_id], index_event_row)?; - let mut builder = TurnDraftBuilder::new(stale_intent_ids); - for row in rows { - let row = row?; - builder.push(&row); + if let Some(current) = folder.finish() { + drafts.push(current); } - Ok(builder.finish()) + drafts } -fn materialized_turn_drafts(drafts: Vec) -> Vec { - let last_index = drafts.len().saturating_sub(1); - drafts - .into_iter() - .enumerate() - .filter_map(|(index, draft)| { - if draft.body_event_count > 0 || index == last_index { - Some(draft) - } else { - None - } - }) - .collect() +fn load_index_event(row: &rusqlite::Row<'_>) -> SqliteResult { + let function_name: Option = row.get(1)?; + let requirements = metadata_projection_requirements(function_name.as_deref()); + let is_user_boundary = matches!( + function_name.as_deref(), + Some(USER_MESSAGE_FUNCTION | IMPORTED_USER_MESSAGE_FUNCTION) + ); + // Boundary metadata is always needed to distinguish optimistic synthetic + // rows and collapse canonical turn-intent ids. Other result bodies are + // materialized only when their typed projection requires them. + let result_json = if is_user_boundary || requirements.needs_result_json() { + row.get(3)? + } else { + String::new() + }; + let starts_turn = is_user_boundary && !is_synthetic_user_input(&result_json); + + Ok(IndexEventRow { + id: if starts_turn { + row.get(0)? + } else { + String::new() + }, + function_name, + args_json: if requirements.needs_args_json() { + row.get(2)? + } else { + String::new() + }, + result_json, + content: if starts_turn { + row.get(4)? + } else { + String::new() + }, + created_at: row.get(5)?, + order_sequence: row.get(6)?, + }) } fn turn_summary_from_row(row: &rusqlite::Row<'_>) -> SqliteResult { @@ -555,32 +527,101 @@ fn turn_summary_from_row(row: &rusqlite::Row<'_>) -> SqliteResult SqliteResult> { +pub fn rebuild_turn_index(session_id: &str) -> SqliteResult<()> { with_sessions_writer(|| rebuild_turn_index_inner(session_id)) } -fn rebuild_turn_index_inner(session_id: &str) -> SqliteResult> { +fn rebuild_turn_index_inner(session_id: &str) -> SqliteResult<()> { let conn = get_connection()?; - backfill_missing_user_events(&conn, session_id)?; - normalize_session_sequences(&conn, session_id)?; // Consult the lifecycle store so the indexer can drop rows whose // intent was retired before it ran (Stale). Read failure // falls back to an empty set, which preserves the legacy behaviour of // building rounds purely from events. - let stale_intent_ids = load_stale_intent_ids(session_id); - let intent_status_overlay = load_intent_status_overlay(session_id); - let drafts = stream_turn_drafts(&conn, session_id, &stale_intent_ids)?; - let (event_count, max_sequence) = event_state(&conn, session_id)?; + let (stale_intent_ids, intent_status_overlay) = load_intent_overlays(session_id); + rebuild_turn_index_on_connection(&conn, session_id, &stale_intent_ids, &intent_status_overlay) +} + +fn insert_turn_draft( + stmt: &mut rusqlite::CachedStatement<'_>, + session_id: &str, + draft: &TurnDraft, + intent_status_overlay: &IntentStatusOverlay, + rebuilt_at: &str, +) -> SqliteResult<()> { + let user_event_ids_json = + serde_json::to_string(&draft.user_event_ids).unwrap_or_else(|_| "[]".to_string()); + let modified_files_json = serde_json::to_string(draft.metadata_accumulator.files()) + .unwrap_or_else(|_| "[]".to_string()); + let resource_interactions_json = + serde_json::to_string(draft.metadata_accumulator.resource_interactions()) + .unwrap_or_else(|_| "[]".to_string()); + let git_artifacts_json = serde_json::to_string(draft.metadata_accumulator.git_artifacts()) + .unwrap_or_else(|_| "[]".to_string()); + // Status derivation: lifecycle store wins when available. Falls back to + // the legacy body-event heuristic for rows without a canonical intent. + let status = draft + .turn_intent_id + .as_ref() + .and_then(|intent_id| intent_status_overlay.get(intent_id)) + .map(|status| match status.as_str() { + "completed" => TURN_STATUS_COMPLETED, + "failed" | "cancelled" => TURN_STATUS_FAILED, + // Running / queued / optimistic all surface as pending. + _ => TURN_STATUS_PENDING, + }) + .unwrap_or_else(|| { + if draft.body_event_count > 0 { + TURN_STATUS_COMPLETED + } else { + TURN_STATUS_PENDING + } + }); + stmt.execute(params![ + session_id, + draft.turn_id, + draft.start_sequence, + draft.end_sequence, + draft.next_turn_id, + draft.started_at, + draft.ended_at, + duration_ms(&draft.started_at, draft.ended_at.as_deref()), + user_event_ids_json, + draft.user_preview, + draft.event_count, + draft.body_event_count, + status, + 0_i64, + rebuilt_at, + modified_files_json, + resource_interactions_json, + git_artifacts_json, + ])?; + Ok(()) +} + +fn rebuild_turn_index_on_connection( + conn: &Connection, + session_id: &str, + stale_intent_ids: &StaleIntentIds, + intent_status_overlay: &IntentStatusOverlay, +) -> SqliteResult<()> { + // Keep the old materialized index visible to other connections until the + // complete replacement and its state row can commit together. Any parse, + // projection, or insertion error drops this transaction and restores the + // previous generation. + let tx = begin_immediate(conn)?; + backfill_missing_user_events(&tx, session_id)?; + normalize_session_sequences(&tx, session_id)?; + let (event_count, max_sequence) = event_state(&tx, session_id)?; let rebuilt_at = Utc::now().to_rfc3339(); - let tx = begin_immediate(&conn)?; tx.execute( "DELETE FROM session_turns WHERE session_id = ?1", [session_id], )?; { - let mut stmt = tx.prepare_cached( + let mut insert_stmt = tx.prepare_cached( "INSERT INTO session_turns (session_id, turn_id, start_sequence, end_sequence, next_turn_id, started_at, ended_at, duration_ms, user_event_ids_json, user_preview, event_count, body_event_count, @@ -588,63 +629,35 @@ fn rebuild_turn_index_inner(session_id: &str) -> SqliteResult 0` heuristic for - // rows that predate the canonical intent id (no row in - // `session_turn_intents`). The lifecycle store is the - // authoritative source for cancelled turns that had zero body - // events and for turns interrupted mid-stream. - let status = draft - .turn_intent_id - .as_ref() - .and_then(|intent_id| intent_status_overlay.get(intent_id)) - .map(|status| match status.as_str() { - "completed" => TURN_STATUS_COMPLETED, - "failed" | "cancelled" => TURN_STATUS_FAILED, - // Running / queued / optimistic all surface as pending - // — the round is open and we don't yet know the - // terminal outcome. - _ => TURN_STATUS_PENDING, - }) - .unwrap_or_else(|| { - if draft.body_event_count > 0 { - TURN_STATUS_COMPLETED - } else { - TURN_STATUS_PENDING - } - }); - stmt.execute(params![ + let mut events_stmt = tx.prepare_cached( + "SELECT id, function_name, args_json, result_json, content, created_at, + history_sequence AS order_sequence + FROM events + WHERE session_id = ?1 + ORDER BY history_sequence ASC, created_at ASC, id ASC", + )?; + let mut rows = events_stmt.query([session_id])?; + let mut folder = TurnDraftFolder::new(stale_intent_ids); + while let Some(row) = rows.next()? { + let event = load_index_event(row)?; + if let Some(completed) = folder.push(&event) { + insert_turn_draft( + &mut insert_stmt, + session_id, + &completed, + intent_status_overlay, + &rebuilt_at, + )?; + } + } + if let Some(current) = folder.finish() { + insert_turn_draft( + &mut insert_stmt, session_id, - draft.turn_id, - draft.start_sequence, - draft.end_sequence, - draft.next_turn_id, - draft.started_at, - draft.ended_at, - duration_ms(&draft.started_at, draft.ended_at.as_deref()), - user_event_ids_json, - draft.user_preview, - draft.event_count, - draft.body_event_count, - status, - 0_i64, - rebuilt_at, - modified_files_json, - resource_interactions_json, - git_artifacts_json, - ])?; + ¤t, + intent_status_overlay, + &rebuilt_at, + )?; } } @@ -666,8 +679,7 @@ fn rebuild_turn_index_inner(session_id: &str) -> SqliteResult SqliteResult<()> { @@ -787,317 +799,4 @@ pub fn get_turn_summary( } #[cfg(test)] -mod tests { - use super::*; - - fn row( - id: &str, - function_name: Option<&str>, - result_json: &str, - sequence: i64, - ) -> IndexEventRow { - IndexEventRow { - id: id.to_string(), - function_name: function_name.map(str::to_string), - args_json: "{}".to_string(), - result_json: result_json.to_string(), - content: id.to_string(), - created_at: "2026-05-27T00:00:00Z".to_string(), - order_sequence: sequence, - } - } - - fn create_backfill_test_tables(conn: &Connection) { - crate::schema::init_session_tables(conn).unwrap(); - conn.execute_batch( - "CREATE TABLE IF NOT EXISTS agent_messages ( - id TEXT PRIMARY KEY, - session_id TEXT NOT NULL, - role TEXT NOT NULL, - content TEXT NOT NULL, - sequence INTEGER NOT NULL, - created_at TEXT NOT NULL, - images TEXT - );", - ) - .unwrap(); - } - - #[test] - fn backfill_missing_user_events_is_idempotent() { - let conn = Connection::open_in_memory().unwrap(); - create_backfill_test_tables(&conn); - conn.execute( - "INSERT INTO agent_messages (id, session_id, role, content, sequence, created_at, images) - VALUES (?1, ?2, 'user', ?3, ?4, ?5, NULL)", - params![ - "message-1", - "session-1", - "hello from persisted user", - 1_i64, - "2026-05-27T00:00:00Z", - ], - ) - .unwrap(); - - assert_eq!(backfill_missing_user_events(&conn, "session-1").unwrap(), 1); - assert_eq!(backfill_missing_user_events(&conn, "session-1").unwrap(), 0); - - let count: i64 = conn - .query_row( - "SELECT COUNT(*) FROM events WHERE session_id = ?1 AND id = ?2", - params!["session-1", "user-message-message-1"], - |row| row.get(0), - ) - .unwrap(); - assert_eq!(count, 1); - } - - #[test] - fn backfill_dedups_long_messages_against_truncated_event_previews() { - // After a transcript rewrite (compaction) agent_messages rows get - // fresh ids, so id-based dedup misses and we fall back to content - // matching. Event rows store content truncated to 500 bytes; the - // full agent_messages content must still match instead of - // re-inserting a duplicate "[Plan approved] …" user bubble. - let conn = Connection::open_in_memory().unwrap(); - create_backfill_test_tables(&conn); - - let long_content = format!( - "[Plan approved] Implement the approved plan now. 计划正文 {}", - "非常长的计划内容 plan body ".repeat(200) - ); - assert!(long_content.len() > 1_000); - - conn.execute( - "INSERT INTO agent_messages (id, session_id, role, content, sequence, created_at, images) - VALUES (?1, ?2, 'user', ?3, ?4, ?5, NULL)", - params![ - "rewritten-id", - "session-1", - &long_content, - 1_i64, - "2026-05-27T00:00:00Z", - ], - ) - .unwrap(); - - // Pre-existing event row from the original submit (different - // message id, content truncated like build_searchable_content). - let truncated = user_content_dedup_key(&long_content); - conn.execute( - "INSERT INTO events - (id, session_id, event_type, function_name, thread_id, args_json, result_json, - content, created_at, meta_json, history_sequence) - VALUES (?1, ?2, 'raw', 'user_message', NULL, '{}', ?3, ?4, ?5, '{}', 1)", - params![ - "user-message-original-id", - "session-1", - r#"{"backendPersisted":true}"#, - format!("user_message {truncated}"), - "2026-05-27T00:00:00Z", - ], - ) - .unwrap(); - - assert_eq!(backfill_missing_user_events(&conn, "session-1").unwrap(), 0); - } - - #[test] - fn synthetic_user_input_does_not_start_turn() { - let rows = vec![ - row( - "user-input-optimistic", - Some(USER_MESSAGE_FUNCTION), - r#"{"syntheticUserInput":true}"#, - 1, - ), - row("assistant-event", Some("assistant_message"), "{}", 2), - row( - "user-message-authoritative", - Some(USER_MESSAGE_FUNCTION), - r#"{"backendPersisted":true}"#, - 3, - ), - ]; - - let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); - - assert_eq!(drafts.len(), 1); - assert_eq!(drafts[0].turn_id, "user-message-authoritative"); - assert_eq!(drafts[0].start_sequence, 3); - } - - #[test] - fn imported_user_alias_starts_turn() { - let rows = vec![ - row( - "imported-user", - Some(IMPORTED_USER_MESSAGE_FUNCTION), - "{}", - 1, - ), - row("assistant-event", Some("assistant"), "{}", 2), - ]; - - let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); - - assert_eq!(drafts.len(), 1); - assert_eq!(drafts[0].turn_id, "imported-user"); - assert_eq!(drafts[0].body_event_count, 1); - } - - #[test] - fn consecutive_user_messages_do_not_materialize_ghost_pending_turns() { - let rows = vec![ - row( - "user-message-queued-ghost", - Some(USER_MESSAGE_FUNCTION), - r#"{"backendPersisted":true}"#, - 1, - ), - row( - "user-message-authoritative", - Some(USER_MESSAGE_FUNCTION), - r#"{"backendPersisted":true}"#, - 2, - ), - row("assistant-event", Some("assistant_message"), "{}", 3), - ]; - - let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); - - assert_eq!(drafts.len(), 1); - assert_eq!(drafts[0].turn_id, "user-message-authoritative"); - assert_eq!(drafts[0].start_sequence, 2); - assert_eq!(drafts[0].body_event_count, 1); - } - - #[test] - fn latest_user_only_turn_still_materializes_as_pending() { - let rows = vec![row( - "user-message-latest", - Some(USER_MESSAGE_FUNCTION), - r#"{"backendPersisted":true}"#, - 1, - )]; - - let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); - - assert_eq!(drafts.len(), 1); - assert_eq!(drafts[0].turn_id, "user-message-latest"); - assert_eq!(drafts[0].body_event_count, 0); - } - - #[test] - fn two_rows_with_same_turn_intent_id_collapse_into_one_round() { - // The optimistic synthetic event is normally filtered out by - // `is_synthetic_user_input`, but a backend can also legitimately - // persist two user_message rows under the same intent (inbox - // transcript followed by main submit). The indexer must collapse - // them into a single round so the user sees one bubble, not two. - let intent = r#"{"backendPersisted":true,"turnIntentId":"intent-A"}"#; - let rows = vec![ - row("user-message-1", Some(USER_MESSAGE_FUNCTION), intent, 1), - row("user-message-2", Some(USER_MESSAGE_FUNCTION), intent, 2), - row("assistant-event", Some("assistant_message"), "{}", 3), - ]; - - let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); - - assert_eq!(drafts.len(), 1); - // The first user_message that opened the round wins as turn_id; - // both user event ids are tracked. - assert_eq!(drafts[0].turn_id, "user-message-1"); - assert_eq!( - drafts[0].user_event_ids, - vec!["user-message-1".to_string(), "user-message-2".to_string()] - ); - assert_eq!(drafts[0].event_count, 3); - assert_eq!(drafts[0].body_event_count, 1); - } - - #[test] - fn rows_with_stale_intent_are_dropped() { - // Reproduces the Stop + model switch + Send Now path: the first - // submit's intent was retired (stale) before its user_message - // row was even persisted. The indexer must not paint a phantom - // round for it. - let stale = r#"{"backendPersisted":true,"turnIntentId":"intent-stale"}"#; - let fresh = r#"{"backendPersisted":true,"turnIntentId":"intent-fresh"}"#; - let rows = vec![ - row("user-message-stale", Some(USER_MESSAGE_FUNCTION), stale, 1), - row("user-message-fresh", Some(USER_MESSAGE_FUNCTION), fresh, 2), - row("assistant-event", Some("assistant_message"), "{}", 3), - ]; - - let mut stale_ids = StaleIntentIds::new(); - stale_ids.insert("intent-stale".to_string()); - let drafts = build_turn_drafts(&rows, &stale_ids); - - assert_eq!(drafts.len(), 1); - assert_eq!(drafts[0].turn_id, "user-message-fresh"); - } - - #[test] - fn rows_with_distinct_turn_intent_ids_stay_separate() { - let intent_a = r#"{"backendPersisted":true,"turnIntentId":"intent-A"}"#; - let intent_b = r#"{"backendPersisted":true,"turnIntentId":"intent-B"}"#; - let rows = vec![ - row("user-message-a", Some(USER_MESSAGE_FUNCTION), intent_a, 1), - row("assistant-1", Some("assistant_message"), "{}", 2), - row("user-message-b", Some(USER_MESSAGE_FUNCTION), intent_b, 3), - row("assistant-2", Some("assistant_message"), "{}", 4), - ]; - - let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); - - assert_eq!(drafts.len(), 2); - assert_eq!(drafts[0].turn_id, "user-message-a"); - assert_eq!(drafts[1].turn_id, "user-message-b"); - } - - #[test] - fn round_metadata_is_projected_by_orgtrack_from_normalized_provider_events() { - let mut read = row("read-1", Some("Read"), "{}", 2); - read.args_json = r#"{"file_path":"src/lib.rs"}"#.to_string(); - let mut search = row( - "search-1", - Some("Grep"), - r#"{"matches":[{"file":"src/lib.rs"},{"file":"src/main.rs"}]}"#, - 3, - ); - search.args_json = r#"{"path":"src"}"#.to_string(); - let rows = vec![ - row( - "user-message-1", - Some(USER_MESSAGE_FUNCTION), - r#"{"backendPersisted":true}"#, - 1, - ), - read, - search, - ]; - - let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); - - assert_eq!(drafts.len(), 1); - assert!(drafts[0] - .metadata_accumulator - .resource_interactions() - .iter() - .any(|item| item.path == "src/lib.rs" && item.action.as_str() == "read")); - // search-rows: the Grep is projected away entirely, so `src/main.rs` — - // named only by that search's matches — never reaches the index. - assert!(!drafts[0] - .metadata_accumulator - .resource_interactions() - .iter() - .any(|item| item.action.as_str() == "search")); - assert!(!drafts[0] - .metadata_accumulator - .resource_interactions() - .iter() - .any(|item| item.path == "src/main.rs")); - } -} +mod tests; diff --git a/src-tauri/crates/session-persistence/src/turn_index/tests.rs b/src-tauri/crates/session-persistence/src/turn_index/tests.rs new file mode 100644 index 000000000..743bb754c --- /dev/null +++ b/src-tauri/crates/session-persistence/src/turn_index/tests.rs @@ -0,0 +1,587 @@ +use super::*; +use rusqlite::types::Value as SqlValue; + +fn row(id: &str, function_name: Option<&str>, result_json: &str, sequence: i64) -> IndexEventRow { + IndexEventRow { + id: id.to_string(), + function_name: function_name.map(str::to_string), + args_json: "{}".to_string(), + result_json: result_json.to_string(), + content: id.to_string(), + created_at: "2026-05-27T00:00:00Z".to_string(), + order_sequence: sequence, + } +} + +fn create_backfill_test_tables(conn: &Connection) { + crate::schema::init_session_tables(conn).unwrap(); + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS agent_messages ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + role TEXT NOT NULL, + content TEXT NOT NULL, + sequence INTEGER NOT NULL, + created_at TEXT NOT NULL, + images TEXT + );", + ) + .unwrap(); +} + +fn insert_index_test_event( + conn: &Connection, + id: &str, + function_name: &str, + args_json: SqlValue, + result_json: SqlValue, + content: SqlValue, + sequence: i64, +) { + conn.execute( + "INSERT INTO events + (id, session_id, event_type, function_name, thread_id, args_json, result_json, + content, created_at, meta_json, history_sequence) + VALUES (?1, 'session-1', 'raw', ?2, NULL, ?3, ?4, ?5, ?6, '{}', ?7)", + params![ + id, + function_name, + args_json, + result_json, + content, + format!("2026-07-20T00:00:{sequence:02}Z"), + sequence, + ], + ) + .unwrap(); +} + +#[test] +fn backfill_missing_user_events_is_idempotent() { + let conn = Connection::open_in_memory().unwrap(); + create_backfill_test_tables(&conn); + conn.execute( + "INSERT INTO agent_messages (id, session_id, role, content, sequence, created_at, images) + VALUES (?1, ?2, 'user', ?3, ?4, ?5, NULL)", + params![ + "message-1", + "session-1", + "hello from persisted user", + 1_i64, + "2026-05-27T00:00:00Z", + ], + ) + .unwrap(); + + assert_eq!(backfill_missing_user_events(&conn, "session-1").unwrap(), 1); + assert_eq!(backfill_missing_user_events(&conn, "session-1").unwrap(), 0); + + let count: i64 = conn + .query_row( + "SELECT COUNT(*) FROM events WHERE session_id = ?1 AND id = ?2", + params!["session-1", "user-message-message-1"], + |row| row.get(0), + ) + .unwrap(); + assert_eq!(count, 1); +} + +#[test] +fn backfill_dedups_long_messages_against_truncated_event_previews() { + // After a transcript rewrite (compaction) agent_messages rows get + // fresh ids, so id-based dedup misses and we fall back to content + // matching. Event rows store content truncated to 500 bytes; the + // full agent_messages content must still match instead of + // re-inserting a duplicate "[Plan approved] …" user bubble. + let conn = Connection::open_in_memory().unwrap(); + create_backfill_test_tables(&conn); + + let long_content = format!( + "[Plan approved] Implement the approved plan now. 计划正文 {}", + "非常长的计划内容 plan body ".repeat(200) + ); + assert!(long_content.len() > 1_000); + + conn.execute( + "INSERT INTO agent_messages (id, session_id, role, content, sequence, created_at, images) + VALUES (?1, ?2, 'user', ?3, ?4, ?5, NULL)", + params![ + "rewritten-id", + "session-1", + &long_content, + 1_i64, + "2026-05-27T00:00:00Z", + ], + ) + .unwrap(); + + // Pre-existing event row from the original submit (different + // message id, content truncated like build_searchable_content). + let truncated = user_content_dedup_key(&long_content); + conn.execute( + "INSERT INTO events + (id, session_id, event_type, function_name, thread_id, args_json, result_json, + content, created_at, meta_json, history_sequence) + VALUES (?1, ?2, 'raw', 'user_message', NULL, '{}', ?3, ?4, ?5, '{}', 1)", + params![ + "user-message-original-id", + "session-1", + r#"{"backendPersisted":true}"#, + format!("user_message {truncated}"), + "2026-05-27T00:00:00Z", + ], + ) + .unwrap(); + + assert_eq!(backfill_missing_user_events(&conn, "session-1").unwrap(), 0); +} + +#[test] +fn rebuild_skips_large_columns_for_known_no_metadata_rows() { + let conn = Connection::open_in_memory().unwrap(); + create_backfill_test_tables(&conn); + insert_index_test_event( + &conn, + "user-1", + USER_MESSAGE_FUNCTION, + SqlValue::Text("{}".to_string()), + SqlValue::Text(r#"{"backendPersisted":true}"#.to_string()), + SqlValue::Text("user_message inspect memory".to_string()), + 1, + ); + + // SQLite's dynamic typing lets this fixture use BLOBs in TEXT + // columns. `Row::get::` would fail, so a successful rebuild + // proves these large columns were not materialized by Rust. + let large_blob = vec![b'x'; 2 * 1024 * 1024]; + for (sequence, id, function_name) in [ + (2, "assistant-1", "assistant"), + (3, "thinking-1", "thinking"), + (4, "node-1", "node_repl"), + ] { + insert_index_test_event( + &conn, + id, + function_name, + SqlValue::Blob(large_blob.clone()), + SqlValue::Blob(large_blob.clone()), + SqlValue::Blob(large_blob.clone()), + sequence, + ); + } + insert_index_test_event( + &conn, + "grep-1", + "Grep", + SqlValue::Text(r#"{"path":"src"}"#.to_string()), + SqlValue::Blob(large_blob), + SqlValue::Blob(vec![b'y'; 1024 * 1024]), + 5, + ); + + rebuild_turn_index_on_connection( + &conn, + "session-1", + &StaleIntentIds::new(), + &IntentStatusOverlay::new(), + ) + .unwrap(); + + let (event_count, body_event_count): (i64, i64) = conn + .query_row( + "SELECT event_count, body_event_count FROM session_turns + WHERE session_id = 'session-1' AND turn_id = 'user-1'", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .unwrap(); + assert_eq!(event_count, 5); + assert_eq!(body_event_count, 4); +} + +#[test] +fn rebuild_rolls_back_old_index_when_conservative_payload_read_fails() { + let conn = Connection::open_in_memory().unwrap(); + create_backfill_test_tables(&conn); + conn.execute( + "INSERT INTO session_turns + (session_id, turn_id, start_sequence, started_at, status, updated_at) + VALUES ('session-1', 'old-turn', 0, '2026-07-19T00:00:00Z', 'completed', + '2026-07-19T00:00:01Z')", + [], + ) + .unwrap(); + conn.execute( + "INSERT INTO session_turn_index_state + (session_id, indexed_event_count, indexed_max_sequence, rebuilt_at, index_version) + VALUES ('session-1', 99, 98, '2026-07-19T00:00:01Z', 11)", + [], + ) + .unwrap(); + insert_index_test_event( + &conn, + "user-1", + USER_MESSAGE_FUNCTION, + SqlValue::Text("{}".to_string()), + SqlValue::Text(r#"{"backendPersisted":true}"#.to_string()), + SqlValue::Text("user_message trigger rebuild".to_string()), + 1, + ); + insert_index_test_event( + &conn, + "future-1", + "future_provider_tool", + SqlValue::Blob(vec![b'x'; 1024]), + SqlValue::Text("{}".to_string()), + SqlValue::Text(String::new()), + 2, + ); + + let error = rebuild_turn_index_on_connection( + &conn, + "session-1", + &StaleIntentIds::new(), + &IntentStatusOverlay::new(), + ) + .unwrap_err(); + assert!(matches!(error, rusqlite::Error::InvalidColumnType(..))); + + let turn_ids: Vec = conn + .prepare( + "SELECT turn_id FROM session_turns WHERE session_id = 'session-1' + ORDER BY turn_id", + ) + .unwrap() + .query_map([], |row| row.get(0)) + .unwrap() + .collect::>>() + .unwrap(); + assert_eq!(turn_ids, vec!["old-turn"]); + let index_version: i64 = conn + .query_row( + "SELECT index_version FROM session_turn_index_state + WHERE session_id = 'session-1'", + [], + |row| row.get(0), + ) + .unwrap(); + assert_eq!(index_version, 11); +} + +#[test] +fn synthetic_user_input_does_not_start_turn() { + let rows = vec![ + row( + "user-input-optimistic", + Some(USER_MESSAGE_FUNCTION), + r#"{"syntheticUserInput":true}"#, + 1, + ), + row("assistant-event", Some("assistant_message"), "{}", 2), + row( + "user-message-authoritative", + Some(USER_MESSAGE_FUNCTION), + r#"{"backendPersisted":true}"#, + 3, + ), + ]; + + let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); + + assert_eq!(drafts.len(), 1); + assert_eq!(drafts[0].turn_id, "user-message-authoritative"); + assert_eq!(drafts[0].start_sequence, 3); +} + +#[test] +fn imported_user_alias_starts_turn() { + let rows = vec![ + row( + "imported-user", + Some(IMPORTED_USER_MESSAGE_FUNCTION), + "{}", + 1, + ), + row("assistant-event", Some("assistant"), "{}", 2), + ]; + + let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); + + assert_eq!(drafts.len(), 1); + assert_eq!(drafts[0].turn_id, "imported-user"); + assert_eq!(drafts[0].body_event_count, 1); +} + +#[test] +fn consecutive_user_messages_do_not_materialize_ghost_pending_turns() { + let rows = vec![ + row( + "user-message-queued-ghost", + Some(USER_MESSAGE_FUNCTION), + r#"{"backendPersisted":true}"#, + 1, + ), + row( + "user-message-authoritative", + Some(USER_MESSAGE_FUNCTION), + r#"{"backendPersisted":true}"#, + 2, + ), + row("assistant-event", Some("assistant_message"), "{}", 3), + ]; + + let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); + + assert_eq!(drafts.len(), 1); + assert_eq!(drafts[0].turn_id, "user-message-authoritative"); + assert_eq!(drafts[0].start_sequence, 2); + assert_eq!(drafts[0].body_event_count, 1); +} + +#[test] +fn latest_user_only_turn_still_materializes_as_pending() { + let rows = vec![row( + "user-message-latest", + Some(USER_MESSAGE_FUNCTION), + r#"{"backendPersisted":true}"#, + 1, + )]; + + let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); + + assert_eq!(drafts.len(), 1); + assert_eq!(drafts[0].turn_id, "user-message-latest"); + assert_eq!(drafts[0].body_event_count, 0); +} + +#[test] +fn two_rows_with_same_turn_intent_id_collapse_into_one_round() { + // The optimistic synthetic event is normally filtered out by + // `is_synthetic_user_input`, but a backend can also legitimately + // persist two user_message rows under the same intent (inbox + // transcript followed by main submit). The indexer must collapse + // them into a single round so the user sees one bubble, not two. + let intent = r#"{"backendPersisted":true,"turnIntentId":"intent-A"}"#; + let rows = vec![ + row("user-message-1", Some(USER_MESSAGE_FUNCTION), intent, 1), + row("user-message-2", Some(USER_MESSAGE_FUNCTION), intent, 2), + row("assistant-event", Some("assistant_message"), "{}", 3), + ]; + + let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); + + assert_eq!(drafts.len(), 1); + // The first user_message that opened the round wins as turn_id; + // both user event ids are tracked. + assert_eq!(drafts[0].turn_id, "user-message-1"); + assert_eq!( + drafts[0].user_event_ids, + vec!["user-message-1".to_string(), "user-message-2".to_string()] + ); + assert_eq!(drafts[0].event_count, 3); + assert_eq!(drafts[0].body_event_count, 1); +} + +#[test] +fn rows_with_stale_intent_are_dropped() { + // Reproduces the Stop + model switch + Send Now path: the first + // submit's intent was retired (stale) before its user_message + // row was even persisted. The indexer must not paint a phantom + // round for it. + let stale = r#"{"backendPersisted":true,"turnIntentId":"intent-stale"}"#; + let fresh = r#"{"backendPersisted":true,"turnIntentId":"intent-fresh"}"#; + let rows = vec![ + row("user-message-stale", Some(USER_MESSAGE_FUNCTION), stale, 1), + row("user-message-fresh", Some(USER_MESSAGE_FUNCTION), fresh, 2), + row("assistant-event", Some("assistant_message"), "{}", 3), + ]; + + let mut stale_ids = StaleIntentIds::new(); + stale_ids.insert("intent-stale".to_string()); + let drafts = build_turn_drafts(&rows, &stale_ids); + + assert_eq!(drafts.len(), 1); + assert_eq!(drafts[0].turn_id, "user-message-fresh"); +} + +#[test] +fn rows_with_distinct_turn_intent_ids_stay_separate() { + let intent_a = r#"{"backendPersisted":true,"turnIntentId":"intent-A"}"#; + let intent_b = r#"{"backendPersisted":true,"turnIntentId":"intent-B"}"#; + let rows = vec![ + row("user-message-a", Some(USER_MESSAGE_FUNCTION), intent_a, 1), + row("assistant-1", Some("assistant_message"), "{}", 2), + row("user-message-b", Some(USER_MESSAGE_FUNCTION), intent_b, 3), + row("assistant-2", Some("assistant_message"), "{}", 4), + ]; + + let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); + + assert_eq!(drafts.len(), 2); + assert_eq!(drafts[0].turn_id, "user-message-a"); + assert_eq!(drafts[1].turn_id, "user-message-b"); +} + +#[test] +fn round_metadata_is_projected_by_orgtrack_from_normalized_provider_events() { + let mut read = row("read-1", Some("Read"), "{}", 2); + read.args_json = r#"{"file_path":"src/lib.rs"}"#.to_string(); + let mut search = row( + "search-1", + Some("Grep"), + r#"{"matches":[{"file":"src/lib.rs"},{"file":"src/main.rs"}]}"#, + 3, + ); + search.args_json = r#"{"path":"src"}"#.to_string(); + let rows = vec![ + row( + "user-message-1", + Some(USER_MESSAGE_FUNCTION), + r#"{"backendPersisted":true}"#, + 1, + ), + read, + search, + ]; + + let drafts = build_turn_drafts(&rows, &StaleIntentIds::new()); + + assert_eq!(drafts.len(), 1); + assert!(drafts[0] + .metadata_accumulator + .resource_interactions() + .iter() + .any(|item| item.path == "src/lib.rs" && item.action.as_str() == "read")); + // search-rows: the Grep is projected away entirely, so `src/main.rs` — + // named only by that search's matches — never reaches the index. + assert!(!drafts[0] + .metadata_accumulator + .resource_interactions() + .iter() + .any(|item| item.action.as_str() == "search")); + assert!(!drafts[0] + .metadata_accumulator + .resource_interactions() + .iter() + .any(|item| item.path == "src/main.rs")); +} + +#[cfg(unix)] +fn peak_rss_bytes() -> usize { + let mut usage = std::mem::MaybeUninit::::zeroed(); + let result = unsafe { libc::getrusage(libc::RUSAGE_SELF, usage.as_mut_ptr()) }; + assert_eq!(result, 0, "getrusage failed"); + let peak = unsafe { usage.assume_init() }.ru_maxrss as usize; + #[cfg(target_os = "macos")] + { + peak + } + #[cfg(not(target_os = "macos"))] + { + peak.saturating_mul(1024) + } +} + +/// Real-machine acceptance harness for the v10/v11 -> current turn-index +/// migration that originally exposed #443's multi-gigabyte peak. The +/// caller must point `ORGII_HOME` at a disposable copy under the operating +/// system temp directory; the safety assertion runs before opening SQLite. +#[cfg(unix)] +#[test] +#[ignore = "#443 real DB migration/RSS acceptance; requires a disposable ORGII_HOME copy"] +fn real_db_turn_index_migrates_and_reopens_with_bounded_rss() { + let session_id = std::env::var("ORGII_ISSUE_443_REAL_SESSION_ID") + .expect("set ORGII_ISSUE_443_REAL_SESSION_ID to the large copied session"); + let orgii_home = std::env::var_os("ORGII_HOME") + .map(std::path::PathBuf::from) + .expect("set ORGII_HOME to a disposable real-DB copy"); + let canonical_home = + std::fs::canonicalize(&orgii_home).expect("canonicalize disposable real-DB copy home"); + let canonical_temp = std::fs::canonicalize(std::env::temp_dir()) + .expect("canonicalize operating system temp directory"); + #[cfg(target_os = "macos")] + let is_macos_private_tmp = canonical_home.starts_with("/private/tmp"); + #[cfg(not(target_os = "macos"))] + let is_macos_private_tmp = false; + assert!( + canonical_home.starts_with(&canonical_temp) || is_macos_private_tmp, + "refusing to mutate a real DB outside the temp directory: {}", + canonical_home.display() + ); + let db_path = canonical_home.join("sessions.db"); + assert!( + db_path.is_file(), + "missing copied DB: {}", + db_path.display() + ); + + let read_only = Connection::open_with_flags( + &db_path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX, + ) + .expect("open copied DB read-only for the pre-migration state"); + let before_version: i64 = read_only + .query_row( + "SELECT index_version FROM session_turn_index_state WHERE session_id = ?1", + [&session_id], + |row| row.get(0), + ) + .expect("read pre-migration turn-index version"); + let event_count: i64 = read_only + .query_row( + "SELECT COUNT(*) FROM events WHERE session_id = ?1", + [&session_id], + |row| row.get(0), + ) + .expect("count copied session events"); + drop(read_only); + + let baseline_peak = peak_rss_bytes(); + let first_started = std::time::Instant::now(); + let first = load_turn_index(&session_id).expect("migrate and load copied turn index"); + let first_elapsed = first_started.elapsed(); + let first_peak = peak_rss_bytes(); + let first_growth = first_peak.saturating_sub(baseline_peak); + assert!(!first.is_empty(), "copied large session must contain turns"); + + let conn = get_connection().expect("reopen copied sessions DB after migration"); + let (indexed_event_count, after_version): (i64, i64) = conn + .query_row( + "SELECT indexed_event_count, index_version + FROM session_turn_index_state + WHERE session_id = ?1", + [&session_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("read migrated turn-index state"); + drop(conn); + assert_eq!(indexed_event_count, event_count); + assert_eq!(after_version, TURN_INDEX_VERSION); + + let reopen_started = std::time::Instant::now(); + let reopened = load_turn_index(&session_id).expect("reopen current copied turn index"); + let reopen_elapsed = reopen_started.elapsed(); + let reopened_peak = peak_rss_bytes(); + assert_eq!(reopened.len(), first.len()); + assert_eq!( + reopened.first().map(|turn| &turn.turn_id), + first.first().map(|turn| &turn.turn_id) + ); + assert_eq!( + reopened.last().map(|turn| &turn.turn_id), + first.last().map(|turn| &turn.turn_id) + ); + assert!( + first_growth <= 400 * 1024 * 1024, + "real DB first open grew peak RSS by {first_growth} bytes" + ); + + eprintln!( + "#443 real DB: session={session_id}, events={event_count}, turns={}, index v{before_version}->v{after_version}, first_open={first_elapsed:?}, reopen={reopen_elapsed:?}, baseline_peak={:.1} MiB, first_peak={:.1} MiB, reopened_peak={:.1} MiB, first_growth={:.1} MiB", + first.len(), + baseline_peak as f64 / (1024.0 * 1024.0), + first_peak as f64 / (1024.0 * 1024.0), + reopened_peak as f64 / (1024.0 * 1024.0), + first_growth as f64 / (1024.0 * 1024.0), + ); +} diff --git a/src-tauri/crates/types/src/session_event.rs b/src-tauri/crates/types/src/session_event.rs index 999620078..6e4e5179d 100644 --- a/src-tauri/crates/types/src/session_event.rs +++ b/src-tauri/crates/types/src/session_event.rs @@ -106,6 +106,26 @@ pub struct PayloadRef { pub preview: String, pub full_size_bytes: usize, pub truncated: bool, + /// External replay ranges can contain either a complete serialized JSON + /// value or decoded UTF-8 string content. Native refs leave this unset so + /// their existing wire shape and persistence semantics remain unchanged. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub replay_encoding: Option, + /// External replay locators point into provider-owned storage rather than + /// the sessions.db event cache. Optional fields preserve native refs. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub replay_source_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub replay_generation: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub replay_source_event_id: Option, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum PayloadRefEncoding { + JsonValue, + Utf8Text, } /// Stable reference to one append-only shell replay artifact. @@ -673,6 +693,30 @@ mod tests { ); } + #[test] + fn replay_payload_encoding_is_optional_without_changing_native_wire_shape() { + let mut payload_ref = PayloadRef { + event_id: "event-1".to_string(), + field_path: "result.output".to_string(), + preview: "preview".to_string(), + full_size_bytes: 1024, + truncated: true, + replay_encoding: None, + replay_source_id: None, + replay_generation: None, + replay_source_event_id: None, + }; + let native_wire = serde_json::to_value(&payload_ref).expect("native payload ref wire"); + assert!(native_wire.get("replayEncoding").is_none()); + + payload_ref.replay_encoding = Some(PayloadRefEncoding::Utf8Text); + let replay_wire = serde_json::to_value(&payload_ref).expect("replay payload ref wire"); + assert_eq!(replay_wire["replayEncoding"], "utf8_text"); + + let legacy: PayloadRef = serde_json::from_value(native_wire).expect("legacy payload ref"); + assert_eq!(legacy.replay_encoding, None); + } + #[test] fn incremental_snapshot_delta_uses_the_expected_wire_fields() { let delta = SnapshotDelta { diff --git a/src-tauri/src/agent_sessions/cli/commands.rs b/src-tauri/src/agent_sessions/cli/commands.rs index 866a9c478..9fc3fb6d1 100644 --- a/src-tauri/src/agent_sessions/cli/commands.rs +++ b/src-tauri/src/agent_sessions/cli/commands.rs @@ -1,13 +1,13 @@ //! Tauri commands for CLI agent session management. //! //! Split into focused submodules: -//! - `create` — `cli_agent_create` session/worktree provisioning +//! - `create` — session/worktree provisioning //! - `launch_profile` — get/update/reset per-agent CLI launch profile -//! - `resume_delete` — `cli_agent_resume` / `cli_agent_delete` -//! - `run` — `cli_agent_run` / `cli_agent_message` / `cli_agent_approval_response` -//! - `status` — status/history/cancel/list queries -//! - `transcript` — native/legacy transcript chunk loading and truncation -//! - `worktree` — `cli_agent_merge` / `cli_agent_worktree_diff` / `cli_agent_worktree_discard` +//! - `resume_delete` — resume and delete lifecycle +//! - `run` — run, message and approval responses +//! - `status` — status, history, cancel and list queries +//! - `transcript` — transcript path lookup and truncation +//! - `worktree` — merge, diff and discard operations mod create; mod launch_profile; @@ -17,9 +17,8 @@ mod status; mod transcript; mod worktree; -// Glob re-exports so each `#[tauri::command]`'s generated `__cmd__` macro is -// re-exported alongside the fn, keeping `commands::` resolvable for -// `generate_handler!` (which references `commands::__cmd__`). +// Glob re-exports keep each `#[tauri::command]` macro-generated +// `__cmd__` symbol reachable from the existing handler paths. pub use create::*; pub use launch_profile::*; pub use resume_delete::*; diff --git a/src-tauri/src/agent_sessions/cli/commands/run.rs b/src-tauri/src/agent_sessions/cli/commands/run.rs index 43e097a5f..461fd3fe7 100644 --- a/src-tauri/src/agent_sessions/cli/commands/run.rs +++ b/src-tauri/src/agent_sessions/cli/commands/run.rs @@ -29,11 +29,9 @@ fn inject_ide_context_into_prompt(user_input: &str, ide_context: Option<&IdeCont /// tab close). Non-TUI sessions and already-terminal rows are left alone. #[tauri::command] pub async fn cli_agent_tui_release(session_id: String) -> Result { - tokio::task::spawn_blocking(move || { - super::super::tui_bridge::release_tui_session(&session_id) - }) - .await - .map_err(|e| format!("Task error: {}", e))? + tokio::task::spawn_blocking(move || super::super::tui_bridge::release_tui_session(&session_id)) + .await + .map_err(|e| format!("Task error: {}", e))? } /// Run a code session (spawn CLI agent in background). diff --git a/src-tauri/src/agent_sessions/cli/commands/transcript.rs b/src-tauri/src/agent_sessions/cli/commands/transcript.rs index c04defaf1..e2ad5792e 100644 --- a/src-tauri/src/agent_sessions/cli/commands/transcript.rs +++ b/src-tauri/src/agent_sessions/cli/commands/transcript.rs @@ -1,53 +1,7 @@ -//! Transcript loading and mutation — native/legacy chunk resolution -//! (`cli_agent_chunks`, `cli_agent_transcript_path`) and message-edit -//! truncation (`cli_agent_truncate_after_chunk`). +//! Transcript location and message-edit truncation. -use super::super::persistence::{self, CodeSession}; +use super::super::persistence; use super::super::session_runner; -use core_types::activity::ActivityChunk; - -/// Resolve and parse a native-mode session's transcript from the CLI's own -/// store through the imported-history loaders. `None` falls back to legacy -/// chunks — covering pre-migration sessions, crash-before-native-write, and -/// a store the reader can't currently open. -fn load_native_transcript_chunks(session: &CodeSession) -> Option> { - use super::super::native_transcript; - if session.transcript_source != native_transcript::TRANSCRIPT_SOURCE_NATIVE { - return None; - } - let agent = session - .cli_agent_type - .as_deref() - .and_then(key_vault::key_store::ModelType::from_str)?; - let binding = native_transcript::native_transcript_binding(&agent)?; - let cli_session_id = - persistence::latest_native_transcript_id(&session.session_id, binding.source) - .ok() - .flatten() - .or_else(|| session.cli_session_id.clone())?; - let imported_id = binding.imported_session_id(&cli_session_id); - let conn = database::db::get_connection().ok()?; - match orgtrack_core::sources::imported_history::load_activity_chunks_for_session( - &conn, - &imported_id, - ) { - Ok(Some(mut chunks)) if !chunks.is_empty() => { - // Loaders stamp the imported id; the frontend event store, - // WS merge, and snapshot keys all key on the managed id. - for chunk in &mut chunks { - chunk.session_id = session.session_id.clone(); - } - Some(chunks) - } - Ok(_) => None, - Err(err) => { - tracing::warn!( - "[cli_agent_chunks] Native transcript load failed for {imported_id}: {err}" - ); - None - } - } -} /// Where a managed session's transcript of record lives, for display /// surfaces (session hover card storage row). @@ -114,68 +68,6 @@ pub async fn cli_agent_transcript_path( .map_err(|e| format!("Task error: {}", e))? } -/// A failed first turn in native mode may leave no readable transcript at -/// all; a synthesized user bubble beats a blank chat. -fn synthesized_user_message_chunk(session: &CodeSession) -> Option { - let user_input = session.user_input.as_deref()?.trim(); - if user_input.is_empty() { - return None; - } - let mut chunk = ActivityChunk::new(&session.session_id, "raw", "user_message"); - chunk.chunk_id = format!("user-input-{}-synthesized", session.session_id); - chunk.created_at = session.created_at.clone(); - chunk.result = serde_json::json!({ - "type": "user", - "message": { "content": user_input, "role": "user" } - }); - Some(chunk) -} - -/// Load persisted chunks for a session (for resume/session switch). -/// Native-transcript sessions route through the imported-history loaders; -/// everything else (and every fallback) reads legacy `code_session_chunks`. -#[tauri::command] -pub async fn cli_agent_chunks(session_id: String) -> Result, String> { - tracing::info!( - "[cli_agent_chunks] Loading chunks for session: {}", - session_id - ); - let result = tokio::task::spawn_blocking(move || { - let session = - persistence::get_session(&session_id).map_err(|e| format!("DB error: {}", e))?; - if let Some(session) = session.as_ref() { - if let Some(chunks) = load_native_transcript_chunks(session) { - return Ok(chunks); - } - } - let chunks = - persistence::load_chunks(&session_id).map_err(|e| format!("DB error: {}", e))?; - if chunks.is_empty() { - if let Some(chunk) = session - .as_ref() - .filter(|session| { - session.transcript_source - == super::super::native_transcript::TRANSCRIPT_SOURCE_NATIVE - }) - .and_then(synthesized_user_message_chunk) - { - return Ok(vec![chunk]); - } - } - Ok(chunks) - }) - .await - .map_err(|e| format!("Task error: {}", e))?; - - match &result { - Ok(chunks) => { - tracing::info!("[cli_agent_chunks] Loaded {} chunks", chunks.len()) - } - Err(ref err) => tracing::error!("[cli_agent_chunks] Failed: {}", err), - } - result -} - /// Truncate chunks at and after a specific timestamp. /// Used for message editing — removes chunks at or after the given timestamp, /// kills the running agent, clears CLI resume state, and optionally restores file snapshots. diff --git a/src-tauri/src/agent_sessions/cli/mod.rs b/src-tauri/src/agent_sessions/cli/mod.rs index a55052372..91c035677 100644 --- a/src-tauri/src/agent_sessions/cli/mod.rs +++ b/src-tauri/src/agent_sessions/cli/mod.rs @@ -2,12 +2,14 @@ //! //! Manages CLI agent sessions (Cursor, Claude Code, Codex, Gemini, Kiro, Copilot). //! -//! SQLite table names: `cli_agent_sessions` and `cli_agent_chunks`. +//! SQLite session metadata lives in `cli_agent_sessions`; readerless managed +//! CLI deltas use the bounded `code_session_chunks` adapter, while CLIs with a +//! native transcript mirror reuse their imported-history source adapter. //! //! ## Components //! //! - `parsers` — Stdout parsers for each CLI agent (Cursor, Claude Code, Codex, Gemini, etc.) -//! - `persistence` — SQLite CRUD for `cli_agent_sessions` + `cli_agent_chunks` tables +//! - `persistence` — session metadata plus bounded `code_session_chunks` writes //! - `session_runner` — Spawns CLI agent subprocess, pipes stdout through parser, broadcasts events //! - `commands` — Tauri commands exposed to the frontend diff --git a/src-tauri/src/agent_sessions/cli/native_transcript.rs b/src-tauri/src/agent_sessions/cli/native_transcript.rs index 5104395ab..ac20ea050 100644 --- a/src-tauri/src/agent_sessions/cli/native_transcript.rs +++ b/src-tauri/src/agent_sessions/cli/native_transcript.rs @@ -3,10 +3,10 @@ //! A managed session whose agent has a binding here persists NO transcript //! chunks of its own (`code_sessions.transcript_source = 'native'`): the CLI //! writes its native store (inside the ORGII profile dirs, which the -//! imported-history readers scan as extra discovery roots) and replay routes -//! through `imported_history::load_activity_chunks_for_session` keyed by +//! imported-history readers scan as extra discovery roots) and bounded replay +//! routes through the source registry keyed by //! ``. Agents without a binding keep the -//! legacy `code_session_chunks` path. +//! bounded SQL adapter over `code_session_chunks`. use key_vault::key_store::ModelType; use orgtrack_core::sources::imported_history::metadata::{ diff --git a/src-tauri/src/agent_sessions/cli/persistence/chunk_ops.rs b/src-tauri/src/agent_sessions/cli/persistence/chunk_ops.rs index 800607d84..5b250e1de 100644 --- a/src-tauri/src/agent_sessions/cli/persistence/chunk_ops.rs +++ b/src-tauri/src/agent_sessions/cli/persistence/chunk_ops.rs @@ -1,10 +1,10 @@ -use rusqlite::{params, Result as SqliteResult}; +use rusqlite::{params, OptionalExtension, Result as SqliteResult}; use agent_core::foundation::session_bridge; use core_types::activity::ActivityChunk; use database::db::get_connection; -use super::session_crud::{clear_cli_resume_state_with_tx, now_iso}; +use super::session_crud::{bump_history_mutation_with_tx, clear_cli_resume_state_with_tx, now_iso}; /// Get the maximum sequence number for a session's chunks. /// Returns -1 if no chunks exist (so base_sequence + 1 == 0 for first run). @@ -20,7 +20,7 @@ pub fn max_chunk_sequence(session_id: &str) -> SqliteResult { /// Store an ActivityChunk. pub fn insert_chunk(chunk: &ActivityChunk, sequence: i64) -> SqliteResult<()> { - let conn = get_connection()?; + let mut conn = get_connection()?; // `serde_json::to_string` on `serde_json::Value` is infallible — the // value tree was already validated when the chunk was constructed. // Using `expect` here (instead of the previous silent fallback to @@ -33,7 +33,53 @@ pub fn insert_chunk(chunk: &ActivityChunk, sequence: i64) -> SqliteResult<()> { let result_str = serde_json::to_string(&chunk.result) .expect("ActivityChunk.result -> JSON string is infallible for Value"); - conn.execute( + let tx = conn.transaction()?; + let previous = tx + .query_row( + "SELECT session_id,action_type,function,args_json,result_json, + thread_id,process_id,sequence,created_at + FROM code_session_chunks WHERE chunk_id=?1", + [&chunk.chunk_id], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, String>(2)?, + row.get::<_, String>(3)?, + row.get::<_, String>(4)?, + row.get::<_, Option>(5)?, + row.get::<_, Option>(6)?, + row.get::<_, i64>(7)?, + row.get::<_, String>(8)?, + )) + }, + ) + .optional()?; + let replacement_changed = previous.as_ref().is_some_and( + |( + old_session_id, + old_action_type, + old_function, + old_args_json, + old_result_json, + old_thread_id, + old_process_id, + old_sequence, + old_created_at, + )| { + old_session_id != &chunk.session_id + || old_action_type != &chunk.action_type + || old_function != &chunk.function + || old_args_json != &args_str + || old_result_json != &result_str + || old_thread_id != &chunk.thread_id + || old_process_id != &chunk.process_id + || *old_sequence != sequence + || old_created_at != &chunk.created_at + }, + ); + + tx.execute( "INSERT OR REPLACE INTO code_session_chunks (chunk_id, session_id, action_type, function, args_json, result_json, thread_id, process_id, sequence, created_at) @@ -52,6 +98,22 @@ pub fn insert_chunk(chunk: &ActivityChunk, sequence: i64) -> SqliteResult<()> { ], )?; + if replacement_changed { + let mut affected_sessions = vec![chunk.session_id.as_str()]; + if let Some((old_session_id, ..)) = previous.as_ref() { + affected_sessions.push(old_session_id.as_str()); + } + affected_sessions.sort_unstable(); + affected_sessions.dedup(); + let mutated_at = now_iso(); + for session_id in affected_sessions { + bump_history_mutation_with_tx(&tx, session_id, "chunk_replaced", &mutated_at)?; + } + } + tx.commit()?; + + // Database replay and its mutation epoch are now committed atomically; + // lineage/subagent side effects remain best-effort and run afterwards. run_chunk_side_effects_with_args(chunk, args_str); Ok(()) @@ -191,62 +253,53 @@ fn truncate_label(s: &str) -> String { } } -/// Load all chunks for a session, ordered by sequence. -pub fn load_chunks(session_id: &str) -> SqliteResult> { +/// Load only the newest user/assistant text needed by the fresh-process +/// context bridge. SQLite extracts and tail-bounds the scalar text before it +/// crosses into Rust, so a large readerless managed-CLI history is never +/// materialized as `Vec` just to keep 24 short messages. +pub fn load_recent_context_messages( + session_id: &str, + max_messages: usize, + max_chars_per_message: usize, +) -> SqliteResult> { let conn = get_connection()?; let mut stmt = conn.prepare( - "SELECT chunk_id, session_id, action_type, function, - args_json, result_json, thread_id, process_id, created_at - FROM code_session_chunks - WHERE session_id = ?1 - ORDER BY sequence ASC", + "WITH candidate AS ( + SELECT sequence, + CASE WHEN function = 'user_message' + THEN 'User' ELSE 'Assistant' END AS role, + CASE + WHEN json_type(result_json, '$.message.content') = 'text' + THEN json_extract(result_json, '$.message.content') + WHEN json_type(result_json, '$.content') = 'text' + THEN json_extract(result_json, '$.content') + WHEN json_type(result_json, '$.observation') = 'text' + THEN json_extract(result_json, '$.observation') + ELSE NULL + END AS text + FROM code_session_chunks + WHERE session_id = ?1 + AND ( + function = 'user_message' + OR action_type IN ( + 'assistant', 'assistant_delta', 'message', 'message_delta' + ) + ) + ) + SELECT role, substr(text, -?3) + FROM candidate + WHERE text IS NOT NULL AND trim(text) <> '' + ORDER BY sequence DESC + LIMIT ?2", )?; - let rows = stmt.query_map([session_id], |row| { - let args_str: String = row.get(4)?; - let result_str: String = row.get(5)?; - // The args/result columns are written as serialized JSON by the - // chunk writer. Silently rendering a corrupt blob as `{}` - // (the previous behaviour) made it impossible to tell whether - // a tool call genuinely had no arguments or whether the row - // had been corrupted out of band — both look identical to the - // frontend, but the second case is a real data-integrity bug - // that would have stayed invisible. Surface a typed - // `FromSqlConversionFailure` instead so the loader returns - // an error and the UI can show a real failure state. - let args = serde_json::from_str::(&args_str).map_err(|err| { - rusqlite::Error::FromSqlConversionFailure( - 4, - rusqlite::types::Type::Text, - format!("invalid args_json for chunk: {err}").into(), - ) - })?; - let result = serde_json::from_str::(&result_str).map_err(|err| { - rusqlite::Error::FromSqlConversionFailure( - 5, - rusqlite::types::Type::Text, - format!("invalid result_json for chunk: {err}").into(), - ) - })?; - Ok(ActivityChunk { - chunk_id: row.get(0)?, - session_id: row.get(1)?, - action_type: row.get(2)?, - function: row.get(3)?, - args, - result, - thread_id: row.get(6)?, - process_id: row.get(7)?, - created_at: row.get(8)?, - broadcast_only: false, - }) + let max_messages = i64::try_from(max_messages).unwrap_or(i64::MAX).max(1); + let max_chars = i64::try_from(max_chars_per_message) + .unwrap_or(i64::MAX) + .max(1); + let rows = stmt.query_map(params![session_id, max_messages, max_chars], |row| { + Ok((row.get(0)?, row.get(1)?)) })?; - let chunks: Vec = rows.collect::>>()?; - tracing::info!( - "[load_chunks] session={}, returned {} chunks", - session_id, - chunks.len() - ); - Ok(chunks) + rows.collect() } /// Truncate chunks at and after a specific timestamp. @@ -285,3 +338,145 @@ pub fn truncate_chunks_after_with_reason( Ok(deleted as i64) } + +#[cfg(test)] +mod tests { + use super::*; + + fn mutation_test_chunk(chunk_id: &str, session_id: &str, content: &str) -> ActivityChunk { + let mut chunk = ActivityChunk::new(session_id, "tool_result", "run_command_line"); + chunk.chunk_id = chunk_id.to_string(); + chunk.args = serde_json::json!({"command":"printf payload"}); + chunk.result = serde_json::json!({"output":content}); + chunk.created_at = "2026-07-23T00:00:00Z".to_string(); + chunk + } + + fn history_epoch(session_id: &str) -> Option { + get_connection() + .expect("history epoch DB") + .query_row( + "SELECT epoch FROM code_session_history_mutations WHERE session_id=?1", + [session_id], + |row| row.get(0), + ) + .optional() + .expect("history epoch") + } + + #[test] + fn chunk_replace_bumps_only_affected_history_epochs_atomically() { + let _sandbox = test_helpers::test_env::sandbox(); + let conn = get_connection().expect("test database"); + crate::agent_sessions::cli::init_cli_agent_tables(&conn).expect("CLI schema"); + for session_id in ["cliagent-old", "cliagent-new"] { + conn.execute( + "INSERT INTO code_sessions(session_id,created_at,updated_at) + VALUES(?1,'2026-07-23T00:00:00Z','2026-07-23T00:00:00Z')", + [session_id], + ) + .expect("chunk test session"); + } + drop(conn); + + let original = mutation_test_chunk("mutable-chunk", "cliagent-old", "AAAA"); + insert_chunk(&original, 7).expect("pure append"); + assert_eq!(history_epoch("cliagent-old"), None, "append is a delta"); + + insert_chunk(&original, 7).expect("idempotent replace"); + assert_eq!( + history_epoch("cliagent-old"), + None, + "byte-identical replace must not reset replay" + ); + + let changed = mutation_test_chunk("mutable-chunk", "cliagent-old", "BBBB"); + assert_eq!( + serde_json::to_string(&original.result) + .expect("old result") + .len(), + serde_json::to_string(&changed.result) + .expect("changed result") + .len() + ); + insert_chunk(&changed, 7).expect("same-length content replacement"); + assert_eq!(history_epoch("cliagent-old"), Some(1)); + + let append = mutation_test_chunk("append-chunk", "cliagent-old", "CCCC"); + insert_chunk(&append, 8).expect("append after replacement"); + assert_eq!( + history_epoch("cliagent-old"), + Some(1), + "later append keeps the current generation" + ); + + let moved = mutation_test_chunk("mutable-chunk", "cliagent-new", "BBBB"); + insert_chunk(&moved, 7).expect("cross-session replacement"); + assert_eq!(history_epoch("cliagent-old"), Some(2)); + assert_eq!(history_epoch("cliagent-new"), Some(1)); + let stored_session = get_connection() + .expect("stored chunk DB") + .query_row( + "SELECT session_id FROM code_session_chunks WHERE chunk_id='mutable-chunk'", + [], + |row| row.get::<_, String>(0), + ) + .expect("moved chunk row"); + assert_eq!(stored_session, "cliagent-new"); + } + + #[test] + fn recent_context_query_is_row_and_text_bounded_before_rust() { + let _sandbox = test_helpers::test_env::sandbox(); + let conn = get_connection().expect("test database"); + crate::agent_sessions::cli::init_cli_agent_tables(&conn).expect("CLI schema"); + conn.execute_batch("PRAGMA foreign_keys=OFF") + .expect("standalone chunk fixture"); + + for sequence in 0..30_i64 { + let function = if sequence % 2 == 0 { + "user_message" + } else { + "assistant_message" + }; + let action = if sequence % 2 == 0 { + "message" + } else { + "assistant" + }; + let content = format!("{}:{sequence}", "x".repeat(20_000)); + conn.execute( + "INSERT INTO code_session_chunks( + chunk_id,session_id,action_type,function,args_json,result_json, + sequence,created_at + ) VALUES(?1,'bounded-context',?2,?3,'{}',?4,?5,'2026-07-22T00:00:00Z')", + params![ + format!("context-{sequence}"), + action, + function, + serde_json::json!({"content": content}).to_string(), + sequence, + ], + ) + .expect("context row"); + } + // A tool payload is outside the projection and must never be parsed. + conn.execute( + "INSERT INTO code_session_chunks( + chunk_id,session_id,action_type,function,args_json,result_json, + sequence,created_at + ) VALUES('irrelevant-tool','bounded-context','tool_result','Bash','{}', + 'not-json',31,'2026-07-22T00:00:00Z')", + [], + ) + .expect("irrelevant malformed payload"); + drop(conn); + + let rows = load_recent_context_messages("bounded-context", 24, 12_000) + .expect("bounded context rows"); + assert_eq!(rows.len(), 24); + assert!(rows[0].1.ends_with(":29")); + assert!(rows[23].1.ends_with(":6")); + assert!(rows.iter().all(|(_, text)| text.chars().count() <= 12_000)); + } +} diff --git a/src-tauri/src/agent_sessions/cli/persistence/mod.rs b/src-tauri/src/agent_sessions/cli/persistence/mod.rs index 6e31bd102..e11c17949 100644 --- a/src-tauri/src/agent_sessions/cli/persistence/mod.rs +++ b/src-tauri/src/agent_sessions/cli/persistence/mod.rs @@ -15,6 +15,10 @@ mod resume_state_tests { use super::*; use crate::test_utils::test_env; use agent_core::foundation::session_bridge; + use orgtrack_core::sources::imported_history::{ + cache::upsert_imported_session_cache_from_conn, + metadata::{ImportedHistoryCacheInput, ImportedHistoryImpactStats, SOURCE_CLAUDE_CODE}, + }; fn create_test_session(session_id: &str, account_id: &str) { create_session( @@ -52,6 +56,37 @@ mod resume_state_tests { .expect("create test CLI session"); } + fn imported_cache_input( + source_session_id: &str, + updated_at_ms: i64, + ) -> ImportedHistoryCacheInput { + ImportedHistoryCacheInput { + source: SOURCE_CLAUDE_CODE, + source_session_id: source_session_id.to_string(), + session_id: format!("claudecodeapp-{source_session_id}"), + source_path: format!("/tmp/{source_session_id}.jsonl"), + source_record_key: source_session_id.to_string(), + source_mtime_ms: updated_at_ms, + source_size_bytes: 100, + source_fingerprint: updated_at_ms.to_string(), + parser_version: 1, + name: format!("Session {source_session_id}"), + created_at_ms: updated_at_ms, + updated_at_ms, + model: Some("claude-sonnet-4-6".to_string()), + input_tokens: 0, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + repo_path: Some("/tmp".to_string()), + branch: Some("main".to_string()), + impact: ImportedHistoryImpactStats::default(), + listable: true, + source_metadata_json: None, + parent_session_id: None, + } + } + #[test] fn cli_resume_state_is_scoped_by_account_and_restored_on_switch_back() { let _sandbox = test_env::sandbox(); @@ -91,6 +126,45 @@ mod resume_state_tests { assert_eq!(session.cli_session_id.as_deref(), Some("native-b-1")); } + #[test] + fn binding_native_id_hides_only_the_literal_imported_suffix() { + let _sandbox = test_env::sandbox(); + let session_id = "cli-resume-literal-wildcards"; + create_test_session(session_id, "account-a"); + let mut conn = database::db::get_connection().expect("open fixture database"); + upsert_imported_session_cache_from_conn( + &mut conn, + &[ + imported_cache_input("rollout-native%_", 100), + imported_cache_input("rollout-nativeXY", 200), + ], + ) + .expect("insert imported cache fixtures"); + drop(conn); + + update_cli_session_id(session_id, "native%_").expect("bind native id"); + + let conn = database::db::get_connection().expect("reopen fixture database"); + let literal_listable: i64 = conn + .query_row( + "SELECT listable FROM imported_history_session_cache + WHERE source = ?1 AND source_session_id = ?2", + rusqlite::params![SOURCE_CLAUDE_CODE, "rollout-native%_"], + |row| row.get(0), + ) + .expect("literal cache row"); + let decoy_listable: i64 = conn + .query_row( + "SELECT listable FROM imported_history_session_cache + WHERE source = ?1 AND source_session_id = ?2", + rusqlite::params![SOURCE_CLAUDE_CODE, "rollout-nativeXY"], + |row| row.get(0), + ) + .expect("decoy cache row"); + assert_eq!(literal_listable, 0); + assert_eq!(decoy_listable, 1); + } + #[test] fn model_switch_on_same_account_preserves_legacy_single_column_resume_id() { let _sandbox = test_env::sandbox(); diff --git a/src-tauri/src/agent_sessions/cli/persistence/session_crud.rs b/src-tauri/src/agent_sessions/cli/persistence/session_crud.rs index 54b456985..8d1c97050 100644 --- a/src-tauri/src/agent_sessions/cli/persistence/session_crud.rs +++ b/src-tauri/src/agent_sessions/cli/persistence/session_crud.rs @@ -166,12 +166,19 @@ pub fn list_sessions() -> SqliteResult> { rows.collect() } -/// One page of sessions ordered by recent activity. Serves the sidebar's -/// paginated category view without loading the whole table. -pub fn list_sessions_page(limit: usize, offset: usize) -> SqliteResult> { +/// One page of top-level sessions ordered by recent activity. +/// +/// This query serves the sidebar category pager. Filtering children before +/// `LIMIT`/`OFFSET` is essential: managed OpenCode and Agent Org rows can +/// otherwise occupy an entire SQL page even though the sidebar hides them. +/// Child-session APIs continue to use [`list_sessions`] or direct parent +/// queries and are unaffected. +pub fn list_root_sessions_page(limit: usize, offset: usize) -> SqliteResult> { let conn = get_connection()?; let query = format!( - "SELECT {} FROM code_sessions cs ORDER BY cs.updated_at DESC LIMIT ?1 OFFSET ?2", + "SELECT {} FROM code_sessions cs + WHERE cs.parent_session_id IS NULL OR cs.parent_session_id = '' + ORDER BY cs.updated_at DESC LIMIT ?1 OFFSET ?2", SESSION_COLUMNS ); let limit = limit.min(i64::MAX as usize) as i64; @@ -341,7 +348,10 @@ pub fn update_cli_session_id_for_account( "UPDATE imported_history_session_cache SET listable = 0 WHERE source = ?1 - AND (source_session_id = ?2 OR source_session_id LIKE '%-' || ?2)", + AND ( + source_session_id = ?2 + OR substr(source_session_id, -(length(?2) + 1)) = ('-' || ?2) + )", params![binding.source, cli_session_id], ) .ok(); @@ -853,3 +863,71 @@ fn row_to_session(row: &rusqlite::Row) -> rusqlite::Result { transcript_source: row.get(41)?, }) } + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_utils::test_env; + + fn insert_sidebar_fixture(session_id: &str, parent_session_id: Option<&str>, updated_at: &str) { + let conn = get_connection().expect("open sandbox database"); + conn.execute( + "INSERT INTO code_sessions + (session_id, name, status, flow, runner, cli_agent_type, + parent_session_id, created_at, updated_at) + VALUES (?1, ?1, 'completed', 'quick', 'local', 'opencode', + ?2, '2026-07-26T00:00:00Z', ?3)", + params![session_id, parent_session_id, updated_at], + ) + .expect("insert CLI sidebar fixture"); + } + + #[test] + fn root_page_offsets_ignore_dense_managed_cli_children() { + let _sandbox = test_env::sandbox(); + + for (root_index, second) in [50, 30, 10].into_iter().enumerate() { + let root_id = format!("cliagent-root-{root_index}"); + insert_sidebar_fixture(&root_id, None, &format!("2026-07-26T12:00:{second:02}Z")); + if root_index < 2 { + for child_index in 1..=9 { + insert_sidebar_fixture( + &format!("opencodeapp-child-{root_index}-{child_index}"), + Some(&root_id), + &format!("2026-07-26T12:00:{:02}Z", second - child_index), + ); + } + } + } + + // Sidebar callers ask for page_size + 1 roots. The extra row makes + // hasMore=true without allowing the 18 child rows to consume slots. + let first_page = list_root_sessions_page(3, 0).expect("load first root-only CLI page"); + assert_eq!( + first_page + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + vec!["cliagent-root-0", "cliagent-root-1", "cliagent-root-2"] + ); + + let second_page = list_root_sessions_page(3, 2).expect("load second root-only CLI page"); + assert_eq!( + second_page + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + vec!["cliagent-root-2"] + ); + + assert_eq!( + list_sessions() + .expect("ordinary CLI listing") + .iter() + .filter(|session| session.parent_session_id.is_some()) + .count(), + 18, + "root pagination must not remove children from the ordinary APIs" + ); + } +} diff --git a/src-tauri/src/agent_sessions/cli/session_runner/context_bridge.rs b/src-tauri/src/agent_sessions/cli/session_runner/context_bridge.rs index 8bbd89ed6..34aed7ac7 100644 --- a/src-tauri/src/agent_sessions/cli/session_runner/context_bridge.rs +++ b/src-tauri/src/agent_sessions/cli/session_runner/context_bridge.rs @@ -1,51 +1,24 @@ //! Context bridge building — injects prior ORGII conversation history into CLI //! sessions that have no native conversation state. -use core_types::activity::ActivityChunk; - use super::super::persistence; const CONTEXT_BRIDGE_MAX_CHARS: usize = 12_000; const CONTEXT_BRIDGE_MAX_MESSAGES: usize = 24; -pub(super) fn chunk_text(chunk: &ActivityChunk) -> Option { - let result = &chunk.result; - let text = result - .get("message") - .and_then(|message| message.get("content")) - .and_then(|value| value.as_str()) - .or_else(|| result.get("content").and_then(|value| value.as_str())) - .or_else(|| result.get("observation").and_then(|value| value.as_str()))?; - let trimmed = text.trim(); - if trimmed.is_empty() { - return None; - } - Some(trimmed.to_string()) -} - -pub(super) fn chunk_role(chunk: &ActivityChunk) -> Option<&'static str> { - if chunk.function == "user_message" { - return Some("User"); - } - match chunk.action_type.as_str() { - "assistant" | "assistant_delta" | "message" | "message_delta" => Some("Assistant"), - _ => None, - } -} - pub(super) fn build_context_bridge(session_id: &str) -> Option { - let chunks = persistence::load_chunks(session_id).ok()?; + let messages = persistence::load_recent_context_messages( + session_id, + CONTEXT_BRIDGE_MAX_MESSAGES, + CONTEXT_BRIDGE_MAX_CHARS, + ) + .ok()?; let mut lines = Vec::new(); - for chunk in chunks.iter().rev() { - if lines.len() >= CONTEXT_BRIDGE_MAX_MESSAGES { - break; - } - let Some(role) = chunk_role(chunk) else { - continue; - }; - let Some(text) = chunk_text(chunk) else { + for (role, text) in messages { + let text = text.trim(); + if text.is_empty() { continue; - }; + } lines.push(format!("{role}: {text}")); } if lines.is_empty() { diff --git a/src-tauri/src/agent_sessions/cli/session_runner/session/transport_app_server.rs b/src-tauri/src/agent_sessions/cli/session_runner/session/transport_app_server.rs index 84db0fb82..7c9c63e5f 100644 --- a/src-tauri/src/agent_sessions/cli/session_runner/session/transport_app_server.rs +++ b/src-tauri/src/agent_sessions/cli/session_runner/session/transport_app_server.rs @@ -7,9 +7,9 @@ use tokio::process::Child; use crate::api::websocket_handler; +use super::super::super::persistence; use super::super::helpers::{emit_chunk, snapshot_cli_file_edit}; use super::super::launch_profiles::ResolvedCliLaunchProfile; -use super::super::super::persistence; pub(super) struct AppServerOutcome { pub(super) exit_code: i32, @@ -71,11 +71,9 @@ pub(super) async fn run_codex_app_server_branch( if cli_session_id_out.is_none() { if let Some(ref tid) = chunk.thread_id { cli_session_id_out = Some(tid.clone()); - if let Err(err) = persistence::update_cli_session_id_for_account( - &session_id, - account_id, - tid, - ) { + if let Err(err) = + persistence::update_cli_session_id_for_account(&session_id, account_id, tid) + { tracing::warn!( "[CodeSession] Failed to bind early cli_session_id: {}", err @@ -106,21 +104,19 @@ pub(super) async fn run_codex_app_server_branch( codex_app_server_turn_ok = result.turn_status != "failed"; if let Some(ref usage) = result.usage { let round_model = usage.model.as_deref().or(model); - if let Err(err) = - session_persistence::token_usage::insert_token_usage_record( - &session_id, - "code", - round_model, - account_id, - usage.input_tokens as i64, - usage.output_tokens as i64, - usage.cache_read_tokens as i64, - usage.cache_write_tokens as i64, - usage.total_tokens as i64, - 0, - None, - ) - { + if let Err(err) = session_persistence::token_usage::insert_token_usage_record( + &session_id, + "code", + round_model, + account_id, + usage.input_tokens as i64, + usage.output_tokens as i64, + usage.cache_read_tokens as i64, + usage.cache_write_tokens as i64, + usage.total_tokens as i64, + 0, + None, + ) { tracing::warn!( "[CodeSession] Failed to insert per-round token usage: {}", err diff --git a/src-tauri/src/agent_sessions/cli/session_runner/session/transport_standard.rs b/src-tauri/src/agent_sessions/cli/session_runner/session/transport_standard.rs index 72146a968..301ee539e 100644 --- a/src-tauri/src/agent_sessions/cli/session_runner/session/transport_standard.rs +++ b/src-tauri/src/agent_sessions/cli/session_runner/session/transport_standard.rs @@ -13,8 +13,13 @@ use tokio::sync::Mutex; use crate::api::websocket_handler; use key_vault::key_store::ModelType; +use super::super::super::persistence; +use super::super::super::persistence::CodeSession; +use super::super::super::types::SessionStatus; use super::super::command::create_parser; -use super::super::helpers::{clear_live_status, emit_chunk, flush_and_broadcast, snapshot_cli_file_edit}; +use super::super::helpers::{ + clear_live_status, emit_chunk, flush_and_broadcast, snapshot_cli_file_edit, +}; use super::super::oauth_setup::{ is_cli_chunk_replay_unsafe, is_cli_oauth_failure_message, is_cli_oauth_stderr_retry_candidate, is_retryable_cli_oauth_failure_chunk, is_retryable_overloaded_chunk, @@ -23,9 +28,6 @@ use super::super::plan_approval::{ create_plan_content_from_chunk, is_successful_mode_tool, plan_candidate_path_from_chunk, register_cli_plan_approval, register_synthetic_cli_plan_approval, }; -use super::super::super::persistence; -use super::super::super::persistence::CodeSession; -use super::super::super::types::SessionStatus; const CLI_PLAN_GATE_NATURAL_EXIT_GRACE_SECS: u64 = 45; diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/analytics.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/analytics.rs index 98ead6ae8..94ea0213a 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/analytics.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/analytics.rs @@ -7,6 +7,7 @@ use tauri::State; use crate::agent_sessions::event_pipeline::analytics::{ self, MultiSessionSummary, SessionAnalytics, }; +use crate::agent_sessions::event_pipeline::session_providers; use crate::agent_sessions::event_pipeline::types::SessionEvent; use session_persistence as sqlite_cache; @@ -32,6 +33,7 @@ pub async fn es_compute_analytics( pub async fn es_compute_cached_session_analytics( session_id: String, ) -> Result { + session_providers::reject_bounded_replay_full_load(&session_id)?; let cached = tokio::task::spawn_blocking(move || sqlite_cache::load_events(&session_id)) .await .map_err(|e| e.to_string())? @@ -46,6 +48,9 @@ pub async fn es_compute_cached_session_analytics( pub async fn es_compute_multi_session_analytics( session_ids: Vec, ) -> Result { + for session_id in &session_ids { + session_providers::reject_bounded_replay_full_load(session_id)?; + } let session_events = tokio::task::spawn_blocking(move || { let mut result: Vec<(String, Vec)> = Vec::new(); for session_id in &session_ids { diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/batch_update.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/batch_update.rs index 7bb4ded8d..0dfd5db6d 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/batch_update.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/batch_update.rs @@ -22,6 +22,7 @@ pub async fn es_complete_last_running( let sid = state.resolve_session_id(session_id)?; let result = state.with_store_mut(&sid, |store| store.complete_last_running()); if result.is_some() { + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); } Ok(result) @@ -37,8 +38,10 @@ pub async fn es_patch_by_ids( patch: SessionEventPatch, ) -> Result { let sid = state.resolve_session_id(session_id)?; + state.validate_bounded_replay_patch(&sid, &ids, &patch)?; let count = state.with_store_mut(&sid, |store| store.patch_by_ids(&ids, &patch)); if count > 0 { + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); } Ok(count) @@ -55,6 +58,7 @@ pub async fn es_remove_by_id_prefix( let sid = state.resolve_session_id(session_id)?; let removed = state.with_store_mut(&sid, |store| store.remove_by_id_prefix(&prefix)); if removed > 0 { + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); } Ok(removed) @@ -70,6 +74,7 @@ pub async fn es_remove_synthetic_user_inputs( let sid = state.resolve_session_id(session_id)?; let removed = state.with_store_mut(&sid, |store| store.remove_synthetic_user_inputs()); if removed > 0 { + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); } Ok(removed) @@ -86,9 +91,12 @@ pub async fn es_replace_and_remove( new_event: SessionEvent, ) -> Result { let sid = state.resolve_session_id(session_id)?; + let incoming_bytes = + state.validate_bounded_replay_input(&sid, std::slice::from_ref(&new_event))?; state.with_store_mut(&sid, |store| { store.replace_and_remove(remove_id.as_deref(), new_event); }); + state.account_bounded_replay_write(&sid, incoming_bytes)?; schedule_notify(&app, &state, &sid); Ok(true) } @@ -106,10 +114,12 @@ pub async fn es_update_active_task_args( let default_names = vec!["task".to_string()]; let names = function_names.unwrap_or(default_names); let names_refs: Vec<&str> = names.iter().map(|s| s.as_str()).collect(); + state.validate_bounded_replay_args_merge(&sid, &names_refs, &merge_args)?; let result = state.with_store_mut(&sid, |store| { store.update_spawning_tool_args(&names_refs, merge_args) }); if result.is_some() { + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); } Ok(result) diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/cache_bridge.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/cache_bridge.rs index ff801360d..0b2ad4122 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/cache_bridge.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/cache_bridge.rs @@ -22,10 +22,6 @@ use super::{ BULK_WRITE_MAX_RETRIES, }; -fn try_load_provider_history_events(session_id: &str) -> Result, String> { - session_providers::load_history_events(session_id) -} - // ============================================================================ // SQLite Bridge Commands // ============================================================================ @@ -41,6 +37,7 @@ pub async fn es_load_from_cache( state: State<'_, EventStoreState>, session_id: String, ) -> Result { + session_providers::reject_bounded_replay_full_load(&session_id)?; let existing_count = state .with_store_opt(&session_id, |store| store.events().len()) .unwrap_or(0); @@ -54,21 +51,11 @@ pub async fn es_load_from_cache( .await .map_err(|e| e.to_string())? .map_err(|e| e.to_string())?; - let mut events: Vec = cached + let events: Vec = cached .into_iter() .map(|ce| cached_event_to_session_event(&ce)) .collect(); - if events.is_empty() { - match try_load_provider_history_events(&session_id) { - Ok(loaded) if !loaded.is_empty() => events = loaded, - Ok(_) => {} - Err(err) => tracing::warn!( - "[cache_bridge] failed to load provider history for {session_id}: {err}" - ), - } - } - let events = prepare_loaded_events(&session_id, events); let count = events.len(); if count > 0 { @@ -217,24 +204,30 @@ pub async fn cache_finalize_session_event_import(session_id: String) -> Result Result, String> { log::debug!("[cache_bridge] cache_load_session_events called for session_id={session_id}"); + session_providers::reject_bounded_replay_full_load(&session_id)?; let sid = session_id.clone(); let cached = tokio::task::spawn_blocking(move || sqlite_cache::load_events(&sid)) .await .map_err(|e| e.to_string())? .map_err(|e| e.to_string())?; - let mut events: Vec = cached.iter().map(cached_event_to_session_event).collect(); - if events.is_empty() { - match try_load_provider_history_events(&session_id) { - Ok(loaded) if !loaded.is_empty() => events = loaded, - Ok(_) => {} - Err(err) => tracing::warn!( - "[cache_bridge] failed to load provider history for {session_id}: {err}" - ), - } - } + let events: Vec = cached.iter().map(cached_event_to_session_event).collect(); Ok(prepare_loaded_events(&session_id, events)) } +/// Compatibility rows for the dormant hosted-key bridge. Keep the wire shape +/// while enforcing the same native-only boundary as the SessionEvent loader; +/// external transcripts must use their bounded replay adapter instead. +#[tauri::command] +pub async fn cache_load_native_cached_events( + session_id: String, +) -> Result, String> { + session_providers::reject_bounded_replay_full_load(&session_id)?; + tokio::task::spawn_blocking(move || sqlite_cache::load_events(&session_id)) + .await + .map_err(|e| e.to_string())? + .map_err(|e| e.to_string()) +} + /// Search events via LIKE substring matching, returning SessionEvents directly. #[tauri::command] pub async fn cache_search_session_events( @@ -372,6 +365,7 @@ pub async fn cache_save_full_session(payload: FullSessionPayload) -> Result<(), pub async fn cache_load_full_session( session_id: String, ) -> Result, String> { + session_providers::reject_bounded_replay_full_load(&session_id)?; let result = tokio::task::spawn_blocking(move || sqlite_cache::load_session(&session_id)) .await .map_err(|e| e.to_string())? @@ -392,717 +386,4 @@ pub async fn cache_load_full_session( } #[cfg(test)] -mod tests { - use super::{ - cached_event_to_session_event, is_synthetic_persistence_artifact, - session_event_to_cached_event, - }; - use crate::agent_sessions::event_pipeline::commands::event_conversion::{ - compact_boundary_row_to_event, dedup_by_call_id, is_ts_placeholder_id, CompactBoundaryRow, - }; - use crate::agent_sessions::event_pipeline::ingestion::prompt_backfill; - use crate::agent_sessions::event_pipeline::types::{ - ActivityStatus, EventDisplayStatus, EventDisplayVariant, EventSource, PayloadRef, - SessionEvent, - }; - use core_types::activity::ActivityChunk; - - const OPENCODE_SUBAGENT_USER_PROMPT: &str = "启动一个（subagent），让它帮我分析当前项目里有多少个 .rs 文件，并生成一份报告。必须要用subagent，然后要让我看到过程"; - const OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT: &str = "在当前工作目录下分析 Rust 源文件数量：统计所有 **/*.rs 文件，排除 target/ 目录；生成一份报告，包含总文件数、按目录分布、最大文件 Top 5，并在过程中持续汇报进展。"; - const FINAL_REPORT_CONTENT: &str = "Now I have all the data. Here is the comprehensive report."; - const FINAL_ASSISTANT_ANSWER: &str = - "Subagent 已完成分析：当前项目共有 260 个 .rs 文件，并已生成报告。"; - - #[test] - fn ts_placeholder_msg_and_think_ids_match() { - assert!(is_ts_placeholder_id("stream-msg-ts-session-1776099853993")); - assert!(is_ts_placeholder_id( - "stream-think-ts-session-1776099853993" - )); - } - - #[test] - fn cached_event_normalizes_legacy_string_result() { - let cached = session_persistence::CachedEvent { - id: "legacy-string-result".to_string(), - session_id: "session-history-regression".to_string(), - event_type: "message".to_string(), - function_name: Some("message".to_string()), - thread_id: None, - args_json: "{}".to_string(), - result_json: "\"loaded historical assistant text\"".to_string(), - content: "loaded historical assistant text".to_string(), - created_at: "2026-05-16T00:00:00.000Z".to_string(), - meta_json: Some( - serde_json::json!({ - "source": "assistant", - "displayText": "loaded historical assistant text", - "displayStatus": "completed", - "displayVariant": "message", - "activityStatus": "agent", - "uiCanonical": "message" - }) - .to_string(), - ), - history_sequence: None, - }; - - let event = cached_event_to_session_event(&cached); - let result = event.result.as_object().expect("result must be normalized"); - assert_eq!( - result.get("content").and_then(|value| value.as_str()), - Some("loaded historical assistant text") - ); - assert_eq!( - result.get("observation").and_then(|value| value.as_str()), - Some("loaded historical assistant text") - ); - } - - #[test] - fn cached_event_normalizes_legacy_string_args() { - let cached = session_persistence::CachedEvent { - id: "legacy-string-args".to_string(), - session_id: "session-history-regression".to_string(), - event_type: "tool_call".to_string(), - function_name: Some("tool_call".to_string()), - thread_id: None, - args_json: "\"legacy arguments\"".to_string(), - result_json: "{}".to_string(), - content: "legacy arguments".to_string(), - created_at: "2026-05-16T00:00:00.000Z".to_string(), - meta_json: Some( - serde_json::json!({ - "source": "assistant", - "displayText": "legacy arguments", - "displayStatus": "completed", - "displayVariant": "tool_call", - "activityStatus": "agent", - "uiCanonical": "tool_call" - }) - .to_string(), - ), - history_sequence: None, - }; - - let event = cached_event_to_session_event(&cached); - let args = event.args.as_object().expect("args must be normalized"); - assert_eq!( - args.get("content").and_then(|value| value.as_str()), - Some("legacy arguments") - ); - assert_eq!( - args.get("observation").and_then(|value| value.as_str()), - Some("legacy arguments") - ); - } - - #[test] - fn rust_authoritative_ids_do_not_match() { - assert!(!is_ts_placeholder_id( - "stream-msg-sdeagent-a91612f3-4f94-4fac-a0c2-f6e85f0c1f63-1" - )); - assert!(!is_ts_placeholder_id( - "stream-think-sdeagent-a91612f3-4f94-4fac-a0c2-f6e85f0c1f63-1" - )); - } - - #[test] - fn unrelated_event_ids_do_not_match() { - assert!(!is_ts_placeholder_id("tool-call-42")); - assert!(!is_ts_placeholder_id("user-msg-1")); - assert!(!is_ts_placeholder_id("")); - // Prefix must be the full "stream-msg-ts-" / "stream-think-ts-" — - // ids like "stream-msg-tsfoo-…" are not placeholders. - assert!(!is_ts_placeholder_id("stream-msg-tsfoo")); - } - - #[test] - fn turn_placeholder_is_synthetic_persistence_artifact() { - let placeholder = make_tool_call( - "turn-placeholder-turn-1", - None, - "turn_placeholder", - serde_json::json!({}), - serde_json::json!({ "unloadedTurn": { "turnId": "turn-1" } }), - ); - assert!(is_synthetic_persistence_artifact(&placeholder)); - - let mut synthetic_header = make_tool_call( - "turn-1", - None, - "user_message", - serde_json::json!({}), - serde_json::json!({ "syntheticTurnHeader": true }), - ); - synthetic_header.source = EventSource::User; - assert!(is_synthetic_persistence_artifact(&synthetic_header)); - - let normal = make_tool_call( - "tool-call-42", - None, - "bash", - serde_json::json!({}), - serde_json::json!({}), - ); - assert!(!is_synthetic_persistence_artifact(&normal)); - } - - #[test] - fn compacted_event_is_synthetic_persistence_artifact() { - let mut compacted = make_tool_call( - "tool-call-compacted", - None, - "bash", - serde_json::json!({ "streamOutput": "preview" }), - serde_json::json!({}), - ); - compacted.payload_refs.push(PayloadRef { - event_id: compacted.id.clone(), - field_path: "args.streamOutput".to_string(), - preview: "preview".to_string(), - full_size_bytes: 128 * 1024, - truncated: true, - }); - - assert!(is_synthetic_persistence_artifact(&compacted)); - } - - #[test] - fn backfill_provider_subagent_prompts_uses_child_assignment_for_real_prompt() { - let mut event = make_tool_call( - "opencode-subagent-real-user-prompt-fixture", - Some("call-opencode-real-user-prompt-fixture"), - "subagent", - serde_json::json!({ - "description": "Task", - "prompt": "Task", - "subagentSessionId": "opencodeapp-child-real-assignment" - }), - serde_json::json!({ - "content": "Now I have all the data. Here is the comprehensive report.", - "summary": "Subagent 已完成分析，结果如下" - }), - ); - event.ui_canonical = "subagent".to_string(); - - let mut events = vec![event]; - prompt_backfill::backfill_subagent_prompts_with_resolver(&mut events, |child_session_id| { - assert_eq!(child_session_id, "opencodeapp-child-real-assignment"); - Some(OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT.to_string()) - }); - - assert_eq!( - events[0].args["prompt"], - OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT - ); - assert_eq!( - events[0].args["description"], - OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT - ); - assert_ne!(events[0].args["prompt"], OPENCODE_SUBAGENT_USER_PROMPT); - assert_ne!(events[0].args["prompt"], "Task"); - assert_ne!( - events[0].args["prompt"], - "Now I have all the data. Here is the comprehensive report." - ); - } - - #[test] - fn cache_roundtrip_preserves_opencode_answer_and_subagent_prompt() { - let mut user = make_tool_call( - "opencode-user-prompt-real-fixture", - None, - "user_message", - serde_json::json!({}), - serde_json::json!({ - "content": OPENCODE_SUBAGENT_USER_PROMPT, - "message": { - "content": OPENCODE_SUBAGENT_USER_PROMPT, - "role": "user" - } - }), - ); - user.source = EventSource::User; - user.display_variant = EventDisplayVariant::Message; - user.display_text = OPENCODE_SUBAGENT_USER_PROMPT.to_string(); - - let mut subagent = make_tool_call( - "opencode-subagent-roundtrip", - Some("call-opencode-roundtrip"), - "subagent", - serde_json::json!({ - "description": OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT, - "prompt": OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT, - "subagentSessionId": "opencodeapp-child-roundtrip" - }), - serde_json::json!({ - "content": FINAL_REPORT_CONTENT, - "summary": "Subagent 已完成分析，结果如下", - "success": true - }), - ); - subagent.ui_canonical = "subagent".to_string(); - - let mut assistant = make_tool_call( - "opencode-assistant-answer-roundtrip", - None, - "assistant", - serde_json::json!({}), - serde_json::json!({ - "content": FINAL_ASSISTANT_ANSWER, - "observation": FINAL_ASSISTANT_ANSWER, - "is_delta": false, - "is_full_content": true - }), - ); - assistant.source = EventSource::Assistant; - assistant.display_variant = EventDisplayVariant::Message; - assistant.display_text = FINAL_ASSISTANT_ANSWER.to_string(); - assistant.is_delta = Some(false); - - let cached = [user, subagent, assistant] - .iter() - .filter(|event| !is_synthetic_persistence_artifact(event)) - .map(session_event_to_cached_event) - .collect::>(); - let mut reloaded = cached - .iter() - .map(cached_event_to_session_event) - .collect::>(); - prompt_backfill::backfill_subagent_prompts_with_resolver(&mut reloaded, |_| { - Some(OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT.to_string()) - }); - - let assistant = reloaded - .iter() - .find(|event| event.id == "opencode-assistant-answer-roundtrip") - .expect("assistant answer should survive reload"); - assert_eq!(assistant.result["content"], FINAL_ASSISTANT_ANSWER); - assert_eq!(assistant.result["observation"], FINAL_ASSISTANT_ANSWER); - assert_eq!(assistant.is_delta, Some(false)); - - let subagent = reloaded - .iter() - .find(|event| event.id == "opencode-subagent-roundtrip") - .expect("subagent event should survive reload"); - assert_eq!(subagent.args["prompt"], OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT); - assert_eq!( - subagent.args["description"], - OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT - ); - assert_ne!(subagent.result["content"], serde_json::Value::Null); - } - - #[test] - fn backfill_provider_subagent_prompts_preserves_existing_real_prompt() { - let mut event = make_tool_call( - "opencode-subagent-real-prompt", - Some("call-opencode-real-prompt"), - "subagent", - serde_json::json!({ - "description": "Task", - "prompt": "Inspect the OpenCode child session and summarize markdown findings.", - "subagentSessionId": "opencodeapp-child-real-prompt" - }), - serde_json::json!({}), - ); - event.ui_canonical = "subagent".to_string(); - - let events = crate::agent_sessions::event_pipeline::commands::prepare_loaded_events( - "opencodeapp-parent", - vec![event], - ); - - assert_eq!( - events[0].args["prompt"], - "Inspect the OpenCode child session and summarize markdown findings." - ); - assert_eq!(events[0].args["description"], "Task"); - } - - #[test] - fn backfill_provider_subagent_prompts_does_not_invent_parent_prompt() { - let parent_prompt = "启动一个子任务（subagent），让它分析项目并生成报告"; - let mut event = make_tool_call( - "opencode-subagent-no-child-prompt", - Some("call-opencode-no-child-prompt"), - "subagent", - serde_json::json!({ - "description": "Task", - "prompt": "Task", - "subagentSessionId": "opencodeapp-child-without-cache-row" - }), - serde_json::json!({}), - ); - event.ui_canonical = "subagent".to_string(); - - let events = crate::agent_sessions::event_pipeline::commands::prepare_loaded_events( - parent_prompt, - vec![event], - ); - - assert_eq!(events[0].args["prompt"], "Task"); - assert_eq!(events[0].args["description"], "Task"); - } - - #[test] - fn prompt_from_history_chunks_prefers_child_user_assignment() { - let mut user = ActivityChunk::new("opencodeapp-child", "raw", "user_message"); - user.result = serde_json::json!({ - "message": { - "content": "请分析当前工作目录下所有 .rs 文件，并生成结构化报告", - "role": "user" - } - }); - let mut assistant = ActivityChunk::new("opencodeapp-child", "assistant", "assistant"); - assistant.result = serde_json::json!({ - "content": "Now I have all the data. Here is the comprehensive report." - }); - - assert_eq!( - prompt_backfill::prompt_from_history_chunks(&[user, assistant]), - Some("请分析当前工作目录下所有 .rs 文件，并生成结构化报告".to_string()) - ); - } - - #[test] - fn opencode_prompt_quality_rejects_result_like_report() { - assert!(!prompt_backfill::is_good_subagent_prompt( - "Now I have all the data. Here is the comprehensive report." - )); - assert_eq!( - prompt_backfill::non_generic_subagent_prompt( - "Now I have all the data. Here is the comprehensive report.".to_string() - ), - None - ); - } - - #[test] - fn opencode_prompt_quality_rejects_paste_placeholder() { - assert!(!prompt_backfill::is_good_subagent_prompt( - "pasted.txt [paste:paste://1782778711175-d8dsv8]" - )); - assert_eq!( - prompt_backfill::non_generic_subagent_prompt( - "pasted.txt [paste:paste://1782778711175-d8dsv8]".to_string() - ), - None - ); - } - - #[test] - fn opencode_prompt_quality_accepts_assignment_title() { - assert!(prompt_backfill::is_good_subagent_prompt( - "Analyze .rs files in project (@explore subagent)" - )); - assert_eq!( - prompt_backfill::non_generic_subagent_prompt( - "Analyze .rs files in project (@explore subagent)".to_string() - ), - Some("Analyze .rs files in project (@explore subagent)".to_string()) - ); - } - - // --- dedup_by_call_id --- - - fn make_tool_call( - id: &str, - call_id: Option<&str>, - function_name: &str, - args: serde_json::Value, - result: serde_json::Value, - ) -> SessionEvent { - SessionEvent { - id: id.to_string(), - chunk_id: None, - session_id: "test-session".to_string(), - created_at: "2026-04-16T00:00:00Z".to_string(), - function_name: function_name.to_string(), - ui_canonical: function_name.to_string(), - action_type: "tool_call".to_string(), - args, - result, - source: EventSource::Assistant, - display_text: format!("Tool call: {function_name}"), - display_status: EventDisplayStatus::Completed, - display_variant: EventDisplayVariant::ToolCall, - activity_status: ActivityStatus::Processed, - thread_id: None, - process_id: None, - call_id: call_id.map(String::from), - file_path: None, - command: None, - is_delta: None, - repo_id: None, - repo_path: None, - extracted: None, - payload_refs: Vec::new(), - shell_replay: None, - shell_replay_bookmarks: None, - last_extract_at: None, - } - } - - /// Regression: when two rows share the same `callId` but each carries only - /// half of the subagent payload — one has the enriched `args` - /// (`subagentSessionId`), the other has the final `result.content` — - /// dedup must preserve BOTH by merging the dropped row into the survivor. - /// - /// This is the exact DB shape observed in `sessions.db` for historical - /// agent spawns: the EventStore write path stamps args but never writes - /// result, and the message-level path persists the tool observation but - /// misses the stamp. Previously the loser was discarded wholesale, which - /// meant the subagent block either lacked nested trajectory (missing - /// `subagentSessionId`) or lacked the final report (missing `result`). - #[test] - fn dedup_merges_split_subagent_rows_on_same_call_id() { - let call_id = "toolu_test_split"; - let message_row = make_tool_call( - "uuid-message-row", - Some(call_id), - "agent", - serde_json::json!({ - "agent_id": "builtin:explore", - "description": "Audit frontend", - "prompt": "audit prompt", - }), - serde_json::json!({ - "content": "final audit report", - "observation": "final audit report", - }), - ); - let eventstore_row = make_tool_call( - &format!("tool-call-{call_id}"), - Some(call_id), - "agent", - serde_json::json!({ - "agent_id": "builtin:explore", - "description": "Audit frontend", - "prompt": "audit prompt", - "action": "delegate", - "subagentSessionId": "agent-builtin:explore-abc123", - }), - serde_json::json!({}), - ); - - let out = dedup_by_call_id(vec![message_row, eventstore_row]); - assert_eq!(out.len(), 1, "expected dedup to collapse two rows into one"); - - let merged = &out[0]; - // Winner is the EventStore row (richer args). - assert_eq!(merged.id, format!("tool-call-{call_id}")); - - let args = merged.args.as_object().expect("args must be an object"); - assert_eq!( - args.get("subagentSessionId").and_then(|v| v.as_str()), - Some("agent-builtin:explore-abc123"), - "subagentSessionId must survive" - ); - assert_eq!( - args.get("action").and_then(|v| v.as_str()), - Some("delegate") - ); - - let result = merged.result.as_object().expect("result must be an object"); - assert_eq!( - result.get("content").and_then(|v| v.as_str()), - Some("final audit report"), - "result.content must be adopted from the dropped message row" - ); - } - - #[test] - fn dedup_merges_tool_result_row_into_matching_tool_call_row() { - let call_id = "toolu_code_search"; - let mut tool_call = make_tool_call( - &format!("tool-call-{call_id}"), - Some(call_id), - "code_search", - serde_json::json!({ - "action": "grep", - "pattern": "interactive terminal", - "max_results": 30, - }), - serde_json::json!({}), - ); - tool_call.display_status = EventDisplayStatus::Running; - tool_call.activity_status = ActivityStatus::Agent; - - let mut tool_result = make_tool_call( - &format!("tool-result-{call_id}"), - Some(call_id), - "code_search", - serde_json::json!({}), - serde_json::json!("src/terminal.ts:12:interactive terminal"), - ); - tool_result.action_type = "tool_result".to_string(); - tool_result.display_status = EventDisplayStatus::Completed; - tool_result.activity_status = ActivityStatus::Processed; - - let out = dedup_by_call_id(vec![tool_call, tool_result]); - assert_eq!(out.len(), 1); - assert_eq!(out[0].id, format!("tool-call-{call_id}")); - assert_eq!(out[0].action_type, "tool_call"); - assert_eq!( - out[0].args.get("pattern").and_then(|value| value.as_str()), - Some("interactive terminal") - ); - assert_eq!( - out[0].result.as_str(), - Some("src/terminal.ts:12:interactive terminal") - ); - assert_eq!(out[0].display_status, EventDisplayStatus::Completed); - assert_eq!(out[0].activity_status, ActivityStatus::Processed); - } - - /// Cross-call_id variant: same logical agent spawn gets written with a - /// `toolu_xxx` id by the message layer and a distinct internal `tool_xxx` - /// id by the EventStore layer. Pass 2 matches them by `args.description` - /// and must merge, not just drop. - #[test] - fn dedup_merges_agent_spawns_with_different_call_ids_by_description() { - let message_row = make_tool_call( - "uuid-msg", - Some("toolu_abc"), - "agent", - serde_json::json!({ - "description": "Refactor auth", - "prompt": "do it", - }), - serde_json::json!({ "content": "refactor report body" }), - ); - let eventstore_row = make_tool_call( - "tool-call-internal", - Some("tool_xyz"), - "agent", - serde_json::json!({ - "description": "Refactor auth", - "prompt": "do it", - "subagentSessionId": "agent-builtin:sde-42", - }), - serde_json::json!({}), - ); - - let out = dedup_by_call_id(vec![message_row, eventstore_row]); - assert_eq!(out.len(), 1); - - let merged = &out[0]; - let args = merged.args.as_object().unwrap(); - assert_eq!( - args.get("subagentSessionId").and_then(|v| v.as_str()), - Some("agent-builtin:sde-42"), - "subagentSessionId must be preserved on the surviving row" - ); - - let result = merged.result.as_object().unwrap(); - assert_eq!( - result.get("content").and_then(|v| v.as_str()), - Some("refactor report body"), - "message row's result.content must be merged into the survivor" - ); - } - - /// Unrelated tool calls with distinct call_ids must pass through untouched. - #[test] - fn dedup_leaves_unique_call_ids_intact() { - let a = make_tool_call( - "a", - Some("call-a"), - "read_file", - serde_json::json!({ "path": "/foo" }), - serde_json::json!({ "content": "ok" }), - ); - let b = make_tool_call( - "b", - Some("call-b"), - "read_file", - serde_json::json!({ "path": "/bar" }), - serde_json::json!({ "content": "ok" }), - ); - - let out = dedup_by_call_id(vec![a, b]); - assert_eq!(out.len(), 2); - assert_eq!(out[0].id, "a"); - assert_eq!(out[1].id, "b"); - } - - /// Winner's existing args keys must NEVER be overwritten by the loser. - /// Only gaps are filled. - #[test] - fn dedup_preserves_winner_args_on_key_conflict() { - let loser = make_tool_call( - "loser", - Some("cid"), - "agent", - serde_json::json!({ - "description": "x", - "prompt": "OLD prompt", - }), - serde_json::json!({}), - ); - let winner = make_tool_call( - "winner", - Some("cid"), - "agent", - serde_json::json!({ - "description": "x", - "prompt": "NEW prompt", - "subagentSessionId": "sid-1", - }), - serde_json::json!({}), - ); - - let out = dedup_by_call_id(vec![loser, winner]); - assert_eq!(out.len(), 1); - assert_eq!(out[0].id, "winner"); - let args = out[0].args.as_object().unwrap(); - assert_eq!( - args.get("prompt").and_then(|v| v.as_str()), - Some("NEW prompt"), - "winner's prompt must not be overwritten by the loser" - ); - } - - #[test] - fn compact_boundary_row_maps_to_context_compacted_event() { - let content = - "[Conversation summary \u{2014} 6 earlier messages compacted]\n\nsummary body"; - let event = compact_boundary_row_to_event( - "session-x", - CompactBoundaryRow { - id: "row-1".to_string(), - content: content.to_string(), - created_at: "2026-07-08T20:51:37Z".to_string(), - tokens_before: Some(10402), - tokens_after: Some(1042), - }, - ); - - assert_eq!(event.function_name, "context_compacted"); - assert_eq!(event.ui_canonical, "context_compacted"); - assert_eq!(event.action_type, "system"); - assert_eq!(event.source, EventSource::System); - assert_eq!(event.display_variant, EventDisplayVariant::Message); - assert_eq!(event.display_status, EventDisplayStatus::Completed); - assert_eq!( - event.result.get("observation").and_then(|v| v.as_str()), - Some("summary body") - ); - assert_eq!( - event.result.get("compactedCount").and_then(|v| v.as_u64()), - Some(6) - ); - assert_eq!( - event.result.get("tokensBefore").and_then(|v| v.as_i64()), - Some(10402) - ); - assert_eq!( - event.result.get("tokensAfter").and_then(|v| v.as_i64()), - Some(1042) - ); - // Must never be mistaken for a synthetic persistence artifact. - assert!(!is_synthetic_persistence_artifact(&event)); - } -} +mod tests; diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/cache_bridge/tests.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/cache_bridge/tests.rs new file mode 100644 index 000000000..291d4a43b --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/cache_bridge/tests.rs @@ -0,0 +1,713 @@ +use super::{ + cached_event_to_session_event, is_synthetic_persistence_artifact, session_event_to_cached_event, +}; +use crate::agent_sessions::event_pipeline::commands::event_conversion::{ + compact_boundary_row_to_event, dedup_by_call_id, is_ts_placeholder_id, CompactBoundaryRow, +}; +use crate::agent_sessions::event_pipeline::ingestion::prompt_backfill; +use crate::agent_sessions::event_pipeline::types::{ + ActivityStatus, EventDisplayStatus, EventDisplayVariant, EventSource, PayloadRef, SessionEvent, +}; +use core_types::activity::ActivityChunk; + +const OPENCODE_SUBAGENT_USER_PROMPT: &str = "启动一个（subagent），让它帮我分析当前项目里有多少个 .rs 文件，并生成一份报告。必须要用subagent，然后要让我看到过程"; +const OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT: &str = "在当前工作目录下分析 Rust 源文件数量：统计所有 **/*.rs 文件，排除 target/ 目录；生成一份报告，包含总文件数、按目录分布、最大文件 Top 5，并在过程中持续汇报进展。"; +const FINAL_REPORT_CONTENT: &str = "Now I have all the data. Here is the comprehensive report."; +const FINAL_ASSISTANT_ANSWER: &str = + "Subagent 已完成分析：当前项目共有 260 个 .rs 文件，并已生成报告。"; + +#[test] +fn ts_placeholder_msg_and_think_ids_match() { + assert!(is_ts_placeholder_id("stream-msg-ts-session-1776099853993")); + assert!(is_ts_placeholder_id( + "stream-think-ts-session-1776099853993" + )); +} + +#[test] +fn cached_event_normalizes_legacy_string_result() { + let cached = session_persistence::CachedEvent { + id: "legacy-string-result".to_string(), + session_id: "session-history-regression".to_string(), + event_type: "message".to_string(), + function_name: Some("message".to_string()), + thread_id: None, + args_json: "{}".to_string(), + result_json: "\"loaded historical assistant text\"".to_string(), + content: "loaded historical assistant text".to_string(), + created_at: "2026-05-16T00:00:00.000Z".to_string(), + meta_json: Some( + serde_json::json!({ + "source": "assistant", + "displayText": "loaded historical assistant text", + "displayStatus": "completed", + "displayVariant": "message", + "activityStatus": "agent", + "uiCanonical": "message" + }) + .to_string(), + ), + history_sequence: None, + }; + + let event = cached_event_to_session_event(&cached); + let result = event.result.as_object().expect("result must be normalized"); + assert_eq!( + result.get("content").and_then(|value| value.as_str()), + Some("loaded historical assistant text") + ); + assert_eq!( + result.get("observation").and_then(|value| value.as_str()), + Some("loaded historical assistant text") + ); +} + +#[test] +fn cached_event_normalizes_legacy_string_args() { + let cached = session_persistence::CachedEvent { + id: "legacy-string-args".to_string(), + session_id: "session-history-regression".to_string(), + event_type: "tool_call".to_string(), + function_name: Some("tool_call".to_string()), + thread_id: None, + args_json: "\"legacy arguments\"".to_string(), + result_json: "{}".to_string(), + content: "legacy arguments".to_string(), + created_at: "2026-05-16T00:00:00.000Z".to_string(), + meta_json: Some( + serde_json::json!({ + "source": "assistant", + "displayText": "legacy arguments", + "displayStatus": "completed", + "displayVariant": "tool_call", + "activityStatus": "agent", + "uiCanonical": "tool_call" + }) + .to_string(), + ), + history_sequence: None, + }; + + let event = cached_event_to_session_event(&cached); + let args = event.args.as_object().expect("args must be normalized"); + assert_eq!( + args.get("content").and_then(|value| value.as_str()), + Some("legacy arguments") + ); + assert_eq!( + args.get("observation").and_then(|value| value.as_str()), + Some("legacy arguments") + ); +} + +#[test] +fn rust_authoritative_ids_do_not_match() { + assert!(!is_ts_placeholder_id( + "stream-msg-sdeagent-a91612f3-4f94-4fac-a0c2-f6e85f0c1f63-1" + )); + assert!(!is_ts_placeholder_id( + "stream-think-sdeagent-a91612f3-4f94-4fac-a0c2-f6e85f0c1f63-1" + )); +} + +#[test] +fn unrelated_event_ids_do_not_match() { + assert!(!is_ts_placeholder_id("tool-call-42")); + assert!(!is_ts_placeholder_id("user-msg-1")); + assert!(!is_ts_placeholder_id("")); + // Prefix must be the full "stream-msg-ts-" / "stream-think-ts-" — + // ids like "stream-msg-tsfoo-…" are not placeholders. + assert!(!is_ts_placeholder_id("stream-msg-tsfoo")); +} + +#[test] +fn turn_placeholder_is_synthetic_persistence_artifact() { + let placeholder = make_tool_call( + "turn-placeholder-turn-1", + None, + "turn_placeholder", + serde_json::json!({}), + serde_json::json!({ "unloadedTurn": { "turnId": "turn-1" } }), + ); + assert!(is_synthetic_persistence_artifact(&placeholder)); + + let mut synthetic_header = make_tool_call( + "turn-1", + None, + "user_message", + serde_json::json!({}), + serde_json::json!({ "syntheticTurnHeader": true }), + ); + synthetic_header.source = EventSource::User; + assert!(is_synthetic_persistence_artifact(&synthetic_header)); + + let normal = make_tool_call( + "tool-call-42", + None, + "bash", + serde_json::json!({}), + serde_json::json!({}), + ); + assert!(!is_synthetic_persistence_artifact(&normal)); +} + +#[test] +fn compacted_event_is_synthetic_persistence_artifact() { + let mut compacted = make_tool_call( + "tool-call-compacted", + None, + "bash", + serde_json::json!({ "streamOutput": "preview" }), + serde_json::json!({}), + ); + compacted.payload_refs.push(PayloadRef { + event_id: compacted.id.clone(), + field_path: "args.streamOutput".to_string(), + preview: "preview".to_string(), + full_size_bytes: 128 * 1024, + truncated: true, + replay_encoding: None, + replay_source_id: None, + replay_generation: None, + replay_source_event_id: None, + }); + + assert!(is_synthetic_persistence_artifact(&compacted)); +} + +#[test] +fn backfill_provider_subagent_prompts_uses_child_assignment_for_real_prompt() { + let mut event = make_tool_call( + "opencode-subagent-real-user-prompt-fixture", + Some("call-opencode-real-user-prompt-fixture"), + "subagent", + serde_json::json!({ + "description": "Task", + "prompt": "Task", + "subagentSessionId": "opencodeapp-child-real-assignment" + }), + serde_json::json!({ + "content": "Now I have all the data. Here is the comprehensive report.", + "summary": "Subagent 已完成分析，结果如下" + }), + ); + event.ui_canonical = "subagent".to_string(); + + let mut events = vec![event]; + prompt_backfill::backfill_subagent_prompts_with_resolver(&mut events, |child_session_id| { + assert_eq!(child_session_id, "opencodeapp-child-real-assignment"); + Some(OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT.to_string()) + }); + + assert_eq!( + events[0].args["prompt"], + OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT + ); + assert_eq!( + events[0].args["description"], + OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT + ); + assert_ne!(events[0].args["prompt"], OPENCODE_SUBAGENT_USER_PROMPT); + assert_ne!(events[0].args["prompt"], "Task"); + assert_ne!( + events[0].args["prompt"], + "Now I have all the data. Here is the comprehensive report." + ); +} + +#[test] +fn cache_roundtrip_preserves_opencode_answer_and_subagent_prompt() { + let mut user = make_tool_call( + "opencode-user-prompt-real-fixture", + None, + "user_message", + serde_json::json!({}), + serde_json::json!({ + "content": OPENCODE_SUBAGENT_USER_PROMPT, + "message": { + "content": OPENCODE_SUBAGENT_USER_PROMPT, + "role": "user" + } + }), + ); + user.source = EventSource::User; + user.display_variant = EventDisplayVariant::Message; + user.display_text = OPENCODE_SUBAGENT_USER_PROMPT.to_string(); + + let mut subagent = make_tool_call( + "opencode-subagent-roundtrip", + Some("call-opencode-roundtrip"), + "subagent", + serde_json::json!({ + "description": OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT, + "prompt": OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT, + "subagentSessionId": "opencodeapp-child-roundtrip" + }), + serde_json::json!({ + "content": FINAL_REPORT_CONTENT, + "summary": "Subagent 已完成分析，结果如下", + "success": true + }), + ); + subagent.ui_canonical = "subagent".to_string(); + + let mut assistant = make_tool_call( + "opencode-assistant-answer-roundtrip", + None, + "assistant", + serde_json::json!({}), + serde_json::json!({ + "content": FINAL_ASSISTANT_ANSWER, + "observation": FINAL_ASSISTANT_ANSWER, + "is_delta": false, + "is_full_content": true + }), + ); + assistant.source = EventSource::Assistant; + assistant.display_variant = EventDisplayVariant::Message; + assistant.display_text = FINAL_ASSISTANT_ANSWER.to_string(); + assistant.is_delta = Some(false); + + let cached = [user, subagent, assistant] + .iter() + .filter(|event| !is_synthetic_persistence_artifact(event)) + .map(session_event_to_cached_event) + .collect::>(); + let mut reloaded = cached + .iter() + .map(cached_event_to_session_event) + .collect::>(); + prompt_backfill::backfill_subagent_prompts_with_resolver(&mut reloaded, |_| { + Some(OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT.to_string()) + }); + + let assistant = reloaded + .iter() + .find(|event| event.id == "opencode-assistant-answer-roundtrip") + .expect("assistant answer should survive reload"); + assert_eq!(assistant.result["content"], FINAL_ASSISTANT_ANSWER); + assert_eq!(assistant.result["observation"], FINAL_ASSISTANT_ANSWER); + assert_eq!(assistant.is_delta, Some(false)); + + let subagent = reloaded + .iter() + .find(|event| event.id == "opencode-subagent-roundtrip") + .expect("subagent event should survive reload"); + assert_eq!(subagent.args["prompt"], OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT); + assert_eq!( + subagent.args["description"], + OPENCODE_SUBAGENT_ASSIGNMENT_PROMPT + ); + assert_ne!(subagent.result["content"], serde_json::Value::Null); +} + +#[test] +fn backfill_provider_subagent_prompts_preserves_existing_real_prompt() { + let mut event = make_tool_call( + "opencode-subagent-real-prompt", + Some("call-opencode-real-prompt"), + "subagent", + serde_json::json!({ + "description": "Task", + "prompt": "Inspect the OpenCode child session and summarize markdown findings.", + "subagentSessionId": "opencodeapp-child-real-prompt" + }), + serde_json::json!({}), + ); + event.ui_canonical = "subagent".to_string(); + + let events = crate::agent_sessions::event_pipeline::commands::prepare_loaded_events( + "opencodeapp-parent", + vec![event], + ); + + assert_eq!( + events[0].args["prompt"], + "Inspect the OpenCode child session and summarize markdown findings." + ); + assert_eq!(events[0].args["description"], "Task"); +} + +#[test] +fn backfill_provider_subagent_prompts_does_not_invent_parent_prompt() { + let parent_prompt = "启动一个子任务（subagent），让它分析项目并生成报告"; + let mut event = make_tool_call( + "opencode-subagent-no-child-prompt", + Some("call-opencode-no-child-prompt"), + "subagent", + serde_json::json!({ + "description": "Task", + "prompt": "Task", + "subagentSessionId": "opencodeapp-child-without-cache-row" + }), + serde_json::json!({}), + ); + event.ui_canonical = "subagent".to_string(); + + let events = crate::agent_sessions::event_pipeline::commands::prepare_loaded_events( + parent_prompt, + vec![event], + ); + + assert_eq!(events[0].args["prompt"], "Task"); + assert_eq!(events[0].args["description"], "Task"); +} + +#[test] +fn prompt_from_history_chunks_prefers_child_user_assignment() { + let mut user = ActivityChunk::new("opencodeapp-child", "raw", "user_message"); + user.result = serde_json::json!({ + "message": { + "content": "请分析当前工作目录下所有 .rs 文件，并生成结构化报告", + "role": "user" + } + }); + let mut assistant = ActivityChunk::new("opencodeapp-child", "assistant", "assistant"); + assistant.result = serde_json::json!({ + "content": "Now I have all the data. Here is the comprehensive report." + }); + + assert_eq!( + prompt_backfill::prompt_from_history_chunks(&[user, assistant]), + Some("请分析当前工作目录下所有 .rs 文件，并生成结构化报告".to_string()) + ); +} + +#[test] +fn opencode_prompt_quality_rejects_result_like_report() { + assert!(!prompt_backfill::is_good_subagent_prompt( + "Now I have all the data. Here is the comprehensive report." + )); + assert_eq!( + prompt_backfill::non_generic_subagent_prompt( + "Now I have all the data. Here is the comprehensive report.".to_string() + ), + None + ); +} + +#[test] +fn opencode_prompt_quality_rejects_paste_placeholder() { + assert!(!prompt_backfill::is_good_subagent_prompt( + "pasted.txt [paste:paste://1782778711175-d8dsv8]" + )); + assert_eq!( + prompt_backfill::non_generic_subagent_prompt( + "pasted.txt [paste:paste://1782778711175-d8dsv8]".to_string() + ), + None + ); +} + +#[test] +fn opencode_prompt_quality_accepts_assignment_title() { + assert!(prompt_backfill::is_good_subagent_prompt( + "Analyze .rs files in project (@explore subagent)" + )); + assert_eq!( + prompt_backfill::non_generic_subagent_prompt( + "Analyze .rs files in project (@explore subagent)".to_string() + ), + Some("Analyze .rs files in project (@explore subagent)".to_string()) + ); +} + +// --- dedup_by_call_id --- + +fn make_tool_call( + id: &str, + call_id: Option<&str>, + function_name: &str, + args: serde_json::Value, + result: serde_json::Value, +) -> SessionEvent { + SessionEvent { + id: id.to_string(), + chunk_id: None, + session_id: "test-session".to_string(), + created_at: "2026-04-16T00:00:00Z".to_string(), + function_name: function_name.to_string(), + ui_canonical: function_name.to_string(), + action_type: "tool_call".to_string(), + args, + result, + source: EventSource::Assistant, + display_text: format!("Tool call: {function_name}"), + display_status: EventDisplayStatus::Completed, + display_variant: EventDisplayVariant::ToolCall, + activity_status: ActivityStatus::Processed, + thread_id: None, + process_id: None, + call_id: call_id.map(String::from), + file_path: None, + command: None, + is_delta: None, + repo_id: None, + repo_path: None, + extracted: None, + payload_refs: Vec::new(), + shell_replay: None, + shell_replay_bookmarks: None, + last_extract_at: None, + } +} + +/// Regression: when two rows share the same `callId` but each carries only +/// half of the subagent payload — one has the enriched `args` +/// (`subagentSessionId`), the other has the final `result.content` — +/// dedup must preserve BOTH by merging the dropped row into the survivor. +/// +/// This is the exact DB shape observed in `sessions.db` for historical +/// agent spawns: the EventStore write path stamps args but never writes +/// result, and the message-level path persists the tool observation but +/// misses the stamp. Previously the loser was discarded wholesale, which +/// meant the subagent block either lacked nested trajectory (missing +/// `subagentSessionId`) or lacked the final report (missing `result`). +#[test] +fn dedup_merges_split_subagent_rows_on_same_call_id() { + let call_id = "toolu_test_split"; + let message_row = make_tool_call( + "uuid-message-row", + Some(call_id), + "agent", + serde_json::json!({ + "agent_id": "builtin:explore", + "description": "Audit frontend", + "prompt": "audit prompt", + }), + serde_json::json!({ + "content": "final audit report", + "observation": "final audit report", + }), + ); + let eventstore_row = make_tool_call( + &format!("tool-call-{call_id}"), + Some(call_id), + "agent", + serde_json::json!({ + "agent_id": "builtin:explore", + "description": "Audit frontend", + "prompt": "audit prompt", + "action": "delegate", + "subagentSessionId": "agent-builtin:explore-abc123", + }), + serde_json::json!({}), + ); + + let out = dedup_by_call_id(vec![message_row, eventstore_row]); + assert_eq!(out.len(), 1, "expected dedup to collapse two rows into one"); + + let merged = &out[0]; + // Winner is the EventStore row (richer args). + assert_eq!(merged.id, format!("tool-call-{call_id}")); + + let args = merged.args.as_object().expect("args must be an object"); + assert_eq!( + args.get("subagentSessionId").and_then(|v| v.as_str()), + Some("agent-builtin:explore-abc123"), + "subagentSessionId must survive" + ); + assert_eq!( + args.get("action").and_then(|v| v.as_str()), + Some("delegate") + ); + + let result = merged.result.as_object().expect("result must be an object"); + assert_eq!( + result.get("content").and_then(|v| v.as_str()), + Some("final audit report"), + "result.content must be adopted from the dropped message row" + ); +} + +#[test] +fn dedup_merges_tool_result_row_into_matching_tool_call_row() { + let call_id = "toolu_code_search"; + let mut tool_call = make_tool_call( + &format!("tool-call-{call_id}"), + Some(call_id), + "code_search", + serde_json::json!({ + "action": "grep", + "pattern": "interactive terminal", + "max_results": 30, + }), + serde_json::json!({}), + ); + tool_call.display_status = EventDisplayStatus::Running; + tool_call.activity_status = ActivityStatus::Agent; + + let mut tool_result = make_tool_call( + &format!("tool-result-{call_id}"), + Some(call_id), + "code_search", + serde_json::json!({}), + serde_json::json!("src/terminal.ts:12:interactive terminal"), + ); + tool_result.action_type = "tool_result".to_string(); + tool_result.display_status = EventDisplayStatus::Completed; + tool_result.activity_status = ActivityStatus::Processed; + + let out = dedup_by_call_id(vec![tool_call, tool_result]); + assert_eq!(out.len(), 1); + assert_eq!(out[0].id, format!("tool-call-{call_id}")); + assert_eq!(out[0].action_type, "tool_call"); + assert_eq!( + out[0].args.get("pattern").and_then(|value| value.as_str()), + Some("interactive terminal") + ); + assert_eq!( + out[0].result.as_str(), + Some("src/terminal.ts:12:interactive terminal") + ); + assert_eq!(out[0].display_status, EventDisplayStatus::Completed); + assert_eq!(out[0].activity_status, ActivityStatus::Processed); +} + +/// Cross-call_id variant: same logical agent spawn gets written with a +/// `toolu_xxx` id by the message layer and a distinct internal `tool_xxx` +/// id by the EventStore layer. Pass 2 matches them by `args.description` +/// and must merge, not just drop. +#[test] +fn dedup_merges_agent_spawns_with_different_call_ids_by_description() { + let message_row = make_tool_call( + "uuid-msg", + Some("toolu_abc"), + "agent", + serde_json::json!({ + "description": "Refactor auth", + "prompt": "do it", + }), + serde_json::json!({ "content": "refactor report body" }), + ); + let eventstore_row = make_tool_call( + "tool-call-internal", + Some("tool_xyz"), + "agent", + serde_json::json!({ + "description": "Refactor auth", + "prompt": "do it", + "subagentSessionId": "agent-builtin:sde-42", + }), + serde_json::json!({}), + ); + + let out = dedup_by_call_id(vec![message_row, eventstore_row]); + assert_eq!(out.len(), 1); + + let merged = &out[0]; + let args = merged.args.as_object().unwrap(); + assert_eq!( + args.get("subagentSessionId").and_then(|v| v.as_str()), + Some("agent-builtin:sde-42"), + "subagentSessionId must be preserved on the surviving row" + ); + + let result = merged.result.as_object().unwrap(); + assert_eq!( + result.get("content").and_then(|v| v.as_str()), + Some("refactor report body"), + "message row's result.content must be merged into the survivor" + ); +} + +/// Unrelated tool calls with distinct call_ids must pass through untouched. +#[test] +fn dedup_leaves_unique_call_ids_intact() { + let a = make_tool_call( + "a", + Some("call-a"), + "read_file", + serde_json::json!({ "path": "/foo" }), + serde_json::json!({ "content": "ok" }), + ); + let b = make_tool_call( + "b", + Some("call-b"), + "read_file", + serde_json::json!({ "path": "/bar" }), + serde_json::json!({ "content": "ok" }), + ); + + let out = dedup_by_call_id(vec![a, b]); + assert_eq!(out.len(), 2); + assert_eq!(out[0].id, "a"); + assert_eq!(out[1].id, "b"); +} + +/// Winner's existing args keys must NEVER be overwritten by the loser. +/// Only gaps are filled. +#[test] +fn dedup_preserves_winner_args_on_key_conflict() { + let loser = make_tool_call( + "loser", + Some("cid"), + "agent", + serde_json::json!({ + "description": "x", + "prompt": "OLD prompt", + }), + serde_json::json!({}), + ); + let winner = make_tool_call( + "winner", + Some("cid"), + "agent", + serde_json::json!({ + "description": "x", + "prompt": "NEW prompt", + "subagentSessionId": "sid-1", + }), + serde_json::json!({}), + ); + + let out = dedup_by_call_id(vec![loser, winner]); + assert_eq!(out.len(), 1); + assert_eq!(out[0].id, "winner"); + let args = out[0].args.as_object().unwrap(); + assert_eq!( + args.get("prompt").and_then(|v| v.as_str()), + Some("NEW prompt"), + "winner's prompt must not be overwritten by the loser" + ); +} + +#[test] +fn compact_boundary_row_maps_to_context_compacted_event() { + let content = "[Conversation summary \u{2014} 6 earlier messages compacted]\n\nsummary body"; + let event = compact_boundary_row_to_event( + "session-x", + CompactBoundaryRow { + id: "row-1".to_string(), + content: content.to_string(), + created_at: "2026-07-08T20:51:37Z".to_string(), + tokens_before: Some(10402), + tokens_after: Some(1042), + }, + ); + + assert_eq!(event.function_name, "context_compacted"); + assert_eq!(event.ui_canonical, "context_compacted"); + assert_eq!(event.action_type, "system"); + assert_eq!(event.source, EventSource::System); + assert_eq!(event.display_variant, EventDisplayVariant::Message); + assert_eq!(event.display_status, EventDisplayStatus::Completed); + assert_eq!( + event.result.get("observation").and_then(|v| v.as_str()), + Some("summary body") + ); + assert_eq!( + event.result.get("compactedCount").and_then(|v| v.as_u64()), + Some(6) + ); + assert_eq!( + event.result.get("tokensBefore").and_then(|v| v.as_i64()), + Some(10402) + ); + assert_eq!( + event.result.get("tokensAfter").and_then(|v| v.as_i64()), + Some(1042) + ); + // Must never be mistaken for a synthetic persistence artifact. + assert!(!is_synthetic_persistence_artifact(&event)); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/mod.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/mod.rs new file mode 100644 index 000000000..6129b692f --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/mod.rs @@ -0,0 +1,339 @@ +//! Byte-bounded Cloud collaboration snapshot ingestion. +//! +//! The renderer passes opaque compressed physical rows, never a complete +//! `SessionEvent[]`. Rows are verified and folded into a token-scoped staging +//! SQLite file. Publishing is a single `sessions.db` transaction, so a +//! malformed page, cancellation, process crash, or failed commit leaves the +//! previous imported snapshot visible. + +use std::fs::{self, File}; +use std::io::{BufReader, BufWriter, Read, Write}; +use std::path::{Path, PathBuf}; +use std::time::{Duration, SystemTime}; + +use base64::engine::general_purpose::STANDARD as BASE64_STANDARD; +use chrono::Utc; +use flate2::read::GzDecoder; +use rusqlite::{params, Connection, OptionalExtension, Transaction}; +use serde::de::{SeqAccess, Visitor}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use uuid::Uuid; + +use crate::agent_sessions::event_pipeline::commands::event_conversion::{ + cached_event_to_session_event, session_event_to_cached_event, +}; +use crate::agent_sessions::event_pipeline::types::SessionEvent; + +use super::replay_cloud_wire::{ + decode_replay_attachment_v2_frame, ReplayAttachmentV2FrameHeader, + CLOUD_PAGE_MAX_BYTES as MAX_PAGE_BYTES, CLOUD_PAGE_MAX_SEGMENTS as MAX_PAGE_SEGMENTS, + CLOUD_SEGMENT_WIRE_MAX_BYTES as MAX_WIRE_BYTES, + LEGACY_V1_MAX_DECOMPRESSED_BYTES as MAX_DECOMPRESSED_V1_BYTES, LEGACY_V1_MAX_WIRE_BYTES, + REPLAY_ATTACHMENT_V2_MAGIC as FRAME_MAGIC, + REPLAY_ATTACHMENT_V2_MAX_DECOMPRESSED_BYTES as MAX_DECOMPRESSED_V2_BYTES, +}; +#[cfg(test)] +use super::replay_cloud_wire::{ + encode_replay_attachment_v2_frame, REPLAY_ATTACHMENT_CHUNK_BYTES as ATTACHMENT_CHUNK_BYTES, +}; + +const IMPORTED_SESSION_PREFIX: &str = "imported-session-"; +const AGENT_SESSION_PREFIX: &str = "agentsession-"; +const COPY_ID_DELIMITER: &str = "~"; +const STAGING_DIR_NAME: &str = "collaboration-snapshot-staging"; +const STAGING_VERSION: i64 = 1; +const STAGING_STALE_AFTER: Duration = Duration::from_secs(24 * 60 * 60); +const HASH_HEX_BYTES: usize = 64; +const HANDOFF_MAX_ITEMS: usize = 80; +const HANDOFF_MAX_ITEM_UTF16: usize = 1_200; +const HANDOFF_SCAN_BYTES: usize = 4 * 1024 * 1024; +const HANDOFF_FIELD_PREVIEW_BYTES: i64 = 8 * 1024; +const HANDOFF_FIELD_PREVIEW_CHARS: i64 = 2 * 1024; +const SNAPSHOT_INVALIDATION_TRIGGER_COUNT: i64 = 6; +const SNAPSHOT_INDEX_COUNT: i64 = 2; +const SECONDARY_MUTATION_TRIGGERS_SQL: &str = + "CREATE TABLE IF NOT EXISTS collaboration_snapshot_secondary_state ( + session_id TEXT PRIMARY KEY, + generation INTEGER NOT NULL DEFAULT 0, + revision INTEGER NOT NULL DEFAULT 0, + reset_revision INTEGER NOT NULL DEFAULT 0, + max_sequence INTEGER NOT NULL DEFAULT -1, + event_count INTEGER NOT NULL DEFAULT 0 + ); + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_native_events_insert_touch + AFTER INSERT ON events + WHEN NEW.session_id GLOB 'agentsession-*' + BEGIN + UPDATE collaboration_snapshot_secondary_state + SET revision=revision+1, + reset_revision=CASE + WHEN NEW.history_sequence IS NULL + OR NEW.history_sequence<=max_sequence + THEN revision+1 ELSE reset_revision END, + max_sequence=CASE + WHEN NEW.history_sequence IS NULL THEN max_sequence + ELSE MAX(max_sequence,NEW.history_sequence) END, + event_count=event_count+1 + WHERE session_id=NEW.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_native_events_delete_touch + AFTER DELETE ON events + WHEN OLD.session_id GLOB 'agentsession-*' + BEGIN + UPDATE collaboration_snapshot_secondary_state + SET revision=revision+1, + reset_revision=revision+1, + max_sequence=COALESCE(( + SELECT history_sequence FROM events + WHERE session_id=OLD.session_id AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC LIMIT 1 + ),-1), + event_count=MAX(event_count-1,0) + WHERE session_id=OLD.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_native_events_update_touch + AFTER UPDATE ON events + WHEN OLD.session_id GLOB 'agentsession-*' + OR NEW.session_id GLOB 'agentsession-*' + BEGIN + UPDATE collaboration_snapshot_secondary_state + SET revision=revision+1, + reset_revision=revision+1, + max_sequence=COALESCE(( + SELECT history_sequence FROM events + WHERE session_id=OLD.session_id AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC LIMIT 1 + ),-1), + event_count=MAX( + event_count-CASE WHEN NEW.session_id!=OLD.session_id THEN 1 ELSE 0 END, + 0 + ) + WHERE session_id=OLD.session_id; + UPDATE collaboration_snapshot_secondary_state + SET revision=revision+1, + reset_revision=revision+1, + max_sequence=COALESCE(( + SELECT history_sequence FROM events + WHERE session_id=NEW.session_id AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC LIMIT 1 + ),-1), + event_count=event_count+1 + WHERE session_id=NEW.session_id AND NEW.session_id!=OLD.session_id; + END;"; +const DROP_SECONDARY_MUTATION_TRIGGERS_SQL: &str = + "DROP TRIGGER IF EXISTS collaboration_snapshot_native_events_insert_touch; + DROP TRIGGER IF EXISTS collaboration_snapshot_native_events_delete_touch; + DROP TRIGGER IF EXISTS collaboration_snapshot_native_events_update_touch;"; +const DELETE_TAIL_EVENTS_SQL: &str = "DELETE FROM events WHERE id IN ( + SELECT event_id FROM collaboration_snapshot_event_map + WHERE session_id=?1 AND is_tail=1 + )"; +const DELETE_TAIL_MAP_SQL: &str = + "DELETE FROM collaboration_snapshot_event_map WHERE session_id=?1 AND is_tail=1"; +const PUBLISH_REPLAY_ACCOUNTING_SQL: &str = "INSERT INTO collaboration_replay_state( + session_id,generation,revision,max_sequence,event_count + ) VALUES(?1,0,?2,?3,?2) + ON CONFLICT(session_id) DO UPDATE SET + generation=collaboration_replay_state.generation+1, + revision=collaboration_replay_state.revision+1, + max_sequence=excluded.max_sequence, + event_count=excluded.event_count"; +const CURSOR_SENTINEL_SQL: &str = "SELECT + EXISTS( + SELECT 1 + FROM collaboration_snapshot_event_map m + INDEXED BY idx_collaboration_snapshot_event_order + JOIN events e ON e.id=m.event_id + WHERE m.session_id=?1 AND m.logical_index=0 AND e.session_id=?1 + ), + EXISTS( + SELECT 1 + FROM collaboration_snapshot_event_map m + INDEXED BY idx_collaboration_snapshot_event_order + JOIN events e ON e.id=m.event_id + WHERE m.session_id=?1 AND m.logical_index=?2 AND e.session_id=?1 + )"; + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotCursor { + pub epoch: i64, + pub frozen_seq: u64, + pub count: u64, + pub frozen_count: u64, + pub tail_hash: Option, +} + +#[derive(Debug, Clone)] +struct CollaborationSnapshotSessionMetadata { + time_range_start: Option, + time_range_end: Option, +} + +/// Constant-space state for secondary consumers of a Cloud-created native +/// fork. This is deliberately separate from imported-session replay state: +/// its presence never opts the native SessionCore open/send path into replay. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct CollaborationSnapshotSecondaryState { + pub generation: String, + pub revision: u64, + pub reset_revision: u64, + pub max_sequence: i64, + pub event_count: u64, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotIngestBeginRequest { + pub local_session_id: String, + pub epoch: i64, + pub expected_count: u64, + pub expected_frozen_seq: u64, + pub tail_hash: Option, + pub replace: bool, + #[serde(default)] + pub previous: Option, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotIngestBeginResult { + pub token: String, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(tag = "direction", rename_all = "lowercase")] +pub enum CollaborationSnapshotWireCursor { + Forward { + #[serde(rename = "afterSeq")] + after_seq: u64, + #[serde(rename = "throughSeq", default)] + through_seq: Option, + }, + Backward { + #[serde(rename = "beforeSeq", default)] + before_seq: Option, + }, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotWire { + pub seq: u64, + pub payload_gz: String, + pub event_count: u64, + pub segment_hash: String, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotIngestPageRequest { + pub token: String, + pub epoch: i64, + pub frozen_seq: u64, + pub count: u64, + pub tail_hash: Option, + pub cursor: CollaborationSnapshotWireCursor, + #[serde(default)] + pub next_cursor: Option, + pub tail_included: bool, + pub has_more: bool, + pub returned_wire_bytes: u64, + pub segments: Vec, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotIngestProgress { + pub accepted_physical_rows: u64, + pub accepted_logical_events: u64, + pub complete: bool, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotIngestTokenRequest { + pub token: String, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotIngestGetCursorRequest { + pub local_session_id: String, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotSecondaryProbeRequest { + pub session_id: String, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct CollaborationSnapshotIngestCommitResult { + pub local_session_id: String, + pub epoch: i64, + pub frozen_seq: u64, + pub event_count: u64, + pub frozen_event_count: u64, + pub tail_hash: Option, + pub handoff_items: Vec, + pub handoff_scanned_bytes: u64, + pub handoff_scanned_events: u64, +} + +mod publish; +mod schema; +mod staging; +mod wire; + +#[tauri::command] +pub async fn collaboration_snapshot_ingest_begin( + request: CollaborationSnapshotIngestBeginRequest, +) -> Result { + staging::collaboration_snapshot_ingest_begin_impl(request).await +} + +#[tauri::command] +pub async fn collaboration_snapshot_ingest_apply_wire_page( + request: CollaborationSnapshotIngestPageRequest, +) -> Result { + wire::collaboration_snapshot_ingest_apply_wire_page_impl(request).await +} + +#[tauri::command] +pub async fn collaboration_snapshot_ingest_get_cursor( + request: CollaborationSnapshotIngestGetCursorRequest, +) -> Result, String> { + schema::collaboration_snapshot_ingest_get_cursor_impl(request).await +} + +#[tauri::command] +pub async fn collaboration_snapshot_secondary_probe( + request: CollaborationSnapshotSecondaryProbeRequest, +) -> Result { + schema::collaboration_snapshot_secondary_probe_impl(request).await +} + +#[tauri::command] +pub async fn collaboration_snapshot_ingest_commit( + request: CollaborationSnapshotIngestTokenRequest, +) -> Result { + publish::collaboration_snapshot_ingest_commit_impl(request).await +} + +#[tauri::command] +pub async fn collaboration_snapshot_ingest_abort( + request: CollaborationSnapshotIngestTokenRequest, +) -> Result<(), String> { + staging::collaboration_snapshot_ingest_abort_impl(request).await +} + +pub(super) use schema::collaboration_snapshot_secondary_state; +#[cfg(test)] +pub(super) use schema::install_snapshot_schema_for_test; +pub(crate) use schema::is_snapshot_backed_native_fork; + +#[cfg(test)] +mod tests; diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/publish.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/publish.rs new file mode 100644 index 000000000..d05855151 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/publish.rs @@ -0,0 +1,670 @@ +use super::schema::*; +use super::staging::*; +use super::wire::*; +use super::*; + +pub(super) fn drop_replay_accounting_triggers(tx: &Transaction<'_>) -> Result<(), String> { + // A full replacement must not emit one generation/accounting mutation per + // row. The ingest transaction publishes the exact aggregate state once + // below; the next replay access reinstalls these + // CREATE-IF-NOT-EXISTS triggers. + tx.execute_batch( + "DROP TRIGGER IF EXISTS collaboration_replay_events_insert; + DROP TRIGGER IF EXISTS collaboration_replay_events_delete; + DROP TRIGGER IF EXISTS collaboration_replay_events_update_old; + DROP TRIGGER IF EXISTS collaboration_replay_events_update_new;", + ) + .map_err(|error| format!("suspend per-row collaboration replay accounting: {error}")) +} + +pub(super) fn publish_replay_accounting_state( + tx: &Transaction<'_>, + session_id: &str, + event_count: i64, + max_sequence: i64, +) -> Result<(), String> { + tx.execute_batch( + "CREATE TABLE IF NOT EXISTS collaboration_replay_state ( + session_id TEXT PRIMARY KEY, + generation INTEGER NOT NULL DEFAULT 0, + revision INTEGER NOT NULL DEFAULT 0, + max_sequence INTEGER NOT NULL DEFAULT -1, + event_count INTEGER NOT NULL DEFAULT 0 + );", + ) + .map_err(|error| format!("initialize collaboration replay accounting state: {error}"))?; + tx.execute( + PUBLISH_REPLAY_ACCOUNTING_SQL, + params![session_id, event_count, max_sequence], + ) + .map_err(|error| format!("publish collaboration replay accounting state: {error}"))?; + Ok(()) +} + +pub(super) fn handoff_text_value(value: &serde_json::Value) -> Option { + match value { + serde_json::Value::String(text) => { + let trimmed = text.trim(); + (!trimmed.is_empty()).then(|| trimmed.to_string()) + } + serde_json::Value::Array(values) => { + let mut joined = String::new(); + for text in values.iter().filter_map(handoff_text_value) { + if !joined.is_empty() { + joined.push('\n'); + } + joined.push_str(&text); + if joined.encode_utf16().count() >= HANDOFF_MAX_ITEM_UTF16 { + break; + } + } + (!joined.is_empty()).then_some(joined) + } + serde_json::Value::Object(object) => ["text", "content", "message", "output", "summary"] + .into_iter() + .find_map(|key| object.get(key).and_then(handoff_text_value)), + _ => None, + } +} + +pub(super) fn truncate_handoff_item(text: &str) -> String { + if text.encode_utf16().count() <= HANDOFF_MAX_ITEM_UTF16 { + return text.to_string(); + } + let budget = HANDOFF_MAX_ITEM_UTF16.saturating_sub(1); + let mut output = String::new(); + let mut units = 0_usize; + for character in text.chars() { + let next = units.saturating_add(character.len_utf16()); + if next > budget { + break; + } + output.push(character); + units = next; + } + output.push('…'); + output +} + +pub(super) fn handoff_item_from_event(event: &SessionEvent) -> Option { + let action_type = event.action_type.as_str(); + let function = event.function_name.as_str(); + if action_type.contains("thinking") + || action_type.contains("reasoning") + || matches!(function, "thinking" | "thinking_delta" | "reasoning") + { + return None; + } + let result = handoff_text_value(&event.result); + let args = handoff_text_value(&event.args); + let content = result.as_deref().or(args.as_deref()).or_else(|| { + let text = event.display_text.trim(); + (!text.is_empty()).then_some(text) + }); + let item = if matches!(action_type, "user" | "user_message") + || matches!(function, "user" | "user_message") + { + content.map(|text| format!("User: {text}")) + } else if matches!( + action_type, + "assistant" | "assistant_message" | "llm_response" + ) || matches!( + function, + "agent_message" | "assistant" | "assistant_message" + ) { + content.map(|text| format!("Assistant: {text}")) + } else if action_type.contains("tool") { + let mut lines = vec![ + "[Imported Collaboration Snapshot action]".to_string(), + format!( + "Tool: {}", + if function.is_empty() { + "unknown_tool" + } else { + function + } + ), + ]; + if let Some(args) = args { + lines.push(format!("Input: {args}")); + } + if let Some(result) = result { + lines.push(format!("Result at that time: {result}")); + } + Some(lines.join("\n")) + } else { + content.map(|text| format!("Assistant context: {text}")) + }?; + Some(truncate_handoff_item(&item)) +} + +pub(super) fn collect_published_handoff( + tx: &Transaction<'_>, + session_id: &str, +) -> Result<(Vec, u64, u64), String> { + let mut statement = tx + .prepare( + "SELECT id,session_id,event_type,function_name,thread_id, + CASE WHEN length(CAST(args_json AS BLOB))<=?2 + THEN args_json + WHEN json_valid(args_json) THEN json_object( + 'content',substr(COALESCE( + json_extract(args_json,'$.content'), + json_extract(args_json,'$.text'), + json_extract(args_json,'$.message'), + json_extract(args_json,'$.command'), + json_extract(args_json,'$.path'), + json_extract(args_json,'$.description'),'' + ),1,?3) + ) ELSE '{}' END, + CASE WHEN length(CAST(result_json AS BLOB))<=?2 + THEN result_json + WHEN json_valid(result_json) THEN json_object( + 'content',substr(COALESCE( + json_extract(result_json,'$.content'), + json_extract(result_json,'$.text'), + json_extract(result_json,'$.message'), + json_extract(result_json,'$.output'), + json_extract(result_json,'$.summary'),'' + ),1,?3) + ) ELSE '{}' END, + '',created_at, + CASE WHEN length(CAST(meta_json AS BLOB))<=?2 + THEN meta_json ELSE json_object( + 'source',CASE WHEN json_valid(meta_json) + THEN json_extract(meta_json,'$.source') END, + 'displayText',CASE WHEN json_valid(meta_json) + THEN substr(json_extract(meta_json,'$.displayText'),1,1200) END, + 'displayStatus',CASE WHEN json_valid(meta_json) + THEN json_extract(meta_json,'$.displayStatus') END, + 'displayVariant',CASE WHEN json_valid(meta_json) + THEN json_extract(meta_json,'$.displayVariant') END, + 'activityStatus',CASE WHEN json_valid(meta_json) + THEN json_extract(meta_json,'$.activityStatus') END, + 'uiCanonical',CASE WHEN json_valid(meta_json) + THEN json_extract(meta_json,'$.uiCanonical') END, + 'chunk_id',CASE WHEN json_valid(meta_json) + THEN json_extract(meta_json,'$.chunk_id') END + ) END, + history_sequence, + MIN(COALESCE(length(CAST(args_json AS BLOB)),0),?2) + + MIN(COALESCE(length(CAST(result_json AS BLOB)),0),?2) + + MIN(COALESCE(length(CAST(meta_json AS BLOB)),0),?2) + + COALESCE(length(CAST(id AS BLOB)),0) + + COALESCE(length(CAST(event_type AS BLOB)),0) + + COALESCE(length(CAST(function_name AS BLOB)),0) + + COALESCE(length(CAST(created_at AS BLOB)),0) + FROM events + WHERE session_id=?1 AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC + LIMIT 400", + ) + .map_err(|error| format!("prepare collaboration handoff fold: {error}"))?; + let mut rows = statement + .query(params![ + session_id, + HANDOFF_FIELD_PREVIEW_BYTES, + HANDOFF_FIELD_PREVIEW_CHARS, + ]) + .map_err(|error| format!("query collaboration handoff fold: {error}"))?; + let mut remaining = HANDOFF_SCAN_BYTES; + let mut scanned_bytes = 0_u64; + let mut scanned_events = 0_u64; + let mut newest_first = Vec::new(); + while let Some(row) = rows + .next() + .map_err(|error| format!("read collaboration handoff row: {error}"))? + { + let row_bytes = row + .get::<_, Option>(11) + .map_err(|error| error.to_string())? + .unwrap_or(0) + .max(0) as usize; + if row_bytes > remaining { + break; + } + remaining -= row_bytes; + scanned_bytes = scanned_bytes.saturating_add(row_bytes as u64); + scanned_events = scanned_events.saturating_add(1); + let cached = session_persistence::CachedEvent { + id: row.get(0).map_err(|error| error.to_string())?, + session_id: row.get(1).map_err(|error| error.to_string())?, + event_type: row.get(2).map_err(|error| error.to_string())?, + function_name: row.get(3).map_err(|error| error.to_string())?, + thread_id: row.get(4).map_err(|error| error.to_string())?, + args_json: row.get(5).map_err(|error| error.to_string())?, + result_json: row.get(6).map_err(|error| error.to_string())?, + content: row.get(7).map_err(|error| error.to_string())?, + created_at: row.get(8).map_err(|error| error.to_string())?, + meta_json: row.get(9).map_err(|error| error.to_string())?, + history_sequence: row.get(10).map_err(|error| error.to_string())?, + }; + let event = cached_event_to_session_event(&cached); + if let Some(item) = handoff_item_from_event(&event) { + newest_first.push(item); + if newest_first.len() >= HANDOFF_MAX_ITEMS { + break; + } + } + if remaining == 0 { + break; + } + } + newest_first.reverse(); + Ok((newest_first, scanned_bytes, scanned_events)) +} + +pub(super) fn extend_time_range( + time_range_start: &mut Option, + time_range_end: &mut Option, + created_at: &str, +) { + if time_range_start + .as_deref() + .is_none_or(|current| created_at < current) + { + *time_range_start = Some(created_at.to_string()); + } + if time_range_end + .as_deref() + .is_none_or(|current| created_at > current) + { + *time_range_end = Some(created_at.to_string()); + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum SnapshotPublishTarget { + ImportedReplay, + FreshNativeFork, + ExistingNativeSnapshot, +} + +fn classify_snapshot_publish_target( + tx: &Transaction<'_>, + session_id: &str, +) -> Result { + if is_imported_snapshot_session(session_id) { + return Ok(SnapshotPublishTarget::ImportedReplay); + } + if !session_id.starts_with(AGENT_SESSION_PREFIX) { + return Err("collaboration snapshot target has an unsupported session prefix".to_string()); + } + let (event_count, has_snapshot_state): (i64, bool) = tx + .query_row( + "SELECT + (SELECT COUNT(*) FROM events WHERE session_id=?1), + EXISTS( + SELECT 1 FROM collaboration_snapshot_ingest_state + WHERE session_id=?1 + )", + [session_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .map_err(|error| format!("classify native snapshot target: {error}"))?; + match (event_count, has_snapshot_state) { + (0, false) => Ok(SnapshotPublishTarget::FreshNativeFork), + (_, true) => Ok(SnapshotPublishTarget::ExistingNativeSnapshot), + (_, false) => Err(format!( + "refusing to replace native session {session_id}: existing events are not a collaboration snapshot" + )), + } +} + +pub(super) fn publish_staged_snapshot( + destination: &Connection, + staging: &Connection, + manifest: &StagingManifest, + final_count: u64, + final_frozen_count: u64, +) -> Result { + // Incremental publication may read destination state only through primary + // keys or bounded/indexed sentinels. History-sized aggregation belongs on + // the token-scoped staging DB, whose size is exactly the incoming delta. + let tx = database::db::begin_immediate(destination) + .map_err(|error| format!("begin collaboration snapshot publish: {error}"))?; + ensure_destination_schema(&tx)?; + let publish_target = classify_snapshot_publish_target(&tx, &manifest.local_session_id)?; + let current = match read_destination_cursor(&tx, &manifest.local_session_id) { + Ok(cursor) => cursor, + Err(_) + if manifest.replace + && manifest.previous.is_none() + && matches!( + publish_target, + SnapshotPublishTarget::ImportedReplay | SnapshotPublishTarget::FreshNativeFork + ) => + { + None + } + Err(error) => return Err(error), + }; + if let Some(previous) = manifest.previous.as_ref() { + if current.as_ref() != Some(previous) { + return Err("local collaboration snapshot changed before commit".to_string()); + } + } else if !manifest.replace { + return Err("incremental collaboration snapshot has no prior cursor".to_string()); + } + let previous_session_metadata = if !manifest.replace { + if !destination_indexes_are_installed(&tx)? { + return Err( + "local collaboration snapshot indexes are missing; rebuild required".to_string(), + ); + } + let current_cursor = current.as_ref().ok_or_else(|| { + "incremental collaboration snapshot has no published base".to_string() + })?; + Some( + destination_snapshot_constant_time_metadata( + &tx, + &manifest.local_session_id, + current_cursor, + )? + .ok_or_else(|| { + "local collaboration snapshot base is incomplete; rebuild required".to_string() + })?, + ) + } else { + None + }; + + let target_cursor = CollaborationSnapshotCursor { + epoch: manifest.epoch, + frozen_seq: manifest.expected_frozen_seq, + count: final_count, + frozen_count: final_frozen_count, + tail_hash: manifest.expected_tail_hash.clone(), + }; + if !manifest.replace && current.as_ref() == Some(&target_cursor) { + let (handoff_items, handoff_scanned_bytes, handoff_scanned_events) = + collect_published_handoff(&tx, &manifest.local_session_id)?; + tx.commit() + .map_err(|error| format!("finish unchanged collaboration snapshot: {error}"))?; + return Ok(CollaborationSnapshotIngestCommitResult { + local_session_id: manifest.local_session_id.clone(), + epoch: target_cursor.epoch, + frozen_seq: target_cursor.frozen_seq, + event_count: target_cursor.count, + frozen_event_count: target_cursor.frozen_count, + tail_hash: target_cursor.tail_hash, + handoff_items, + handoff_scanned_bytes, + handoff_scanned_events, + }); + } + + let imported_snapshot = publish_target == SnapshotPublishTarget::ImportedReplay; + let native_snapshot = matches!( + publish_target, + SnapshotPublishTarget::FreshNativeFork | SnapshotPublishTarget::ExistingNativeSnapshot + ); + if imported_snapshot { + drop_replay_accounting_triggers(&tx)?; + } + if native_snapshot { + // Replacing a large fork must not issue one state UPDATE per inherited + // event. Rollback restores the previous triggers on failure; success + // reinstalls them after publishing the new aggregate state below. + drop_secondary_mutation_triggers(&tx)?; + } + if manifest.replace { + tx.execute( + "DELETE FROM events WHERE session_id=?1", + [&manifest.local_session_id], + ) + .map_err(|error| format!("clear prior collaboration snapshot events: {error}"))?; + tx.execute( + "DELETE FROM collaboration_snapshot_event_map WHERE session_id=?1", + [&manifest.local_session_id], + ) + .map_err(|error| format!("clear prior collaboration snapshot event map: {error}"))?; + create_destination_indexes(&tx)?; + } else { + tx.execute(DELETE_TAIL_EVENTS_SQL, [&manifest.local_session_id]) + .map_err(|error| format!("replace prior collaboration snapshot tail: {error}"))?; + tx.execute(DELETE_TAIL_MAP_SQL, [&manifest.local_session_id]) + .map_err(|error| format!("clear prior collaboration snapshot tail map: {error}"))?; + } + + let base_logical_index = if manifest.replace { + 0_i64 + } else { + i64::try_from( + manifest + .previous + .as_ref() + .map_or(0, |value| value.frozen_count), + ) + .map_err(|_| "previous frozen event count is too large")? + }; + let (mut time_start, mut time_end) = previous_session_metadata + .map(|metadata| (metadata.time_range_start, metadata.time_range_end)) + .unwrap_or_default(); + let mut statement = staging + .prepare( + "SELECT normalized_id,original_id,physical_seq,event_index,is_tail,event_type, + function_name,thread_id,args_json,result_json,content,created_at,meta_json + FROM staged_events ORDER BY is_tail ASC,physical_seq ASC,event_index ASC", + ) + .map_err(|error| format!("prepare staged collaboration events: {error}"))?; + let mut rows = statement + .query([]) + .map_err(|error| format!("query staged collaboration events: {error}"))?; + let mut offset = 0_i64; + while let Some(row) = rows + .next() + .map_err(|error| format!("read staged collaboration event: {error}"))? + { + let event_id: String = row.get(0).map_err(|error| error.to_string())?; + let original_id: String = row.get(1).map_err(|error| error.to_string())?; + let physical_seq: i64 = row.get(2).map_err(|error| error.to_string())?; + let event_index: i64 = row.get(3).map_err(|error| error.to_string())?; + let is_tail: bool = row.get::<_, i64>(4).map_err(|error| error.to_string())? != 0; + let logical_index = base_logical_index + .checked_add(offset) + .ok_or_else(|| "published logical event index overflow".to_string())?; + let created_at: String = row.get(11).map_err(|error| error.to_string())?; + extend_time_range(&mut time_start, &mut time_end, &created_at); + tx.execute( + "INSERT INTO events( + id,session_id,event_type,function_name,thread_id,args_json,result_json, + content,created_at,meta_json,history_sequence + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11)", + params![ + event_id, + manifest.local_session_id, + row.get::<_, String>(5).map_err(|error| error.to_string())?, + row.get::<_, Option>(6) + .map_err(|error| error.to_string())?, + row.get::<_, Option>(7) + .map_err(|error| error.to_string())?, + row.get::<_, String>(8).map_err(|error| error.to_string())?, + row.get::<_, String>(9).map_err(|error| error.to_string())?, + row.get::<_, String>(10) + .map_err(|error| error.to_string())?, + created_at, + row.get::<_, Option>(12) + .map_err(|error| error.to_string())?, + logical_index, + ], + ) + .map_err(|error| format!("publish collaboration event {event_id}: {error}"))?; + tx.execute( + "INSERT INTO collaboration_snapshot_event_map( + session_id,event_id,original_id,physical_seq,event_index,logical_index,is_tail + ) VALUES(?1,?2,?3,?4,?5,?6,?7)", + params![ + manifest.local_session_id, + event_id, + original_id, + physical_seq, + event_index, + logical_index, + is_tail, + ], + ) + .map_err(|error| format!("publish collaboration event map: {error}"))?; + offset += 1; + } + + let published_logical_count = base_logical_index + .checked_add(offset) + .and_then(|value| u64::try_from(value).ok()) + .ok_or_else(|| "published logical event count overflow".to_string())?; + if published_logical_count != final_count { + return Err(format!( + "published logical event count mismatch: expected {final_count}, got {published_logical_count}" + )); + } + + let now = Utc::now().timestamp(); + let final_event_count = + i64::try_from(final_count).map_err(|_| "final event count is too large")?; + let final_frozen_event_count = + i64::try_from(final_frozen_count).map_err(|_| "final frozen event count is too large")?; + let final_max_sequence = if final_event_count == 0 { + -1 + } else { + final_event_count - 1 + }; + tx.execute( + "INSERT INTO sessions( + session_id,event_count,cached_at,time_range_start,time_range_end,specs_json + ) VALUES(?1,?2,?3,?4,?5,NULL) + ON CONFLICT(session_id) DO UPDATE SET + event_count=excluded.event_count, + cached_at=excluded.cached_at, + time_range_start=excluded.time_range_start, + time_range_end=excluded.time_range_end", + params![ + manifest.local_session_id, + final_event_count, + now, + time_start, + time_end, + ], + ) + .map_err(|error| format!("publish collaboration session metadata: {error}"))?; + tx.execute( + "INSERT INTO collaboration_snapshot_ingest_state( + session_id,epoch,frozen_seq,event_count,frozen_event_count,tail_hash,updated_at + ) VALUES(?1,?2,?3,?4,?5,?6,?7) + ON CONFLICT(session_id) DO UPDATE SET + epoch=excluded.epoch, + frozen_seq=excluded.frozen_seq, + event_count=excluded.event_count, + frozen_event_count=excluded.frozen_event_count, + tail_hash=excluded.tail_hash, + updated_at=excluded.updated_at", + params![ + manifest.local_session_id, + manifest.epoch, + i64::try_from(manifest.expected_frozen_seq) + .map_err(|_| "final frozen sequence is too large")?, + final_event_count, + final_frozen_event_count, + manifest.expected_tail_hash, + now, + ], + ) + .map_err(|error| format!("publish collaboration snapshot cursor: {error}"))?; + if native_snapshot { + tx.execute( + "INSERT INTO collaboration_snapshot_secondary_state( + session_id,generation,revision,reset_revision,max_sequence,event_count + ) VALUES(?1,0,0,0,?2,?3) + ON CONFLICT(session_id) DO UPDATE SET + generation=collaboration_snapshot_secondary_state.generation+1, + revision=collaboration_snapshot_secondary_state.revision+1, + reset_revision=collaboration_snapshot_secondary_state.revision+1, + max_sequence=excluded.max_sequence, + event_count=excluded.event_count", + params![ + manifest.local_session_id, + final_max_sequence, + final_event_count, + ], + ) + .map_err(|error| format!("publish native fork secondary replay state: {error}"))?; + ensure_secondary_mutation_triggers(&tx)?; + } + tx.execute( + "DELETE FROM session_turns WHERE session_id=?1", + [&manifest.local_session_id], + ) + .map_err(|error| format!("invalidate collaboration turn summaries: {error}"))?; + tx.execute( + "DELETE FROM session_turn_index_state WHERE session_id=?1", + [&manifest.local_session_id], + ) + .map_err(|error| format!("invalidate collaboration turn index state: {error}"))?; + if imported_snapshot { + publish_replay_accounting_state( + &tx, + &manifest.local_session_id, + final_event_count, + final_max_sequence, + )?; + } + let (handoff_items, handoff_scanned_bytes, handoff_scanned_events) = + collect_published_handoff(&tx, &manifest.local_session_id)?; + tx.commit() + .map_err(|error| format!("commit collaboration snapshot publish: {error}"))?; + + Ok(CollaborationSnapshotIngestCommitResult { + local_session_id: manifest.local_session_id.clone(), + epoch: manifest.epoch, + frozen_seq: manifest.expected_frozen_seq, + event_count: final_count, + frozen_event_count: final_frozen_count, + tail_hash: manifest.expected_tail_hash.clone(), + handoff_items, + handoff_scanned_bytes, + handoff_scanned_events, + }) +} + +pub(super) fn commit_at_root_with_connection( + root: &Path, + token: &str, + destination: &Connection, +) -> Result { + let path = staging_path(root, token)?; + let result = (|| { + let mut staging = open_staging(&path)?; + let manifest = load_manifest(&staging)?; + if manifest.token != token { + return Err("snapshot ingest token does not match its manifest".to_string()); + } + finalize_attachments(&mut staging, root, &manifest)?; + let (final_count, final_frozen_count) = validate_complete_staging(&staging, &manifest)?; + publish_staged_snapshot( + destination, + &staging, + &manifest, + final_count, + final_frozen_count, + ) + })(); + // A commit token is single-use. Validation and SQLite failures are + // fail-closed and cannot be repaired by replaying an ambiguous suffix. + remove_staging_files(&path); + remove_token_temp_files(root, token); + result +} + +pub(super) async fn collaboration_snapshot_ingest_commit_impl( + request: CollaborationSnapshotIngestTokenRequest, +) -> Result { + let root = staging_root()?; + tokio::task::spawn_blocking(move || { + database::db::with_sessions_writer(|| { + let destination = database::db::get_connection() + .map_err(|error| format!("open sessions.db for snapshot publish: {error}"))?; + commit_at_root_with_connection(&root, &request.token, &destination) + }) + }) + .await + .map_err(|error| error.to_string())? +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/schema.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/schema.rs new file mode 100644 index 000000000..c07378efa --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/schema.rs @@ -0,0 +1,551 @@ +use super::staging::*; +use super::wire::sha256_hex; +use super::*; + +pub(super) fn ensure_destination_schema(tx: &Transaction<'_>) -> Result<(), String> { + tx.execute_batch( + "CREATE TABLE IF NOT EXISTS collaboration_snapshot_ingest_state ( + session_id TEXT PRIMARY KEY, + epoch INTEGER NOT NULL, + frozen_seq INTEGER NOT NULL, + event_count INTEGER NOT NULL, + frozen_event_count INTEGER NOT NULL, + tail_hash TEXT, + updated_at INTEGER NOT NULL + ); + CREATE TABLE IF NOT EXISTS collaboration_snapshot_event_map ( + session_id TEXT NOT NULL, + event_id TEXT NOT NULL, + original_id TEXT NOT NULL, + physical_seq INTEGER NOT NULL, + event_index INTEGER NOT NULL, + logical_index INTEGER NOT NULL, + is_tail INTEGER NOT NULL, + PRIMARY KEY(session_id,event_id) + ); + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_events_insert_invalidate + AFTER INSERT ON events + WHEN NEW.session_id GLOB 'imported-session-*' BEGIN + DELETE FROM collaboration_snapshot_ingest_state + WHERE session_id=NEW.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_events_delete_invalidate + AFTER DELETE ON events + WHEN OLD.session_id GLOB 'imported-session-*' BEGIN + DELETE FROM collaboration_snapshot_ingest_state + WHERE session_id=OLD.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_events_update_invalidate + AFTER UPDATE ON events + WHEN OLD.session_id GLOB 'imported-session-*' + OR NEW.session_id GLOB 'imported-session-*' BEGIN + DELETE FROM collaboration_snapshot_ingest_state + WHERE session_id IN (OLD.session_id,NEW.session_id); + END; + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_map_insert_invalidate + AFTER INSERT ON collaboration_snapshot_event_map + WHEN NEW.session_id GLOB 'imported-session-*' BEGIN + DELETE FROM collaboration_snapshot_ingest_state + WHERE session_id=NEW.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_map_delete_invalidate + AFTER DELETE ON collaboration_snapshot_event_map + WHEN OLD.session_id GLOB 'imported-session-*' BEGIN + DELETE FROM collaboration_snapshot_ingest_state + WHERE session_id=OLD.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_snapshot_map_update_invalidate + AFTER UPDATE ON collaboration_snapshot_event_map + WHEN OLD.session_id GLOB 'imported-session-*' + OR NEW.session_id GLOB 'imported-session-*' BEGIN + DELETE FROM collaboration_snapshot_ingest_state + WHERE session_id IN (OLD.session_id,NEW.session_id); + END;", + ) + .map_err(|error| format!("initialize collaboration snapshot destination schema: {error}"))?; + ensure_secondary_mutation_triggers(tx) +} + +pub(super) fn ensure_secondary_mutation_triggers(conn: &Connection) -> Result<(), String> { + conn.execute_batch(SECONDARY_MUTATION_TRIGGERS_SQL) + .map_err(|error| format!("initialize native fork snapshot mutation tracking: {error}")) +} + +pub(super) fn drop_secondary_mutation_triggers(conn: &Connection) -> Result<(), String> { + conn.execute_batch(DROP_SECONDARY_MUTATION_TRIGGERS_SQL) + .map_err(|error| format!("suspend native fork snapshot mutation tracking: {error}")) +} + +pub(super) fn create_destination_indexes(tx: &Transaction<'_>) -> Result<(), String> { + tx.execute_batch( + "CREATE INDEX IF NOT EXISTS idx_collaboration_snapshot_event_order + ON collaboration_snapshot_event_map(session_id,logical_index); + CREATE INDEX IF NOT EXISTS idx_collaboration_snapshot_event_tail + ON collaboration_snapshot_event_map(session_id,is_tail,event_id);", + ) + .map_err(|error| format!("initialize collaboration snapshot destination indexes: {error}")) +} + +#[cfg(test)] +pub(in crate::agent_sessions::event_pipeline::commands) fn install_snapshot_schema_for_test( + conn: &mut Connection, +) -> Result<(), String> { + let tx = conn + .transaction() + .map_err(|error| format!("begin collaboration snapshot test schema: {error}"))?; + ensure_destination_schema(&tx)?; + create_destination_indexes(&tx)?; + tx.commit() + .map_err(|error| format!("commit collaboration snapshot test schema: {error}")) +} + +pub(super) fn destination_indexes_are_installed(conn: &Connection) -> Result { + let installed: i64 = conn + .query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name IN ( + 'idx_collaboration_snapshot_event_order', + 'idx_collaboration_snapshot_event_tail' + )", + [], + |row| row.get(0), + ) + .map_err(|error| format!("inspect collaboration snapshot indexes: {error}"))?; + Ok(installed == SNAPSHOT_INDEX_COUNT) +} + +pub(super) fn read_destination_cursor( + conn: &Connection, + session_id: &str, +) -> Result, String> { + let raw = conn + .query_row( + "SELECT epoch,frozen_seq,event_count,frozen_event_count,tail_hash + FROM collaboration_snapshot_ingest_state WHERE session_id=?1", + [session_id], + |row| { + Ok(( + row.get::<_, i64>(0)?, + row.get::<_, i64>(1)?, + row.get::<_, i64>(2)?, + row.get::<_, i64>(3)?, + row.get::<_, Option>(4)?, + )) + }, + ) + .optional() + .map_err(|error| format!("read current collaboration snapshot cursor: {error}"))?; + let Some((epoch, frozen_seq, count, frozen_count, tail_hash)) = raw else { + return Ok(None); + }; + if epoch < 0 || frozen_seq < 0 || count < 0 || frozen_count < 0 { + return Err("current collaboration snapshot cursor contains negative values".to_string()); + } + Ok(Some(CollaborationSnapshotCursor { + epoch, + frozen_seq: frozen_seq as u64, + count: count as u64, + frozen_count: frozen_count as u64, + tail_hash, + })) +} + +pub(super) fn destination_snapshot_has_sentinels( + conn: &Connection, + session_id: &str, + event_count: u64, +) -> Result { + if event_count == 0 { + return Ok(true); + } + let last_index = i64::try_from(event_count - 1) + .map_err(|_| "collaboration snapshot event count is too large".to_string())?; + let (has_first, has_last): (i64, i64) = conn + .query_row( + CURSOR_SENTINEL_SQL, + params![session_id, last_index], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .map_err(|error| format!("validate collaboration snapshot sentinels: {error}"))?; + Ok(has_first == 1 && has_last == 1) +} + +pub(super) fn destination_snapshot_constant_time_metadata( + conn: &Connection, + session_id: &str, + cursor: &CollaborationSnapshotCursor, +) -> Result, String> { + let session_metadata = conn + .query_row( + "SELECT event_count,time_range_start,time_range_end + FROM sessions WHERE session_id=?1", + [session_id], + |row| { + Ok(( + row.get::<_, i64>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, Option>(2)?, + )) + }, + ) + .optional() + .map_err(|error| format!("read collaboration snapshot session metadata: {error}"))?; + let Some((session_event_count, time_range_start, time_range_end)) = session_metadata else { + return Ok(None); + }; + if session_event_count < 0 || session_event_count as u64 != cursor.count { + return Ok(None); + } + if !destination_snapshot_has_sentinels(conn, session_id, cursor.count)? { + return Ok(None); + } + Ok(Some(CollaborationSnapshotSessionMetadata { + time_range_start, + time_range_end, + })) +} + +pub(super) fn get_cursor_from_connection( + conn: &Connection, + local_session_id: &str, +) -> Result, String> { + validate_session_id(local_session_id)?; + if !is_imported_snapshot_session(local_session_id) { + return Err( + "only imported-session collaboration snapshots expose an ingest cursor".to_string(), + ); + } + let (required_tables, required_triggers, required_indexes): (i64, i64, i64) = conn + .query_row( + "SELECT + COALESCE(SUM(CASE WHEN type='table' AND name IN ( + 'collaboration_snapshot_ingest_state', + 'collaboration_snapshot_event_map', + 'events', + 'sessions' + ) THEN 1 ELSE 0 END),0), + COALESCE(SUM(CASE WHEN type='trigger' AND name IN ( + 'collaboration_snapshot_events_insert_invalidate', + 'collaboration_snapshot_events_delete_invalidate', + 'collaboration_snapshot_events_update_invalidate', + 'collaboration_snapshot_map_insert_invalidate', + 'collaboration_snapshot_map_delete_invalidate', + 'collaboration_snapshot_map_update_invalidate' + ) THEN 1 ELSE 0 END),0), + COALESCE(SUM(CASE WHEN type='index' AND name IN ( + 'idx_collaboration_snapshot_event_order', + 'idx_collaboration_snapshot_event_tail' + ) THEN 1 ELSE 0 END),0) + FROM sqlite_master", + [], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .map_err(|error| format!("inspect collaboration snapshot cursor schema: {error}"))?; + if required_tables != 4 + || required_triggers != SNAPSHOT_INVALIDATION_TRIGGER_COUNT + || required_indexes != SNAPSHOT_INDEX_COUNT + { + return Ok(None); + } + let cursor = match read_destination_cursor(conn, local_session_id) { + Ok(Some(cursor)) => cursor, + Ok(None) => return Ok(None), + Err(_) => return Ok(None), + }; + if cursor.epoch < 0 + || cursor.frozen_count > cursor.count + || cursor + .tail_hash + .as_deref() + .is_some_and(|hash| validate_hash("tailHash", hash).is_err()) + { + return Ok(None); + } + match destination_snapshot_constant_time_metadata(conn, local_session_id, &cursor) { + Ok(Some(_)) => Ok(Some(cursor)), + Ok(None) | Err(_) => Ok(None), + } +} + +pub(super) async fn collaboration_snapshot_ingest_get_cursor_impl( + request: CollaborationSnapshotIngestGetCursorRequest, +) -> Result, String> { + tokio::task::spawn_blocking(move || { + let conn = database::db::get_connection() + .map_err(|error| format!("open sessions.db for snapshot cursor: {error}"))?; + get_cursor_from_connection(&conn, &request.local_session_id) + }) + .await + .map_err(|error| error.to_string())? +} + +pub(super) fn has_snapshot_backed_native_fork( + conn: &Connection, + session_id: &str, +) -> Result { + if !session_id.starts_with(AGENT_SESSION_PREFIX) + || session_id.len() <= AGENT_SESSION_PREFIX.len() + { + return Ok(false); + } + validate_session_id(session_id)?; + if !destination_indexes_are_installed(conn)? { + return Ok(false); + } + let Some(cursor) = read_destination_cursor(conn, session_id)? else { + return Ok(false); + }; + if cursor.frozen_count > cursor.count + || cursor + .tail_hash + .as_deref() + .is_some_and(|hash| validate_hash("tailHash", hash).is_err()) + { + return Ok(false); + } + let session_event_count = conn + .query_row( + "SELECT event_count FROM sessions WHERE session_id=?1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|error| format!("read snapshot-backed fork metadata: {error}"))?; + let Some(session_event_count) = session_event_count else { + return Ok(false); + }; + if session_event_count < 0 || (session_event_count as u64) < cursor.count { + return Ok(false); + } + destination_snapshot_has_sentinels(conn, session_id, cursor.count) +} + +pub(super) fn has_native_snapshot_marker( + conn: &Connection, + session_id: &str, +) -> Result { + if !session_id.starts_with(AGENT_SESSION_PREFIX) + || session_id.len() <= AGENT_SESSION_PREFIX.len() + { + return Ok(false); + } + validate_session_id(session_id)?; + let state_table_exists = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master + WHERE type='table' AND name='collaboration_snapshot_ingest_state')", + [], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("inspect native fork snapshot marker schema: {error}"))? + != 0; + if !state_table_exists { + return Ok(false); + } + conn.query_row( + "SELECT EXISTS(SELECT 1 FROM collaboration_snapshot_ingest_state + WHERE session_id=?1)", + [session_id], + |row| row.get::<_, i64>(0), + ) + .map(|exists| exists != 0) + .map_err(|error| format!("read native fork snapshot marker: {error}")) +} + +/// Cheap crate-local origin check for background/native consumers. +/// +/// The prefix guard avoids opening SQLite for OS/SDE ids that can never be a +/// Cloud-created Agent fork. This deliberately checks the persisted snapshot +/// marker, not full sentinel integrity: a damaged inherited prefix must still +/// fail closed instead of triggering a history-sized native turn rebuild. +pub(crate) fn is_snapshot_backed_native_fork(session_id: &str) -> Result { + if !session_id.starts_with(AGENT_SESSION_PREFIX) + || session_id.len() <= AGENT_SESSION_PREFIX.len() + { + return Ok(false); + } + let conn = database::db::get_connection() + .map_err(|error| format!("open sessions.db for native fork snapshot probe: {error}"))?; + has_native_snapshot_marker(&conn, session_id) +} + +/// Resolve a snapshot-backed native fork without scanning or materializing +/// its inherited history. The `events` table remains the canonical view, so +/// `event_count` and `max_sequence` include native events appended after the +/// inherited map frontier. Append advances only the wire revision; destructive +/// mutations advance `reset_revision`; a newly published inherited snapshot +/// advances the generation. +pub(in crate::agent_sessions::event_pipeline::commands) fn collaboration_snapshot_secondary_state( + conn: &Connection, + session_id: &str, +) -> Result, String> { + if !session_id.starts_with(AGENT_SESSION_PREFIX) + || session_id.len() <= AGENT_SESSION_PREFIX.len() + { + return Ok(None); + } + validate_session_id(session_id)?; + ensure_secondary_mutation_triggers(conn)?; + if !has_snapshot_backed_native_fork(conn, session_id)? { + return Ok(None); + } + let cursor = read_destination_cursor(conn, session_id)?.ok_or_else(|| { + "snapshot-backed native fork lost its ingest cursor after validation".to_string() + })?; + let session_event_count = conn + .query_row( + "SELECT event_count FROM sessions WHERE session_id=?1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("read native fork secondary replay state: {error}"))?; + if session_event_count < 0 { + return Err("native fork secondary replay state contains negative values".to_string()); + } + let mut mutation_state = conn + .query_row( + "SELECT generation,revision,reset_revision,max_sequence,event_count + FROM collaboration_snapshot_secondary_state WHERE session_id=?1", + [session_id], + |row| { + Ok(( + row.get::<_, i64>(0)?, + row.get::<_, i64>(1)?, + row.get::<_, i64>(2)?, + row.get::<_, i64>(3)?, + row.get::<_, i64>(4)?, + )) + }, + ) + .optional() + .map_err(|error| format!("read native fork mutation cursor: {error}"))?; + if mutation_state.is_none() { + let max_sequence = conn + .query_row( + "SELECT history_sequence FROM events + WHERE session_id=?1 AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC LIMIT 1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|error| format!("seed native fork replay frontier: {error}"))? + .unwrap_or(-1); + conn.execute( + "INSERT INTO collaboration_snapshot_secondary_state( + session_id,generation,revision,reset_revision,max_sequence,event_count + ) VALUES(?1,0,0,0,?2,?3) + ON CONFLICT(session_id) DO NOTHING", + params![session_id, max_sequence, session_event_count], + ) + .map_err(|error| format!("seed native fork mutation cursor: {error}"))?; + mutation_state = conn + .query_row( + "SELECT generation,revision,reset_revision,max_sequence,event_count + FROM collaboration_snapshot_secondary_state WHERE session_id=?1", + [session_id], + |row| { + Ok(( + row.get::<_, i64>(0)?, + row.get::<_, i64>(1)?, + row.get::<_, i64>(2)?, + row.get::<_, i64>(3)?, + row.get::<_, i64>(4)?, + )) + }, + ) + .optional() + .map_err(|error| format!("reload native fork mutation cursor: {error}"))?; + } + let ( + mut secondary_generation, + mut mutation_revision, + mut reset_revision, + mut max_sequence, + tracked_event_count, + ) = mutation_state.ok_or_else(|| "native fork mutation cursor is unavailable".to_string())?; + if secondary_generation < 0 + || mutation_revision < 0 + || reset_revision < 0 + || tracked_event_count < 0 + { + return Err("native fork mutation cursor contains negative values".to_string()); + } + if tracked_event_count != session_event_count { + max_sequence = conn + .query_row( + "SELECT history_sequence FROM events + WHERE session_id=?1 AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC LIMIT 1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|error| format!("repair native fork replay frontier: {error}"))? + .unwrap_or(-1); + secondary_generation = secondary_generation.saturating_add(1); + mutation_revision = mutation_revision.saturating_add(1); + reset_revision = mutation_revision; + conn.execute( + "UPDATE collaboration_snapshot_secondary_state + SET generation=?2,revision=?3,reset_revision=?3, + max_sequence=?4,event_count=?5 + WHERE session_id=?1", + params![ + session_id, + secondary_generation, + mutation_revision, + max_sequence, + session_event_count, + ], + ) + .map_err(|error| format!("repair native fork mutation cursor: {error}"))?; + } + let has_unsequenced = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM events + WHERE session_id=?1 AND history_sequence IS NULL)", + [session_id], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("validate native fork replay sequences: {error}"))? + != 0; + if has_unsequenced { + return Err("snapshot-backed native fork contains an unsequenced event".to_string()); + } + let generation_material = format!( + "v2|{}|{}|{}|{}|{}|{}", + cursor.epoch, + cursor.frozen_seq, + cursor.count, + cursor.frozen_count, + cursor.tail_hash.as_deref().unwrap_or("-"), + secondary_generation, + ); + Ok(Some(CollaborationSnapshotSecondaryState { + generation: format!( + "collaboration-fork-v2-{}", + sha256_hex(generation_material.as_bytes()) + ), + revision: mutation_revision as u64, + reset_revision: reset_revision as u64, + max_sequence, + event_count: session_event_count as u64, + })) +} + +/// Secondary-consumer capability probe for a Cloud-created native fork. +/// +/// This does not opt the session into external replay for execution or the +/// SessionCore open path. It only proves that the immutable inherited prefix +/// still has its atomically published snapshot state and indexed sentinels. +pub(super) async fn collaboration_snapshot_secondary_probe_impl( + request: CollaborationSnapshotSecondaryProbeRequest, +) -> Result { + tokio::task::spawn_blocking(move || { + let conn = database::db::get_connection() + .map_err(|error| format!("open sessions.db for fork snapshot probe: {error}"))?; + has_snapshot_backed_native_fork(&conn, &request.session_id) + }) + .await + .map_err(|error| error.to_string())? +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/staging.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/staging.rs new file mode 100644 index 000000000..de31846c9 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/staging.rs @@ -0,0 +1,341 @@ +use super::*; + +#[derive(Debug, Clone)] +pub(super) struct StagingManifest { + pub(super) token: String, + pub(super) local_session_id: String, + pub(super) epoch: i64, + pub(super) expected_count: u64, + pub(super) expected_frozen_seq: u64, + pub(super) expected_tail_hash: Option, + pub(super) replace: bool, + pub(super) previous: Option, + pub(super) page_count: u64, + pub(super) next_cursor_json: Option, + pub(super) page_chain_complete: bool, +} + +pub(super) fn validate_session_id(session_id: &str) -> Result<(), String> { + let valid_prefix = [IMPORTED_SESSION_PREFIX, AGENT_SESSION_PREFIX] + .into_iter() + .find(|prefix| session_id.starts_with(prefix) && session_id.len() > prefix.len()); + if valid_prefix.is_some() && !session_id.contains(['/', '\\']) { + return Ok(()); + } + Err(format!( + "collaboration snapshot target must start with {IMPORTED_SESSION_PREFIX} or {AGENT_SESSION_PREFIX}" + )) +} + +pub(super) fn is_imported_snapshot_session(session_id: &str) -> bool { + session_id.starts_with(IMPORTED_SESSION_PREFIX) +} + +pub(super) fn validate_hash(label: &str, hash: &str) -> Result<(), String> { + if hash.len() == HASH_HEX_BYTES && hash.bytes().all(|byte| byte.is_ascii_hexdigit()) { + Ok(()) + } else { + Err(format!("{label} must be a 64-character SHA-256 hex digest")) + } +} + +pub(super) fn namespace_copy_id(local_session_id: &str, original_id: &str) -> String { + let prefix = format!("{local_session_id}{COPY_ID_DELIMITER}"); + if original_id.starts_with(&prefix) { + original_id.to_string() + } else { + format!("{prefix}{original_id}") + } +} + +pub(super) fn normalize_event( + mut event: SessionEvent, + local_session_id: &str, +) -> Result { + if event.id.is_empty() { + return Err("collaboration snapshot event id cannot be empty".to_string()); + } + event.id = namespace_copy_id(local_session_id, &event.id); + event.chunk_id = event + .chunk_id + .take() + .map(|id| namespace_copy_id(local_session_id, &id)); + event.session_id = local_session_id.to_string(); + Ok(event) +} + +pub(super) fn staging_root() -> Result { + let db = app_paths::sessions_db(); + let parent = db + .parent() + .ok_or_else(|| "sessions.db has no parent directory".to_string())?; + Ok(parent.join(STAGING_DIR_NAME)) +} + +pub(super) fn validate_token(token: &str) -> Result { + let parsed = Uuid::parse_str(token).map_err(|_| "invalid snapshot ingest token".to_string())?; + if parsed.to_string() != token { + return Err("snapshot ingest token is not canonical".to_string()); + } + Ok(parsed) +} + +pub(super) fn staging_path(root: &Path, token: &str) -> Result { + validate_token(token)?; + Ok(root.join(format!("{token}.sqlite"))) +} + +pub(super) fn remove_staging_files(path: &Path) { + let _ = fs::remove_file(path); + let mut wal = path.as_os_str().to_os_string(); + wal.push("-wal"); + let _ = fs::remove_file(PathBuf::from(wal)); + let mut shm = path.as_os_str().to_os_string(); + shm.push("-shm"); + let _ = fs::remove_file(PathBuf::from(shm)); +} + +pub(super) fn remove_token_temp_files(root: &Path, token: &str) { + let prefix = format!("{token}-"); + let Ok(entries) = fs::read_dir(root) else { + return; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path + .file_name() + .and_then(|value| value.to_str()) + .is_some_and(|name| name.starts_with(&prefix) && name.ends_with(".tmp")) + { + let _ = fs::remove_file(path); + } + } +} + +pub(super) fn cleanup_stale_staging(root: &Path) -> Result<(), String> { + fs::create_dir_all(root).map_err(|error| format!("create snapshot staging dir: {error}"))?; + let now = SystemTime::now(); + let entries = + fs::read_dir(root).map_err(|error| format!("read snapshot staging dir: {error}"))?; + for entry in entries.flatten() { + let path = entry.path(); + let is_sqlite = path.extension().and_then(|value| value.to_str()) == Some("sqlite"); + let is_temp = path.extension().and_then(|value| value.to_str()) == Some("tmp"); + let modified = entry.metadata().and_then(|meta| meta.modified()); + let is_stale = modified + .ok() + .and_then(|value| now.duration_since(value).ok()) + .is_some_and(|age| age >= STAGING_STALE_AFTER); + if is_temp && is_stale { + let _ = fs::remove_file(path); + } else if is_sqlite && is_stale { + if let Some(token) = path.file_stem().and_then(|value| value.to_str()) { + remove_token_temp_files(root, token); + } + remove_staging_files(&path); + } + } + Ok(()) +} + +pub(super) fn configure_staging_connection(conn: &Connection) -> Result<(), String> { + conn.execute_batch( + "PRAGMA journal_mode=WAL; + PRAGMA synchronous=FULL; + PRAGMA temp_store=FILE; + CREATE TABLE IF NOT EXISTS manifest ( + singleton INTEGER PRIMARY KEY CHECK(singleton=1), + version INTEGER NOT NULL, + token TEXT NOT NULL, + local_session_id TEXT NOT NULL, + epoch INTEGER NOT NULL, + expected_count INTEGER NOT NULL, + expected_frozen_seq INTEGER NOT NULL, + expected_tail_hash TEXT, + replace_snapshot INTEGER NOT NULL, + previous_json TEXT, + page_count INTEGER NOT NULL DEFAULT 0, + next_cursor_json TEXT, + page_chain_complete INTEGER NOT NULL DEFAULT 0 + ); + CREATE TABLE IF NOT EXISTS staged_wires ( + seq INTEGER PRIMARY KEY, + segment_hash TEXT NOT NULL, + event_count INTEGER NOT NULL, + is_tail INTEGER NOT NULL + ); + CREATE TABLE IF NOT EXISTS staged_events ( + normalized_id TEXT PRIMARY KEY, + original_id TEXT NOT NULL, + physical_seq INTEGER NOT NULL, + event_index INTEGER NOT NULL, + is_tail INTEGER NOT NULL, + event_type TEXT NOT NULL, + function_name TEXT, + thread_id TEXT, + args_json TEXT NOT NULL, + result_json TEXT NOT NULL, + content TEXT NOT NULL, + created_at TEXT NOT NULL, + meta_json TEXT + ); + CREATE UNIQUE INDEX IF NOT EXISTS idx_staged_events_order + ON staged_events(is_tail, physical_seq, event_index); + CREATE TABLE IF NOT EXISTS attachment_parts ( + attachment_id TEXT NOT NULL, + part_index INTEGER NOT NULL, + physical_seq INTEGER NOT NULL UNIQUE, + chunk_offset INTEGER NOT NULL, + chunk BLOB NOT NULL, + final_part INTEGER NOT NULL, + event_bytes INTEGER, + attachment_hash TEXT, + PRIMARY KEY(attachment_id,part_index) + );", + ) + .map_err(|error| format!("initialize snapshot staging database: {error}"))?; + Ok(()) +} + +pub(super) fn open_staging(path: &Path) -> Result { + if !path.is_file() { + return Err("snapshot ingest token is missing or expired".to_string()); + } + let conn = + Connection::open(path).map_err(|error| format!("open snapshot staging db: {error}"))?; + configure_staging_connection(&conn)?; + Ok(conn) +} + +pub(super) fn load_manifest(conn: &Connection) -> Result { + conn.query_row( + "SELECT token,local_session_id,epoch,expected_count,expected_frozen_seq, + expected_tail_hash,replace_snapshot,previous_json,page_count, + next_cursor_json,page_chain_complete + FROM manifest WHERE singleton=1 AND version=?1", + [STAGING_VERSION], + |row| { + let previous_json: Option = row.get(7)?; + let previous = previous_json + .as_deref() + .map(serde_json::from_str) + .transpose() + .map_err(|error| { + rusqlite::Error::FromSqlConversionFailure( + 7, + rusqlite::types::Type::Text, + Box::new(error), + ) + })?; + Ok(StagingManifest { + token: row.get(0)?, + local_session_id: row.get(1)?, + epoch: row.get(2)?, + expected_count: row.get::<_, i64>(3)?.max(0) as u64, + expected_frozen_seq: row.get::<_, i64>(4)?.max(0) as u64, + expected_tail_hash: row.get(5)?, + replace: row.get::<_, i64>(6)? != 0, + previous, + page_count: row.get::<_, i64>(8)?.max(0) as u64, + next_cursor_json: row.get(9)?, + page_chain_complete: row.get::<_, i64>(10)? != 0, + }) + }, + ) + .map_err(|error| format!("read snapshot staging manifest: {error}")) +} + +pub(super) fn begin_at_root( + root: &Path, + request: CollaborationSnapshotIngestBeginRequest, +) -> Result { + validate_session_id(&request.local_session_id)?; + if request.epoch < 0 { + return Err("snapshot epoch must be non-negative".to_string()); + } + if let Some(hash) = request.tail_hash.as_deref() { + validate_hash("tailHash", hash)?; + } + if !request.replace && request.previous.is_none() { + return Err("incremental snapshot ingest requires a previous cursor".to_string()); + } + if request.local_session_id.starts_with(AGENT_SESSION_PREFIX) + && (!request.replace || request.previous.is_some()) + { + return Err( + "native fork snapshot ingest must be an unconditional full replacement".to_string(), + ); + } + if let Some(previous) = request.previous.as_ref() { + if previous.epoch < 0 { + return Err("previous snapshot epoch must be non-negative".to_string()); + } + if let Some(hash) = previous.tail_hash.as_deref() { + validate_hash("previous.tailHash", hash)?; + } + if !request.replace && previous.epoch != request.epoch { + return Err("incremental snapshot ingest cannot change epoch".to_string()); + } + if !request.replace && previous.frozen_seq > request.expected_frozen_seq { + return Err("incremental snapshot frozen sequence moved backwards".to_string()); + } + } + + cleanup_stale_staging(root)?; + let token = Uuid::new_v4().to_string(); + let path = staging_path(root, &token)?; + let conn = + Connection::open(&path).map_err(|error| format!("create snapshot staging db: {error}"))?; + configure_staging_connection(&conn)?; + let previous_json = request + .previous + .as_ref() + .map(serde_json::to_string) + .transpose() + .map_err(|error| format!("serialize previous snapshot cursor: {error}"))?; + conn.execute( + "INSERT INTO manifest( + singleton,version,token,local_session_id,epoch,expected_count, + expected_frozen_seq,expected_tail_hash,replace_snapshot,previous_json + ) VALUES(1,?1,?2,?3,?4,?5,?6,?7,?8,?9)", + params![ + STAGING_VERSION, + token, + request.local_session_id, + request.epoch, + i64::try_from(request.expected_count).map_err(|_| "expectedCount is too large")?, + i64::try_from(request.expected_frozen_seq) + .map_err(|_| "expectedFrozenSeq is too large")?, + request.tail_hash, + request.replace, + previous_json, + ], + ) + .map_err(|error| format!("write snapshot staging manifest: {error}"))?; + Ok(CollaborationSnapshotIngestBeginResult { token }) +} + +pub(super) async fn collaboration_snapshot_ingest_begin_impl( + request: CollaborationSnapshotIngestBeginRequest, +) -> Result { + let root = staging_root()?; + tokio::task::spawn_blocking(move || begin_at_root(&root, request)) + .await + .map_err(|error| error.to_string())? +} + +pub(super) fn abort_at_root(root: &Path, token: &str) -> Result<(), String> { + let path = staging_path(root, token)?; + remove_staging_files(&path); + remove_token_temp_files(root, token); + Ok(()) +} + +pub(super) async fn collaboration_snapshot_ingest_abort_impl( + request: CollaborationSnapshotIngestTokenRequest, +) -> Result<(), String> { + let root = staging_root()?; + tokio::task::spawn_blocking(move || abort_at_root(&root, &request.token)) + .await + .map_err(|error| error.to_string())? +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/tests/mod.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/tests/mod.rs new file mode 100644 index 000000000..1ad7824db --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/tests/mod.rs @@ -0,0 +1,1119 @@ +use std::collections::HashMap; +use std::io::Write as _; + +use base64::Engine as _; +use flate2::write::GzEncoder; +use flate2::Compression; +use tempfile::TempDir; + +use super::*; +use super::{publish::*, schema::*, staging::*, wire::*}; +use crate::agent_sessions::event_pipeline::types::{ + ActivityStatus, EventDisplayStatus, EventDisplayVariant, EventSource, +}; + +fn test_event(id: &str, source: EventSource, text: &str) -> SessionEvent { + let (action_type, function_name) = match source { + EventSource::User => ("user_message", "user_message"), + EventSource::Assistant => ("assistant_message", "agent_message"), + EventSource::System => ("tool_call", "read"), + }; + SessionEvent { + id: id.to_string(), + chunk_id: Some(format!("chunk-{id}")), + session_id: "source-session".to_string(), + created_at: "2026-07-22T00:00:00.000Z".to_string(), + function_name: function_name.to_string(), + ui_canonical: function_name.to_string(), + action_type: action_type.to_string(), + args: serde_json::json!({ "content": text }), + result: serde_json::json!({}), + source, + display_text: text.to_string(), + display_status: EventDisplayStatus::Completed, + display_variant: EventDisplayVariant::Message, + activity_status: ActivityStatus::Agent, + thread_id: None, + process_id: None, + call_id: None, + file_path: None, + command: None, + is_delta: None, + repo_id: None, + repo_path: None, + extracted: None, + payload_refs: Vec::new(), + shell_replay: None, + shell_replay_bookmarks: Some(HashMap::new()), + last_extract_at: None, + } +} + +fn gzip_base64(bytes: &[u8]) -> String { + let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); + encoder.write_all(bytes).expect("gzip bytes"); + BASE64_STANDARD.encode(encoder.finish().expect("finish gzip")) +} + +fn v1_wire(seq: u64, events: &[SessionEvent]) -> CollaborationSnapshotWire { + let bytes = serde_json::to_vec(events).expect("serialize event segment"); + CollaborationSnapshotWire { + seq, + payload_gz: gzip_base64(&bytes), + event_count: events.len() as u64, + segment_hash: sha256_hex(&bytes), + } +} + +fn page_bytes(segments: &[CollaborationSnapshotWire]) -> u64 { + segments + .iter() + .map(|wire| serde_json::to_vec(wire).expect("measure wire").len() as u64) + .sum() +} + +fn deterministic_incompressible_text(bytes: usize) -> String { + const ALPHABET: &[u8] = b"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; + let mut state = 0x9e37_79b9_u32; + let mut text = String::with_capacity(bytes); + for _ in 0..bytes { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + text.push(ALPHABET[state as usize % ALPHABET.len()] as char); + } + text +} + +fn backward_page( + token: &str, + epoch: i64, + frozen_seq: u64, + count: u64, + before_seq: Option, + next_before_seq: Option, + segments: Vec, +) -> CollaborationSnapshotIngestPageRequest { + let has_more = next_before_seq.is_some(); + CollaborationSnapshotIngestPageRequest { + token: token.to_string(), + epoch, + frozen_seq, + count, + tail_hash: None, + cursor: CollaborationSnapshotWireCursor::Backward { before_seq }, + next_cursor: next_before_seq.map(|before_seq| CollaborationSnapshotWireCursor::Backward { + before_seq: Some(before_seq), + }), + tail_included: false, + has_more, + returned_wire_bytes: page_bytes(&segments), + segments, + } +} + +fn forward_page( + token: &str, + epoch: i64, + frozen_seq: u64, + count: u64, + after_seq: u64, + segments: Vec, + has_more: bool, +) -> CollaborationSnapshotIngestPageRequest { + let last = segments + .iter() + .map(|wire| wire.seq) + .max() + .unwrap_or(after_seq); + CollaborationSnapshotIngestPageRequest { + token: token.to_string(), + epoch, + frozen_seq, + count, + tail_hash: None, + cursor: CollaborationSnapshotWireCursor::Forward { + after_seq, + through_seq: Some(frozen_seq), + }, + next_cursor: has_more.then_some(CollaborationSnapshotWireCursor::Forward { + after_seq: last, + through_seq: Some(frozen_seq), + }), + tail_included: false, + has_more, + returned_wire_bytes: page_bytes(&segments), + segments, + } +} + +fn destination() -> Connection { + let conn = Connection::open_in_memory().expect("destination db"); + session_persistence::init_session_tables(&conn).expect("session schema"); + conn +} + +fn begin_replace(root: &Path, session_id: &str, epoch: i64, count: u64, frozen_seq: u64) -> String { + begin_at_root( + root, + CollaborationSnapshotIngestBeginRequest { + local_session_id: session_id.to_string(), + epoch, + expected_count: count, + expected_frozen_seq: frozen_seq, + tail_hash: None, + replace: true, + previous: None, + }, + ) + .expect("begin ingest") + .token +} + +#[test] +fn v1_backward_pages_publish_atomically_and_namespace_ids() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "imported-session-v1-pages"; + let first = test_event("first", EventSource::User, "hello"); + let second = test_event("second", EventSource::Assistant, "world"); + let token = begin_replace(root, session_id, 4, 2, 2); + let second_wire = v1_wire(2, std::slice::from_ref(&second)); + apply_page_at_root( + root, + backward_page(&token, 4, 2, 2, None, Some(2), vec![second_wire]), + ) + .expect("newest page"); + let first_wire = v1_wire(1, std::slice::from_ref(&first)); + apply_page_at_root( + root, + backward_page(&token, 4, 2, 2, Some(2), None, vec![first_wire]), + ) + .expect("oldest page"); + + let result = + commit_at_root_with_connection(root, &token, &destination).expect("publish snapshot"); + assert_eq!(result.event_count, 2); + assert_eq!(result.frozen_event_count, 2); + assert_eq!( + result.handoff_items, + vec!["User: hello", "Assistant: world"] + ); + let ids = destination + .prepare("SELECT id FROM events WHERE session_id=?1 ORDER BY history_sequence") + .expect("prepare ids") + .query_map([session_id], |row| row.get::<_, String>(0)) + .expect("query ids") + .collect::>>() + .expect("collect ids"); + assert_eq!( + ids, + vec![ + format!("{session_id}~first"), + format!("{session_id}~second") + ] + ); + assert!(!staging_path(root, &token).expect("stage path").exists()); +} + +#[test] +fn one_oversized_legacy_v1_wire_remains_importable() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "imported-session-legacy-large-row"; + let payload = deterministic_incompressible_text(512 * 1024); + let event = test_event("legacy-large", EventSource::Assistant, &payload); + let wire = v1_wire(1, std::slice::from_ref(&event)); + let wire_bytes = page_bytes(std::slice::from_ref(&wire)) as usize; + assert!(wire_bytes > MAX_WIRE_BYTES); + assert!(wire_bytes <= LEGACY_V1_MAX_WIRE_BYTES); + + let token = begin_replace(root, session_id, 5, 1, 1); + apply_page_at_root(root, backward_page(&token, 5, 1, 1, None, None, vec![wire])) + .expect("stage pre-V2 oversized row"); + let result = + commit_at_root_with_connection(root, &token, &destination).expect("publish legacy row"); + assert_eq!(result.event_count, 1); + let args_json: String = destination + .query_row( + "SELECT args_json FROM events WHERE session_id=?1", + [session_id], + |row| row.get(0), + ) + .expect("load migrated args"); + let args: serde_json::Value = serde_json::from_str(&args_json).expect("parse migrated args"); + assert_eq!(args["content"].as_str(), Some(payload.as_str())); +} + +#[test] +fn oversized_attachment_v2_wire_cannot_use_the_legacy_allowance() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let chunk = deterministic_incompressible_text(384 * 1024).into_bytes(); + let attachment_hash = sha256_hex(&chunk); + let header = ReplayAttachmentV2FrameHeader { + kind: "event".to_string(), + attachment_id: attachment_hash.clone(), + part_index: 0, + chunk_offset: 0, + chunk_bytes: chunk.len() as u64, + final_part: true, + event_bytes: Some(chunk.len() as u64), + attachment_hash: Some(attachment_hash), + }; + let frame = encode_replay_attachment_v2_frame(&header, &chunk).expect("encode large V2 frame"); + let wire = CollaborationSnapshotWire { + seq: 1, + payload_gz: gzip_base64(&frame), + event_count: 1, + segment_hash: sha256_hex(&frame), + }; + assert!(page_bytes(std::slice::from_ref(&wire)) as usize > MAX_WIRE_BYTES); + let token = begin_replace(root, "imported-session-v2-over-budget", 6, 1, 1); + let error = apply_page_at_root(root, backward_page(&token, 6, 1, 1, None, None, vec![wire])) + .expect_err("new V2 rows must keep the 256 KiB physical limit"); + assert!(error.contains("Attachment V2 physical wire")); +} + +#[test] +fn native_fork_snapshot_publishes_without_external_replay_accounting() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "agentsession-cloud-fork"; + let event = test_event("source-event", EventSource::User, "fork context"); + let token = begin_replace(root, session_id, 9, 1, 1); + let wire = v1_wire(1, std::slice::from_ref(&event)); + apply_page_at_root(root, backward_page(&token, 9, 1, 1, None, None, vec![wire])) + .expect("stage native fork snapshot"); + let result = commit_at_root_with_connection(root, &token, &destination) + .expect("publish native fork snapshot"); + assert_eq!(result.local_session_id, session_id); + assert_eq!(result.handoff_items, vec!["User: fork context"]); + let event_id: String = destination + .query_row( + "SELECT id FROM events WHERE session_id=?1", + [session_id], + |row| row.get(0), + ) + .expect("native fork event"); + assert_eq!(event_id, format!("{session_id}~source-event")); + let replay_state_table: i64 = destination + .query_row( + "SELECT COUNT(*) FROM sqlite_master + WHERE type='table' AND name='collaboration_replay_state'", + [], + |row| row.get(0), + ) + .expect("replay state table count"); + assert_eq!(replay_state_table, 0); + assert!(has_snapshot_backed_native_fork(&destination, session_id) + .expect("probe intact native fork")); + assert!(has_native_snapshot_marker(&destination, session_id) + .expect("read native fork origin marker")); + let initial_state = collaboration_snapshot_secondary_state(&destination, session_id) + .expect("read initial native fork secondary state") + .expect("native fork has secondary state"); + assert_eq!(initial_state.revision, 0); + assert_eq!(initial_state.reset_revision, 0); + assert_eq!(initial_state.max_sequence, 0); + assert_eq!(initial_state.event_count, 1); + + let replacement = test_event("source-event", EventSource::User, "new fork context"); + let replacement_token = begin_replace(root, session_id, 10, 1, 1); + let replacement_wire = v1_wire(1, std::slice::from_ref(&replacement)); + apply_page_at_root( + root, + backward_page( + &replacement_token, + 10, + 1, + 1, + None, + None, + vec![replacement_wire], + ), + ) + .expect("stage replacement native fork snapshot"); + commit_at_root_with_connection(root, &replacement_token, &destination) + .expect("replace native fork snapshot"); + let replaced_state = collaboration_snapshot_secondary_state(&destination, session_id) + .expect("read replaced native fork secondary state") + .expect("replaced native fork remains snapshot-backed"); + assert_ne!(replaced_state.generation, initial_state.generation); + assert_eq!(replaced_state.revision, initial_state.revision + 1); + assert_eq!(replaced_state.reset_revision, replaced_state.revision); + assert_eq!(replaced_state.max_sequence, 0); + assert_eq!(replaced_state.event_count, 1); + + destination + .execute( + "INSERT INTO events( + id,session_id,event_type,function_name,args_json,result_json, + content,created_at,history_sequence + ) VALUES(?1,?2,'assistant_message','agent_message','{}','{}', + 'native suffix','2026-07-22T00:00:01.000Z',1)", + params![format!("{session_id}~native-suffix"), session_id], + ) + .expect("append native suffix event"); + destination + .execute( + "UPDATE sessions SET event_count=2 WHERE session_id=?1", + [session_id], + ) + .expect("publish native suffix count"); + assert!(has_snapshot_backed_native_fork(&destination, session_id) + .expect("probe native fork with suffix")); + let appended_state = collaboration_snapshot_secondary_state(&destination, session_id) + .expect("read appended native fork secondary state") + .expect("native fork remains snapshot-backed"); + assert_eq!(appended_state.generation, replaced_state.generation); + assert_eq!(appended_state.revision, replaced_state.revision + 1); + assert_eq!(appended_state.reset_revision, replaced_state.reset_revision); + assert_eq!(appended_state.max_sequence, 1); + assert_eq!(appended_state.event_count, 2); + + destination + .execute( + "UPDATE events SET result_json='{\"content\":\"updated suffix\"}' + WHERE id=?1", + [format!("{session_id}~native-suffix")], + ) + .expect("update native suffix event"); + let updated_state = collaboration_snapshot_secondary_state(&destination, session_id) + .expect("read updated native fork secondary state") + .expect("updated native fork remains snapshot-backed"); + assert_eq!(updated_state.generation, appended_state.generation); + assert_eq!(updated_state.revision, appended_state.revision + 1); + assert_eq!(updated_state.reset_revision, updated_state.revision); + assert_eq!(updated_state.max_sequence, 1); + assert_eq!(updated_state.event_count, 2); + + destination + .execute( + "DELETE FROM events WHERE id=?1", + [format!("{session_id}~source-event")], + ) + .expect("remove inherited sentinel"); + assert!(!has_snapshot_backed_native_fork(&destination, session_id) + .expect("reject hollow native fork")); + assert!(has_native_snapshot_marker(&destination, session_id) + .expect("damaged snapshot still fails closed for background consumers")); + assert!( + !has_snapshot_backed_native_fork(&destination, "agentsession-native") + .expect("ordinary native session has no snapshot") + ); + assert!( + !has_snapshot_backed_native_fork(&destination, "sdeagent-native") + .expect("SDE session is never snapshot-backed") + ); +} + +#[test] +fn native_snapshot_commit_refuses_to_replace_an_ordinary_agent_session() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "agentsession-existing-native"; + let token = begin_replace(root, session_id, 1, 1, 1); + let incoming = test_event("incoming", EventSource::User, "cloud context"); + let wire = v1_wire(1, std::slice::from_ref(&incoming)); + apply_page_at_root(root, backward_page(&token, 1, 1, 1, None, None, vec![wire])) + .expect("stage native replacement"); + + // Seed after begin to prove the commit-time check and destructive write + // share the same BEGIN IMMEDIATE transaction. + destination + .execute( + "INSERT INTO events( + id,session_id,event_type,function_name,args_json,result_json, + content,created_at,history_sequence + ) VALUES( + 'native-event',?1,'user_message','user_message','{}','{}', + 'must survive','2026-07-25T00:00:00.000Z',0 + )", + [session_id], + ) + .expect("seed ordinary native event"); + + let error = commit_at_root_with_connection(root, &token, &destination) + .expect_err("ordinary native session must not be replaced"); + assert!(error.contains("existing events are not a collaboration snapshot")); + assert_eq!( + destination + .query_row( + "SELECT content FROM events WHERE id='native-event'", + [], + |row| row.get::<_, String>(0), + ) + .expect("ordinary native event remains"), + "must survive" + ); + assert!(!staging_path(root, &token).expect("stage path").exists()); +} + +#[test] +fn cursor_query_returns_only_an_intact_imported_snapshot() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "imported-session-cursor"; + let event = test_event("cursor-event", EventSource::User, "cursor payload"); + let token = begin_replace(root, session_id, 12, 1, 1); + let wire = v1_wire(1, std::slice::from_ref(&event)); + apply_page_at_root( + root, + backward_page(&token, 12, 1, 1, None, None, vec![wire]), + ) + .expect("stage cursor snapshot"); + commit_at_root_with_connection(root, &token, &destination).expect("publish cursor snapshot"); + + let intact_cursor = get_cursor_from_connection(&destination, session_id) + .expect("read intact cursor") + .expect("intact cursor exists"); + assert_eq!( + intact_cursor, + CollaborationSnapshotCursor { + epoch: 12, + frozen_seq: 1, + count: 1, + frozen_count: 1, + tail_hash: None, + } + ); + assert_eq!( + serde_json::to_value(&intact_cursor).expect("serialize cursor wire value"), + serde_json::json!({ + "epoch": 12, + "frozenSeq": 1, + "count": 1, + "frozenCount": 1, + "tailHash": null, + }) + ); + + let trigger_count: i64 = destination + .query_row( + "SELECT COUNT(*) FROM sqlite_master + WHERE type='trigger' AND name LIKE 'collaboration_snapshot_%_invalidate'", + [], + |row| row.get(0), + ) + .expect("snapshot invalidation trigger count"); + assert_eq!(trigger_count, SNAPSHOT_INVALIDATION_TRIGGER_COUNT); + let query_plan = destination + .prepare(&format!("EXPLAIN QUERY PLAN {CURSOR_SENTINEL_SQL}")) + .expect("prepare sentinel query plan") + .query_map(params![session_id, 0_i64], |row| row.get::<_, String>(3)) + .expect("query sentinel plan") + .collect::>>() + .expect("collect sentinel plan"); + assert!(query_plan + .iter() + .any(|detail| detail.contains("idx_collaboration_snapshot_event_order"))); + assert!(!query_plan + .iter() + .any(|detail| detail.contains("SCAN m") || detail.contains("SCAN e"))); + + destination + .execute("DELETE FROM events WHERE session_id=?1", [session_id]) + .expect("make snapshot hollow"); + let state_rows: i64 = destination + .query_row( + "SELECT COUNT(*) FROM collaboration_snapshot_ingest_state + WHERE session_id=?1", + [session_id], + |row| row.get(0), + ) + .expect("count invalidated state"); + assert_eq!(state_rows, 0); + assert_eq!( + get_cursor_from_connection(&destination, session_id).expect("read hollow cursor"), + None + ); + + destination + .execute( + "INSERT INTO collaboration_snapshot_ingest_state( + session_id,epoch,frozen_seq,event_count,frozen_event_count,tail_hash,updated_at + ) VALUES(?1,12,1,1,1,NULL,0)", + [session_id], + ) + .expect("restore stale cursor state"); + assert_eq!( + get_cursor_from_connection(&destination, session_id) + .expect("sentinel rejects hollow cursor"), + None + ); + destination + .execute( + "UPDATE collaboration_snapshot_ingest_state SET event_count=-1 + WHERE session_id=?1", + [session_id], + ) + .expect("corrupt cursor state"); + assert_eq!( + get_cursor_from_connection(&destination, session_id).expect("read invalid cursor"), + None + ); + + let repair_event = test_event("repair-event", EventSource::Assistant, "repaired"); + let repair_token = begin_replace(root, session_id, 13, 1, 1); + let repair_wire = v1_wire(1, std::slice::from_ref(&repair_event)); + apply_page_at_root( + root, + backward_page(&repair_token, 13, 1, 1, None, None, vec![repair_wire]), + ) + .expect("stage repaired cursor snapshot"); + commit_at_root_with_connection(root, &repair_token, &destination) + .expect("full replacement repairs an invalid cursor"); + assert_eq!( + get_cursor_from_connection(&destination, session_id).expect("read repaired cursor"), + Some(CollaborationSnapshotCursor { + epoch: 13, + frozen_seq: 1, + count: 1, + frozen_count: 1, + tail_hash: None, + }) + ); +} + +#[test] +fn cursor_query_rejects_native_agent_sessions() { + let destination = destination(); + let error = get_cursor_from_connection(&destination, "agentsession-native-fork") + .expect_err("native sessions must not expose external snapshot cursors"); + assert!(error.contains("only imported-session")); +} + +#[test] +fn incremental_destination_queries_use_indexes_on_a_large_map() { + let mut destination = destination(); + let session_id = "imported-session-large-cursor-map"; + let tx = destination.transaction().expect("large map transaction"); + ensure_destination_schema(&tx).expect("snapshot schema"); + create_destination_indexes(&tx).expect("snapshot indexes"); + tx.execute_batch( + "WITH digits(n) AS ( + VALUES(0),(1),(2),(3),(4),(5),(6),(7),(8),(9) + ), sequence(i) AS ( + SELECT a.n + 10*b.n + 100*c.n + 1000*d.n + FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d + ) + INSERT INTO events( + id,session_id,event_type,function_name,args_json,result_json, + content,created_at,history_sequence + ) + SELECT printf('imported-session-large-cursor-map~event-%d',i), + 'imported-session-large-cursor-map','user_message','user_message', + '{}','{}','','2026-07-22T00:00:00.000Z',i + FROM sequence; + + WITH digits(n) AS ( + VALUES(0),(1),(2),(3),(4),(5),(6),(7),(8),(9) + ), sequence(i) AS ( + SELECT a.n + 10*b.n + 100*c.n + 1000*d.n + FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d + ) + INSERT INTO collaboration_snapshot_event_map( + session_id,event_id,original_id,physical_seq,event_index,logical_index,is_tail + ) + SELECT 'imported-session-large-cursor-map', + printf('imported-session-large-cursor-map~event-%d',i), + printf('event-%d',i),i+1,0,i,0 + FROM sequence; + + INSERT INTO sessions(session_id,event_count,cached_at) + VALUES('imported-session-large-cursor-map',10000,0); + INSERT INTO collaboration_snapshot_ingest_state( + session_id,epoch,frozen_seq,event_count,frozen_event_count,tail_hash,updated_at + ) VALUES('imported-session-large-cursor-map',1,10000,10000,10000,NULL,0);", + ) + .expect("seed large cursor map"); + tx.commit().expect("commit large cursor map"); + + let cursor = get_cursor_from_connection(&destination, session_id) + .expect("read large cursor map") + .expect("large cursor remains healthy"); + assert_eq!(cursor.count, 10_000); + let query_plan = destination + .prepare(&format!("EXPLAIN QUERY PLAN {CURSOR_SENTINEL_SQL}")) + .expect("prepare large sentinel query plan") + .query_map(params![session_id, 9_999_i64], |row| { + row.get::<_, String>(3) + }) + .expect("query large sentinel plan") + .collect::>>() + .expect("collect large sentinel plan"); + assert!(query_plan + .iter() + .any(|detail| detail.contains("idx_collaboration_snapshot_event_order"))); + assert!(!query_plan + .iter() + .any(|detail| detail.contains("SCAN m") || detail.contains("SCAN e"))); + + let delete_tail_plan = destination + .prepare(&format!("EXPLAIN QUERY PLAN {DELETE_TAIL_EVENTS_SQL}")) + .expect("prepare indexed tail delete plan") + .query_map([session_id], |row| row.get::<_, String>(3)) + .expect("query indexed tail delete plan") + .collect::>>() + .expect("collect indexed tail delete plan"); + assert!(delete_tail_plan + .iter() + .any(|detail| detail.contains("idx_collaboration_snapshot_event_tail"))); + assert!(!delete_tail_plan.iter().any(|detail| { + let detail = detail.to_ascii_uppercase(); + detail.contains("SCAN EVENTS") || detail.contains("SCAN COLLABORATION_SNAPSHOT_EVENT_MAP") + })); + let delete_tail_map_plan = destination + .prepare(&format!("EXPLAIN QUERY PLAN {DELETE_TAIL_MAP_SQL}")) + .expect("prepare indexed tail map delete plan") + .query_map([session_id], |row| row.get::<_, String>(3)) + .expect("query indexed tail map delete plan") + .collect::>>() + .expect("collect indexed tail map delete plan"); + assert!(delete_tail_map_plan + .iter() + .any(|detail| detail.contains("idx_collaboration_snapshot_event_tail"))); + assert!(!delete_tail_map_plan.iter().any(|detail| { + detail + .to_ascii_uppercase() + .contains("SCAN COLLABORATION_SNAPSHOT_EVENT_MAP") + })); + + let handoff_plan = destination + .prepare( + "EXPLAIN QUERY PLAN + SELECT id FROM events + WHERE session_id=?1 AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC LIMIT 400", + ) + .expect("prepare bounded handoff plan") + .query_map([session_id], |row| row.get::<_, String>(3)) + .expect("query bounded handoff plan") + .collect::>>() + .expect("collect bounded handoff plan"); + assert!(handoff_plan + .iter() + .any(|detail| detail.contains("idx_events_session_sequence"))); + assert!(!handoff_plan + .iter() + .any(|detail| { detail.to_ascii_uppercase().contains("SCAN EVENTS") })); + + let replay_sql = PUBLISH_REPLAY_ACCOUNTING_SQL.to_ascii_uppercase(); + assert!(!replay_sql.contains("SELECT")); + assert!(!replay_sql.contains("FROM EVENTS")); + assert!(!replay_sql.contains("COUNT(")); + assert!(!replay_sql.contains("MAX(")); +} + +#[test] +fn commit_handoff_is_last_80_bounded_items_and_skips_thinking() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "agentsession-bounded-handoff"; + let mut events = (0..82) + .map(|index| { + test_event( + &format!("user-{index}"), + EventSource::User, + &format!("{index}-{}", "x".repeat(2_000)), + ) + }) + .collect::>(); + let mut thinking = test_event("thinking", EventSource::Assistant, "private reasoning"); + thinking.action_type = "thinking".to_string(); + thinking.function_name = "reasoning".to_string(); + thinking.display_variant = EventDisplayVariant::Thinking; + events.push(thinking); + let token = begin_replace(root, session_id, 1, events.len() as u64, 1); + let wire = v1_wire(1, &events); + apply_page_at_root( + root, + backward_page(&token, 1, 1, events.len() as u64, None, None, vec![wire]), + ) + .expect("stage handoff events"); + let result = + commit_at_root_with_connection(root, &token, &destination).expect("publish handoff events"); + assert_eq!(result.handoff_items.len(), HANDOFF_MAX_ITEMS); + assert!(result + .handoff_items + .iter() + .all(|item| item.encode_utf16().count() <= HANDOFF_MAX_ITEM_UTF16)); + assert!(result + .handoff_items + .iter() + .all(|item| !item.contains("private reasoning"))); + assert!(result.handoff_scanned_bytes <= HANDOFF_SCAN_BYTES as u64); +} + +#[test] +fn incremental_publish_preserves_frozen_prefix_and_replaces_tail() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "imported-session-incremental"; + let mut first = test_event("first", EventSource::User, "first"); + first.created_at = "2026-07-20T00:00:00.000Z".to_string(); + let mut old_tail = test_event("old-tail", EventSource::Assistant, "old tail"); + old_tail.created_at = "2026-07-21T00:00:00.000Z".to_string(); + let first_wire = v1_wire(1, std::slice::from_ref(&first)); + let old_tail_wire = v1_wire(0, std::slice::from_ref(&old_tail)); + let old_tail_hash = old_tail_wire.segment_hash.clone(); + let token = begin_at_root( + root, + CollaborationSnapshotIngestBeginRequest { + local_session_id: session_id.to_string(), + epoch: 1, + expected_count: 2, + expected_frozen_seq: 1, + tail_hash: Some(old_tail_hash.clone()), + replace: true, + previous: None, + }, + ) + .expect("begin initial snapshot") + .token; + let segments = vec![first_wire, old_tail_wire]; + apply_page_at_root( + root, + CollaborationSnapshotIngestPageRequest { + token: token.clone(), + epoch: 1, + frozen_seq: 1, + count: 2, + tail_hash: Some(old_tail_hash.clone()), + cursor: CollaborationSnapshotWireCursor::Backward { before_seq: None }, + next_cursor: None, + tail_included: true, + has_more: false, + returned_wire_bytes: page_bytes(&segments), + segments, + }, + ) + .expect("stage initial snapshot"); + commit_at_root_with_connection(root, &token, &destination).expect("publish initial snapshot"); + + let previous = CollaborationSnapshotCursor { + epoch: 1, + frozen_seq: 1, + count: 2, + frozen_count: 1, + tail_hash: Some(old_tail_hash), + }; + let mut second = test_event("second", EventSource::User, "second"); + second.created_at = "2026-07-22T00:00:00.000Z".to_string(); + let mut new_tail = test_event("new-tail", EventSource::Assistant, "new tail"); + new_tail.created_at = "2026-07-23T00:00:00.000Z".to_string(); + let second_wire = v1_wire(2, std::slice::from_ref(&second)); + let new_tail_wire = v1_wire(0, std::slice::from_ref(&new_tail)); + let new_tail_hash = new_tail_wire.segment_hash.clone(); + let token = begin_at_root( + root, + CollaborationSnapshotIngestBeginRequest { + local_session_id: session_id.to_string(), + epoch: 1, + expected_count: 3, + expected_frozen_seq: 2, + tail_hash: Some(new_tail_hash.clone()), + replace: false, + previous: Some(previous), + }, + ) + .expect("begin incremental snapshot") + .token; + let segments = vec![second_wire, new_tail_wire]; + apply_page_at_root( + root, + CollaborationSnapshotIngestPageRequest { + token: token.clone(), + epoch: 1, + frozen_seq: 2, + count: 3, + tail_hash: Some(new_tail_hash.clone()), + cursor: CollaborationSnapshotWireCursor::Forward { + after_seq: 1, + through_seq: Some(2), + }, + next_cursor: None, + tail_included: true, + has_more: false, + returned_wire_bytes: page_bytes(&segments), + segments, + }, + ) + .expect("stage incremental snapshot"); + let result = commit_at_root_with_connection(root, &token, &destination) + .expect("publish incremental snapshot"); + assert_eq!(result.event_count, 3); + assert_eq!(result.frozen_event_count, 2); + assert_eq!(result.tail_hash.as_deref(), Some(new_tail_hash.as_str())); + let ids = destination + .prepare("SELECT id FROM events WHERE session_id=?1 ORDER BY history_sequence") + .expect("prepare ids") + .query_map([session_id], |row| row.get::<_, String>(0)) + .expect("query ids") + .collect::>>() + .expect("collect ids"); + assert_eq!( + ids, + vec![ + format!("{session_id}~first"), + format!("{session_id}~second"), + format!("{session_id}~new-tail"), + ] + ); + let session_time_range: (Option, Option) = destination + .query_row( + "SELECT time_range_start,time_range_end FROM sessions WHERE session_id=?1", + [session_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("incremental session time range"); + assert_eq!( + session_time_range, + ( + Some("2026-07-20T00:00:00.000Z".to_string()), + Some("2026-07-23T00:00:00.000Z".to_string()), + ) + ); + + let replay_accounting_before: (i64, i64, i64, i64) = destination + .query_row( + "SELECT generation,revision,max_sequence,event_count + FROM collaboration_replay_state WHERE session_id=?1", + [session_id], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), + ) + .expect("replay accounting before no-op"); + assert_eq!(replay_accounting_before, (1, 3, 2, 3)); + let unchanged_cursor = CollaborationSnapshotCursor { + epoch: 1, + frozen_seq: 2, + count: 3, + frozen_count: 2, + tail_hash: Some(new_tail_hash.clone()), + }; + let unchanged_tail_wire = v1_wire(0, std::slice::from_ref(&new_tail)); + let token = begin_at_root( + root, + CollaborationSnapshotIngestBeginRequest { + local_session_id: session_id.to_string(), + epoch: 1, + expected_count: 3, + expected_frozen_seq: 2, + tail_hash: Some(new_tail_hash.clone()), + replace: false, + previous: Some(unchanged_cursor), + }, + ) + .expect("begin unchanged snapshot") + .token; + let segments = vec![unchanged_tail_wire]; + apply_page_at_root( + root, + CollaborationSnapshotIngestPageRequest { + token: token.clone(), + epoch: 1, + frozen_seq: 2, + count: 3, + tail_hash: Some(new_tail_hash), + cursor: CollaborationSnapshotWireCursor::Forward { + after_seq: 2, + through_seq: Some(2), + }, + next_cursor: None, + tail_included: true, + has_more: false, + returned_wire_bytes: page_bytes(&segments), + segments, + }, + ) + .expect("stage unchanged snapshot"); + commit_at_root_with_connection(root, &token, &destination).expect("commit unchanged snapshot"); + let replay_accounting_after: (i64, i64, i64, i64) = destination + .query_row( + "SELECT generation,revision,max_sequence,event_count + FROM collaboration_replay_state WHERE session_id=?1", + [session_id], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), + ) + .expect("replay accounting after no-op"); + assert_eq!(replay_accounting_after, replay_accounting_before); +} + +fn v2_wires(event: &SessionEvent) -> Vec { + let event_bytes = serde_json::to_vec(event).expect("serialize attachment event"); + let attachment_hash = sha256_hex(&event_bytes); + let attachment_id = sha256_hex(event.id.as_bytes()); + let chunk_bytes = ATTACHMENT_CHUNK_BYTES; + event_bytes + .chunks(chunk_bytes) + .enumerate() + .map(|(part_index, chunk)| { + let chunk_offset = part_index * chunk_bytes; + let final_part = chunk_offset + chunk.len() == event_bytes.len(); + let header = ReplayAttachmentV2FrameHeader { + kind: "event".to_string(), + attachment_id: attachment_id.clone(), + part_index: part_index as u64, + chunk_offset: chunk_offset as u64, + chunk_bytes: chunk.len() as u64, + final_part, + event_bytes: final_part.then_some(event_bytes.len() as u64), + attachment_hash: final_part.then_some(attachment_hash.clone()), + }; + let frame = encode_replay_attachment_v2_frame(&header, chunk) + .expect("encode shared attachment frame"); + CollaborationSnapshotWire { + seq: part_index as u64 + 1, + payload_gz: gzip_base64(&frame), + event_count: u64::from(final_part), + segment_hash: sha256_hex(&frame), + } + }) + .collect() +} + +#[test] +fn v2_ten_mib_event_stages_parts_and_restores_exact_payload() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "imported-session-v2-large"; + let payload = "x".repeat(10 * 1024 * 1024); + let mut event = test_event("large", EventSource::Assistant, "large result"); + event.result = serde_json::json!({ "content": payload.clone() }); + let wires = v2_wires(&event); + assert!(wires.iter().all(|wire| { + serde_json::to_vec(wire).expect("measure v2 wire").len() <= MAX_WIRE_BYTES + })); + let frozen_seq = wires.len() as u64; + let token = begin_replace(root, session_id, 7, 1, frozen_seq); + let mut after_seq = 0_u64; + for (index, chunk) in wires.chunks(12).enumerate() { + let has_more = (index + 1) * 12 < wires.len(); + let page = forward_page( + &token, + 7, + frozen_seq, + 1, + after_seq, + chunk.to_vec(), + has_more, + ); + after_seq = chunk.last().expect("wire chunk").seq; + apply_page_at_root(root, page).expect("stage v2 page"); + } + let result = + commit_at_root_with_connection(root, &token, &destination).expect("publish v2 snapshot"); + assert_eq!(result.event_count, 1); + let result_json: String = destination + .query_row( + "SELECT result_json FROM events WHERE session_id=?1", + [session_id], + |row| row.get(0), + ) + .expect("load large result"); + let restored: serde_json::Value = serde_json::from_str(&result_json).expect("parse result"); + let restored_payload = restored + .get("content") + .and_then(serde_json::Value::as_str) + .expect("content"); + assert_eq!(restored_payload.len(), 10 * 1024 * 1024); + assert_eq!( + sha256_hex(restored_payload.as_bytes()), + sha256_hex(payload.as_bytes()) + ); +} + +#[test] +fn hash_gap_and_abort_fail_closed() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "imported-session-fail-closed"; + let event = test_event("event", EventSource::User, "payload"); + + let bad_token = begin_replace(root, session_id, 1, 1, 1); + let mut bad_wire = v1_wire(1, std::slice::from_ref(&event)); + bad_wire.segment_hash = "0".repeat(64); + let error = apply_page_at_root( + root, + backward_page(&bad_token, 1, 1, 1, None, None, vec![bad_wire]), + ) + .expect_err("hash mismatch must fail"); + assert!(error.contains("hash mismatch")); + abort_at_root(root, &bad_token).expect("abort bad token"); + assert!(!staging_path(root, &bad_token) + .expect("bad stage path") + .exists()); + + let gap_token = begin_replace(root, session_id, 2, 1, 2); + let wire = v1_wire(2, std::slice::from_ref(&event)); + apply_page_at_root( + root, + backward_page(&gap_token, 2, 2, 1, None, None, vec![wire]), + ) + .expect("stage gapped page"); + let error = commit_at_root_with_connection(root, &gap_token, &destination) + .expect_err("missing physical row must fail"); + assert!(error.contains("incomplete")); + assert_eq!( + destination + .query_row("SELECT COUNT(*) FROM events", [], |row| row + .get::<_, i64>(0)) + .expect("count destination"), + 0 + ); + assert!(!staging_path(root, &gap_token) + .expect("gap stage path") + .exists()); +} + +#[test] +fn commit_failure_rolls_back_old_snapshot_and_cleans_staging() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let destination = destination(); + let session_id = "imported-session-rollback"; + destination + .execute( + "INSERT INTO events( + id,session_id,event_type,args_json,result_json,content,created_at,history_sequence + ) VALUES('old',?1,'user_message','{}','{}','old','2026-01-01',0)", + [session_id], + ) + .expect("seed old event"); + destination + .execute_batch( + "CREATE TRIGGER reject_new_snapshot BEFORE INSERT ON events + WHEN NEW.id LIKE '%~new' + BEGIN SELECT RAISE(ABORT,'forced commit failure'); END;", + ) + .expect("failure trigger"); + let token = begin_replace(root, session_id, 3, 1, 1); + let wire = v1_wire( + 1, + &[test_event("new", EventSource::Assistant, "replacement")], + ); + apply_page_at_root(root, backward_page(&token, 3, 1, 1, None, None, vec![wire])) + .expect("stage replacement"); + let error = commit_at_root_with_connection(root, &token, &destination) + .expect_err("forced publish failure"); + assert!(error.contains("forced commit failure")); + let ids = destination + .prepare("SELECT id FROM events WHERE session_id=?1") + .expect("prepare retained ids") + .query_map([session_id], |row| row.get::<_, String>(0)) + .expect("query retained ids") + .collect::>>() + .expect("collect retained ids"); + assert_eq!(ids, vec!["old"]); + assert!(!staging_path(root, &token).expect("stage path").exists()); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/wire.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/wire.rs new file mode 100644 index 000000000..c16e463b7 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/collaboration_snapshot_ingest/wire.rs @@ -0,0 +1,838 @@ +use super::staging::*; +use super::*; + +pub(super) struct DecodedWireFile { + path: PathBuf, + bytes: u64, + hash: String, + is_v2: bool, +} + +pub(super) struct TempFileGuard { + path: PathBuf, + armed: bool, +} + +impl TempFileGuard { + fn new(path: PathBuf) -> Self { + Self { path, armed: true } + } + + fn disarm(mut self) -> PathBuf { + self.armed = false; + self.path.clone() + } +} + +impl Drop for TempFileGuard { + fn drop(&mut self) { + if self.armed { + let _ = fs::remove_file(&self.path); + } + } +} + +impl Drop for DecodedWireFile { + fn drop(&mut self) { + let _ = fs::remove_file(&self.path); + } +} + +pub(super) fn sha256_hex(bytes: &[u8]) -> String { + format!("{:x}", Sha256::digest(bytes)) +} + +#[derive(Default)] +struct CountingWriter { + bytes: usize, +} + +impl Write for CountingWriter { + fn write(&mut self, buffer: &[u8]) -> std::io::Result { + self.bytes = self + .bytes + .checked_add(buffer.len()) + .ok_or_else(|| std::io::Error::other("wire byte count overflow"))?; + Ok(buffer.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +struct Sha256Writer(Sha256); + +impl Write for Sha256Writer { + fn write(&mut self, buffer: &[u8]) -> std::io::Result { + self.0.update(buffer); + Ok(buffer.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +fn serialized_wire_bytes(wire: &CollaborationSnapshotWire) -> Result { + let mut writer = CountingWriter::default(); + serde_json::to_writer(&mut writer, wire) + .map_err(|error| format!("measure physical wire {}: {error}", wire.seq))?; + Ok(writer.bytes) +} + +fn serialized_page_hash(segments: &[CollaborationSnapshotWire]) -> Result { + let mut writer = Sha256Writer(Sha256::new()); + serde_json::to_writer(&mut writer, segments) + .map_err(|error| format!("hash wire page: {error}"))?; + Ok(format!("{:x}", writer.0.finalize())) +} + +pub(super) fn decode_wire_to_file( + root: &Path, + token: &str, + wire: &CollaborationSnapshotWire, +) -> Result { + validate_hash("segmentHash", &wire.segment_hash)?; + let temp_path = root.join(format!("{token}-wire-{}-{}.tmp", wire.seq, Uuid::new_v4())); + let temp_guard = TempFileGuard::new(temp_path.clone()); + let file = File::create(&temp_path) + .map_err(|error| format!("create decoded wire staging file: {error}"))?; + let mut output = BufWriter::with_capacity(64 * 1024, file); + // Decode base64 directly into the gzip reader. Legacy V1 compatibility + // can admit one physical row larger than the normal IPC budget; avoiding + // a second compressed Vec keeps that one-time migration path bounded by + // the incoming wire string plus the 64 KiB streaming buffers. + let base64_reader = + base64::read::DecoderReader::new(wire.payload_gz.as_bytes(), &BASE64_STANDARD); + let mut decoder = GzDecoder::new(base64_reader); + let mut hasher = Sha256::new(); + let mut total = 0_u64; + let mut prefix = Vec::with_capacity(FRAME_MAGIC.len()); + let mut buffer = [0_u8; 64 * 1024]; + loop { + let read = decoder + .read(&mut buffer) + .map_err(|error| format!("gunzip segment {}: {error}", wire.seq))?; + if read == 0 { + break; + } + if prefix.len() < FRAME_MAGIC.len() { + let remaining = FRAME_MAGIC.len() - prefix.len(); + prefix.extend_from_slice(&buffer[..read.min(remaining)]); + } + total = total + .checked_add(read as u64) + .ok_or_else(|| "decoded segment byte count overflow".to_string())?; + // Before the magic is known, use the V1 ceiling. Once the prefix is + // complete, V2's much smaller physical-frame ceiling applies. + let is_v2 = prefix.len() == FRAME_MAGIC.len() && prefix == FRAME_MAGIC; + let limit = if is_v2 { + MAX_DECOMPRESSED_V2_BYTES + } else { + MAX_DECOMPRESSED_V1_BYTES + }; + if total > limit { + return Err(format!( + "decoded segment {} exceeds the {} byte limit", + wire.seq, limit + )); + } + hasher.update(&buffer[..read]); + output + .write_all(&buffer[..read]) + .map_err(|error| format!("stage decoded segment {}: {error}", wire.seq))?; + } + output + .flush() + .map_err(|error| format!("flush decoded segment {}: {error}", wire.seq))?; + let hash = format!("{:x}", hasher.finalize()); + if hash != wire.segment_hash.to_ascii_lowercase() { + return Err(format!("segment {} content hash mismatch", wire.seq)); + } + Ok(DecodedWireFile { + path: temp_guard.disarm(), + bytes: total, + hash, + is_v2: prefix == FRAME_MAGIC, + }) +} + +pub(super) fn stage_cached_event( + tx: &Transaction<'_>, + event: SessionEvent, + local_session_id: &str, + physical_seq: u64, + event_index: u64, + is_tail: bool, +) -> Result<(), String> { + let original_id = event.id.clone(); + let normalized = normalize_event(event, local_session_id)?; + let cached = session_event_to_cached_event(&normalized); + tx.execute( + "INSERT INTO staged_events( + normalized_id,original_id,physical_seq,event_index,is_tail,event_type, + function_name,thread_id,args_json,result_json,content,created_at,meta_json + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13)", + params![ + cached.id, + original_id, + i64::try_from(physical_seq).map_err(|_| "physical sequence is too large")?, + i64::try_from(event_index).map_err(|_| "event index is too large")?, + is_tail, + cached.event_type, + cached.function_name, + cached.thread_id, + cached.args_json, + cached.result_json, + cached.content, + cached.created_at, + cached.meta_json, + ], + ) + .map_err(|error| format!("stage event {original_id}: {error}"))?; + Ok(()) +} + +pub(super) struct StreamingEventArrayVisitor<'a> { + tx: &'a Transaction<'a>, + local_session_id: &'a str, + physical_seq: u64, + is_tail: bool, +} + +impl<'de> Visitor<'de> for StreamingEventArrayVisitor<'_> { + type Value = u64; + + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("a legacy replay SessionEvent array") + } + + fn visit_seq(self, mut sequence: A) -> Result + where + A: SeqAccess<'de>, + { + let mut count = 0_u64; + while let Some(event) = sequence.next_element::()? { + stage_cached_event( + self.tx, + event, + self.local_session_id, + self.physical_seq, + count, + self.is_tail, + ) + .map_err(serde::de::Error::custom)?; + count += 1; + } + Ok(count) + } +} + +pub(super) fn stage_v1_wire( + tx: &Transaction<'_>, + decoded: &DecodedWireFile, + wire: &CollaborationSnapshotWire, + local_session_id: &str, +) -> Result<(), String> { + let file = File::open(&decoded.path) + .map_err(|error| format!("open decoded segment {}: {error}", wire.seq))?; + let mut deserializer = serde_json::Deserializer::from_reader(BufReader::new(file)); + let count = serde::Deserializer::deserialize_seq( + &mut deserializer, + StreamingEventArrayVisitor { + tx, + local_session_id, + physical_seq: wire.seq, + is_tail: wire.seq == 0, + }, + ) + .map_err(|error| format!("decode legacy segment {}: {error}", wire.seq))?; + deserializer + .end() + .map_err(|error| format!("legacy segment {} has trailing data: {error}", wire.seq))?; + if count != wire.event_count { + return Err(format!( + "segment {} declared {} events but decoded {count}", + wire.seq, wire.event_count + )); + } + Ok(()) +} + +pub(super) fn stage_v2_wire( + tx: &Transaction<'_>, + decoded: &DecodedWireFile, + wire: &CollaborationSnapshotWire, +) -> Result<(), String> { + if wire.seq == 0 { + return Err("Replay Attachment V2 cannot be used for the mutable tail".to_string()); + } + let bytes = fs::read(&decoded.path) + .map_err(|error| format!("read attachment frame {}: {error}", wire.seq))?; + let decoded_frame = decode_replay_attachment_v2_frame(&bytes) + .map_err(|error| format!("attachment frame {} {error}", wire.seq))?; + let ReplayAttachmentV2FrameHeader { + attachment_id, + part_index, + chunk_offset, + final_part, + event_bytes, + attachment_hash, + .. + } = decoded_frame.header; + let chunk = decoded_frame.chunk; + if wire.event_count != u64::from(final_part) { + return Err(format!( + "attachment frame {} eventCount is inconsistent", + wire.seq + )); + } + tx.execute( + "INSERT INTO attachment_parts( + attachment_id,part_index,physical_seq,chunk_offset,chunk,final_part, + event_bytes,attachment_hash + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8)", + params![ + attachment_id, + i64::try_from(part_index).map_err(|_| "attachment part index too large")?, + i64::try_from(wire.seq).map_err(|_| "physical sequence too large")?, + i64::try_from(chunk_offset).map_err(|_| "attachment offset too large")?, + chunk, + final_part, + event_bytes + .map(i64::try_from) + .transpose() + .map_err(|_| "attachment event size too large")?, + attachment_hash, + ], + ) + .map_err(|error| format!("stage attachment frame {}: {error}", wire.seq))?; + Ok(()) +} + +pub(super) fn cursor_json(cursor: &CollaborationSnapshotWireCursor) -> Result { + serde_json::to_string(cursor).map_err(|error| format!("serialize wire cursor: {error}")) +} + +pub(super) fn validate_page_contract( + manifest: &StagingManifest, + request: &CollaborationSnapshotIngestPageRequest, +) -> Result<(), String> { + if request.epoch != manifest.epoch + || request.frozen_seq != manifest.expected_frozen_seq + || request.count != manifest.expected_count + || request.tail_hash != manifest.expected_tail_hash + { + return Err("wire page snapshot summary changed during ingestion".to_string()); + } + if request.has_more != request.next_cursor.is_some() { + return Err("wire page hasMore and nextCursor disagree".to_string()); + } + if request.segments.len() > MAX_PAGE_SEGMENTS { + return Err(format!( + "wire page has {} rows (limit {MAX_PAGE_SEGMENTS})", + request.segments.len() + )); + } + let mut seen_sequences = std::collections::HashSet::new(); + let mut legacy_v1_candidates = 0_usize; + let actual_wire_bytes = request.segments.iter().try_fold(0_usize, |total, wire| { + if !seen_sequences.insert(wire.seq) { + return Err(format!("wire page repeats physical seq {}", wire.seq)); + } + let bytes = serialized_wire_bytes(wire)?; + if bytes > MAX_WIRE_BYTES { + if bytes > LEGACY_V1_MAX_WIRE_BYTES { + return Err(format!( + "physical wire {} is {bytes} bytes (legacy V1 limit {LEGACY_V1_MAX_WIRE_BYTES})", + wire.seq + )); + } + legacy_v1_candidates += 1; + if legacy_v1_candidates > 1 { + return Err( + "wire page contains more than one oversized legacy V1 candidate".to_string(), + ); + } + } + total + .checked_add(bytes) + .ok_or_else(|| "wire page byte count overflow".to_string()) + })?; + let page_limit = if legacy_v1_candidates == 0 { + MAX_PAGE_BYTES + } else { + MAX_PAGE_BYTES + .checked_add(LEGACY_V1_MAX_WIRE_BYTES) + .ok_or_else(|| "legacy wire page limit overflow".to_string())? + }; + if actual_wire_bytes > page_limit || request.returned_wire_bytes != actual_wire_bytes as u64 { + return Err(format!( + "wire page byte count is {actual_wire_bytes}, reported {} (limit {page_limit})", + request.returned_wire_bytes + )); + } + if manifest.page_count > 0 { + let expected = manifest + .next_cursor_json + .as_deref() + .ok_or_else(|| "wire page chain was already complete".to_string())?; + if cursor_json(&request.cursor)? != expected { + return Err("wire page cursor does not continue the prior page".to_string()); + } + } + match &request.cursor { + CollaborationSnapshotWireCursor::Forward { + after_seq, + through_seq, + } => { + if through_seq.is_some_and(|through| through != manifest.expected_frozen_seq) { + return Err("forward wire cursor changed the frozen high-water mark".to_string()); + } + for wire in request.segments.iter().filter(|wire| wire.seq > 0) { + if wire.seq <= *after_seq || through_seq.is_some_and(|through| wire.seq > through) { + return Err(format!( + "forward page row {} is outside its cursor", + wire.seq + )); + } + } + } + CollaborationSnapshotWireCursor::Backward { before_seq } => { + for wire in request.segments.iter().filter(|wire| wire.seq > 0) { + if before_seq.is_some_and(|before| wire.seq >= before) { + return Err(format!( + "backward page row {} is outside its cursor", + wire.seq + )); + } + } + } + } + if request.tail_included != request.segments.iter().any(|wire| wire.seq == 0) { + return Err("wire page tailIncluded does not match seq 0 presence".to_string()); + } + if request.tail_included && manifest.expected_tail_hash.is_none() { + return Err("wire page returned an unpinned mutable tail".to_string()); + } + if let Some(tail) = request.segments.iter().find(|wire| wire.seq == 0) { + if Some(tail.segment_hash.as_str()) != manifest.expected_tail_hash.as_deref() { + return Err("wire page mutable tail hash mismatch".to_string()); + } + } + if let Some(next) = request.next_cursor.as_ref() { + if std::mem::discriminant(next) != std::mem::discriminant(&request.cursor) { + return Err("wire page continuation changes direction".to_string()); + } + } + let mut frozen_sequences = request + .segments + .iter() + .filter_map(|wire| (wire.seq > 0).then_some(wire.seq)) + .collect::>(); + frozen_sequences.sort_unstable(); + match (&request.cursor, request.next_cursor.as_ref()) { + ( + CollaborationSnapshotWireCursor::Forward { .. }, + Some(CollaborationSnapshotWireCursor::Forward { + after_seq, + through_seq, + }), + ) => { + if frozen_sequences.last().copied() != Some(*after_seq) + || *through_seq != Some(manifest.expected_frozen_seq) + { + return Err( + "forward page continuation does not advance to its last row".to_string() + ); + } + } + ( + CollaborationSnapshotWireCursor::Backward { .. }, + Some(CollaborationSnapshotWireCursor::Backward { before_seq }), + ) => { + if frozen_sequences.first().copied() != *before_seq { + return Err( + "backward page continuation does not continue before its first row".to_string(), + ); + } + } + (_, None) => {} + _ => return Err("wire page continuation changes direction".to_string()), + } + Ok(()) +} + +pub(super) fn page_progress( + conn: &Connection, + complete: bool, +) -> Result { + let (rows, events): (i64, i64) = conn + .query_row( + "SELECT COUNT(*),COALESCE(SUM(event_count),0) FROM staged_wires", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .map_err(|error| format!("read snapshot ingest progress: {error}"))?; + Ok(CollaborationSnapshotIngestProgress { + accepted_physical_rows: rows.max(0) as u64, + accepted_logical_events: events.max(0) as u64, + complete, + }) +} + +pub(super) fn apply_page_at_root( + root: &Path, + request: CollaborationSnapshotIngestPageRequest, +) -> Result { + let path = staging_path(root, &request.token)?; + let mut conn = open_staging(&path)?; + let manifest = load_manifest(&conn)?; + if manifest.token != request.token { + return Err("snapshot ingest token does not match its manifest".to_string()); + } + validate_page_contract(&manifest, &request)?; + + let page_cursor_json = cursor_json(&request.cursor)?; + let next_cursor_json = request.next_cursor.as_ref().map(cursor_json).transpose()?; + let page_hash = serialized_page_hash(&request.segments)?; + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS staged_pages( + cursor_json TEXT PRIMARY KEY, + page_hash TEXT NOT NULL, + next_cursor_json TEXT, + complete INTEGER NOT NULL + );", + ) + .map_err(|error| format!("initialize snapshot page receipts: {error}"))?; + let prior_page: Option<(String, Option, bool)> = conn + .query_row( + "SELECT page_hash,next_cursor_json,complete FROM staged_pages WHERE cursor_json=?1", + [&page_cursor_json], + |row| Ok((row.get(0)?, row.get(1)?, row.get::<_, i64>(2)? != 0)), + ) + .optional() + .map_err(|error| format!("read snapshot page receipt: {error}"))?; + if let Some((prior_hash, prior_next, prior_complete)) = prior_page { + if prior_hash != page_hash + || prior_next != next_cursor_json + || prior_complete == request.has_more + { + return Err("wire page retry differs from the accepted page".to_string()); + } + return page_progress(&conn, !request.has_more); + } + + let tx = conn + .transaction() + .map_err(|error| format!("begin snapshot page transaction: {error}"))?; + for wire in &request.segments { + let prior: Option<(String, i64)> = tx + .query_row( + "SELECT segment_hash,event_count FROM staged_wires WHERE seq=?1", + [i64::try_from(wire.seq).map_err(|_| "physical sequence is too large")?], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .optional() + .map_err(|error| format!("read staged physical row {}: {error}", wire.seq))?; + if let Some((hash, count)) = prior { + if hash == wire.segment_hash && count == wire.event_count as i64 { + continue; + } + return Err(format!( + "physical row {} changed during ingestion", + wire.seq + )); + } + let decoded = decode_wire_to_file(root, &request.token, wire)?; + let physical_wire_bytes = serialized_wire_bytes(wire)?; + if physical_wire_bytes > MAX_WIRE_BYTES && decoded.is_v2 { + return Err(format!( + "Attachment V2 physical wire {} exceeds the {MAX_WIRE_BYTES} byte limit", + wire.seq + )); + } + if decoded.hash != wire.segment_hash.to_ascii_lowercase() || decoded.bytes == 0 { + return Err(format!( + "physical row {} decoded to invalid content", + wire.seq + )); + } + if decoded.is_v2 { + stage_v2_wire(&tx, &decoded, wire)?; + } else { + stage_v1_wire(&tx, &decoded, wire, &manifest.local_session_id)?; + } + tx.execute( + "INSERT INTO staged_wires(seq,segment_hash,event_count,is_tail) + VALUES(?1,?2,?3,?4)", + params![ + i64::try_from(wire.seq).map_err(|_| "physical sequence is too large")?, + wire.segment_hash.to_ascii_lowercase(), + i64::try_from(wire.event_count).map_err(|_| "event count is too large")?, + wire.seq == 0, + ], + ) + .map_err(|error| format!("record physical row {}: {error}", wire.seq))?; + } + tx.execute( + "INSERT INTO staged_pages(cursor_json,page_hash,next_cursor_json,complete) + VALUES(?1,?2,?3,?4)", + params![ + page_cursor_json, + page_hash, + next_cursor_json, + !request.has_more, + ], + ) + .map_err(|error| format!("record accepted wire page: {error}"))?; + tx.execute( + "UPDATE manifest SET page_count=page_count+1,next_cursor_json=?1, + page_chain_complete=?2 WHERE singleton=1", + params![next_cursor_json, !request.has_more], + ) + .map_err(|error| format!("advance snapshot wire cursor: {error}"))?; + tx.commit() + .map_err(|error| format!("commit snapshot wire page: {error}"))?; + page_progress(&conn, !request.has_more) +} + +pub(super) async fn collaboration_snapshot_ingest_apply_wire_page_impl( + request: CollaborationSnapshotIngestPageRequest, +) -> Result { + let root = staging_root()?; + tokio::task::spawn_blocking(move || apply_page_at_root(&root, request)) + .await + .map_err(|error| error.to_string())? +} + +pub(super) fn finalize_one_attachment( + tx: &Transaction<'_>, + root: &Path, + token: &str, + local_session_id: &str, + attachment_id: &str, +) -> Result<(), String> { + let event_path = root.join(format!("{token}-event-{}.tmp", Uuid::new_v4())); + let _event_guard = TempFileGuard::new(event_path.clone()); + let event_file = File::create(&event_path) + .map_err(|error| format!("create attachment event staging file: {error}"))?; + let mut output = BufWriter::with_capacity(64 * 1024, event_file); + let mut hasher = Sha256::new(); + let mut expected_part = 0_i64; + let mut expected_offset = 0_i64; + let mut final_metadata: Option<(i64, String, i64)> = None; + { + let mut statement = tx + .prepare( + "SELECT part_index,physical_seq,chunk_offset,chunk,final_part, + event_bytes,attachment_hash + FROM attachment_parts WHERE attachment_id=?1 ORDER BY part_index ASC", + ) + .map_err(|error| format!("prepare attachment {attachment_id}: {error}"))?; + let mut rows = statement + .query([attachment_id]) + .map_err(|error| format!("query attachment {attachment_id}: {error}"))?; + while let Some(row) = rows + .next() + .map_err(|error| format!("read attachment {attachment_id}: {error}"))? + { + let part_index: i64 = row.get(0).map_err(|error| error.to_string())?; + let physical_seq: i64 = row.get(1).map_err(|error| error.to_string())?; + let offset: i64 = row.get(2).map_err(|error| error.to_string())?; + let chunk: Vec = row.get(3).map_err(|error| error.to_string())?; + let final_part = row.get::<_, i64>(4).map_err(|error| error.to_string())? != 0; + let event_bytes: Option = row.get(5).map_err(|error| error.to_string())?; + let attachment_hash: Option = row.get(6).map_err(|error| error.to_string())?; + if part_index != expected_part || offset != expected_offset { + return Err(format!( + "attachment {attachment_id} has missing or reordered parts" + )); + } + if final_metadata.is_some() { + return Err(format!( + "attachment {attachment_id} has data after its final part" + )); + } + output + .write_all(&chunk) + .map_err(|error| format!("write attachment {attachment_id}: {error}"))?; + hasher.update(&chunk); + expected_part += 1; + expected_offset = expected_offset + .checked_add(chunk.len() as i64) + .ok_or_else(|| "attachment byte count overflow".to_string())?; + if final_part { + let size = event_bytes + .ok_or_else(|| format!("attachment {attachment_id} final size is missing"))?; + let hash = attachment_hash + .ok_or_else(|| format!("attachment {attachment_id} final hash is missing"))?; + final_metadata = Some((size, hash, physical_seq)); + } else if event_bytes.is_some() || attachment_hash.is_some() { + return Err(format!( + "attachment {attachment_id} has premature final metadata" + )); + } + } + } + output + .flush() + .map_err(|error| format!("flush attachment {attachment_id}: {error}"))?; + drop(output); + let result = (|| { + let (event_bytes, expected_hash, physical_seq) = final_metadata + .ok_or_else(|| format!("attachment {attachment_id} is missing its final part"))?; + if event_bytes != expected_offset { + return Err(format!("attachment {attachment_id} total size mismatch")); + } + let actual_hash = format!("{:x}", hasher.finalize()); + if actual_hash != expected_hash.to_ascii_lowercase() { + return Err(format!( + "attachment {attachment_id} complete event hash mismatch" + )); + } + let file = File::open(&event_path) + .map_err(|error| format!("open assembled attachment {attachment_id}: {error}"))?; + let mut deserializer = serde_json::Deserializer::from_reader(BufReader::new(file)); + let event = SessionEvent::deserialize(&mut deserializer) + .map_err(|error| format!("parse attachment {attachment_id} event: {error}"))?; + deserializer.end().map_err(|error| { + format!("attachment {attachment_id} event has trailing data: {error}") + })?; + stage_cached_event(tx, event, local_session_id, physical_seq as u64, 0, false)?; + tx.execute( + "DELETE FROM attachment_parts WHERE attachment_id=?1", + [attachment_id], + ) + .map_err(|error| format!("clear attachment {attachment_id} parts: {error}"))?; + Ok(()) + })(); + result +} + +pub(super) fn finalize_attachments( + conn: &mut Connection, + root: &Path, + manifest: &StagingManifest, +) -> Result<(), String> { + let tx = conn + .transaction() + .map_err(|error| format!("begin attachment finalization: {error}"))?; + loop { + let attachment_id: Option = tx + .query_row( + "SELECT attachment_id FROM attachment_parts ORDER BY physical_seq ASC LIMIT 1", + [], + |row| row.get(0), + ) + .optional() + .map_err(|error| format!("find pending attachment: {error}"))?; + let Some(attachment_id) = attachment_id else { + break; + }; + finalize_one_attachment( + &tx, + root, + &manifest.token, + &manifest.local_session_id, + &attachment_id, + )?; + } + tx.commit() + .map_err(|error| format!("commit attachment finalization: {error}")) +} + +pub(super) fn validate_complete_staging( + conn: &Connection, + manifest: &StagingManifest, +) -> Result<(u64, u64), String> { + if manifest.page_count == 0 || !manifest.page_chain_complete { + return Err("snapshot wire page chain is incomplete".to_string()); + } + let base_frozen_seq = if manifest.replace { + 0 + } else { + manifest + .previous + .as_ref() + .map_or(0, |cursor| cursor.frozen_seq) + }; + let expected_physical = manifest + .expected_frozen_seq + .checked_sub(base_frozen_seq) + .ok_or_else(|| "snapshot frozen sequence moved backwards".to_string())?; + let (physical_count, min_seq, max_seq): (i64, Option, Option) = conn + .query_row( + "SELECT COUNT(*),MIN(seq),MAX(seq) FROM staged_wires WHERE seq>0", + [], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .map_err(|error| format!("validate frozen physical rows: {error}"))?; + if physical_count.max(0) as u64 != expected_physical + || (expected_physical > 0 + && (min_seq != Some((base_frozen_seq + 1) as i64) + || max_seq != Some(manifest.expected_frozen_seq as i64))) + { + return Err(format!( + "snapshot frozen rows are incomplete: expected {}..={}, got count={} min={min_seq:?} max={max_seq:?}", + base_frozen_seq + 1, + manifest.expected_frozen_seq, + physical_count + )); + } + let has_tail: bool = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM staged_wires WHERE seq=0)", + [], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("validate snapshot tail: {error}"))? + != 0; + if has_tail != manifest.expected_tail_hash.is_some() { + return Err("snapshot mutable tail is incomplete".to_string()); + } + let staged_logical: i64 = conn + .query_row( + "SELECT COALESCE(SUM(event_count),0) FROM staged_wires", + [], + |row| row.get(0), + ) + .map_err(|error| format!("count staged logical events: {error}"))?; + let staged_events: i64 = conn + .query_row("SELECT COUNT(*) FROM staged_events", [], |row| row.get(0)) + .map_err(|error| format!("count staged event rows: {error}"))?; + if staged_events != staged_logical { + return Err(format!( + "snapshot decoded event count {staged_events} does not match physical rows {staged_logical}" + )); + } + let base_frozen_count = if manifest.replace { + 0 + } else { + manifest + .previous + .as_ref() + .map_or(0, |cursor| cursor.frozen_count) + }; + let final_count = base_frozen_count + .checked_add(staged_logical.max(0) as u64) + .ok_or_else(|| "snapshot logical event count overflow".to_string())?; + if final_count != manifest.expected_count { + return Err(format!( + "snapshot logical count is {final_count}, expected {}", + manifest.expected_count + )); + } + let staged_frozen: i64 = conn + .query_row( + "SELECT COUNT(*) FROM staged_events WHERE is_tail=0", + [], + |row| row.get(0), + ) + .map_err(|error| format!("count staged frozen events: {error}"))?; + let final_frozen_count = base_frozen_count + .checked_add(staged_frozen.max(0) as u64) + .ok_or_else(|| "snapshot frozen event count overflow".to_string())?; + Ok((final_count, final_frozen_count)) +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/cloud.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/cloud.rs new file mode 100644 index 000000000..abd9b28e8 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/cloud.rs @@ -0,0 +1,1016 @@ +use super::*; + +static CLOUD_SPOOL_PRUNE_TASK_RUNNING: AtomicBool = AtomicBool::new(false); + +pub(super) fn cloud_spools() -> &'static Mutex> { + static SPOOLS: OnceLock>> = OnceLock::new(); + SPOOLS.get_or_init(|| { + #[cfg(not(test))] + { + if let Err(error) = cleanup_orphan_cloud_spool_files() { + log::warn!("[external-replay] {error}"); + } + } + Mutex::new(HashMap::new()) + }) +} + +pub(super) fn ensure_private_cloud_spool_root(root: &std::path::Path) -> Result<(), String> { + fs::create_dir_all(root) + .map_err(|err| format!("create replay cloud spool directory: {err}"))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + + fs::set_permissions(root, fs::Permissions::from_mode(0o700)) + .map_err(|err| format!("protect replay cloud spool directory: {err}"))?; + } + Ok(()) +} + +fn cloud_spool_root() -> Result { + let root = app_paths::orgii_root() + .join(".runtime") + .join("replay-cloud-spools"); + ensure_private_cloud_spool_root(&root)?; + Ok(root) +} + +pub(super) fn cloud_spool_paths(root: &std::path::Path, token: &str) -> (PathBuf, PathBuf) { + let final_path = root.join(format!("orgii-replay-cloud-{token}.sqlite")); + let partial_path = root.join(format!("orgii-replay-cloud-{token}.sqlite-part")); + (final_path, partial_path) +} + +pub(super) fn create_private_cloud_spool_file(path: &std::path::Path) -> Result<(), String> { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + drop( + options + .open(path) + .map_err(|err| format!("create private replay cloud spool: {err}"))?, + ); + app_paths::set_sensitive_file_permissions(path) + .map_err(|err| format!("protect replay cloud spool: {err}")) +} + +#[cfg(not(test))] +fn cleanup_orphan_cloud_spool_files() -> Result<(), String> { + let root = cloud_spool_root()?; + let entries = fs::read_dir(&root) + .map_err(|err| format!("enumerate replay cloud spool directory: {err}"))?; + for entry in entries { + let entry = entry.map_err(|err| format!("read replay cloud spool entry: {err}"))?; + let path = entry.path(); + let is_spool = path + .file_name() + .and_then(|name| name.to_str()) + .is_some_and(|name| { + name.starts_with("orgii-replay-cloud-") + && (name.ends_with(".sqlite") || name.ends_with(".sqlite-part")) + }); + if !is_spool || !path.is_file() { + continue; + } + if let Err(error) = fs::remove_file(&path) { + log::warn!( + "[external-replay] cannot remove orphan cloud spool {}: {error}", + path.display() + ); + } + } + Ok(()) +} + +fn schedule_cloud_spool_prune() { + if CLOUD_SPOOL_PRUNE_TASK_RUNNING + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + return; + } + tauri::async_runtime::spawn(async { + loop { + tokio::time::sleep(CLOUD_SPOOL_TTL).await; + cleanup_cloud_spools(); + let is_empty = cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .is_empty(); + if !is_empty { + continue; + } + + CLOUD_SPOOL_PRUNE_TASK_RUNNING.store(false, Ordering::Release); + let needs_restart = !cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .is_empty(); + if !needs_restart { + break; + } + if CLOUD_SPOOL_PRUNE_TASK_RUNNING + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + break; + } + } + }); +} + +#[derive(Default)] +pub(super) struct BoundedCloudGzipBuffer { + bytes: Vec, +} + +impl Write for BoundedCloudGzipBuffer { + fn write(&mut self, input: &[u8]) -> std::io::Result { + if self.bytes.len().saturating_add(input.len()) > CLOUD_SEGMENT_GZIP_MAX_BYTES { + return Err(std::io::Error::other( + "compressed replay event exceeds the cloud segment wire budget; the current SessionEvent[] RPC has no attachment/continuation type", + )); + } + self.bytes.extend_from_slice(input); + Ok(input.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +pub(super) struct CloudSegmentEncoder { + gzip: GzEncoder, + digest: Sha256, +} + +impl CloudSegmentEncoder { + fn new() -> Self { + Self { + gzip: GzEncoder::new(BoundedCloudGzipBuffer::default(), Compression::default()), + digest: Sha256::new(), + } + } + + fn finish(self, event_count: u64) -> Result { + let segment_hash = format!("{:x}", self.digest.finalize()); + let compressed = self.gzip.finish().map_err(cloud_segment_write_error)?.bytes; + let payload_gz = BASE64_STANDARD.encode(compressed); + #[derive(Serialize)] + #[serde(rename_all = "camelCase")] + struct BudgetWire<'a> { + seq: u64, + payload_gz: &'a str, + event_count: u64, + segment_hash: &'a str, + } + let wire_bytes = serde_json::to_vec(&BudgetWire { + // The spool is reused across orgs/cursors, so reserve the largest + // possible sequence representation before publication. + seq: u64::MAX, + payload_gz: &payload_gz, + event_count, + segment_hash: &segment_hash, + }) + .map_err(|err| format!("measure replay cloud wire segment: {err}"))? + .len(); + if wire_bytes > CLOUD_SEGMENT_WIRE_MAX_BYTES { + return Err(format!( + "Cloud replay cannot represent this event without loss: encoded segment is {wire_bytes} bytes (limit {CLOUD_SEGMENT_WIRE_MAX_BYTES}). The current SessionEvent[] RPC has no attachment/continuation type; upload requires the versioned replay-attachment protocol." + )); + } + Ok(ExternalReplayCloudSegment { + payload_gz, + event_count, + segment_hash, + wire_bytes: wire_bytes as u64, + }) + } +} + +impl Write for CloudSegmentEncoder { + fn write(&mut self, input: &[u8]) -> std::io::Result { + self.gzip.write_all(input)?; + self.digest.update(input); + Ok(input.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.gzip.flush() + } +} + +pub(super) struct DigestingWriter<'a, W: Write> { + inner: &'a mut W, + digest: &'a mut Sha256, +} + +impl Write for DigestingWriter<'_, W> { + fn write(&mut self, input: &[u8]) -> std::io::Result { + self.inner.write_all(input)?; + self.digest.update(input); + Ok(input.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.inner.flush() + } +} + +pub(super) fn cloud_segment_write_error(error: std::io::Error) -> String { + if error + .to_string() + .contains("exceeds the cloud segment wire budget") + { + return format!( + "Cloud replay cannot represent this event without loss: compressed payload exceeds the {CLOUD_SEGMENT_WIRE_MAX_BYTES}-byte wire limit. The current SessionEvent[] RPC has no attachment/continuation type; upload requires the versioned replay-attachment protocol." + ); + } + format!("encode replay cloud segment: {error}") +} + +pub(super) fn encode_cloud_frozen_event( + event: &SessionEvent, + read_payload: &mut dyn FnMut(&PayloadRef, u64) -> Result, +) -> Result<(ExternalReplayCloudSegment, String), String> { + let mut encoder = CloudSegmentEncoder::new(); + encoder.write_all(b"[").map_err(cloud_segment_write_error)?; + let mut event_digest = Sha256::new(); + { + let mut event_writer = DigestingWriter { + inner: &mut encoder, + digest: &mut event_digest, + }; + write_hydrated_event_json(&mut event_writer, event, read_payload)?; + } + encoder.write_all(b"]").map_err(cloud_segment_write_error)?; + Ok((encoder.finish(1)?, format!("{:x}", event_digest.finalize()))) +} + +pub(super) fn encode_cloud_attachment_frame( + header: &CloudAttachmentFrameHeader, + chunk: &[u8], + event_count: u64, +) -> Result { + let frame = encode_replay_attachment_v2_frame(header, chunk)?; + let mut encoder = CloudSegmentEncoder::new(); + encoder + .write_all(&frame) + .map_err(cloud_segment_write_error)?; + encoder.finish(event_count) +} + +pub(super) struct StreamingCloudAttachmentEncoder<'a> { + attachment_id: String, + chunk: Vec, + part_index: u64, + total_bytes: u64, + digest: Sha256, + emit: &'a mut dyn FnMut(ExternalReplayCloudSegment) -> Result<(), String>, +} + +impl<'a> StreamingCloudAttachmentEncoder<'a> { + fn new( + event_id: &str, + emit: &'a mut dyn FnMut(ExternalReplayCloudSegment) -> Result<(), String>, + ) -> Self { + Self { + attachment_id: sha256_hex(event_id.as_bytes()), + chunk: Vec::with_capacity(CLOUD_ATTACHMENT_CHUNK_BYTES), + part_index: 0, + total_bytes: 0, + digest: Sha256::new(), + emit, + } + } + + fn emit_chunk(&mut self, final_part: bool) -> Result<(), String> { + if self.chunk.is_empty() { + return Err("Replay attachment V2 cannot emit an empty part".to_string()); + } + let chunk_bytes = self.chunk.len() as u64; + let chunk_offset = self.total_bytes.saturating_sub(chunk_bytes); + let attachment_hash = final_part.then(|| format!("{:x}", self.digest.clone().finalize())); + let header = CloudAttachmentFrameHeader { + kind: "event".to_string(), + attachment_id: self.attachment_id.clone(), + part_index: self.part_index, + chunk_offset, + chunk_bytes, + final_part, + event_bytes: final_part.then_some(self.total_bytes), + attachment_hash, + }; + let segment = + encode_cloud_attachment_frame(&header, &self.chunk, if final_part { 1 } else { 0 })?; + (self.emit)(segment)?; + self.part_index = self.part_index.saturating_add(1); + self.chunk.clear(); + Ok(()) + } + + fn finish(mut self) -> Result { + self.emit_chunk(true)?; + Ok(format!("{:x}", self.digest.finalize())) + } +} + +impl Write for StreamingCloudAttachmentEncoder<'_> { + fn write(&mut self, mut input: &[u8]) -> std::io::Result { + let input_len = input.len(); + while !input.is_empty() { + if self.chunk.len() == CLOUD_ATTACHMENT_CHUNK_BYTES { + self.emit_chunk(false).map_err(std::io::Error::other)?; + } + let available = CLOUD_ATTACHMENT_CHUNK_BYTES.saturating_sub(self.chunk.len()); + let take = available.min(input.len()); + let bytes = &input[..take]; + self.chunk.extend_from_slice(bytes); + self.digest.update(bytes); + self.total_bytes = self.total_bytes.saturating_add(take as u64); + input = &input[take..]; + } + Ok(input_len) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +pub(super) fn is_cloud_segment_budget_error(error: &str) -> bool { + error.contains("cloud segment wire budget") + || error.contains("cannot represent this event without loss") + || error.contains("compressed payload exceeds") +} + +pub(super) fn encode_cloud_event_segments( + event: &SessionEvent, + read_payload: &mut dyn FnMut(&PayloadRef, u64) -> Result, + emit: &mut dyn FnMut(ExternalReplayCloudSegment) -> Result<(), String>, +) -> Result { + match encode_cloud_frozen_event(event, read_payload) { + Ok((segment, event_hash)) => { + emit(segment)?; + Ok(event_hash) + } + Err(error) if is_cloud_segment_budget_error(&error) => { + let mut encoder = StreamingCloudAttachmentEncoder::new(&event.id, emit); + write_hydrated_event_json(&mut encoder, event, read_payload)?; + encoder.finish() + } + Err(error) => Err(error), + } +} + +pub(super) fn prepare_cloud_spool( + source_id: &str, + session_id: &str, +) -> Result { + cleanup_cloud_spools(); + let token = uuid::Uuid::new_v4().to_string(); + let root = cloud_spool_root()?; + let (final_path, partial_path) = cloud_spool_paths(&root, &token); + create_private_cloud_spool_file(&partial_path)?; + let prepared = (|| { + let mut spool = rusqlite::Connection::open(&partial_path) + .map_err(|err| format!("create replay cloud spool: {err}"))?; + spool + .execute_batch( + "PRAGMA journal_mode=OFF; + PRAGMA synchronous=OFF; + CREATE TABLE events ( + event_index INTEGER PRIMARY KEY, + event_hash TEXT NOT NULL, + frozen_chain_hash TEXT NOT NULL + ); + CREATE TABLE frozen_segments ( + segment_index INTEGER PRIMARY KEY, + event_index INTEGER NOT NULL, + payload_gz TEXT NOT NULL, + event_count INTEGER NOT NULL, + segment_hash TEXT NOT NULL, + wire_bytes INTEGER NOT NULL + ); + CREATE INDEX frozen_segments_event_idx + ON frozen_segments(event_index, segment_index); + CREATE TABLE tail_segment ( + singleton INTEGER PRIMARY KEY CHECK(singleton = 1), + payload_gz TEXT NOT NULL, + event_count INTEGER NOT NULL, + segment_hash TEXT NOT NULL, + wire_bytes INTEGER NOT NULL + );", + ) + .map_err(|err| format!("initialize replay cloud spool: {err}"))?; + let tx = spool + .transaction() + .map_err(|err| format!("start replay cloud spool transaction: {err}"))?; + let mut total_count = 0_u64; + let mut frozen_event_count = 0_u64; + let mut frozen_segment_count = 0_u64; + let mut frozen_chain = Sha256::new(); + let generation = + stream_replay_cloud_events(source_id, session_id, |event, read_payload| { + let event_index = total_count; + let mut emit = |segment: ExternalReplayCloudSegment| { + tx.execute( + "INSERT INTO frozen_segments ( + segment_index, event_index, payload_gz, event_count, + segment_hash, wire_bytes + ) VALUES (?1, ?2, ?3, ?4, ?5, ?6)", + rusqlite::params![ + frozen_segment_count as i64, + event_index as i64, + segment.payload_gz, + segment.event_count as i64, + segment.segment_hash, + segment.wire_bytes as i64, + ], + ) + .map_err(|err| format!("write replay cloud frozen segment: {err}"))?; + frozen_segment_count = frozen_segment_count.saturating_add(1); + Ok(()) + }; + let event_hash = encode_cloud_event_segments(event, read_payload, &mut emit)?; + if frozen_event_count > 0 { + frozen_chain.update(b"\n"); + } + frozen_chain.update(event_hash.as_bytes()); + let chain_hash = format!("{:x}", frozen_chain.clone().finalize()); + tx.execute( + "INSERT INTO events (event_index, event_hash, frozen_chain_hash) + VALUES (?1, ?2, ?3)", + rusqlite::params![event_index as i64, event_hash, chain_hash], + ) + .map_err(|err| format!("write replay cloud event hash: {err}"))?; + // External source events are immutable within one generation. + // Publishing all of them as a frozen prefix lets oversized + // V2 events use continuation rows; an in-place source change + // changes the logical prefix hash and forces an epoch rewrite. + frozen_event_count = frozen_event_count.saturating_add(1); + total_count = total_count.saturating_add(1); + Ok(()) + })?; + tx.commit() + .map_err(|err| format!("commit replay cloud spool: {err}"))?; + drop(spool); + fs::rename(&partial_path, &final_path) + .map_err(|err| format!("publish replay cloud spool: {err}"))?; + app_paths::set_sensitive_file_permissions(&final_path) + .map_err(|err| format!("protect published replay cloud spool: {err}"))?; + Ok::<_, String>(ExternalReplayCloudManifest { + token: token.clone(), + generation, + total_count, + frozen_event_count, + tail_event_count: 0, + frozen_chain_hash: format!("{:x}", frozen_chain.finalize()), + tail_hash: None, + }) + })(); + let manifest = match prepared { + Ok(manifest) => manifest, + Err(error) => { + let _ = fs::remove_file(&partial_path); + let _ = fs::remove_file(&final_path); + return Err(error); + } + }; + cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .insert( + token, + CloudSpoolEntry { + path: final_path, + manifest: manifest.clone(), + last_used: Instant::now(), + lease_count: 1, + owner_released: false, + }, + ); + schedule_cloud_spool_prune(); + Ok(manifest) +} + +pub(super) fn read_cloud_spool_batch( + token: &str, + start_event_index: u64, + end_event_index: u64, + start_segment_index: Option, + max_bytes: Option, +) -> Result { + let entry = acquire_cloud_spool_read(token)?; + let end = end_event_index.min(entry.manifest.total_count); + if start_event_index > end { + return Err("Replay cloud batch range is reversed".to_string()); + } + if start_event_index == end { + return Ok(ExternalReplayCloudBatch { + segments: Vec::new(), + start_event_index, + next_event_index: start_event_index, + start_segment_index: start_segment_index.unwrap_or(0), + next_segment_index: start_segment_index.unwrap_or(0), + eof: true, + serialized_bytes: 0, + }); + } + let byte_limit = max_bytes + .unwrap_or(STREAM_BATCH_MAX_BYTES) + .clamp(1, CLOUD_SEGMENT_WIRE_MAX_BYTES); + let conn = rusqlite::Connection::open_with_flags( + &entry.path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY, + ) + .map_err(|err| format!("open replay cloud spool: {err}"))?; + let reading_tail = start_event_index == entry.manifest.frozen_event_count + && entry.manifest.tail_event_count > 0; + let resolved_segment_index = if let Some(index) = start_segment_index { + index + } else if reading_tail { + 0 + } else { + conn.query_row( + "SELECT MIN(segment_index) FROM frozen_segments + WHERE event_index >= ?1 AND event_index < ?2", + rusqlite::params![ + start_event_index.min(i64::MAX as u64) as i64, + end.min(i64::MAX as u64) as i64, + ], + |row| row.get::<_, Option>(0), + ) + .map_err(|err| format!("locate replay cloud physical cursor: {err}"))? + .ok_or_else(|| "Replay cloud spool has no segment for the requested event".to_string())? + .max(0) as u64 + }; + let (query, start, limit) = if reading_tail { + ( + "SELECT ?1, payload_gz, event_count, segment_hash, wire_bytes + FROM tail_segment WHERE singleton = 1", + resolved_segment_index.min(i64::MAX as u64) as i64, + 1_i64, + ) + } else { + ( + "SELECT segment_index, payload_gz, event_count, segment_hash, wire_bytes + FROM frozen_segments + WHERE segment_index >= ?1 AND event_index >= ?2 AND event_index < ?3 + ORDER BY segment_index ASC LIMIT ?4", + resolved_segment_index.min(i64::MAX as u64) as i64, + STREAM_BATCH_MAX_EVENTS as i64, + ) + }; + let mut stmt = conn + .prepare(query) + .map_err(|err| format!("prepare replay cloud batch: {err}"))?; + let mut rows = if reading_tail { + stmt.query([start]) + } else { + stmt.query(rusqlite::params![ + start, + start_event_index.min(i64::MAX as u64) as i64, + end.min(i64::MAX as u64) as i64, + limit, + ]) + } + .map_err(|err| format!("query replay cloud batch: {err}"))?; + let mut segments = Vec::new(); + let mut serialized_bytes = 0_usize; + let mut consumed_events = 0_u64; + let mut next_segment_index = resolved_segment_index; + while let Some(row) = rows + .next() + .map_err(|err| format!("read replay cloud batch row: {err}"))? + { + let row_bytes = row.get::<_, i64>(4).unwrap_or_default().max(0) as usize; + if !segments.is_empty() && serialized_bytes.saturating_add(row_bytes) > byte_limit { + break; + } + if row_bytes > CLOUD_SEGMENT_WIRE_MAX_BYTES { + return Err("Replay cloud spool contains an over-budget wire segment".to_string()); + } + let physical_index = row.get::<_, i64>(0).unwrap_or_default().max(0) as u64; + let event_count = row.get::<_, i64>(2).unwrap_or_default().max(0) as u64; + segments.push(ExternalReplayCloudSegment { + payload_gz: row.get(1).map_err(|err| err.to_string())?, + event_count, + segment_hash: row.get(3).map_err(|err| err.to_string())?, + wire_bytes: row_bytes as u64, + }); + serialized_bytes = serialized_bytes.saturating_add(row_bytes); + consumed_events = consumed_events.saturating_add(event_count); + next_segment_index = physical_index.saturating_add(1); + } + if segments.is_empty() { + return Err("Replay cloud physical batch cursor did not resolve a segment".to_string()); + } + let next_event_index = start_event_index.saturating_add(consumed_events).min(end); + Ok(ExternalReplayCloudBatch { + segments, + start_event_index, + next_event_index, + start_segment_index: resolved_segment_index, + next_segment_index, + eof: next_event_index >= end, + serialized_bytes: serialized_bytes as u64, + }) +} + +pub(super) fn cloud_spool_prefix_hash( + token: &str, + event_count: u64, +) -> Result { + let entry = acquire_cloud_spool_read(token)?; + if event_count > entry.manifest.frozen_event_count { + return Err("Requested prefix crosses the replay mutable tail".to_string()); + } + let frozen_chain_hash = if event_count == 0 { + sha256_hex(b"") + } else { + let conn = rusqlite::Connection::open_with_flags( + &entry.path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY, + ) + .map_err(|err| format!("open replay cloud spool: {err}"))?; + conn.query_row( + "SELECT frozen_chain_hash FROM events WHERE event_index=?1", + [event_count.saturating_sub(1).min(i64::MAX as u64) as i64], + |row| row.get(0), + ) + .map_err(|err| format!("read replay cloud prefix hash: {err}"))? + }; + Ok(ExternalReplayCloudPrefixHash { + event_count, + frozen_chain_hash, + }) +} + +pub(super) fn acquire_cloud_spool_read(token: &str) -> Result { + let mut spools = cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let entry = spools + .get_mut(token) + .ok_or_else(|| "Replay cloud spool expired; prepare it again".to_string())?; + if entry.owner_released { + return Err("Replay cloud spool was released; prepare it again".to_string()); + } + entry.last_used = Instant::now(); + entry.lease_count = entry.lease_count.saturating_add(1); + Ok(CloudSpoolReadLease { + token: token.to_string(), + path: entry.path.clone(), + manifest: entry.manifest.clone(), + }) +} + +pub(super) fn release_cloud_spool(token: &str) -> Result<(), String> { + let path = { + let mut spools = cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let Some(entry) = spools.get_mut(token) else { + return Ok(()); + }; + if entry.owner_released { + return Ok(()); + } + entry.owner_released = true; + entry.lease_count = entry.lease_count.saturating_sub(1); + if entry.lease_count == 0 { + spools.remove(token).map(|entry| entry.path) + } else { + None + } + }; + if let Some(path) = path { + remove_cloud_spool_file(&path)?; + } + Ok(()) +} + +pub(super) fn release_cloud_spool_read_lease(token: &str) { + let path = { + let mut spools = cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let Some(entry) = spools.get_mut(token) else { + return; + }; + entry.lease_count = entry.lease_count.saturating_sub(1); + if entry.owner_released && entry.lease_count == 0 { + spools.remove(token).map(|entry| entry.path) + } else { + None + } + }; + if let Some(path) = path { + if let Err(error) = remove_cloud_spool_file(&path) { + log::warn!("[external-replay] {error}"); + } + } +} + +pub(super) fn remove_cloud_spool_file(path: &PathBuf) -> Result<(), String> { + match fs::remove_file(path) { + Ok(()) => Ok(()), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(err) => Err(format!("remove replay cloud spool: {err}")), + } +} + +pub(super) fn cleanup_cloud_spools() { + let paths = { + let mut spools = cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let now = Instant::now(); + let expired = spools + .iter() + .filter(|(_, entry)| { + // `lease_count == 1` is the renderer's idle owner lease. An + // in-flight reader raises it above one and must never be + // evicted merely because another session prepared a spool. + !entry.owner_released + && entry.lease_count == 1 + && now.duration_since(entry.last_used) >= CLOUD_SPOOL_TTL + }) + .map(|(token, _)| token.clone()) + .collect::>(); + expired + .into_iter() + .filter_map(|token| spools.remove(&token).map(|entry| entry.path)) + .collect::>() + }; + for path in paths { + if let Err(error) = remove_cloud_spool_file(&path) { + log::warn!("[external-replay] {error}"); + } + } +} + +#[cfg(test)] +pub(super) fn write_stable_json( + writer: &mut impl Write, + value: &serde_json::Value, +) -> Result<(), String> { + match value { + serde_json::Value::Object(object) => { + writer.write_all(b"{").map_err(|err| err.to_string())?; + let mut keys = object.keys().collect::>(); + keys.sort_unstable(); + for (index, key) in keys.into_iter().enumerate() { + if index > 0 { + writer.write_all(b",").map_err(|err| err.to_string())?; + } + serde_json::to_writer(&mut *writer, key).map_err(|err| err.to_string())?; + writer.write_all(b":").map_err(|err| err.to_string())?; + write_stable_json(writer, &object[key])?; + } + writer.write_all(b"}").map_err(|err| err.to_string())?; + } + serde_json::Value::Array(array) => { + writer.write_all(b"[").map_err(|err| err.to_string())?; + for (index, item) in array.iter().enumerate() { + if index > 0 { + writer.write_all(b",").map_err(|err| err.to_string())?; + } + write_stable_json(writer, item)?; + } + writer.write_all(b"]").map_err(|err| err.to_string())?; + } + primitive => { + serde_json::to_writer(writer, primitive).map_err(|err| err.to_string())?; + } + } + Ok(()) +} + +pub(super) fn sha256_hex(bytes: &[u8]) -> String { + format!("{:x}", Sha256::digest(bytes)) +} + +pub(super) fn stream_replay_cloud_events( + source_id: &str, + session_id: &str, + mut consume: impl FnMut( + &SessionEvent, + &mut dyn FnMut(&PayloadRef, u64) -> Result, + ) -> Result<(), String>, +) -> Result { + match resolve_secondary_consumer_target(source_id, session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => { + let limits = ReplayLimits { + max_turns: 10, + max_events: STREAM_BATCH_MAX_EVENTS, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }; + let prepared = + prepare_sessions_stream_replay_snapshot(source, &imported_session_id, limits)?; + let expected_generation = prepared.generation; + let expected_revision = prepared.revision; + let mut payload_conn = database::db::get_connection() + .map_err(|err| format!("open replay stream payload DB: {err}"))?; + let mut after_sequence = -1_i64; + loop { + let scan = with_sessions_replay_writer("replay cloud scan", |conn| { + replay::scan_window_after_generation( + conn, + source, + &imported_session_id, + &expected_generation, + expected_revision, + after_sequence, + limits, + ) + })?; + let next_sequence = scan.cursor.through_sequence; + let has_more = scan.has_more; + let (events, _) = normalize_indexed_chunks( + scan.chunks, + session_id, + source.as_str(), + &scan.cursor.generation, + ); + for event in &events { + let mut read_payload = |payload_ref: &PayloadRef, offset: u64| { + replay::read_payload_range( + &mut payload_conn, + source, + &imported_session_id, + &scan.cursor.generation, + payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id), + &payload_ref.field_path, + offset, + Some(EXPORT_PAYLOAD_RANGE_BYTES), + ) + }; + consume(event, &mut read_payload)?; + } + if !has_more { + break; + } + if next_sequence <= after_sequence { + return Err("Replay stream cursor did not advance".to_string()); + } + after_sequence = next_sequence; + } + let final_scan = with_sessions_replay_writer("replay cloud finalization", |conn| { + replay::scan_window_after( + conn, + source, + &imported_session_id, + after_sequence, + ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }, + ) + })?; + validate_stream_replay_cursor( + &expected_generation, + expected_revision, + &final_scan.cursor, + "finalizing cloud replay", + )?; + Ok(expected_generation) + } + ResolvedReplayTarget::CollaborationSnapshot => { + let conn = database::db::get_connection() + .map_err(|err| format!("open collaboration replay stream DB: {err}"))?; + let state = collaboration_snapshot_state(&conn, session_id)?; + let limits = ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: STREAM_BATCH_MAX_EVENTS, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }; + let mut after_sequence = -1_i64; + loop { + let indexed = query_collaboration_snapshot_events( + &conn, + session_id, + &state.generation, + after_sequence, + state.max_sequence.saturating_add(1), + limits, + false, + )?; + if indexed.is_empty() { + break; + } + let next_sequence = indexed + .last() + .map_or(after_sequence, |(sequence, _)| *sequence); + for (_, event) in &indexed { + let mut read_payload = |payload_ref: &PayloadRef, offset: u64| { + collaboration_snapshot_payload_range_from_conn( + &conn, + session_id, + &state.generation, + payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id), + &payload_ref.field_path, + offset, + EXPORT_PAYLOAD_RANGE_BYTES, + ) + }; + consume(event, &mut read_payload)?; + } + if next_sequence >= state.max_sequence { + break; + } + if next_sequence <= after_sequence { + return Err("Collaboration replay stream cursor did not advance".to_string()); + } + after_sequence = next_sequence; + } + let current = collaboration_snapshot_state(&conn, session_id)?; + validate_query_apply_version( + &state.generation, + state.revision, + ¤t.generation, + current.revision, + )?; + Ok(state.generation) + } + ResolvedReplayTarget::ManagedChunkStore => { + let conn = database::db::get_connection() + .map_err(|err| format!("open managed replay stream DB: {err}"))?; + stream_managed_chunk_replay_events_from_conn( + &conn, + session_id, + "streaming managed cloud replay", + |event, read_payload| consume(event, read_payload), + ) + } + ResolvedReplayTarget::NotReady => { + Err("Managed native transcript is not bound yet".to_string()) + } + } +} + +#[tauri::command] +pub async fn external_replay_cloud_prepare( + source_id: String, + session_id: String, +) -> Result { + let manifest = + tokio::task::spawn_blocking(move || prepare_cloud_spool(&source_id, &session_id)) + .await + .map_err(|err| format!("join replay cloud prepare task: {err}"))??; + schedule_replay_cache_prune(); + Ok(manifest) +} + +#[tauri::command] +pub async fn external_replay_cloud_read_batch( + token: String, + start_event_index: u64, + end_event_index: u64, + start_segment_index: Option, + max_bytes: Option, +) -> Result { + tokio::task::spawn_blocking(move || { + read_cloud_spool_batch( + &token, + start_event_index, + end_event_index, + start_segment_index, + max_bytes, + ) + }) + .await + .map_err(|err| format!("join replay cloud batch task: {err}"))? +} + +#[tauri::command] +pub async fn external_replay_cloud_prefix_hash( + token: String, + event_count: u64, +) -> Result { + tokio::task::spawn_blocking(move || cloud_spool_prefix_hash(&token, event_count)) + .await + .map_err(|err| format!("join replay cloud prefix task: {err}"))? +} + +#[tauri::command] +pub async fn external_replay_cloud_release(token: String) -> Result<(), String> { + release_cloud_spool(&token) +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/collaboration.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/collaboration.rs new file mode 100644 index 000000000..f84c01837 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/collaboration.rs @@ -0,0 +1,1029 @@ +use super::*; + +// ------------------------------------------------------------------------- +// ORGII-owned collaboration snapshot bounded SQL driver. +// ------------------------------------------------------------------------- + +#[derive(Debug, Clone)] +pub(super) struct CollaborationSnapshotState { + pub(super) generation: String, + pub(super) revision: u64, + pub(super) reset_revision: u64, + pub(super) max_sequence: i64, + pub(super) event_count: u64, +} + +pub(super) fn validate_collaboration_snapshot_session_id(session_id: &str) -> Result<(), String> { + if session_id.starts_with(COLLABORATION_SNAPSHOT_SESSION_PREFIX) + && session_id.len() > COLLABORATION_SNAPSHOT_SESSION_PREFIX.len() + && !session_id.contains(['/', '\\']) + { + return Ok(()); + } + Err(format!( + "collaboration snapshot session id must start with {COLLABORATION_SNAPSHOT_SESSION_PREFIX}" + )) +} + +/// Install mutation accounting once, then seed a pre-existing imported copy. +/// INSERTs at the append frontier keep the generation and advance revision; +/// UPDATE/DELETE or out-of-order INSERTs force a generation reset. The +/// triggers observe writes made by the normal cache path without copying any +/// transcript body into a second store. +pub(super) fn ensure_collaboration_snapshot_state( + conn: &rusqlite::Connection, + session_id: &str, +) -> Result<(), String> { + validate_collaboration_snapshot_session_id(session_id)?; + let state_table_exists = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master + WHERE type='table' AND name='collaboration_replay_state')", + [], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("inspect collaboration replay state schema: {error}"))? + != 0; + let trigger_count = conn + .query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type='trigger' AND name IN( + 'collaboration_replay_events_insert', + 'collaboration_replay_events_delete', + 'collaboration_replay_events_update_old', + 'collaboration_replay_events_update_new' + )", + [], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("inspect collaboration replay triggers: {error}"))?; + let state_exists = state_table_exists + && conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM collaboration_replay_state WHERE session_id=?1)", + [session_id], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("inspect collaboration replay session state: {error}"))? + != 0; + if trigger_count == 4 && state_exists { + return Ok(()); + } + conn.execute_batch( + "CREATE TABLE IF NOT EXISTS collaboration_replay_state ( + session_id TEXT PRIMARY KEY, + generation INTEGER NOT NULL DEFAULT 0, + revision INTEGER NOT NULL DEFAULT 0, + max_sequence INTEGER NOT NULL DEFAULT -1, + event_count INTEGER NOT NULL DEFAULT 0 + ); + CREATE TRIGGER IF NOT EXISTS collaboration_replay_events_insert + AFTER INSERT ON events + WHEN NEW.session_id GLOB 'imported-session-*' + BEGIN + INSERT INTO collaboration_replay_state( + session_id,generation,revision,max_sequence,event_count + ) VALUES( + NEW.session_id,0,1,COALESCE(NEW.history_sequence,NEW.rowid),1 + ) + ON CONFLICT(session_id) DO UPDATE SET + generation = collaboration_replay_state.generation + + CASE WHEN collaboration_replay_state.max_sequence=-2 THEN 0 + WHEN COALESCE(NEW.history_sequence,NEW.rowid) > + collaboration_replay_state.max_sequence + THEN 0 ELSE 1 END, + revision = collaboration_replay_state.revision + 1, + max_sequence = CASE + WHEN collaboration_replay_state.max_sequence=-2 THEN -2 + ELSE MAX( + collaboration_replay_state.max_sequence, + COALESCE(NEW.history_sequence,NEW.rowid) + ) END, + event_count = collaboration_replay_state.event_count + 1; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_replay_events_delete + AFTER DELETE ON events + WHEN OLD.session_id GLOB 'imported-session-*' + BEGIN + UPDATE collaboration_replay_state + SET generation = generation + CASE WHEN max_sequence=-2 THEN 0 ELSE 1 END, + revision = revision + CASE WHEN max_sequence=-2 THEN 0 ELSE 1 END, + max_sequence = -2, + event_count = MAX(event_count - 1,0) + WHERE session_id=OLD.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_replay_events_update_old + AFTER UPDATE ON events + WHEN OLD.session_id GLOB 'imported-session-*' + BEGIN + UPDATE collaboration_replay_state + SET generation = generation + CASE WHEN max_sequence=-2 THEN 0 ELSE 1 END, + revision = revision + CASE WHEN max_sequence=-2 THEN 0 ELSE 1 END, + max_sequence = -2, + event_count = MAX( + event_count - CASE WHEN NEW.session_id != OLD.session_id THEN 1 ELSE 0 END, + 0 + ) + WHERE session_id=OLD.session_id; + END; + CREATE TRIGGER IF NOT EXISTS collaboration_replay_events_update_new + AFTER UPDATE ON events + WHEN NEW.session_id GLOB 'imported-session-*' + AND NEW.session_id != OLD.session_id + BEGIN + INSERT INTO collaboration_replay_state( + session_id,generation,revision,max_sequence,event_count + ) + VALUES( + NEW.session_id,0,1,COALESCE(NEW.history_sequence,NEW.rowid),1 + ) + ON CONFLICT(session_id) DO UPDATE SET + generation = collaboration_replay_state.generation + + CASE WHEN collaboration_replay_state.max_sequence=-2 THEN 0 ELSE 1 END, + revision = collaboration_replay_state.revision + + CASE WHEN collaboration_replay_state.max_sequence=-2 THEN 0 ELSE 1 END, + max_sequence = -2, + event_count = collaboration_replay_state.event_count + 1; + END;", + ) + .map_err(|error| format!("initialize collaboration replay state: {error}"))?; + conn.execute( + "INSERT INTO collaboration_replay_state( + session_id,generation,revision,max_sequence,event_count + ) + SELECT ?1,0,COUNT(*), + COALESCE(MAX(COALESCE(history_sequence,rowid)),-1),COUNT(*) + FROM events WHERE session_id=?1 + ON CONFLICT(session_id) DO NOTHING", + [session_id], + ) + .map_err(|error| format!("seed collaboration replay state: {error}"))?; + Ok(()) +} + +pub(super) fn collaboration_snapshot_state( + conn: &rusqlite::Connection, + session_id: &str, +) -> Result { + if session_id.starts_with(COLLABORATION_SNAPSHOT_FORK_PREFIX) { + return crate::agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_secondary_state(conn, session_id)? + .map(|state| CollaborationSnapshotState { + generation: state.generation, + revision: state.revision, + reset_revision: state.reset_revision, + max_sequence: state.max_sequence, + event_count: state.event_count, + }) + .ok_or_else(|| { + format!( + "Native Agent session is not backed by an intact collaboration snapshot: {session_id}" + ) + }); + } + ensure_collaboration_snapshot_state(conn, session_id)?; + let mut state = conn + .query_row( + "SELECT generation,revision,max_sequence,event_count + FROM collaboration_replay_state WHERE session_id=?1", + [session_id], + |row| { + let generation = row.get::<_, i64>(0)?.max(0); + Ok(CollaborationSnapshotState { + generation: format!( + "collaboration-v{COLLABORATION_SNAPSHOT_DRIVER_VERSION}-{generation}" + ), + revision: row.get::<_, i64>(1)?.max(0) as u64, + reset_revision: 0, + max_sequence: row.get(2)?, + event_count: row.get::<_, i64>(3)?.max(0) as u64, + }) + }, + ) + .map_err(|error| format!("read collaboration replay state: {error}"))?; + if state.max_sequence == -2 { + conn.execute( + "UPDATE collaboration_replay_state + SET max_sequence=COALESCE(( + SELECT MAX(history_sequence) FROM events WHERE session_id=?1 + ),-1), + event_count=( + SELECT COUNT(*) FROM events WHERE session_id=?1 + ) + WHERE session_id=?1", + [session_id], + ) + .map_err(|error| format!("refresh dirty collaboration replay state: {error}"))?; + let (max_sequence, event_count) = conn + .query_row( + "SELECT max_sequence,event_count FROM collaboration_replay_state + WHERE session_id=?1", + [session_id], + |row| Ok((row.get::<_, i64>(0)?, row.get::<_, i64>(1)?)), + ) + .map_err(|error| format!("reload collaboration replay state: {error}"))?; + state.max_sequence = max_sequence; + state.event_count = event_count.max(0) as u64; + } + let has_unsequenced_rows = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM events + WHERE session_id=?1 AND history_sequence IS NULL)", + [session_id], + |row| row.get::<_, i64>(0), + ) + .map_err(|error| format!("validate collaboration replay sequences: {error}"))? + != 0; + if has_unsequenced_rows { + return Err( + "Collaboration snapshot contains an event without history_sequence; retry the atomic import" + .to_string(), + ); + } + Ok(state) +} + +pub(super) fn snapshot_user_predicate() -> &'static str { + "(event_type IN ('user','user_message') + OR function_name IN ('user','user_message') + OR CASE WHEN json_valid(meta_json) + THEN json_extract(meta_json,'$.source')='user' + ELSE 0 END)" +} + +pub(super) fn collaboration_snapshot_turn_count( + conn: &rusqlite::Connection, + session_id: &str, + event_count: u64, +) -> Result { + let sql = format!( + "SELECT COUNT(*) FROM events WHERE session_id=?1 AND {}", + snapshot_user_predicate() + ); + let user_count = conn + .query_row(&sql, [session_id], |row| row.get::<_, i64>(0)) + .map_err(|error| format!("count collaboration replay turns: {error}"))? + .max(0) as u64; + Ok(if user_count == 0 && event_count > 0 { + 1 + } else { + user_count + }) +} + +pub(super) fn collaboration_snapshot_turn_sequence( + conn: &rusqlite::Connection, + session_id: &str, + turn_index: i64, +) -> Result, String> { + if turn_index < 0 { + return Ok(None); + } + let sql = format!( + "SELECT history_sequence FROM events + WHERE session_id=?1 AND {} + AND history_sequence IS NOT NULL + ORDER BY history_sequence ASC,id ASC + LIMIT 1 OFFSET ?2", + snapshot_user_predicate() + ); + conn.query_row(&sql, rusqlite::params![session_id, turn_index], |row| { + row.get(0) + }) + .optional() + .map_err(|error| format!("resolve collaboration replay turn index: {error}")) +} + +pub(super) fn collaboration_snapshot_turn_id_sequence( + conn: &rusqlite::Connection, + session_id: &str, + turn_id: &str, +) -> Result, String> { + conn.query_row( + "SELECT history_sequence FROM events + WHERE session_id=?1 AND id=?2 AND history_sequence IS NOT NULL", + rusqlite::params![session_id, turn_id], + |row| row.get(0), + ) + .optional() + .map_err(|error| format!("resolve collaboration replay turn id: {error}")) +} + +pub(super) fn collaboration_snapshot_latest_turn_start( + conn: &rusqlite::Connection, + session_id: &str, + max_turns: usize, +) -> Result, String> { + let sql = format!( + "SELECT history_sequence FROM events + WHERE session_id=?1 AND {} + AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC,id DESC + LIMIT 1 OFFSET ?2", + snapshot_user_predicate() + ); + conn.query_row( + &sql, + rusqlite::params![session_id, max_turns.saturating_sub(1) as i64], + |row| row.get(0), + ) + .optional() + .map_err(|error| format!("resolve latest collaboration replay turn: {error}")) +} + +pub(super) fn snapshot_root_preview(raw_prefix: Option) -> serde_json::Value { + serde_json::json!({ + "_replayTruncated": true, + "_preview": raw_prefix.unwrap_or_else(|| "[payload truncated]".to_string()), + }) +} + +pub(super) fn snapshot_payload_ref( + event_id: &str, + field_path: &str, + preview: String, + full_size_bytes: i64, + generation: &str, + encoding: PayloadRefEncoding, +) -> PayloadRef { + PayloadRef { + event_id: event_id.to_string(), + field_path: field_path.to_string(), + preview, + full_size_bytes: full_size_bytes.max(0) as usize, + truncated: true, + replay_encoding: Some(encoding), + replay_source_id: Some(COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID.to_string()), + replay_generation: Some(generation.to_string()), + replay_source_event_id: Some(event_id.to_string()), + } +} + +pub(super) fn query_collaboration_snapshot_events( + conn: &rusqlite::Connection, + session_id: &str, + generation: &str, + lower_exclusive: i64, + upper_exclusive: i64, + limits: ReplayLimits, + newest_first: bool, +) -> Result, String> { + let limits = limits.bounded(); + let order = if newest_first { "DESC" } else { "ASC" }; + // args are never loaded past the normal preview boundary; result allows + // the Shell preview boundary. Oversized roots are reconstructed only by + // `read_payload_range`, never by the ordinary window query. + let sql = format!( + "SELECT id,session_id,event_type,function_name,thread_id, + CASE WHEN length(CAST(args_json AS BLOB))<=?4 + THEN args_json ELSE '{{}}' END, + CASE WHEN length(CAST(result_json AS BLOB))<=?5 + THEN result_json ELSE '{{}}' END, + created_at, + CASE WHEN json_valid(meta_json) THEN + CASE WHEN length(CAST(json_extract(meta_json,'$.displayText') AS BLOB))>?4 + THEN json_set(meta_json,'$.displayText', + substr(json_extract(meta_json,'$.displayText'),1,2048)) + ELSE meta_json END + ELSE meta_json END, + history_sequence, + length(CAST(args_json AS BLOB)), + length(CAST(result_json AS BLOB)), + CASE WHEN length(CAST(args_json AS BLOB))>?4 + THEN substr(args_json,1,2048) END, + CASE WHEN length(CAST(result_json AS BLOB))>?5 + THEN substr(result_json,1,8192) END, + CASE WHEN json_valid(meta_json) + THEN length(CAST(json_extract(meta_json,'$.displayText') AS BLOB)) + ELSE 0 END + FROM events + WHERE session_id=?1 + AND history_sequence>?2 + AND history_sequence>(10) + .map_err(|e| e.to_string())? + .unwrap_or(0); + let result_size: i64 = row + .get::<_, Option>(11) + .map_err(|e| e.to_string())? + .unwrap_or(0); + let args_prefix: Option = row.get(12).map_err(|error| error.to_string())?; + let result_prefix: Option = row.get(13).map_err(|error| error.to_string())?; + let display_size: i64 = row + .get::<_, Option>(14) + .map_err(|e| e.to_string())? + .unwrap_or(0); + let cached = session_persistence::CachedEvent { + id: row.get(0).map_err(|error| error.to_string())?, + session_id: row.get(1).map_err(|error| error.to_string())?, + event_type: row.get(2).map_err(|error| error.to_string())?, + function_name: row.get(3).map_err(|error| error.to_string())?, + thread_id: row.get(4).map_err(|error| error.to_string())?, + args_json: row.get(5).map_err(|error| error.to_string())?, + result_json: row.get(6).map_err(|error| error.to_string())?, + content: String::new(), + created_at: row.get(7).map_err(|error| error.to_string())?, + meta_json: row.get(8).map_err(|error| error.to_string())?, + history_sequence: Some(sequence), + }; + let mut event = cached_event_to_session_event(&cached); + event.payload_refs.clear(); + if args_size as usize > replay::NORMAL_PAYLOAD_PREVIEW_BYTES { + event.args = snapshot_root_preview(args_prefix); + event.payload_refs.push(snapshot_payload_ref( + &event.id, + "args", + json_field_preview(&event, "args"), + args_size, + generation, + PayloadRefEncoding::JsonValue, + )); + } + let result_limit = if event.ui_canonical == core_types::tool_names::RUN_SHELL { + replay::SHELL_PAYLOAD_PREVIEW_BYTES + } else { + replay::NORMAL_PAYLOAD_PREVIEW_BYTES + }; + if result_size as usize > result_limit { + event.result = snapshot_root_preview(result_prefix); + event.payload_refs.push(snapshot_payload_ref( + &event.id, + "result", + json_field_preview(&event, "result"), + result_size, + generation, + PayloadRefEncoding::JsonValue, + )); + } + if display_size as usize > replay::NORMAL_PAYLOAD_PREVIEW_BYTES { + event.payload_refs.push(snapshot_payload_ref( + &event.id, + "displayText", + event.display_text.clone(), + display_size, + generation, + PayloadRefEncoding::Utf8Text, + )); + } + // Extraction must see compact values. This prevents a deferred root + // from being copied into a second large rendering envelope. + event.extracted = None; + event.recompute_extracted(); + let next_bytes = serde_json::to_vec(&event) + .map_err(|error| format!("measure collaboration replay event: {error}"))? + .len(); + if !indexed.is_empty() && wire_bytes.saturating_add(next_bytes) > limits.max_ipc_bytes { + break; + } + if indexed.is_empty() && next_bytes > limits.max_ipc_bytes { + return Err(format!( + "Collaboration replay event {} exceeds the {} byte compact window budget", + event.id, limits.max_ipc_bytes + )); + } + wire_bytes = wire_bytes.saturating_add(next_bytes); + indexed.push((sequence, event)); + } + if newest_first { + indexed.reverse(); + } + Ok(indexed) +} + +pub(super) fn collaboration_snapshot_turn_headers( + conn: &rusqlite::Connection, + session_id: &str, + events: &[(i64, SessionEvent)], +) -> Result, String> { + if events.is_empty() { + return Ok(Vec::new()); + } + let mut starts = events + .iter() + .enumerate() + .filter(|(_, (_, event))| event.source == EventSource::User) + .map(|(offset, (sequence, event))| (offset, *sequence, event)) + .collect::>(); + if starts.is_empty() { + starts.push((0, events[0].0, &events[0].1)); + } + let mut headers = Vec::with_capacity(starts.len()); + for (position, (offset, start_sequence, event)) in starts.iter().enumerate() { + let next_offset = starts + .get(position + 1) + .map_or(events.len(), |(next, _, _)| *next); + let end = events.get(next_offset.saturating_sub(1)); + let sql = format!( + "SELECT COUNT(*) FROM events WHERE session_id=?1 + AND history_sequence(0) + }) + .map_err(|error| format!("index collaboration replay turn: {error}"))?; + headers.push(ReplayTurnHeader { + turn_id: event.id.clone(), + turn_index, + start_sequence: *start_sequence, + end_sequence: end.map(|(sequence, _)| *sequence), + started_at: event.created_at.clone(), + ended_at: end.map(|(_, event)| event.created_at.clone()), + event_count: next_offset.saturating_sub(*offset) as u64, + }); + } + Ok(headers) +} + +pub(super) fn collaboration_snapshot_read_window_from_conn( + conn: &rusqlite::Connection, + session_id: &str, + before_sequence: Option, + turn_id: Option<&str>, + turn_index: Option, + limits: ReplayLimits, +) -> Result { + let limits = limits.bounded(); + let exact_turn = turn_id.is_some() || turn_index.is_some(); + let state = collaboration_snapshot_state(conn, session_id)?; + let (lower_exclusive, upper_exclusive) = if let Some(turn_id) = turn_id { + let start = collaboration_snapshot_turn_id_sequence(conn, session_id, turn_id)? + .ok_or_else(|| format!("Collaboration replay turn is unavailable: {turn_id}"))?; + let next = conn + .query_row( + &format!( + "SELECT history_sequence FROM events + WHERE session_id=?1 AND history_sequence>?2 AND {} + ORDER BY history_sequence ASC,id ASC LIMIT 1", + snapshot_user_predicate() + ), + rusqlite::params![session_id, start], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|error| format!("resolve next collaboration replay turn: {error}"))? + .unwrap_or(i64::MAX); + (start.saturating_sub(1), next) + } else if let Some(turn_index) = turn_index { + let start = collaboration_snapshot_turn_sequence(conn, session_id, turn_index)? + .ok_or_else(|| { + format!("Collaboration replay turn index is unavailable: {turn_index}") + })?; + let next = collaboration_snapshot_turn_sequence(conn, session_id, turn_index + 1)? + .unwrap_or(i64::MAX); + (start.saturating_sub(1), next) + } else if let Some(before_sequence) = before_sequence { + (-1, before_sequence) + } else { + let start = collaboration_snapshot_latest_turn_start(conn, session_id, limits.max_turns)? + .unwrap_or(-1); + (start.saturating_sub(1), i64::MAX) + }; + let (indexed, window_start_sequence) = if exact_turn { + read_collaboration_exact_turn_events( + conn, + session_id, + &state.generation, + lower_exclusive, + upper_exclusive, + limits, + )? + } else { + let indexed = query_collaboration_snapshot_events( + conn, + session_id, + &state.generation, + lower_exclusive, + upper_exclusive, + limits, + true, + )?; + let window_start_sequence = indexed.first().map(|(sequence, _)| *sequence); + (indexed, window_start_sequence) + }; + let through_sequence = indexed.last().map_or(-1, |(sequence, _)| *sequence); + let has_older = if let Some(continuation_sequence) = window_start_sequence { + conn.query_row( + "SELECT EXISTS(SELECT 1 FROM events + WHERE session_id=?1 AND history_sequence(0), + ) + .map_err(|error| format!("query older collaboration replay events: {error}"))? + != 0 + } else { + false + }; + let turn_headers = if exact_turn { + collaboration_snapshot_exact_turn_header( + conn, + session_id, + lower_exclusive.saturating_add(1), + upper_exclusive, + )? + .into_iter() + .collect() + } else { + collaboration_snapshot_turn_headers(conn, session_id, &indexed)? + }; + let total_turn_count = collaboration_snapshot_turn_count(conn, session_id, state.event_count)?; + let parsed_rows = indexed.len() as u64; + let events = indexed + .into_iter() + .map(|(_, event)| event) + .collect::>(); + let ipc_bytes = serde_json::to_vec(&events).map_or(0, |bytes| bytes.len()) as u64; + let current = collaboration_snapshot_state(conn, session_id)?; + validate_query_apply_version( + &state.generation, + state.revision, + ¤t.generation, + current.revision, + )?; + Ok(ExternalReplayWindow { + cursor: ReplayCursor { + source_id: COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence, + }, + events, + window_start_sequence, + turn_headers, + total_turn_count, + total_event_count: state.event_count, + has_older, + stats: ReplayStats { + parsed_rows, + normalized_events: parsed_rows, + ipc_bytes, + ..ReplayStats::default() + }, + watcher_available: false, + }) +} + +fn read_collaboration_exact_turn_events( + conn: &rusqlite::Connection, + session_id: &str, + generation: &str, + lower_exclusive: i64, + upper_exclusive: i64, + limits: ReplayLimits, +) -> Result<(Vec<(i64, SessionEvent)>, Option), String> { + let anchor_limits = ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: limits.max_ipc_bytes, + }; + let mut indexed = query_collaboration_snapshot_events( + conn, + session_id, + generation, + lower_exclusive, + upper_exclusive, + anchor_limits, + false, + )?; + let Some(anchor_sequence) = indexed.first().map(|(sequence, _)| *sequence) else { + return Ok((indexed, None)); + }; + let anchor_bytes = indexed + .first() + .and_then(|(_, event)| serde_json::to_vec(event).ok()) + .map_or(0, |bytes| bytes.len()); + let Some(tail_limits) = limits.after_exact_turn_anchor(anchor_bytes) else { + return Ok((indexed, Some(anchor_sequence))); + }; + let mut tail = query_collaboration_snapshot_events( + conn, + session_id, + generation, + anchor_sequence, + upper_exclusive, + tail_limits, + true, + )?; + let tail_start_sequence = tail.first().map(|(sequence, _)| *sequence); + let has_gap = if let Some(tail_start_sequence) = tail_start_sequence { + conn.query_row( + "SELECT EXISTS( + SELECT 1 FROM events + WHERE session_id=?1 AND history_sequence>?2 AND history_sequence(0), + ) + .map_err(|error| format!("query exact collaboration replay turn gap: {error}"))? + != 0 + } else { + false + }; + let window_start_sequence = if has_gap { + tail_start_sequence + } else { + Some(anchor_sequence) + }; + indexed.append(&mut tail); + Ok((indexed, window_start_sequence)) +} + +fn collaboration_snapshot_exact_turn_header( + conn: &rusqlite::Connection, + session_id: &str, + start_sequence: i64, + upper_exclusive: i64, +) -> Result, String> { + let anchor = conn + .query_row( + "SELECT id,created_at FROM events + WHERE session_id=?1 AND history_sequence=?2 + ORDER BY id ASC LIMIT 1", + rusqlite::params![session_id, start_sequence], + |row| Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)), + ) + .optional() + .map_err(|error| format!("query collaboration replay turn anchor: {error}"))?; + let Some((turn_id, started_at)) = anchor else { + return Ok(None); + }; + let turn_index = conn + .query_row( + &format!( + "SELECT COUNT(*) FROM events WHERE session_id=?1 + AND history_sequence(0), + ) + .map_err(|error| format!("index collaboration replay exact turn: {error}"))?; + let (end_sequence, ended_at, event_count) = conn + .query_row( + "SELECT MAX(history_sequence), + (SELECT tail.created_at FROM events AS tail + WHERE tail.session_id=?1 AND tail.history_sequence>=?2 + AND tail.history_sequence=?2 AND history_sequence>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, i64>(2)?, + )) + }, + ) + .map_err(|error| format!("summarize collaboration replay exact turn: {error}"))?; + Ok(Some(ReplayTurnHeader { + turn_id, + turn_index, + start_sequence, + end_sequence, + started_at, + ended_at, + event_count: event_count.max(0) as u64, + })) +} + +pub(super) fn collaboration_snapshot_poll_delta_from_conn( + conn: &rusqlite::Connection, + session_id: &str, + cursor: &ReplayCursor, + limits: ReplayLimits, +) -> Result { + let state = collaboration_snapshot_state(conn, session_id)?; + if state.generation != cursor.generation { + return Ok(ExternalReplayDelta { + cursor: ReplayCursor { + source_id: COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence: -1, + }, + events: Vec::new(), + removed_event_ids: Vec::new(), + reset_required: true, + stats: ReplayStats::default(), + watcher_available: false, + }); + } + if state.reset_revision > cursor.revision { + return Ok(ExternalReplayDelta { + cursor: ReplayCursor { + source_id: COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence: -1, + }, + events: Vec::new(), + removed_event_ids: Vec::new(), + reset_required: true, + stats: ReplayStats::default(), + watcher_available: false, + }); + } + if state.revision == cursor.revision && cursor.through_sequence >= state.max_sequence { + return Ok(ExternalReplayDelta { + cursor: cursor.clone(), + events: Vec::new(), + removed_event_ids: Vec::new(), + reset_required: false, + stats: ReplayStats::default(), + watcher_available: false, + }); + } + let indexed = query_collaboration_snapshot_events( + conn, + session_id, + &state.generation, + cursor.through_sequence, + state.max_sequence.saturating_add(1), + limits, + false, + )?; + if indexed.is_empty() && state.revision != cursor.revision { + return Ok(ExternalReplayDelta { + cursor: ReplayCursor { + source_id: COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence: -1, + }, + events: Vec::new(), + removed_event_ids: Vec::new(), + reset_required: true, + stats: ReplayStats::default(), + watcher_available: false, + }); + } + let through_sequence = indexed + .last() + .map_or(cursor.through_sequence, |(sequence, _)| *sequence); + let parsed_rows = indexed.len() as u64; + let events = indexed + .into_iter() + .map(|(_, event)| event) + .collect::>(); + let ipc_bytes = serde_json::to_vec(&events).map_or(0, |bytes| bytes.len()) as u64; + let current = collaboration_snapshot_state(conn, session_id)?; + validate_query_apply_version( + &state.generation, + state.revision, + ¤t.generation, + current.revision, + )?; + Ok(ExternalReplayDelta { + cursor: ReplayCursor { + source_id: COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation: state.generation, + revision: state.revision, + through_sequence, + }, + events, + removed_event_ids: Vec::new(), + reset_required: false, + stats: ReplayStats { + parsed_rows, + normalized_events: parsed_rows, + ipc_bytes, + ..ReplayStats::default() + }, + watcher_available: false, + }) +} + +pub(super) fn collaboration_snapshot_payload_range_from_conn( + conn: &rusqlite::Connection, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + let state = collaboration_snapshot_state(conn, session_id)?; + if state.generation != generation { + return Err(format!( + "Collaboration replay generation changed: requested {generation}, current {}", + state.generation + )); + } + let (root, path) = field_path.split_once('.').unwrap_or((field_path, "")); + let (column, json_path) = match root { + "args" => ( + "args_json", + (!path.is_empty()).then(|| replay_sqlite_json_path(path)), + ), + "result" => ( + "result_json", + (!path.is_empty()).then(|| replay_sqlite_json_path(path)), + ), + "displayText" if path.is_empty() => ( + "meta_json", + Some(Ok::("$.displayText".to_string())), + ), + _ => return Err("fieldPath must start with args, result or displayText".to_string()), + }; + let start = offset.min(i64::MAX as u64) as i64; + let read_bytes = max_bytes.saturating_add(4).min(i64::MAX as usize) as i64; + let (total_bytes, bytes): (Option, Option>) = if let Some(path) = json_path { + let path = path?; + let sql = format!( + "SELECT length(CAST(json_extract({column},?3) AS BLOB)), + substr(CAST(json_extract({column},?3) AS BLOB),?4,?5) + FROM events WHERE session_id=?1 AND id=?2" + ); + conn.query_row( + &sql, + rusqlite::params![ + session_id, + event_id, + path, + start.saturating_add(1), + read_bytes + ], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + } else { + let sql = format!( + "SELECT length(CAST({column} AS BLOB)), + substr(CAST({column} AS BLOB),?3,?4) + FROM events WHERE session_id=?1 AND id=?2" + ); + conn.query_row( + &sql, + rusqlite::params![session_id, event_id, start.saturating_add(1), read_bytes], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + } + .map_err(|error| format!("load collaboration replay payload range: {error}"))?; + let total_bytes = total_bytes + .ok_or_else(|| format!("collaboration replay payload field not found: {field_path}"))? + .max(0) as u64; + let bytes = bytes.unwrap_or_default(); + let mut take = max_bytes.min(bytes.len()); + while take > 0 && std::str::from_utf8(&bytes[..take]).is_err() { + take -= 1; + } + if take == 0 && !bytes.is_empty() { + return Err(format!( + "collaboration replay range starts inside invalid UTF-8: {field_path} at {offset}" + )); + } + let text = String::from_utf8(bytes[..take].to_vec()) + .map_err(|error| format!("decode collaboration replay payload range: {error}"))?; + let next_offset = offset.saturating_add(take as u64).min(total_bytes); + let range = ReplayPayloadRange { + event_id: event_id.to_string(), + field_path: field_path.to_string(), + offset: offset.min(total_bytes), + next_offset, + eof: next_offset >= total_bytes, + total_bytes, + text, + }; + let current = collaboration_snapshot_state(conn, session_id)?; + if current.generation != generation { + return Err(format!( + "Collaboration replay generation changed during payload read: requested {generation}, current {}", + current.generation + )); + } + Ok(range) +} + +// ------------------------------------------------------------------------- +// Readerless managed CLI bounded SQL driver. +// ------------------------------------------------------------------------- diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/export.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/export.rs new file mode 100644 index 000000000..7298255b5 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/export.rs @@ -0,0 +1,859 @@ +use super::*; +use tauri_plugin_dialog::DialogExt; + +fn authorized_export_destination(destination: &std::path::Path) -> Result { + if !destination.is_absolute() { + return Err("Replay export destination must be an absolute path".to_string()); + } + let parent = destination + .parent() + .ok_or_else(|| "Replay export destination has no parent directory".to_string())?; + let file_name = destination + .file_name() + .ok_or_else(|| "Replay export destination has no file name".to_string())?; + let canonical_parent = parent + .canonicalize() + .map_err(|err| format!("open replay export directory {}: {err}", parent.display()))?; + if !canonical_parent.is_dir() { + return Err(format!( + "Replay export parent is not a directory: {}", + canonical_parent.display() + )); + } + let authorized = canonical_parent.join(file_name); + match fs::symlink_metadata(&authorized) { + Ok(metadata) if metadata.file_type().is_symlink() => { + return Err(format!( + "Replay export refuses a symbolic-link destination: {}", + authorized.display() + )); + } + Ok(metadata) if metadata.is_dir() => { + return Err(format!( + "Replay export destination is a directory: {}", + authorized.display() + )); + } + Ok(_) => {} + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => { + return Err(format!( + "inspect replay export destination {}: {err}", + authorized.display() + )); + } + } + Ok(authorized) +} + +fn create_private_export_temp(path: &std::path::Path) -> Result { + let mut options = fs::OpenOptions::new(); + options.create_new(true).write(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + options + .open(path) + .map_err(|err| format!("create replay export {}: {err}", path.display())) +} + +#[cfg(not(windows))] +fn atomic_replace_export( + temp: &std::path::Path, + destination: &std::path::Path, +) -> std::io::Result<()> { + fs::rename(temp, destination) +} + +#[cfg(windows)] +fn atomic_replace_export( + temp: &std::path::Path, + destination: &std::path::Path, +) -> std::io::Result<()> { + use std::os::windows::ffi::OsStrExt; + + const MOVEFILE_REPLACE_EXISTING: u32 = 0x0000_0001; + const MOVEFILE_WRITE_THROUGH: u32 = 0x0000_0008; + #[link(name = "Kernel32")] + unsafe extern "system" { + fn MoveFileExW(existing: *const u16, replacement: *const u16, flags: u32) -> i32; + } + + let existing = temp + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + let replacement = destination + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + // SAFETY: both buffers are live, NUL-terminated UTF-16 strings. The API + // does not retain either pointer and performs a same-volume replacement. + let moved = unsafe { + MoveFileExW( + existing.as_ptr(), + replacement.as_ptr(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH, + ) + }; + if moved == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } +} + +pub fn stream_replay_export_to_path( + source_id: &str, + session_id: &str, + destination_path: &std::path::Path, + format: ReplayExportFormat, + orgii_envelope: Option<&OrgiiSessionExportEnvelope>, +) -> Result { + let destination = authorized_export_destination(destination_path)?; + let parent = destination + .parent() + .ok_or_else(|| "Replay export destination has no parent directory".to_string())?; + let destination_name = destination + .file_name() + .map(|value| value.to_string_lossy()) + .unwrap_or_else(|| "replay-export".into()); + let temporary = parent.join(format!( + ".{destination_name}.orgii-{}.part", + uuid::Uuid::new_v4() + )); + let result = (|| -> Result { + let file = create_private_export_temp(&temporary)?; + let mut writer = + HashingWriter::new(BufWriter::with_capacity(EXPORT_WRITER_BUFFER_BYTES, file)); + match format { + ReplayExportFormat::Json => writer.write_all(b"[\n").map_err(|err| err.to_string())?, + ReplayExportFormat::OrgiiSessionJson => { + let envelope = orgii_envelope.ok_or_else(|| { + "orgii_session_json export requires the small session envelope".to_string() + })?; + writer + .write_all( + b"{\"format\":\"orgii.session.export\",\"version\":1,\"exportedAt\":", + ) + .map_err(|err| err.to_string())?; + serde_json::to_writer(&mut writer, &envelope.exported_at) + .map_err(|err| format!("serialize replay export timestamp: {err}"))?; + writer + .write_all(b",\"session\":") + .map_err(|err| err.to_string())?; + serde_json::to_writer(&mut writer, &envelope.session) + .map_err(|err| format!("serialize replay export session: {err}"))?; + writer + .write_all(b",\"payload\":{\"events\":[\n") + .map_err(|err| err.to_string())?; + } + ReplayExportFormat::Markdown => {} + } + let summary = stream_replay_export_events(source_id, session_id, &mut writer, format)?; + let count = summary.event_count; + let first_created_at = summary.first_created_at; + let last_created_at = summary.last_created_at; + match format { + ReplayExportFormat::Json => { + writer.write_all(b"\n]\n").map_err(|err| err.to_string())? + } + ReplayExportFormat::OrgiiSessionJson => { + let envelope = orgii_envelope.expect("validated above"); + writer + .write_all(b"\n],\"specs\":") + .map_err(|err| err.to_string())?; + serde_json::to_writer(&mut writer, &envelope.specs) + .map_err(|err| format!("serialize replay export specs: {err}"))?; + let fallback_start = envelope + .session + .get("created_at") + .and_then(serde_json::Value::as_str) + .unwrap_or_default(); + let fallback_end = envelope + .session + .get("updated_at") + .and_then(serde_json::Value::as_str) + .unwrap_or(fallback_start); + writer + .write_all(b",\"timeRange\":{\"start\":") + .map_err(|err| err.to_string())?; + serde_json::to_writer( + &mut writer, + first_created_at.as_deref().unwrap_or(fallback_start), + ) + .map_err(|err| format!("serialize replay export time range: {err}"))?; + writer + .write_all(b",\"end\":") + .map_err(|err| err.to_string())?; + serde_json::to_writer( + &mut writer, + last_created_at.as_deref().unwrap_or(fallback_end), + ) + .map_err(|err| format!("serialize replay export time range: {err}"))?; + writer + .write_all(b"}},\"metadata\":{\"originalCategory\":") + .map_err(|err| err.to_string())?; + serde_json::to_writer(&mut writer, &envelope.original_category) + .map_err(|err| format!("serialize replay export category: {err}"))?; + writer + .write_all(format!(",\"eventCount\":{count}}}}}\n").as_bytes()) + .map_err(|err| err.to_string())?; + } + ReplayExportFormat::Markdown => {} + } + writer + .flush() + .map_err(|err| format!("flush replay export: {err}"))?; + let (bytes_written, sha256, mut inner) = writer.finish(); + inner + .flush() + .map_err(|err| format!("flush replay export file: {err}"))?; + inner + .get_ref() + .sync_all() + .map_err(|err| format!("sync replay export file: {err}"))?; + drop(inner); + atomic_replace_export(&temporary, &destination).map_err(|err| { + format!( + "publish replay export {} -> {}: {err}", + temporary.display(), + destination.display() + ) + })?; + if let Ok(directory) = fs::File::open(parent) { + let _ = directory.sync_all(); + } + Ok(ReplayExportResult { + destination_path: destination.to_string_lossy().into_owned(), + bytes_written, + event_count: count, + sha256, + }) + })(); + if result.is_err() { + let _ = fs::remove_file(&temporary); + } + result +} + +pub(super) fn sanitize_export_file_name( + suggested_file_name: Option<&str>, + session_id: &str, + format: ReplayExportFormat, +) -> String { + let fallback_extension = match format { + ReplayExportFormat::Json | ReplayExportFormat::OrgiiSessionJson => "json", + ReplayExportFormat::Markdown => "md", + }; + let fallback = format!( + "session-{}.{}", + session_id + .chars() + .map(|character| { + if character.is_ascii_alphanumeric() || matches!(character, '-' | '_') { + character + } else { + '-' + } + }) + .take(96) + .collect::(), + fallback_extension + ); + let Some(file_name) = suggested_file_name + .and_then(|value| std::path::Path::new(value).file_name()) + .and_then(|value| value.to_str()) + else { + return fallback; + }; + let sanitized = file_name + .chars() + .map(|character| { + if character.is_control() + || matches!( + character, + '/' | '\\' | '?' | '%' | '*' | ':' | '|' | '"' | '<' | '>' + ) + { + '-' + } else { + character + } + }) + .take(160) + .collect::(); + let sanitized = sanitized.trim_matches([' ', '.']).to_string(); + if sanitized.is_empty() { + fallback + } else { + sanitized + } +} + +fn choose_replay_export_destination( + app_handle: &AppHandle, + suggested_file_name: Option<&str>, + session_id: &str, + format: ReplayExportFormat, +) -> Result, String> { + let file_name = sanitize_export_file_name(suggested_file_name, session_id, format); + let dialog = app_handle.dialog().file().set_file_name(file_name); + let dialog = match format { + ReplayExportFormat::Json | ReplayExportFormat::OrgiiSessionJson => { + dialog.add_filter("JSON", &["json"]) + } + ReplayExportFormat::Markdown => dialog.add_filter("Markdown", &["md", "markdown"]), + }; + dialog + .blocking_save_file() + .map(|path| { + path.into_path() + .map_err(|err| format!("resolve replay export destination: {err}")) + }) + .transpose() +} + +#[derive(Default)] +pub(super) struct ReplayExportSummary { + event_count: u64, + first_created_at: Option, + last_created_at: Option, +} + +impl ReplayExportSummary { + fn observe(&mut self, event: &SessionEvent) { + if self + .first_created_at + .as_ref() + .is_none_or(|first| event.created_at < *first) + { + self.first_created_at = Some(event.created_at.clone()); + } + if self + .last_created_at + .as_ref() + .is_none_or(|last| event.created_at > *last) + { + self.last_created_at = Some(event.created_at.clone()); + } + self.event_count = self.event_count.saturating_add(1); + } +} + +pub(super) fn prepare_stream_replay_snapshot( + conn: &mut Connection, + source: ImportedHistorySourceId, + imported_session_id: &str, + limits: ReplayLimits, +) -> Result { + replay::prepare_pinned_scan(conn, source, imported_session_id, limits) +} + +pub(super) fn prepare_sessions_stream_replay_snapshot( + source: ImportedHistorySourceId, + imported_session_id: &str, + limits: ReplayLimits, +) -> Result { + with_sessions_replay_writer("replay stream preparation", |conn| { + prepare_stream_replay_snapshot(conn, source, imported_session_id, limits) + }) +} + +/// Export-only source scan. Unlike the cloud spool iterator, this deliberately +/// keeps events compact and gives the writer a range reader for each deferred +/// payload. A single 10 MiB output therefore never becomes a 10 MiB `String`. +pub(super) fn stream_replay_export_events( + source_id: &str, + session_id: &str, + writer: &mut impl Write, + format: ReplayExportFormat, +) -> Result { + let mut summary = ReplayExportSummary::default(); + match resolve_secondary_consumer_target(source_id, session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => { + let limits = ReplayLimits { + max_turns: 10, + max_events: STREAM_BATCH_MAX_EVENTS, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }; + let prepared = + prepare_sessions_stream_replay_snapshot(source, &imported_session_id, limits)?; + let expected_generation = prepared.generation; + let expected_revision = prepared.revision; + let mut payload_conn = database::db::get_connection() + .map_err(|err| format!("open replay export payload DB: {err}"))?; + let mut after_sequence = -1_i64; + loop { + let scan = with_sessions_replay_writer("replay export scan", |conn| { + replay::scan_window_after_generation( + conn, + source, + &imported_session_id, + &expected_generation, + expected_revision, + after_sequence, + limits, + ) + })?; + let next_sequence = scan.cursor.through_sequence; + let has_more = scan.has_more; + let (events, _) = normalize_indexed_chunks( + scan.chunks, + session_id, + source.as_str(), + &scan.cursor.generation, + ); + for event in events { + write_replay_export_event( + writer, + &event, + format, + summary.event_count, + |payload_ref, offset| { + replay::read_payload_range( + &mut payload_conn, + source, + &imported_session_id, + &scan.cursor.generation, + payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id), + &payload_ref.field_path, + offset, + Some(EXPORT_PAYLOAD_RANGE_BYTES), + ) + }, + )?; + summary.observe(&event); + } + if !has_more { + break; + } + if next_sequence <= after_sequence { + return Err("Replay export cursor did not advance".to_string()); + } + after_sequence = next_sequence; + } + let final_scan = with_sessions_replay_writer("replay export finalization", |conn| { + replay::scan_window_after( + conn, + source, + &imported_session_id, + after_sequence, + ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }, + ) + })?; + validate_stream_replay_cursor( + &expected_generation, + expected_revision, + &final_scan.cursor, + "finalizing replay export", + )?; + } + ResolvedReplayTarget::CollaborationSnapshot => { + let conn = database::db::get_connection() + .map_err(|err| format!("open collaboration replay export DB: {err}"))?; + let state = collaboration_snapshot_state(&conn, session_id)?; + let limits = ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: STREAM_BATCH_MAX_EVENTS, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }; + let mut after_sequence = -1_i64; + loop { + let indexed = query_collaboration_snapshot_events( + &conn, + session_id, + &state.generation, + after_sequence, + state.max_sequence.saturating_add(1), + limits, + false, + )?; + if indexed.is_empty() { + break; + } + let next_sequence = indexed + .last() + .map_or(after_sequence, |(sequence, _)| *sequence); + for (_, event) in indexed { + write_replay_export_event( + writer, + &event, + format, + summary.event_count, + |payload_ref, offset| { + collaboration_snapshot_payload_range_from_conn( + &conn, + session_id, + &state.generation, + payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id), + &payload_ref.field_path, + offset, + EXPORT_PAYLOAD_RANGE_BYTES, + ) + }, + )?; + summary.observe(&event); + } + if next_sequence >= state.max_sequence { + break; + } + if next_sequence <= after_sequence { + return Err("Collaboration replay export cursor did not advance".to_string()); + } + after_sequence = next_sequence; + } + let current = collaboration_snapshot_state(&conn, session_id)?; + validate_query_apply_version( + &state.generation, + state.revision, + ¤t.generation, + current.revision, + )?; + } + ResolvedReplayTarget::ManagedChunkStore => { + let conn = database::db::get_connection() + .map_err(|err| format!("open managed replay export DB: {err}"))?; + stream_managed_chunk_replay_events_from_conn( + &conn, + session_id, + "exporting managed replay", + |event, read_payload| { + write_replay_export_event( + writer, + event, + format, + summary.event_count, + |payload_ref, offset| read_payload(payload_ref, offset), + )?; + summary.observe(event); + Ok(()) + }, + )?; + } + ResolvedReplayTarget::NotReady => { + return Err("Managed native transcript is not bound yet".to_string()) + } + } + Ok(summary) +} + +pub(super) fn write_replay_export_event( + writer: &mut impl Write, + event: &SessionEvent, + format: ReplayExportFormat, + event_index: u64, + mut read_payload: impl FnMut(&PayloadRef, u64) -> Result, +) -> Result<(), String> { + match format { + ReplayExportFormat::Json | ReplayExportFormat::OrgiiSessionJson => { + if event_index > 0 { + writer.write_all(b",\n").map_err(|err| err.to_string())?; + } + write_hydrated_event_json(writer, event, &mut read_payload) + } + ReplayExportFormat::Markdown => { + write_markdown_event_streaming(writer, event, &mut read_payload) + } + } +} + +#[derive(Clone, Copy)] +pub(super) enum PayloadMarkerEncoding { + JsonString, + RawJson, +} + +pub(super) struct PayloadMarker { + encoded_marker: Vec, + payload_ref: PayloadRef, + encoding: PayloadMarkerEncoding, +} + +pub(super) fn write_hydrated_event_json( + writer: &mut impl Write, + event: &SessionEvent, + read_payload: &mut dyn FnMut(&PayloadRef, u64) -> Result, +) -> Result<(), String> { + if event.payload_refs.is_empty() { + return serde_json::to_writer(writer, event) + .map_err(|err| format!("serialize replay export event: {err}")); + } + + let mut compact = event.clone(); + let payload_refs = std::mem::take(&mut compact.payload_refs); + let nonce = uuid::Uuid::new_v4().simple().to_string(); + let mut markers = Vec::with_capacity(payload_refs.len().saturating_mul(2)); + for (index, payload_ref) in payload_refs.into_iter().enumerate() { + let field_marker = format!("__ORGII_REPLAY_{nonce}_FIELD_{index}__"); + if set_event_payload_marker(&mut compact, &payload_ref.field_path, &field_marker) { + markers.push(PayloadMarker { + encoded_marker: serde_json::to_vec(&field_marker) + .map_err(|err| format!("encode replay export marker: {err}"))?, + encoding: match payload_ref.replay_encoding { + Some(PayloadRefEncoding::JsonValue) => PayloadMarkerEncoding::RawJson, + Some(PayloadRefEncoding::Utf8Text) => PayloadMarkerEncoding::JsonString, + None if matches!(payload_ref.field_path.as_str(), "args" | "result") => { + PayloadMarkerEncoding::RawJson + } + None => PayloadMarkerEncoding::JsonString, + }, + payload_ref: payload_ref.clone(), + }); + } + if compact.display_text == payload_ref.preview { + let display_marker = format!("__ORGII_REPLAY_{nonce}_DISPLAY_{index}__"); + compact.display_text = display_marker.clone(); + markers.push(PayloadMarker { + encoded_marker: serde_json::to_vec(&display_marker) + .map_err(|err| format!("encode replay display marker: {err}"))?, + payload_ref, + encoding: PayloadMarkerEncoding::JsonString, + }); + } + } + + let encoded = serde_json::to_vec(&compact) + .map_err(|err| format!("serialize compact replay export event: {err}"))?; + let mut position = 0_usize; + while position < encoded.len() { + let next = markers + .iter() + .enumerate() + .filter_map(|(index, marker)| { + find_bytes(&encoded[position..], &marker.encoded_marker) + .map(|offset| (position + offset, index)) + }) + .min_by_key(|(offset, _)| *offset); + let Some((offset, marker_index)) = next else { + writer + .write_all(&encoded[position..]) + .map_err(|err| format!("write compact replay export event: {err}"))?; + break; + }; + writer + .write_all(&encoded[position..offset]) + .map_err(|err| format!("write replay export marker prefix: {err}"))?; + let marker = &markers[marker_index]; + stream_payload_to_writer(writer, &marker.payload_ref, marker.encoding, read_payload)?; + position = offset.saturating_add(marker.encoded_marker.len()); + } + Ok(()) +} + +pub(super) fn find_bytes(haystack: &[u8], needle: &[u8]) -> Option { + if needle.is_empty() { + return Some(0); + } + haystack + .windows(needle.len()) + .position(|candidate| candidate == needle) +} + +pub(super) fn set_event_payload_marker( + event: &mut SessionEvent, + field_path: &str, + marker: &str, +) -> bool { + match field_path { + "args" => { + event.args = serde_json::Value::String(marker.to_string()); + true + } + "result" => { + event.result = serde_json::Value::String(marker.to_string()); + true + } + _ => { + let Some((root, path)) = field_path.split_once('.') else { + return false; + }; + let value = match root { + "args" => &mut event.args, + "result" => &mut event.result, + _ => return false, + }; + set_json_string_path(value, path, marker.to_string()); + json_value_at_path(value, path).is_some_and(|value| value.as_str() == Some(marker)) + } + } +} + +pub(super) fn json_value_at_path<'a>( + mut value: &'a serde_json::Value, + path: &str, +) -> Option<&'a serde_json::Value> { + for segment in path.split('.') { + value = match value { + serde_json::Value::Object(object) => object.get(segment)?, + serde_json::Value::Array(array) => array.get(segment.parse::().ok()?)?, + _ => return None, + }; + } + Some(value) +} + +pub(super) fn stream_payload_to_writer( + writer: &mut impl Write, + payload_ref: &PayloadRef, + encoding: PayloadMarkerEncoding, + read_payload: &mut dyn FnMut(&PayloadRef, u64) -> Result, +) -> Result<(), String> { + if matches!(encoding, PayloadMarkerEncoding::JsonString) { + writer.write_all(b"\"").map_err(|err| err.to_string())?; + } + let mut offset = 0_u64; + loop { + let range = read_payload(payload_ref, offset)?; + match encoding { + PayloadMarkerEncoding::RawJson => writer + .write_all(range.text.as_bytes()) + .map_err(|err| format!("write raw replay export payload: {err}"))?, + PayloadMarkerEncoding::JsonString => { + let escaped = serde_json::to_vec(&range.text) + .map_err(|err| format!("escape replay export payload range: {err}"))?; + if escaped.len() < 2 { + return Err("Encoded replay payload range was not a JSON string".to_string()); + } + writer + .write_all(&escaped[1..escaped.len() - 1]) + .map_err(|err| format!("write escaped replay export payload: {err}"))?; + } + } + if range.eof { + break; + } + if range.next_offset <= offset { + return Err("Replay export payload cursor did not advance".to_string()); + } + offset = range.next_offset; + } + if matches!(encoding, PayloadMarkerEncoding::JsonString) { + writer.write_all(b"\"").map_err(|err| err.to_string())?; + } + Ok(()) +} + +pub(super) fn write_markdown_event_streaming( + writer: &mut impl Write, + event: &SessionEvent, + read_payload: &mut dyn FnMut(&PayloadRef, u64) -> Result, +) -> Result<(), String> { + let heading = match event.source { + EventSource::User => "**User**\n\n", + EventSource::Assistant if event.ui_canonical == "agent_message" => "**Assistant**\n\n", + EventSource::Assistant | EventSource::System => return Ok(()), + }; + let display_payload = event + .payload_refs + .iter() + .find(|payload_ref| event.display_text == payload_ref.preview); + if display_payload.is_none() && event.display_text.trim().is_empty() { + return Ok(()); + } + writer + .write_all(heading.as_bytes()) + .map_err(|err| format!("write replay Markdown heading: {err}"))?; + if let Some(payload_ref) = display_payload { + stream_payload_to_writer( + writer, + payload_ref, + PayloadMarkerEncoding::RawJson, + read_payload, + )?; + } else { + writer + .write_all(event.display_text.trim().as_bytes()) + .map_err(|err| format!("write replay Markdown event: {err}"))?; + } + writer + .write_all(b"\n\n---\n\n") + .map_err(|err| format!("finish replay Markdown event: {err}")) +} + +pub(super) struct HashingWriter { + inner: W, + digest: Sha256, + bytes: u64, +} + +impl HashingWriter { + pub(super) fn new(inner: W) -> Self { + Self { + inner, + digest: Sha256::new(), + bytes: 0, + } + } + + pub(super) fn finish(self) -> (u64, String, W) { + let hash = self.digest.finalize(); + (self.bytes, format!("{hash:x}"), self.inner) + } +} + +impl Write for HashingWriter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + let written = self.inner.write(bytes)?; + self.digest.update(&bytes[..written]); + self.bytes = self.bytes.saturating_add(written as u64); + Ok(written) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.inner.flush() + } +} + +/// Ask the native OS for an export destination, then stream the current +/// generation directly to that authorized file. The renderer can suggest only +/// a file name; it never supplies a filesystem path. +#[tauri::command] +pub async fn external_replay_stream_export( + app_handle: AppHandle, + source_id: String, + session_id: String, + suggested_file_name: Option, + format: ReplayExportFormat, + orgii_envelope: Option, +) -> Result, String> { + let Some(destination_path) = choose_replay_export_destination( + &app_handle, + suggested_file_name.as_deref(), + &session_id, + format, + )? + else { + return Ok(None); + }; + let result = tokio::task::spawn_blocking(move || { + stream_replay_export_to_path( + &source_id, + &session_id, + &destination_path, + format, + orgii_envelope.as_ref(), + ) + }) + .await + .map_err(|err| format!("join replay export task: {err}"))??; + schedule_replay_cache_prune(); + Ok(Some(result)) +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/handoff.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/handoff.rs new file mode 100644 index 000000000..9f5ae80ba --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/handoff.rs @@ -0,0 +1,398 @@ +use super::*; + +pub(super) enum ResolvedReplayWindow { + Imported(ReplayChunkWindow), + ManagedChunks(ReplayChunkWindow), + CollaborationSnapshot(ExternalReplayWindow), + NotReady, +} + +pub(super) enum ResolvedReplayDelta { + Imported(ReplayChunkDelta), + ManagedChunks(ReplayChunkDelta), + CollaborationSnapshot(ExternalReplayDelta), + NotReady, +} + +pub(super) fn load_external_replay_handoff( + source_id: &str, + session_id: &str, + source_name: &str, +) -> Result { + let source_name = source_name.trim(); + if source_name.is_empty() { + return Err("external replay handoff sourceName is required".to_string()); + } + if source_name.encode_utf16().count() > 200 { + return Err("external replay handoff sourceName exceeds 200 characters".to_string()); + } + if matches!( + resolve_target(source_id, session_id)?, + ResolvedReplayTarget::CollaborationSnapshot + ) { + return collect_collaboration_snapshot_handoff(session_id, source_name); + } + collect_external_replay_handoff(source_name, |before_sequence, turn_index, limits| { + load_replay_query_window( + source_id, + session_id, + before_sequence, + None, + turn_index, + limits, + ) + }) +} + +pub(super) fn collect_external_replay_handoff( + source_name: &str, + mut load_page: impl FnMut( + Option, + Option, + ReplayLimits, + ) -> Result, +) -> Result { + let mut before_sequence = None; + let mut requested_turn_index = None; + let mut generation: Option = None; + let mut revision: Option = None; + let mut remaining_bytes = EXTERNAL_REPLAY_HANDOFF_SCAN_BYTES; + let mut scanned_bytes = 0_u64; + let mut scanned_events = 0_u64; + let mut items = Vec::new(); + + while remaining_bytes > 0 && items.len() < EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS { + let page = load_page( + before_sequence, + requested_turn_index, + ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: replay::HARD_MAX_EVENTS, + max_ipc_bytes: remaining_bytes, + }, + )?; + let (window, imported) = match page { + ResolvedReplayWindow::Imported(window) => (window, true), + ResolvedReplayWindow::ManagedChunks(window) => (window, false), + ResolvedReplayWindow::CollaborationSnapshot(_) => { + return Err( + "collaboration snapshot handoff must use its direct bounded SQL fold" + .to_string(), + ) + } + ResolvedReplayWindow::NotReady => { + return Ok(ExternalReplayHandoff { + items: Vec::new(), + generation: "pending".to_string(), + scanned_bytes: 0, + scanned_events: 0, + }) + } + }; + if let Some(expected) = generation.as_deref() { + if expected != window.cursor.generation { + return Err(format!( + "External replay changed generation while building Fork handoff: expected {expected}, found {}; retry the Fork from the new generation", + window.cursor.generation + )); + } + } else { + generation = Some(window.cursor.generation.clone()); + } + if let Some(expected) = revision { + if expected != window.cursor.revision { + return Err(format!( + "External replay changed revision while building Fork handoff: expected {expected}, found {}; retry the Fork from a consistent replay snapshot", + window.cursor.revision + )); + } + } else { + revision = Some(window.cursor.revision); + } + + let compact_bytes = compact_handoff_page_bytes(&window.chunks); + let page_bytes = compact_bytes.max(window.stats.ipc_bytes as usize); + if page_bytes > remaining_bytes { + return Err(format!( + "External replay handoff page exceeded its remaining {remaining_bytes} byte scan budget" + )); + } + remaining_bytes = remaining_bytes.saturating_sub(page_bytes); + scanned_bytes = scanned_bytes.saturating_add(page_bytes as u64); + scanned_events = scanned_events.saturating_add(window.chunks.len() as u64); + + let oldest_sequence = window.chunks.iter().map(|chunk| chunk.sequence).min(); + let oldest_turn_index = window + .turn_headers + .iter() + .map(|header| header.turn_index) + .min(); + let mut page_items = window + .chunks + .iter() + .filter_map(|indexed| handoff_item_from_chunk(&indexed.chunk, source_name)) + .collect::>(); + page_items.append(&mut items); + if page_items.len() > EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS { + page_items.drain(..page_items.len() - EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS); + } + items = page_items; + + let has_older_compact_turn = imported && oldest_turn_index.is_some_and(|index| index > 0); + if items.len() >= EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS + || (!window.has_older && !has_older_compact_turn) + || remaining_bytes == 0 + { + break; + } + if let Some(oldest_turn_index) = oldest_turn_index.filter(|index| imported && *index > 0) { + let next_turn_index = oldest_turn_index - 1; + if requested_turn_index.is_some_and(|previous| next_turn_index >= previous) { + return Err( + "External replay handoff turn cursor did not advance to an older turn" + .to_string(), + ); + } + requested_turn_index = Some(next_turn_index); + before_sequence = None; + continue; + } + let Some(next_before) = oldest_sequence else { + return Err("External replay handoff hasOlder page contained no events".to_string()); + }; + if before_sequence.is_some_and(|previous| next_before >= previous) { + return Err( + "External replay handoff cursor did not advance to older events".to_string(), + ); + } + before_sequence = Some(next_before); + requested_turn_index = None; + } + + Ok(ExternalReplayHandoff { + items, + generation: generation.unwrap_or_else(|| "empty".to_string()), + scanned_bytes, + scanned_events, + }) +} + +pub(super) fn collect_collaboration_snapshot_handoff( + session_id: &str, + source_name: &str, +) -> Result { + let conn = database::db::get_connection() + .map_err(|error| format!("open collaboration handoff DB: {error}"))?; + let state = collaboration_snapshot_state(&conn, session_id)?; + let mut upper_exclusive = i64::MAX; + let mut remaining_bytes = EXTERNAL_REPLAY_HANDOFF_SCAN_BYTES; + let mut scanned_bytes = 0_u64; + let mut scanned_events = 0_u64; + let mut items = Vec::new(); + while remaining_bytes > 0 && items.len() < EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS { + let page = query_collaboration_snapshot_events( + &conn, + session_id, + &state.generation, + -1, + upper_exclusive, + ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: replay::HARD_MAX_EVENTS, + max_ipc_bytes: remaining_bytes, + }, + true, + )?; + if page.is_empty() { + break; + } + let page_bytes = page.iter().try_fold(0_usize, |total, (_, event)| { + serde_json::to_vec(event) + .map(|bytes| total.saturating_add(bytes.len())) + .map_err(|error| format!("measure collaboration handoff page: {error}")) + })?; + if page_bytes > remaining_bytes { + return Err(format!( + "Collaboration replay handoff page exceeded its remaining {remaining_bytes} byte scan budget" + )); + } + remaining_bytes = remaining_bytes.saturating_sub(page_bytes); + scanned_bytes = scanned_bytes.saturating_add(page_bytes as u64); + scanned_events = scanned_events.saturating_add(page.len() as u64); + let oldest_sequence = page.first().map(|(sequence, _)| *sequence).unwrap_or(-1); + let mut page_items = page + .iter() + .filter_map(|(_, event)| { + let chunk = ActivityChunk { + chunk_id: event.id.clone(), + session_id: event.session_id.clone(), + action_type: event.action_type.clone(), + function: event.function_name.clone(), + args: event.args.clone(), + result: event.result.clone(), + thread_id: event.thread_id.clone(), + process_id: event.process_id.clone(), + created_at: event.created_at.clone(), + broadcast_only: false, + }; + handoff_item_from_chunk(&chunk, source_name) + }) + .collect::>(); + page_items.append(&mut items); + if page_items.len() > EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS { + page_items.drain(..page_items.len() - EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS); + } + items = page_items; + if items.len() >= EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS || oldest_sequence < 0 { + break; + } + let has_older = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM events + WHERE session_id=?1 AND history_sequence(0), + ) + .map_err(|error| format!("query older collaboration handoff rows: {error}"))? + != 0; + if !has_older { + break; + } + if oldest_sequence >= upper_exclusive { + return Err("Collaboration replay handoff cursor did not advance".to_string()); + } + upper_exclusive = oldest_sequence; + } + let current = collaboration_snapshot_state(&conn, session_id)?; + validate_query_apply_version( + &state.generation, + state.revision, + ¤t.generation, + current.revision, + )?; + Ok(ExternalReplayHandoff { + items, + generation: state.generation, + scanned_bytes, + scanned_events, + }) +} + +pub(super) fn compact_handoff_page_bytes(chunks: &[ReplayIndexedChunk]) -> usize { + chunks.iter().fold(0_usize, |total, indexed| { + total + .saturating_add(serde_json::to_vec(&indexed.chunk).map_or(0, |bytes| bytes.len())) + .saturating_add(serde_json::to_vec(&indexed.payloads).map_or(0, |bytes| bytes.len())) + }) +} + +pub(super) fn handoff_item_from_chunk(chunk: &ActivityChunk, source_name: &str) -> Option { + let action_type = chunk.action_type.as_str(); + let function = chunk.function.as_str(); + if action_type.contains("thinking") + || action_type.contains("reasoning") + || matches!(function, "thinking" | "thinking_delta" | "reasoning") + { + return None; + } + + let result_text = handoff_text_value(&chunk.result); + let args_text = handoff_text_value(&chunk.args); + let content = result_text.as_deref().or(args_text.as_deref()); + let item = if matches!(action_type, "user" | "user_message") + || matches!(function, "user" | "user_message") + { + content.map(|text| format!("User: {text}")) + } else if matches!( + action_type, + "assistant" | "assistant_message" | "llm_response" + ) || matches!( + function, + "agent_message" | "assistant" | "assistant_message" + ) { + content.map(|text| format!("Assistant: {text}")) + } else if action_type.contains("tool") { + let mut lines = vec![ + format!("[Imported {source_name} action]"), + format!( + "Tool: {}", + if function.is_empty() { + "unknown_tool" + } else { + function + } + ), + ]; + if let Some(args) = args_text { + lines.push(format!("Input: {args}")); + } + if let Some(result) = result_text { + lines.push(format!("Result at that time: {result}")); + } + Some(lines.join("\n")) + } else { + content.map(|text| format!("Assistant context: {text}")) + }?; + Some(truncate_handoff_utf16(&item)) +} + +pub(super) fn handoff_text_value(value: &serde_json::Value) -> Option { + match value { + serde_json::Value::String(text) => { + let trimmed = text.trim(); + (!trimmed.is_empty()).then(|| trimmed.to_string()) + } + serde_json::Value::Array(values) => { + let joined = values + .iter() + .filter_map(handoff_text_value) + .collect::>() + .join("\n"); + (!joined.is_empty()).then_some(joined) + } + serde_json::Value::Object(object) => ["text", "content", "message", "output", "summary"] + .into_iter() + .find_map(|key| object.get(key).and_then(handoff_text_value)), + _ => None, + } +} + +pub(super) fn truncate_handoff_utf16(text: &str) -> String { + if text.encode_utf16().count() <= EXTERNAL_REPLAY_HANDOFF_MAX_TEXT_UTF16 { + return text.to_string(); + } + let content_budget = EXTERNAL_REPLAY_HANDOFF_MAX_TEXT_UTF16.saturating_sub(1); + let mut output = String::new(); + let mut units = 0_usize; + for character in text.chars() { + let next = units.saturating_add(character.len_utf16()); + if next > content_budget { + break; + } + output.push(character); + units = next; + } + output.push('…'); + output +} + +/// Build the last usable imported-history handoff items entirely in Rust. +/// +/// Like `external_replay_query_window`, this may bring ORGII's rebuildable +/// compact index up to date. It intentionally has no AppHandle, EventStore +/// State, episode id, watcher lease, notification, or request-token side +/// effect. Cross-page reads are pinned to one source generation and revision. +#[tauri::command] +pub async fn external_replay_handoff( + source_id: String, + session_id: String, + source_name: String, +) -> Result { + let handoff = tokio::task::spawn_blocking(move || { + load_external_replay_handoff(&source_id, &session_id, &source_name) + }) + .await + .map_err(|err| format!("join pure replay handoff task: {err}"))??; + schedule_replay_cache_prune(); + Ok(handoff) +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/managed_chunks.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/managed_chunks.rs new file mode 100644 index 000000000..ea42eff53 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/managed_chunks.rs @@ -0,0 +1,1397 @@ +use super::*; + +pub(super) fn managed_chunk_generation( + conn: &rusqlite::Connection, + session_id: &str, +) -> Result { + let epoch = conn + .query_row( + "SELECT epoch FROM code_session_history_mutations WHERE session_id=?1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .optional() + .map_err(|err| format!("read managed history generation: {err}"))? + .unwrap_or(0); + Ok(format!("chunks-{epoch}")) +} + +pub(super) fn managed_chunk_stream_cursor( + conn: &rusqlite::Connection, + session_id: &str, +) -> Result<(String, i64), String> { + let generation = managed_chunk_generation(conn, session_id)?; + let max_sequence = conn + .query_row( + "SELECT COALESCE(MAX(sequence), -1) FROM code_session_chunks WHERE session_id=?1", + [session_id], + |row| row.get::<_, i64>(0), + ) + .map_err(|err| format!("read managed replay stream revision: {err}"))?; + Ok((generation, max_sequence)) +} + +pub(super) fn validate_managed_chunk_stream_cursor( + expected_generation: &str, + expected_max_sequence: i64, + current: &(String, i64), + operation: &str, +) -> Result<(), String> { + if current.0 == expected_generation && current.1 == expected_max_sequence { + return Ok(()); + } + Err(format!( + "Managed replay changed while {operation}: expected {expected_generation}@{expected_max_sequence}, found {}@{}; retry from the new replay cursor", + current.0, current.1 + )) +} + +/// Shared readerless managed-CLI scan used by both streamed export and Cloud +/// spooling. Keeping the database cursor and the bounded payload reader on the +/// same connection makes it impossible for either consumer to reintroduce a +/// full args/result materialization behind a separate code path. +pub(super) fn stream_managed_chunk_replay_events_from_conn( + conn: &rusqlite::Connection, + session_id: &str, + operation: &str, + mut consume: impl FnMut( + &SessionEvent, + &mut dyn FnMut(&PayloadRef, u64) -> Result, + ) -> Result<(), String>, +) -> Result { + let (generation, max_sequence) = managed_chunk_stream_cursor(conn, session_id)?; + let limits = ReplayLimits { + max_turns: 10, + max_events: STREAM_BATCH_MAX_EVENTS, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }; + let mut after_sequence = -1_i64; + loop { + if managed_chunk_generation(conn, session_id)? != generation { + return Err(format!( + "Managed replay changed while {operation}; retry from the new generation" + )); + } + let chunks = query_managed_chunks( + conn, + session_id, + "sequence > ?2", + after_sequence, + Some(max_sequence), + limits, + false, + )?; + if chunks.is_empty() { + break; + } + let next_sequence = chunks.last().map_or(after_sequence, |chunk| chunk.sequence); + let (events, _) = normalize_indexed_chunks( + chunks, + session_id, + MANAGED_CLI_REPLAY_TARGET_ID, + &generation, + ); + for event in &events { + let mut read_payload = |payload_ref: &PayloadRef, offset: u64| { + managed_chunk_payload_range_from_conn( + conn, + session_id, + payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id), + &payload_ref.field_path, + offset, + EXPORT_PAYLOAD_RANGE_BYTES, + ) + }; + consume(event, &mut read_payload)?; + } + if next_sequence <= after_sequence { + return Err(format!( + "Managed replay cursor did not advance while {operation}" + )); + } + after_sequence = next_sequence; + } + validate_managed_chunk_stream_cursor( + &generation, + max_sequence, + &managed_chunk_stream_cursor(conn, session_id)?, + operation, + )?; + Ok(generation) +} + +pub(super) fn managed_chunk_open_window( + session_id: &str, + limits: ReplayLimits, +) -> Result { + managed_chunk_read_window(session_id, None, None, None, limits) +} + +pub(super) fn managed_chunk_read_window( + session_id: &str, + before_sequence: Option, + turn_id: Option<&str>, + turn_index: Option, + limits: ReplayLimits, +) -> Result { + let conn = + database::db::get_connection().map_err(|err| format!("open managed chunks DB: {err}"))?; + managed_chunk_read_window_from_conn( + &conn, + session_id, + before_sequence, + turn_id, + turn_index, + limits, + ) +} + +pub(super) fn managed_chunk_read_window_from_conn( + conn: &rusqlite::Connection, + session_id: &str, + before_sequence: Option, + turn_id: Option<&str>, + turn_index: Option, + limits: ReplayLimits, +) -> Result { + let limits = limits.bounded(); + let exact_turn = turn_id.is_some() || turn_index.is_some(); + let (generation, source_revision) = managed_chunk_stream_cursor(conn, session_id)?; + let source_revision = source_revision.max(0) as u64; + let (total_event_count, total_turn_count) = managed_chunk_total_counts(conn, session_id)?; + if total_event_count == 0 { + return Ok(ReplayChunkWindow { + cursor: ReplayCursor { + source_id: MANAGED_CLI_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation, + revision: source_revision, + through_sequence: -1, + }, + chunks: Vec::new(), + window_start_sequence: None, + turn_headers: Vec::new(), + total_turn_count: 0, + total_event_count: 0, + has_older: false, + stats: ReplayStats::default(), + }); + } + + let newest_turn_index = if let Some(turn_id) = turn_id { + Some(managed_chunk_turn_index_for_id( + conn, + session_id, + turn_id, + total_turn_count, + )?) + } else if let Some(turn_index) = turn_index { + if turn_index < 0 || turn_index >= total_turn_count as i64 { + return Err(format!( + "Managed replay turn index is no longer available: {turn_index}" + )); + } + Some(turn_index) + } else { + managed_chunk_latest_turn_index_before( + conn, + session_id, + before_sequence.unwrap_or(i64::MAX), + )? + }; + + let Some(newest_turn_index) = newest_turn_index else { + return Ok(ReplayChunkWindow { + cursor: ReplayCursor { + source_id: MANAGED_CLI_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation, + revision: source_revision, + through_sequence: -1, + }, + chunks: Vec::new(), + window_start_sequence: None, + turn_headers: Vec::new(), + total_turn_count, + total_event_count, + has_older: false, + stats: ReplayStats::default(), + }); + }; + let oldest_turn_index = if exact_turn { + newest_turn_index + } else { + newest_turn_index + .saturating_sub(limits.max_turns.saturating_sub(1) as i64) + .max(0) + }; + let mut turn_headers = Vec::with_capacity( + newest_turn_index + .saturating_sub(oldest_turn_index) + .saturating_add(1) as usize, + ); + for index in oldest_turn_index..=newest_turn_index { + turn_headers.push(managed_chunk_turn_header_at_index( + conn, + session_id, + index, + total_turn_count, + )?); + } + let start_sequence = turn_headers + .first() + .map(|header| header.start_sequence) + .unwrap_or(0); + let mut end_sequence = turn_headers + .last() + .and_then(|header| header.end_sequence) + .unwrap_or(start_sequence); + if let Some(before_sequence) = before_sequence { + end_sequence = end_sequence.min(before_sequence.saturating_sub(1)); + } + let (mut chunks, window_start_sequence) = if exact_turn { + read_managed_exact_turn_chunks( + conn, + session_id, + turn_headers + .first() + .ok_or_else(|| "Managed replay exact turn header disappeared".to_string())?, + limits, + )? + } else { + let chunks = query_managed_chunks( + conn, + session_id, + "sequence >= ?2", + start_sequence, + Some(end_sequence), + limits, + true, + )?; + let window_start_sequence = chunks.first().map(|chunk| chunk.sequence); + (chunks, window_start_sequence) + }; + for chunk in &mut chunks { + if let Some(header) = turn_headers.iter().find(|header| { + chunk.sequence >= header.start_sequence + && chunk.sequence <= header.end_sequence.unwrap_or(header.start_sequence) + }) { + chunk.turn_index = header.turn_index; + } + } + let through_sequence = chunks.last().map_or(-1, |chunk| chunk.sequence); + let has_older = if let Some(continuation_sequence) = window_start_sequence { + conn.query_row( + "SELECT EXISTS( + SELECT 1 FROM code_session_chunks + WHERE session_id=?1 AND sequence(0), + ) + .map_err(|err| format!("query older managed replay events: {err}"))? + != 0 + } else { + false + }; + Ok(ReplayChunkWindow { + cursor: ReplayCursor { + source_id: MANAGED_CLI_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation, + // `revision` identifies the source snapshot, while + // `through_sequence` identifies this page. Older pages from the + // same snapshot must therefore keep one stable revision. + revision: source_revision, + through_sequence, + }, + chunks, + window_start_sequence, + turn_headers, + total_turn_count, + total_event_count, + has_older, + stats: ReplayStats::default(), + }) +} + +fn read_managed_exact_turn_chunks( + conn: &rusqlite::Connection, + session_id: &str, + turn: &ReplayTurnHeader, + limits: ReplayLimits, +) -> Result<(Vec, Option), String> { + let end_sequence = turn.end_sequence.unwrap_or(turn.start_sequence); + let anchor_limits = ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: limits.max_ipc_bytes, + }; + let mut chunks = query_managed_chunks( + conn, + session_id, + "sequence >= ?2", + turn.start_sequence, + Some(end_sequence), + anchor_limits, + false, + )?; + let Some(anchor_sequence) = chunks.first().map(|chunk| chunk.sequence) else { + return Ok((chunks, None)); + }; + let anchor_bytes = chunks.first().map_or(0, managed_indexed_chunk_bytes); + let Some(tail_limits) = limits.after_exact_turn_anchor(anchor_bytes) else { + return Ok((chunks, Some(anchor_sequence))); + }; + let mut tail = query_managed_chunks( + conn, + session_id, + "sequence > ?2", + anchor_sequence, + Some(end_sequence), + tail_limits, + true, + )?; + let tail_start_sequence = tail.first().map(|chunk| chunk.sequence); + let has_gap = if let Some(tail_start_sequence) = tail_start_sequence { + conn.query_row( + "SELECT EXISTS( + SELECT 1 FROM code_session_chunks + WHERE session_id=?1 AND sequence>?2 AND sequence(0), + ) + .map_err(|err| format!("query exact managed replay turn gap: {err}"))? + != 0 + } else { + false + }; + let window_start_sequence = if has_gap { + tail_start_sequence + } else { + Some(anchor_sequence) + }; + chunks.append(&mut tail); + Ok((chunks, window_start_sequence)) +} + +fn managed_indexed_chunk_bytes(indexed: &ReplayIndexedChunk) -> usize { + serde_json::to_vec(&indexed.chunk) + .map_or(0, |bytes| bytes.len()) + .saturating_add(serde_json::to_vec(&indexed.payloads).map_or(0, |bytes| bytes.len())) +} + +pub(super) fn managed_chunk_total_counts( + conn: &rusqlite::Connection, + session_id: &str, +) -> Result<(u64, u64), String> { + let (event_count, user_turn_count) = conn + .query_row( + "SELECT COUNT(*), + SUM(CASE WHEN function='user_message' THEN 1 ELSE 0 END) + FROM code_session_chunks WHERE session_id=?1", + [session_id], + |row| Ok((row.get::<_, i64>(0)?, row.get::<_, Option>(1)?)), + ) + .map_err(|err| format!("count managed replay turns: {err}"))?; + let event_count = event_count.max(0) as u64; + let turn_count = if event_count == 0 { + 0 + } else { + user_turn_count.unwrap_or(0).max(1) as u64 + }; + Ok((event_count, turn_count)) +} + +pub(super) fn managed_chunk_user_turn_anchor_at_index( + conn: &rusqlite::Connection, + session_id: &str, + turn_index: i64, +) -> Result, String> { + conn.query_row( + "SELECT sequence,chunk_id,created_at + FROM code_session_chunks + WHERE session_id=?1 AND function='user_message' + ORDER BY sequence ASC LIMIT 1 OFFSET ?2", + rusqlite::params![session_id, turn_index], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .optional() + .map_err(|err| format!("query managed replay turn {turn_index}: {err}")) +} + +pub(super) fn managed_chunk_turn_header_at_index( + conn: &rusqlite::Connection, + session_id: &str, + turn_index: i64, + total_turn_count: u64, +) -> Result { + if turn_index < 0 || turn_index >= total_turn_count as i64 { + return Err(format!( + "Managed replay turn index is no longer available: {turn_index}" + )); + } + let anchor = managed_chunk_user_turn_anchor_at_index(conn, session_id, turn_index)?; + let (start_sequence, turn_id, started_at) = match anchor { + Some(anchor) => anchor, + None if turn_index == 0 && total_turn_count == 1 => conn + .query_row( + "SELECT sequence,chunk_id,created_at FROM code_session_chunks + WHERE session_id=?1 ORDER BY sequence ASC LIMIT 1", + [session_id], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .map_err(|err| format!("query managed replay fallback turn: {err}"))?, + None => { + return Err(format!( + "Managed replay turn index is no longer available: {turn_index}" + )) + } + }; + let next_start = managed_chunk_user_turn_anchor_at_index(conn, session_id, turn_index + 1)? + .map(|anchor| anchor.0); + let (end_sequence, ended_at, event_count) = conn + .query_row( + "SELECT MAX(sequence), + (SELECT tail.created_at FROM code_session_chunks AS tail + WHERE tail.session_id=?1 AND tail.sequence>=?2 + AND (?3 IS NULL OR tail.sequence=?2 + AND (?3 IS NULL OR sequence>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, i64>(2)?, + )) + }, + ) + .map_err(|err| format!("summarize managed replay turn {turn_index}: {err}"))?; + Ok(ReplayTurnHeader { + turn_id, + turn_index, + start_sequence, + end_sequence, + started_at, + ended_at, + event_count: event_count.max(0) as u64, + }) +} + +pub(super) fn managed_chunk_turn_index_for_id( + conn: &rusqlite::Connection, + session_id: &str, + turn_id: &str, + total_turn_count: u64, +) -> Result { + let row = conn + .query_row( + "SELECT sequence,function FROM code_session_chunks + WHERE session_id=?1 AND chunk_id=?2", + rusqlite::params![session_id, turn_id], + |row| Ok((row.get::<_, i64>(0)?, row.get::<_, String>(1)?)), + ) + .optional() + .map_err(|err| format!("query managed replay turn id {turn_id}: {err}"))? + .ok_or_else(|| format!("Managed replay turn is no longer available: {turn_id}"))?; + if row.1 != "user_message" && total_turn_count != 1 { + return Err(format!( + "Managed replay turn is no longer available: {turn_id}" + )); + } + conn.query_row( + "SELECT COUNT(*) FROM code_session_chunks + WHERE session_id=?1 AND function='user_message' AND sequence(0), + ) + .map_err(|err| format!("resolve managed replay turn id {turn_id}: {err}")) +} + +pub(super) fn managed_chunk_latest_turn_index_before( + conn: &rusqlite::Connection, + session_id: &str, + ceiling: i64, +) -> Result, String> { + let latest_user_sequence = conn + .query_row( + "SELECT sequence FROM code_session_chunks + WHERE session_id=?1 AND function='user_message' AND sequence(0), + ) + .optional() + .map_err(|err| format!("resolve managed replay window turn: {err}"))?; + if let Some(sequence) = latest_user_sequence { + let preceding_users = conn + .query_row( + "SELECT COUNT(*) FROM code_session_chunks + WHERE session_id=?1 AND function='user_message' AND sequence(0), + ) + .map_err(|err| format!("index managed replay window turn: {err}"))?; + return Ok(Some(preceding_users.max(0))); + } + let has_rows = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM code_session_chunks + WHERE session_id=?1 AND sequence(0), + ) + .map_err(|err| format!("query managed replay fallback window: {err}"))? + != 0; + Ok(has_rows.then_some(0)) +} + +pub(super) fn managed_chunk_poll_delta( + session_id: &str, + cursor: &ReplayCursor, + limits: ReplayLimits, +) -> Result { + let limits = limits.bounded(); + let conn = + database::db::get_connection().map_err(|err| format!("open managed chunks DB: {err}"))?; + let generation = managed_chunk_generation(&conn, session_id)?; + if generation != cursor.generation { + return Ok(ReplayChunkDelta { + cursor: ReplayCursor { + source_id: MANAGED_CLI_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation, + revision: 0, + through_sequence: -1, + }, + chunks: Vec::new(), + removed_event_ids: Vec::new(), + reset_required: true, + stats: ReplayStats::default(), + }); + } + let chunks = query_managed_chunks( + &conn, + session_id, + "sequence > ?2", + cursor.through_sequence, + None, + limits, + false, + )?; + let through_sequence = chunks + .last() + .map_or(cursor.through_sequence, |chunk| chunk.sequence); + Ok(ReplayChunkDelta { + cursor: ReplayCursor { + source_id: MANAGED_CLI_REPLAY_TARGET_ID.to_string(), + session_id: session_id.to_string(), + generation, + revision: through_sequence.max(0) as u64, + through_sequence, + }, + chunks, + removed_event_ids: Vec::new(), + reset_required: false, + stats: ReplayStats::default(), + }) +} + +pub(super) fn query_managed_chunks( + conn: &rusqlite::Connection, + session_id: &str, + predicate: &str, + sequence: i64, + through_sequence: Option, + limits: ReplayLimits, + newest_first: bool, +) -> Result, String> { + let order = if newest_first { "DESC" } else { "ASC" }; + let upper_bound = through_sequence + .map(|_| " AND sequence <= ?3") + .unwrap_or_default(); + let normal_preview_read = + replay::NORMAL_PAYLOAD_PREVIEW_BYTES.saturating_add(MANAGED_CHUNK_UTF8_BOUNDARY_BYTES); + let shell_preview_read = + replay::SHELL_PAYLOAD_PREVIEW_BYTES.saturating_add(MANAGED_CHUNK_UTF8_BOUNDARY_BYTES); + let sql = format!( + "SELECT sequence, chunk_id, action_type, function, + length(CAST(args_json AS BLOB)), json_valid(args_json), + CASE WHEN length(CAST(args_json AS BLOB)) <= {MANAGED_CHUNK_INLINE_JSON_MAX_BYTES} + THEN args_json END, + CASE WHEN length(CAST(args_json AS BLOB)) > {MANAGED_CHUNK_INLINE_JSON_MAX_BYTES} + THEN substr( + CAST(args_json AS BLOB), + 1, + CASE WHEN function IN ('run_command_line', 'shell') + THEN {shell_preview_read} ELSE {normal_preview_read} END + ) END, + length(CAST(result_json AS BLOB)), json_valid(result_json), + CASE WHEN length(CAST(result_json AS BLOB)) <= {MANAGED_CHUNK_INLINE_JSON_MAX_BYTES} + THEN result_json END, + CASE WHEN length(CAST(result_json AS BLOB)) > {MANAGED_CHUNK_INLINE_JSON_MAX_BYTES} + THEN CASE WHEN function IN ('run_command_line', 'shell') + THEN substr( + CAST(result_json AS BLOB), + MAX(1, length(CAST(result_json AS BLOB)) - {shell_preview_read} + 1), + {shell_preview_read} + ) + ELSE substr(CAST(result_json AS BLOB), 1, {normal_preview_read}) + END END, + thread_id, process_id, created_at + FROM code_session_chunks WHERE session_id=?1 AND {predicate}{upper_bound} + ORDER BY sequence {order} LIMIT {}", + limits.max_events + ); + let mut stmt = conn + .prepare(&sql) + .map_err(|err| format!("prepare bounded managed chunks: {err}"))?; + let mut rows = match through_sequence { + Some(through_sequence) => { + stmt.query(rusqlite::params![session_id, sequence, through_sequence]) + } + None => stmt.query(rusqlite::params![session_id, sequence]), + } + .map_err(|err| format!("query bounded managed chunks: {err}"))?; + let mut chunks = Vec::new(); + let mut bytes = 0usize; + while let Some(row) = rows + .next() + .map_err(|err| format!("read bounded managed chunk: {err}"))? + { + let chunk_id: String = row.get(1).map_err(|err| err.to_string())?; + let function: String = row.get(3).map_err(|err| err.to_string())?; + let args_inline: Option = row.get(6).map_err(|err| err.to_string())?; + let args_root_preview: Option> = row.get(7).map_err(|err| err.to_string())?; + let result_inline: Option = row.get(10).map_err(|err| err.to_string())?; + let result_root_preview: Option> = row.get(11).map_err(|err| err.to_string())?; + for field_bytes in [ + args_inline.as_ref().map(String::len), + args_root_preview.as_ref().map(Vec::len), + result_inline.as_ref().map(String::len), + result_root_preview.as_ref().map(Vec::len), + ] + .into_iter() + .flatten() + { + observe_managed_chunk_db_json_field(field_bytes); + } + let shell = function == "run_command_line" || function == "shell"; + let preview_limit = if shell { + replay::SHELL_PAYLOAD_PREVIEW_BYTES + } else { + replay::NORMAL_PAYLOAD_PREVIEW_BYTES + }; + let (args, mut payloads) = load_managed_chunk_json_field( + conn, + session_id, + &chunk_id, + ManagedChunkJsonColumn::Args, + row.get::<_, i64>(4).map_err(|err| err.to_string())?, + row.get::<_, i64>(5).map_err(|err| err.to_string())? != 0, + args_inline, + args_root_preview, + preview_limit, + false, + &function, + )?; + let (result, result_payloads) = load_managed_chunk_json_field( + conn, + session_id, + &chunk_id, + ManagedChunkJsonColumn::Result, + row.get::<_, i64>(8).map_err(|err| err.to_string())?, + row.get::<_, i64>(9).map_err(|err| err.to_string())? != 0, + result_inline, + result_root_preview, + preview_limit, + shell, + &function, + )?; + payloads.extend(result_payloads); + let chunk = ActivityChunk { + chunk_id, + session_id: session_id.to_string(), + action_type: row.get(2).map_err(|err| err.to_string())?, + function, + args, + result, + thread_id: row.get(12).map_err(|err| err.to_string())?, + process_id: row.get(13).map_err(|err| err.to_string())?, + created_at: row.get(14).map_err(|err| err.to_string())?, + broadcast_only: false, + }; + let indexed = ReplayIndexedChunk { + sequence: row.get(0).map_err(|err| err.to_string())?, + turn_index: 0, + chunk, + payloads, + }; + let next_bytes = managed_indexed_chunk_bytes(&indexed); + if !chunks.is_empty() && bytes.saturating_add(next_bytes) > limits.max_ipc_bytes { + break; + } + if chunks.is_empty() && next_bytes > limits.max_ipc_bytes { + return Err(format!( + "Managed replay event {} exceeds the {} byte compact window budget", + indexed.chunk.chunk_id, limits.max_ipc_bytes + )); + } + bytes = bytes.saturating_add(next_bytes); + chunks.push(indexed); + } + if newest_first { + chunks.reverse(); + } + Ok(chunks) +} + +#[derive(Clone, Copy)] +pub(super) enum ManagedChunkJsonColumn { + Args, + Result, +} + +impl ManagedChunkJsonColumn { + fn column_name(self) -> &'static str { + match self { + Self::Args => "args_json", + Self::Result => "result_json", + } + } + + fn root(self) -> &'static str { + match self { + Self::Args => "args", + Self::Result => "result", + } + } +} + +#[derive(Clone, Copy)] +pub(super) enum ManagedChunkJsonContainer { + Object, + Array, +} + +#[derive(Clone)] +pub(super) enum ManagedChunkJsonPathSegment { + Key(String), + Index(usize), +} + +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +pub(super) fn load_managed_chunk_json_field( + conn: &rusqlite::Connection, + session_id: &str, + chunk_id: &str, + column: ManagedChunkJsonColumn, + total_bytes: i64, + valid_json: bool, + inline: Option, + root_preview: Option>, + preview_limit: usize, + tail: bool, + function_name: &str, +) -> Result<(serde_json::Value, Vec), String> { + let root = column.root(); + let total_bytes = total_bytes.max(0) as u64; + if let Some(inline) = inline { + let mut value = serde_json::from_str(&inline) + .map_err(|err| format!("decode managed chunk {root}: {err}"))?; + let mut payloads = Vec::new(); + compact_managed_chunk_json_value( + &mut value, + root, + true, + preview_limit, + tail, + chunk_id, + managed_chunk_payload_kind(function_name, root), + &mut payloads, + ); + return Ok((value, payloads)); + } + if !valid_json { + return Err(format!( + "decode managed chunk {root}: invalid JSON in oversized {root} payload" + )); + } + + if let Some(projected) = project_managed_chunk_json_field( + conn, + session_id, + chunk_id, + column, + total_bytes, + preview_limit, + tail, + function_name, + )? { + return Ok(projected); + } + + let root_preview = root_preview + .ok_or_else(|| format!("bounded managed chunk {root} preview is missing for {chunk_id}"))?; + let preview = if tail { + utf8_tail_preview_bytes(&root_preview, preview_limit)? + } else { + utf8_head_preview_bytes(&root_preview, preview_limit)? + }; + Ok(( + serde_json::Value::String(preview.clone()), + vec![ReplayPayloadDescriptor { + field_path: root.to_string(), + kind: managed_chunk_payload_kind(function_name, root), + encoding: ReplayPayloadEncoding::JsonValue, + body_projection: Some(ReplayPayloadBodyProjection { + field_path: root.to_string(), + text: preview.clone(), + truncated: true, + }), + spans: Vec::new(), + total_bytes, + source_ordinal: None, + source_key: Some(chunk_id.to_string()), + }], + )) +} + +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +pub(super) fn project_managed_chunk_json_field( + conn: &rusqlite::Connection, + session_id: &str, + chunk_id: &str, + column: ManagedChunkJsonColumn, + root_total_bytes: u64, + preview_limit: usize, + tail: bool, + function_name: &str, +) -> Result)>, String> { + let column_name = column.column_name(); + let read_bytes = preview_limit + .saturating_add(MANAGED_CHUNK_UTF8_BOUNDARY_BYTES) + .min(i64::MAX as usize) as i64; + let sql = format!( + "SELECT tree.id, tree.parent, + length(CAST(tree.key AS BLOB)), + substr(CAST(tree.key AS BLOB), 1, {}), + tree.type, + CASE WHEN tree.atom IS NULL THEN 0 + ELSE length(CAST(tree.atom AS BLOB)) END, + CASE WHEN tree.atom IS NULL THEN NULL + WHEN tree.type='text' AND ?3<>0 THEN substr( + CAST(tree.atom AS BLOB), + MAX(1, length(CAST(tree.atom AS BLOB)) - ?4 + 1), + ?4 + ) + ELSE substr(CAST(tree.atom AS BLOB), 1, ?4) END + FROM code_session_chunks AS chunks, + json_tree(chunks.{column_name}) AS tree + WHERE chunks.session_id=?1 AND chunks.chunk_id=?2 + ORDER BY tree.id + LIMIT {}", + MANAGED_CHUNK_JSON_KEY_MAX_BYTES.saturating_add(MANAGED_CHUNK_UTF8_BOUNDARY_BYTES), + MANAGED_CHUNK_JSON_PROJECTION_MAX_NODES.saturating_add(1) + ); + let mut stmt = conn + .prepare(&sql) + .map_err(|err| format!("prepare managed {column_name} projection: {err}"))?; + let mut rows = stmt + .query(rusqlite::params![ + session_id, + chunk_id, + if tail { 1_i64 } else { 0_i64 }, + read_bytes + ]) + .map_err(|err| format!("query managed {column_name} projection: {err}"))?; + let mut root_value: Option = None; + let mut containers = + HashMap::, ManagedChunkJsonContainer)>::new(); + let mut payloads = Vec::new(); + let mut projected_bytes = 0_usize; + let mut node_count = 0_usize; + while let Some(row) = rows + .next() + .map_err(|err| format!("read managed {column_name} projection: {err}"))? + { + node_count = node_count.saturating_add(1); + if node_count > MANAGED_CHUNK_JSON_PROJECTION_MAX_NODES { + return Ok(None); + } + let id: i64 = row.get(0).map_err(|err| err.to_string())?; + let parent: Option = row.get(1).map_err(|err| err.to_string())?; + let key_bytes_total = row + .get::<_, Option>(2) + .map_err(|err| err.to_string())? + .unwrap_or_default() + .max(0) as usize; + let key_bytes: Option> = row.get(3).map_err(|err| err.to_string())?; + let node_type: String = row.get(4).map_err(|err| err.to_string())?; + let atom_total = row.get::<_, i64>(5).map_err(|err| err.to_string())?.max(0) as u64; + let atom_bytes: Option> = row.get(6).map_err(|err| err.to_string())?; + if let Some(key_bytes) = &key_bytes { + observe_managed_chunk_db_json_field(key_bytes.len()); + projected_bytes = projected_bytes.saturating_add(key_bytes.len()); + } + if let Some(atom_bytes) = &atom_bytes { + observe_managed_chunk_db_json_field(atom_bytes.len()); + projected_bytes = projected_bytes.saturating_add(atom_bytes.len()); + } + if projected_bytes > MANAGED_CHUNK_JSON_PROJECTION_MAX_BYTES + || key_bytes_total > MANAGED_CHUNK_JSON_KEY_MAX_BYTES + { + return Ok(None); + } + + let path = if let Some(parent) = parent { + let Some((parent_path, parent_kind)) = containers.get(&parent) else { + return Ok(None); + }; + let key_bytes = key_bytes.as_deref().unwrap_or_default(); + let key = std::str::from_utf8(key_bytes).ok(); + let segment = match parent_kind { + ManagedChunkJsonContainer::Object => { + let Some(key) = key else { + return Ok(None); + }; + if key.is_empty() + || key.contains('.') + || key.bytes().all(|byte| byte.is_ascii_digit()) + { + return Ok(None); + } + ManagedChunkJsonPathSegment::Key(key.to_string()) + } + ManagedChunkJsonContainer::Array => { + let Some(index) = key.and_then(|key| key.parse::().ok()) else { + return Ok(None); + }; + ManagedChunkJsonPathSegment::Index(index) + } + }; + let mut path = parent_path.clone(); + path.push(segment); + path + } else { + if root_value.is_some() { + return Ok(None); + } + Vec::new() + }; + let field_path = managed_chunk_field_path(column.root(), &path); + let kind = if field_path.to_ascii_lowercase().contains("diff") { + ReplayPayloadKind::ToolDiff + } else { + managed_chunk_payload_kind(function_name, column.root()) + }; + let (value, container) = match node_type.as_str() { + "object" => ( + serde_json::Value::Object(serde_json::Map::new()), + Some(ManagedChunkJsonContainer::Object), + ), + "array" => ( + serde_json::Value::Array(Vec::new()), + Some(ManagedChunkJsonContainer::Array), + ), + "text" => { + let Some(atom_bytes) = atom_bytes.as_deref() else { + return Ok(None); + }; + let text = if atom_total > preview_limit as u64 { + let preview = if tail { + utf8_tail_preview_bytes(atom_bytes, preview_limit)? + } else { + utf8_head_preview_bytes(atom_bytes, preview_limit)? + }; + payloads.push(ReplayPayloadDescriptor { + field_path: field_path.clone(), + kind, + encoding: if path.is_empty() { + ReplayPayloadEncoding::JsonValue + } else { + ReplayPayloadEncoding::Utf8Text + }, + body_projection: path.is_empty().then(|| ReplayPayloadBodyProjection { + field_path: field_path.clone(), + text: preview.clone(), + truncated: true, + }), + spans: Vec::new(), + total_bytes: if path.is_empty() { + root_total_bytes + } else { + atom_total + }, + source_ordinal: None, + source_key: Some(chunk_id.to_string()), + }); + preview + } else { + String::from_utf8(atom_bytes.to_vec()) + .map_err(|err| format!("decode managed {field_path}: {err}"))? + }; + (serde_json::Value::String(text), None) + } + "integer" | "real" => { + let Some(atom_bytes) = atom_bytes.as_deref() else { + return Ok(None); + }; + let Ok(value) = serde_json::from_slice::(atom_bytes) else { + return Ok(None); + }; + (value, None) + } + "true" => (serde_json::Value::Bool(true), None), + "false" => (serde_json::Value::Bool(false), None), + "null" => (serde_json::Value::Null, None), + _ => return Ok(None), + }; + if path.is_empty() { + root_value = Some(value); + } else { + let Some(root) = root_value.as_mut() else { + return Ok(None); + }; + if !insert_managed_chunk_json_node(root, &path, value) { + return Ok(None); + } + } + if let Some(container) = container { + containers.insert(id, (path, container)); + } + } + Ok(root_value.map(|value| (value, payloads))) +} + +pub(super) fn managed_chunk_field_path(root: &str, path: &[ManagedChunkJsonPathSegment]) -> String { + let mut output = root.to_string(); + for segment in path { + output.push('.'); + match segment { + ManagedChunkJsonPathSegment::Key(key) => output.push_str(key), + ManagedChunkJsonPathSegment::Index(index) => output.push_str(&index.to_string()), + } + } + output +} + +pub(super) fn insert_managed_chunk_json_node( + root: &mut serde_json::Value, + path: &[ManagedChunkJsonPathSegment], + value: serde_json::Value, +) -> bool { + let Some((last, parents)) = path.split_last() else { + *root = value; + return true; + }; + let mut current = root; + for segment in parents { + current = match (current, segment) { + (serde_json::Value::Object(object), ManagedChunkJsonPathSegment::Key(key)) => { + let Some(next) = object.get_mut(key) else { + return false; + }; + next + } + (serde_json::Value::Array(array), ManagedChunkJsonPathSegment::Index(index)) => { + let Some(next) = array.get_mut(*index) else { + return false; + }; + next + } + _ => return false, + }; + } + match (current, last) { + (serde_json::Value::Object(object), ManagedChunkJsonPathSegment::Key(key)) => { + object.insert(key.clone(), value); + true + } + (serde_json::Value::Array(array), ManagedChunkJsonPathSegment::Index(index)) => { + if *index > array.len() { + return false; + } + if *index == array.len() { + array.push(value); + } else { + array[*index] = value; + } + true + } + _ => false, + } +} + +pub(super) fn observe_managed_chunk_db_json_field(_bytes: usize) { + #[cfg(test)] + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.fetch_max(_bytes, Ordering::Relaxed); +} + +pub(super) fn utf8_head_preview_bytes(bytes: &[u8], max_bytes: usize) -> Result { + let mut end = bytes.len().min(max_bytes); + while end > 0 && std::str::from_utf8(&bytes[..end]).is_err() { + end -= 1; + } + let text = std::str::from_utf8(&bytes[..end]) + .map_err(|err| format!("decode managed payload preview: {err}"))?; + Ok(format!("{text}\n… [payload truncated]")) +} + +pub(super) fn utf8_tail_preview_bytes(bytes: &[u8], max_bytes: usize) -> Result { + let mut start = bytes.len().saturating_sub(max_bytes); + while start < bytes.len() && std::str::from_utf8(&bytes[start..]).is_err() { + start = start.saturating_add(1); + } + let text = std::str::from_utf8(&bytes[start..]) + .map_err(|err| format!("decode managed payload preview: {err}"))?; + Ok(format!("[payload truncated] …\n{text}")) +} + +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +pub(super) fn compact_managed_chunk_json_value( + value: &mut serde_json::Value, + field_path: &str, + is_root: bool, + limit: usize, + tail: bool, + source_key: &str, + kind: ReplayPayloadKind, + payloads: &mut Vec, +) { + match value { + serde_json::Value::String(text) if text.len() > limit => { + let total_bytes = if is_root { + serde_json::to_string(text).map_or(text.len(), |encoded| encoded.len()) + } else { + text.len() + } as u64; + let preview = if tail { + utf8_tail_preview(text, limit) + } else { + utf8_head_preview(text, limit) + }; + *text = preview.clone(); + payloads.push(ReplayPayloadDescriptor { + field_path: field_path.to_string(), + kind, + encoding: if is_root { + ReplayPayloadEncoding::JsonValue + } else { + ReplayPayloadEncoding::Utf8Text + }, + body_projection: is_root.then(|| ReplayPayloadBodyProjection { + field_path: field_path.to_string(), + text: preview.clone(), + truncated: true, + }), + spans: Vec::new(), + total_bytes, + source_ordinal: None, + source_key: Some(source_key.to_string()), + }); + } + serde_json::Value::Array(items) => { + for (index, item) in items.iter_mut().enumerate() { + compact_managed_chunk_json_value( + item, + &format!("{field_path}.{index}"), + false, + limit, + tail, + source_key, + kind, + payloads, + ); + } + } + serde_json::Value::Object(object) => { + for (key, item) in object { + // Dot-separated paths mirror the existing PayloadRef wire + // contract. Provider payload keys containing dots are rare; + // keep such values inline rather than publish an ambiguous + // range locator. + if key.contains('.') { + continue; + } + let child_path = format!("{field_path}.{key}"); + let child_kind = if child_path.to_ascii_lowercase().contains("diff") { + ReplayPayloadKind::ToolDiff + } else { + kind + }; + compact_managed_chunk_json_value( + item, + &child_path, + false, + limit, + tail, + source_key, + child_kind, + payloads, + ); + } + } + _ => {} + } +} + +pub(super) fn managed_chunk_payload_kind(function_name: &str, root: &str) -> ReplayPayloadKind { + match function_name { + "user" | "user_message" => ReplayPayloadKind::UserMessage, + "assistant" | "assistant_message" | "agent_message" => ReplayPayloadKind::AssistantContent, + "thinking" | "reasoning" => ReplayPayloadKind::Reasoning, + _ if root == "args" => ReplayPayloadKind::ToolArguments, + _ => ReplayPayloadKind::ToolOutput, + } +} + +pub(super) fn utf8_head_preview(text: &str, max_bytes: usize) -> String { + let mut end = text.len().min(max_bytes); + while end > 0 && !text.is_char_boundary(end) { + end -= 1; + } + format!("{}\n… [payload truncated]", &text[..end]) +} + +pub(super) fn utf8_tail_preview(text: &str, max_bytes: usize) -> String { + let mut start = text.len().saturating_sub(max_bytes); + while start < text.len() && !text.is_char_boundary(start) { + start += 1; + } + format!("[payload truncated] …\n{}", &text[start..]) +} + +pub(super) fn managed_chunk_payload_range( + session_id: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: Option, +) -> Result { + let max_bytes = max_bytes + .unwrap_or(replay::DEFAULT_PAYLOAD_RANGE_BYTES) + .clamp(1, replay::HARD_MAX_PAYLOAD_RANGE_BYTES); + let conn = + database::db::get_connection().map_err(|err| format!("open managed chunks DB: {err}"))?; + managed_chunk_payload_range_from_conn( + &conn, session_id, event_id, field_path, offset, max_bytes, + ) +} + +pub(super) fn managed_chunk_payload_range_from_conn( + conn: &rusqlite::Connection, + session_id: &str, + event_id: &str, + field_path: &str, + offset: u64, + max_bytes: usize, +) -> Result { + let (root, path) = field_path.split_once('.').unwrap_or((field_path, "")); + let column = match root { + "args" => "args_json", + "result" => "result_json", + _ => return Err("fieldPath must start with args or result".to_string()), + }; + let start = offset.min(i64::MAX as u64) as i64; + let read_bytes = max_bytes.saturating_add(4).min(i64::MAX as usize) as i64; + let (total_bytes, bytes): (Option, Option>) = if path.is_empty() { + let sql = format!( + "SELECT length(CAST({column} AS BLOB)), + substr(CAST({column} AS BLOB), ?3, ?4) + FROM code_session_chunks WHERE session_id=?1 AND chunk_id=?2" + ); + conn.query_row( + &sql, + rusqlite::params![session_id, event_id, start.saturating_add(1), read_bytes], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + } else { + let json_path = replay_sqlite_json_path(path)?; + let sql = format!( + "SELECT length(CAST(json_extract({column}, ?3) AS BLOB)), + substr(CAST(json_extract({column}, ?3) AS BLOB), ?4, ?5) + FROM code_session_chunks WHERE session_id=?1 AND chunk_id=?2" + ); + conn.query_row( + &sql, + rusqlite::params![ + session_id, + event_id, + json_path, + start.saturating_add(1), + read_bytes + ], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + } + .map_err(|err| format!("load managed payload range: {err}"))?; + let total_bytes = total_bytes + .ok_or_else(|| format!("managed payload field not found: {field_path}"))? + .max(0) as u64; + let bytes = bytes.unwrap_or_default(); + observe_managed_chunk_db_json_field(bytes.len()); + let mut take = max_bytes.min(bytes.len()); + while take > 0 && std::str::from_utf8(&bytes[..take]).is_err() { + take -= 1; + } + if take == 0 && !bytes.is_empty() { + return Err(format!( + "managed payload range starts inside invalid UTF-8: {field_path} at {offset}" + )); + } + let text = String::from_utf8(bytes[..take].to_vec()) + .map_err(|err| format!("decode managed payload range: {err}"))?; + let next_offset = offset.saturating_add(take as u64).min(total_bytes); + Ok(ReplayPayloadRange { + event_id: event_id.to_string(), + field_path: field_path.to_string(), + offset: offset.min(total_bytes), + next_offset, + eof: next_offset >= total_bytes, + total_bytes, + text, + }) +} + +pub(super) fn replay_sqlite_json_path(path: &str) -> Result { + let mut output = "$".to_string(); + for segment in path.split('.') { + if segment.is_empty() { + return Err("managed payload path contains an empty segment".to_string()); + } + if segment.bytes().all(|byte| byte.is_ascii_digit()) { + output.push('['); + output.push_str(segment); + output.push(']'); + } else { + output.push_str(".\""); + for character in segment.chars() { + match character { + '\\' => output.push_str("\\\\"), + '"' => output.push_str("\\\""), + _ => output.push(character), + } + } + output.push('"'); + } + } + Ok(output) +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/mod.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/mod.rs new file mode 100644 index 000000000..c3d42d597 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/mod.rs @@ -0,0 +1,389 @@ +//! Thin Tauri bridge for imported/managed-CLI bounded replay. +//! +//! Source indexing stays in `orgtrack_core`; this module owns the only +//! dependency on the app EventStore. Native SDE Agent cache/set/merge paths +//! are untouched. + +use std::collections::{HashMap, HashSet}; +use std::fs; +use std::io::{BufWriter, Write}; +use std::path::PathBuf; +#[cfg(test)] +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::{Mutex, OnceLock}; +use std::time::{Duration, Instant}; + +use agent_core::tools::impls::coding::exec::{ + external_replay::external_shell_inline_segments, + shell_replay::{ + ShellReplayFrame, ShellReplayRange, ShellReplayStream, SHELL_REPLAY_FORMAT_VERSION, + SHELL_REPLAY_FRAME_MAX_BYTES, SHELL_REPLAY_PREVIEW_BYTES, SHELL_REPLAY_RANGE_MAX_BYTES, + }, +}; +use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _}; +use core_types::activity::ActivityChunk; +use core_types::session_event::{ + PayloadRefEncoding, ShellReplayBookmark, ShellReplayRef, ShellReplayState, ShellReplayStatus, +}; +use flate2::write::GzEncoder; +use flate2::Compression; +use orgtrack_core::sources::imported_history::replay::{ + self, ImportedHistorySourceId, ReplayChunkDelta, ReplayChunkWindow, ReplayCursor, + ReplayIndexedChunk, ReplayLimits, ReplayPayloadBodyProjection, ReplayPayloadDescriptor, + ReplayPayloadEncoding, ReplayPayloadKind, ReplayPayloadRange, ReplayStats, ReplayTurnHeader, +}; +use rusqlite::{params, Connection, DatabaseName, OptionalExtension, Transaction}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use tauri::{AppHandle, State}; + +use crate::agent_sessions::event_pipeline::ingestion::{self, types::RawActivityChunk}; +use crate::agent_sessions::event_pipeline::store::EventStore; +use crate::agent_sessions::event_pipeline::types::{EventSource, PayloadRef, SessionEvent}; + +#[cfg(test)] +use super::replay_cloud_wire::REPLAY_ATTACHMENT_V2_MAGIC as CLOUD_ATTACHMENT_V2_MAGIC; +use super::{ + event_conversion::cached_event_to_session_event, + external_replay_cache::schedule_replay_cache_prune, + external_replay_watcher, + replay_cloud_wire::{ + encode_replay_attachment_v2_frame, + ReplayAttachmentV2FrameHeader as CloudAttachmentFrameHeader, CLOUD_PAGE_MAX_SEGMENTS, + CLOUD_SEGMENT_WIRE_MAX_BYTES, + REPLAY_ATTACHMENT_CHUNK_BYTES as CLOUD_ATTACHMENT_CHUNK_BYTES, + }, + schedule_notify, EventStoreState, +}; + +pub const MANAGED_CLI_REPLAY_TARGET_ID: &str = "managed_cli"; +/// ORGII-owned collaboration snapshots persisted under `imported-session-*`. +/// This is intentionally outside `ImportedHistorySourceId`: it is not a +/// vendor adapter and must not change the exhaustive 15-source registry. +pub const COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID: &str = "collaboration_snapshot"; +const COLLABORATION_SNAPSHOT_SESSION_PREFIX: &str = "imported-session-"; +const COLLABORATION_SNAPSHOT_FORK_PREFIX: &str = "agentsession-"; +const COLLABORATION_SNAPSHOT_DRIVER_VERSION: u32 = 1; +// One replay response is capped at 4 MiB. Keep a separately bounded resident +// budget for the latest turn plus the selected older turn and its ±1 prefetch +// neighbours; otherwise the newest-suffix cap can evict the page immediately +// after `read_window` applies it. +const EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS: usize = 80; +const EXTERNAL_REPLAY_HANDOFF_MAX_TEXT_UTF16: usize = 1_200; +const EXTERNAL_REPLAY_HANDOFF_SCAN_BYTES: usize = 4 * 1024 * 1024; + +// Readerless managed-CLI rows live in ORGII's SQLite database, but their +// args/result columns can be arbitrarily large. Keep the database-to-Rust +// boundary independently bounded from the replay IPC budget: small JSON can +// still use the exact managed-chunk serde path, while large JSON is projected by +// SQLite into compact nodes and payload locators. +const MANAGED_CHUNK_INLINE_JSON_MAX_BYTES: usize = 64 * 1024; +const MANAGED_CHUNK_JSON_PROJECTION_MAX_BYTES: usize = 64 * 1024; +const MANAGED_CHUNK_JSON_PROJECTION_MAX_NODES: usize = 256; +const MANAGED_CHUNK_JSON_KEY_MAX_BYTES: usize = 1024; +const MANAGED_CHUNK_UTF8_BOUNDARY_BYTES: usize = 4; + +fn with_sessions_replay_writer( + context: &str, + operation: impl FnOnce(&mut Connection) -> Result, +) -> Result { + database::db::with_sessions_writer(|| { + let mut conn = database::db::get_connection() + .map_err(|error| format!("open {context} DB: {error}"))?; + operation(&mut conn) + }) +} + +/// Open a short-lived connection for visible replay work without queueing +/// behind the process-wide catalog-writer mutex. +/// +/// Source discovery holds that mutex while it walks every installed provider, +/// including intervals where it owns no SQLite write transaction. Foreground +/// Shell materialization and delta polls must not wait for that unrelated +/// filesystem work. Replay synchronization still begins an IMMEDIATE +/// transaction, so SQLite itself remains the serialization boundary and the +/// configured busy timeout continues to coordinate a second ORGII process. +fn with_foreground_replay_connection( + context: &str, + operation: impl FnOnce(&mut Connection) -> Result, +) -> Result { + let mut conn = database::db::get_connection() + .map_err(|error| format!("open foreground {context} DB: {error}"))?; + operation(&mut conn) +} + +#[cfg(test)] +static MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES: AtomicUsize = AtomicUsize::new(0); +#[cfg(test)] +static EXTERNAL_SHELL_MANIFEST_DB_PROBES: AtomicUsize = AtomicUsize::new(0); + +#[cfg(debug_assertions)] +const TEST_REPLAY_LIMITS: ReplayLimits = ReplayLimits { + max_turns: 10, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, +}; + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExternalReplayWindow { + pub cursor: ReplayCursor, + pub events: Vec, + /// Smallest source/index sequence scanned into this page. This differs + /// from the first turn header when a single large turn spans pages and + /// remains present even if normalization filters every scanned chunk. + pub window_start_sequence: Option, + pub turn_headers: Vec, + pub total_turn_count: u64, + pub total_event_count: u64, + pub has_older: bool, + pub stats: ReplayStats, + /// False until the backend watcher service is attached. Frontends must + /// keep their visible/active bounded polling fallback while this is false. + pub watcher_available: bool, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExternalReplayDelta { + pub cursor: ReplayCursor, + pub events: Vec, + pub removed_event_ids: Vec, + pub reset_required: bool, + pub stats: ReplayStats, + pub watcher_available: bool, +} + +/// Compact, prompt-ready imported-history handoff. Rust folds the source +/// pages directly so the renderer never receives a transient SessionEvent[] +/// transcript merely to create a new ORGII session. +#[derive(Debug, Clone, Serialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub struct ExternalReplayHandoff { + pub items: Vec, + pub generation: String, + pub scanned_bytes: u64, + pub scanned_events: u64, +} + +/// Debug HTTP probes use the same source resolution and bounded storage +/// drivers as the renderer, without requiring a Tauri `AppHandle`/`State`. +/// The helper deliberately does not apply to EventStore: callers only inspect +/// the returned window and can never resurrect the removed full-history IPC. +#[cfg(debug_assertions)] +pub(crate) async fn test_open_managed_replay_window( + session_id: String, +) -> Result { + let display_session_id = session_id.clone(); + tokio::task::spawn_blocking(move || { + let window = match resolve_target(MANAGED_CLI_REPLAY_TARGET_ID, &session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => open_foreground_imported_window(source, &imported_session_id, TEST_REPLAY_LIMITS) + .map(ResolvedReplayWindow::Imported)?, + ResolvedReplayTarget::CollaborationSnapshot => { + ResolvedReplayWindow::CollaborationSnapshot( + collaboration_snapshot_read_window_from_conn( + &database::db::get_connection() + .map_err(|err| format!("open collaboration replay DB: {err}"))?, + &session_id, + None, + None, + None, + TEST_REPLAY_LIMITS, + )?, + ) + } + ResolvedReplayTarget::ManagedChunkStore => { + ResolvedReplayWindow::ManagedChunks(managed_chunk_open_window( + &session_id, + replay_storage_limits_with_normalization_headroom(TEST_REPLAY_LIMITS), + )?) + } + ResolvedReplayTarget::NotReady => ResolvedReplayWindow::NotReady, + }; + let mut response = match window { + ResolvedReplayWindow::Imported(window) + | ResolvedReplayWindow::ManagedChunks(window) => { + normalize_window(window, &display_session_id) + } + ResolvedReplayWindow::CollaborationSnapshot(window) => window, + ResolvedReplayWindow::NotReady => { + not_ready_window(MANAGED_CLI_REPLAY_TARGET_ID, &display_session_id) + } + }; + remap_cursor( + &mut response.cursor, + MANAGED_CLI_REPLAY_TARGET_ID, + &display_session_id, + ); + response.events = persist_shell_replays_bounded( + MANAGED_CLI_REPLAY_TARGET_ID, + &display_session_id, + &response.cursor.generation, + response.cursor.revision, + response.events, + )?; + finalize_window_wire_budget(&mut response, TEST_REPLAY_LIMITS.max_ipc_bytes)?; + Ok(response) + }) + .await + .map_err(|err| format!("join test replay open task: {err}"))? +} + +#[derive(Debug, Clone, Copy, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ReplayExportFormat { + Json, + Markdown, + OrgiiSessionJson, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct OrgiiSessionExportEnvelope { + pub exported_at: String, + pub session: serde_json::Value, + pub original_category: String, + #[serde(default)] + pub specs: Vec, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ReplayExportResult { + pub destination_path: String, + pub bytes_written: u64, + pub event_count: u64, + pub sha256: String, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExternalReplayCloudManifest { + pub token: String, + pub generation: String, + pub total_count: u64, + pub frozen_event_count: u64, + pub tail_event_count: u64, + pub frozen_chain_hash: String, + pub tail_hash: Option, +} + +/// One already encoded cloud segment. Rust owns canonical serialization, +/// hashing and gzip so the renderer never materializes a full transcript (or +/// a large event) merely to forward it to the cloud RPC. +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExternalReplayCloudSegment { + pub payload_gz: String, + pub event_count: u64, + pub segment_hash: String, + /// Exact JSON bytes of this object after the caller adds a worst-case seq. + /// Every emitted segment is therefore within the actual RPC wire budget, + /// not merely an approximate pre-gzip event-size budget. + pub wire_bytes: u64, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExternalReplayCloudBatch { + pub segments: Vec, + pub start_event_index: u64, + pub next_event_index: u64, + /// Physical frozen-row cursor. One logical V2 event can span many rows, + /// including zero-event continuation rows, so event indexes alone cannot + /// advance a bounded IPC batch. + pub start_segment_index: u64, + pub next_segment_index: u64, + pub eof: bool, + pub serialized_bytes: u64, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExternalReplayCloudPrefixHash { + pub event_count: u64, + pub frozen_chain_hash: String, +} + +#[derive(Debug)] +struct CloudSpoolEntry { + path: PathBuf, + manifest: ExternalReplayCloudManifest, + last_used: Instant, + /// One owner lease is created by `prepare`; each in-flight read adds a + /// short-lived lease. The spool file is never removed while a reader owns + /// it, even if the renderer releases the token concurrently. + lease_count: u64, + owner_released: bool, +} + +#[derive(Debug)] +struct CloudSpoolReadLease { + token: String, + path: PathBuf, + manifest: ExternalReplayCloudManifest, +} + +impl Drop for CloudSpoolReadLease { + fn drop(&mut self) { + release_cloud_spool_read_lease(&self.token); + } +} + +enum ResolvedReplayTarget { + Imported { + source: ImportedHistorySourceId, + imported_session_id: String, + }, + CollaborationSnapshot, + ManagedChunkStore, + NotReady, +} + +const STREAM_BATCH_MAX_EVENTS: usize = CLOUD_PAGE_MAX_SEGMENTS; +const STREAM_BATCH_MAX_BYTES: usize = CLOUD_SEGMENT_WIRE_MAX_BYTES; +const EXPORT_PAYLOAD_RANGE_BYTES: usize = 64 * 1024; +const EXPORT_WRITER_BUFFER_BYTES: usize = 256 * 1024; +/// Base64 expands binary by 4/3. Keeping the gzip sink below this bound also +/// bounds the temporary binary and base64 buffers while the exact wire check +/// below accounts for JSON field overhead. +const CLOUD_SEGMENT_GZIP_MAX_BYTES: usize = 191 * 1024; +const CLOUD_SPOOL_TTL: Duration = Duration::from_secs(10 * 60); + +mod cloud; +mod collaboration; +mod export; +mod handoff; +mod managed_chunks; +mod payload; +mod request_guard; +mod shell; +mod target; +mod window; +mod wire_budget; + +pub use cloud::*; +use collaboration::*; +pub use export::*; +pub use handoff::*; +use managed_chunks::*; +pub use payload::*; +use request_guard::{ + apply_foreground_delta_if_current, apply_foreground_window_if_current, + apply_prewarm_window_if_current, begin_validated_foreground_request, + begin_validated_prewarm_request, is_current_prewarm_request, is_current_replay_request, + release_replay_watch_if_stale_episode, release_session_runtime_if_episode, ReplayWindowPublish, +}; +pub(super) use request_guard::{cancel_prewarm_requests, release_session_runtime}; +pub use shell::*; +use target::*; +use window::validate_query_apply_version; +pub use window::*; +use wire_budget::*; + +#[cfg(test)] +mod tests; diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/payload.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/payload.rs new file mode 100644 index 000000000..b0f20e047 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/payload.rs @@ -0,0 +1,145 @@ +use super::*; + +pub(super) fn validate_stream_replay_cursor( + expected_generation: &str, + expected_revision: u64, + current: &ReplayCursor, + operation: &str, +) -> Result<(), String> { + if current.generation == expected_generation && current.revision == expected_revision { + return Ok(()); + } + Err(format!( + "Replay source changed while {operation}: expected {expected_generation}@{expected_revision}, found {}@{}; retry from the new replay cursor", + current.generation, current.revision + )) +} + +#[cfg(test)] +pub(super) fn replace_event_payload(event: &mut SessionEvent, field_path: &str, text: String) { + let preview = json_field_preview(event, field_path); + if event.display_text == preview { + event.display_text = text.clone(); + } + if field_path == "args" { + event.args = serde_json::from_str(&text).unwrap_or(serde_json::Value::String(text)); + return; + } + if field_path == "result" { + event.result = serde_json::from_str(&text).unwrap_or(serde_json::Value::String(text)); + return; + } + let Some((root, path)) = field_path.split_once('.') else { + return; + }; + let value = match root { + "args" => &mut event.args, + "result" => &mut event.result, + _ => return, + }; + set_json_string_path(value, path, text); +} + +pub(super) fn set_json_string_path(value: &mut serde_json::Value, path: &str, text: String) { + let mut current = value; + let mut segments = path.split('.').peekable(); + while let Some(segment) = segments.next() { + if segments.peek().is_none() { + match current { + serde_json::Value::Object(object) => { + object.insert(segment.to_string(), serde_json::Value::String(text)); + } + serde_json::Value::Array(array) => { + let Ok(index) = segment.parse::() else { + return; + }; + let Some(value) = array.get_mut(index) else { + return; + }; + *value = serde_json::Value::String(text); + } + _ => {} + } + return; + } + current = match current { + serde_json::Value::Object(object) => { + let Some(next) = object.get_mut(segment) else { + return; + }; + next + } + serde_json::Value::Array(array) => { + let Ok(index) = segment.parse::() else { + return; + }; + let Some(next) = array.get_mut(index) else { + return; + }; + next + } + _ => return, + }; + } +} + +#[tauri::command] +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +pub async fn external_replay_read_payload_range( + source_id: String, + session_id: String, + generation: String, + event_id: String, + field_path: String, + offset: u64, + max_bytes: Option, +) -> Result { + tokio::task::spawn_blocking(move || { + match resolve_secondary_consumer_target(&source_id, &session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => { + let mut conn = database::db::get_connection() + .map_err(|err| format!("open replay index DB: {err}"))?; + replay::read_payload_range( + &mut conn, + source, + &imported_session_id, + &generation, + &event_id, + &field_path, + offset, + max_bytes, + ) + } + ResolvedReplayTarget::CollaborationSnapshot => { + let max_bytes = max_bytes + .unwrap_or(replay::DEFAULT_PAYLOAD_RANGE_BYTES) + .clamp(1, replay::HARD_MAX_PAYLOAD_RANGE_BYTES); + let conn = database::db::get_connection() + .map_err(|err| format!("open collaboration replay DB: {err}"))?; + collaboration_snapshot_payload_range_from_conn( + &conn, + &session_id, + &generation, + &event_id, + &field_path, + offset, + max_bytes, + ) + } + ResolvedReplayTarget::ManagedChunkStore => { + managed_chunk_payload_range(&session_id, &event_id, &field_path, offset, max_bytes) + } + ResolvedReplayTarget::NotReady => { + Err("Managed native transcript is not bound yet".into()) + } + } + }) + .await + .map_err(|err| format!("join replay payload task: {err}"))? +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/request_guard.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/request_guard.rs new file mode 100644 index 000000000..7416219e7 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/request_guard.rs @@ -0,0 +1,544 @@ +use super::*; + +#[derive(Debug, Clone)] +pub(super) struct ReplayRequestState { + request_token: u64, + episode_id: u64, + published_generation: Option, + touched_at: Instant, +} + +#[derive(Debug, Clone, Copy)] +pub(super) struct PrewarmRequestState { + request_token: u64, + episode_id: u64, + active: bool, + touched_at: Instant, +} + +trait TouchedRequestState { + fn touched_at(&self) -> Instant; +} + +impl TouchedRequestState for ReplayRequestState { + fn touched_at(&self) -> Instant { + self.touched_at + } +} + +impl TouchedRequestState for PrewarmRequestState { + fn touched_at(&self) -> Instant { + self.touched_at + } +} + +#[derive(Debug, Default)] +struct ReplayRequestRegistry { + foreground: HashMap, + prewarm: HashMap, +} + +fn replay_request_registry() -> &'static Mutex { + static REQUEST_REGISTRY: OnceLock> = OnceLock::new(); + REQUEST_REGISTRY.get_or_init(|| Mutex::new(ReplayRequestRegistry::default())) +} + +fn prune_expired_request_states( + entries: &mut HashMap, + now: Instant, +) { + entries.retain(|_, entry| now.duration_since(entry.touched_at()) < REPLAY_REQUEST_STATE_TTL); +} + +fn reserve_request_slot( + entries: &mut HashMap, + session_id: &str, +) { + if entries.contains_key(session_id) || entries.len() < MAX_REPLAY_REQUEST_STATES { + return; + } + if let Some(oldest) = entries + .iter() + .min_by_key(|(_, entry)| entry.touched_at()) + .map(|(session_id, _)| session_id.clone()) + { + entries.remove(&oldest); + } +} + +fn next_request_token() -> u64 { + static NEXT_REQUEST_TOKEN: AtomicU64 = AtomicU64::new(0); + NEXT_REQUEST_TOKEN + .fetch_add(1, Ordering::Relaxed) + .saturating_add(1) +} + +pub(super) fn begin_validated_prewarm_request( + source_id: &str, + session_id: &str, + episode_id: u64, +) -> Result { + validate_primary_replay_target_identity(source_id, session_id)?; + begin_prewarm_request(session_id, episode_id) +} + +pub(super) fn begin_prewarm_request(session_id: &str, episode_id: u64) -> Result { + let now = Instant::now(); + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + prune_expired_request_states(&mut registry.prewarm, now); + if let Some(current) = registry.prewarm.get(session_id) { + if current.episode_id > episode_id || (!current.active && current.episode_id >= episode_id) + { + return Err(format!( + "stale external replay prewarm episode {episode_id}; current episode is {}", + current.episode_id + )); + } + } + reserve_request_slot(&mut registry.prewarm, session_id); + let request_token = next_request_token(); + registry.prewarm.insert( + session_id.to_string(), + PrewarmRequestState { + request_token, + episode_id, + active: true, + touched_at: now, + }, + ); + Ok(request_token) +} + +pub(super) fn is_current_prewarm_request( + session_id: &str, + episode_id: u64, + request_token: u64, +) -> bool { + replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .prewarm + .get(session_id) + .is_some_and(|current| { + current.active + && current.episode_id == episode_id + && current.request_token == request_token + }) +} + +/// Mark the latest prewarm episode as cancelled without dropping its episode +/// floor. Keeping a short-lived tombstone prevents a delayed IPC invocation +/// from recreating the just-closed A episode after an A -> B switch. +pub(in crate::agent_sessions::event_pipeline::commands) fn cancel_prewarm_requests( + session_id: &str, +) { + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if let Some(current) = registry.prewarm.get_mut(session_id) { + current.active = false; + current.touched_at = Instant::now(); + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum ReplayWindowPublish { + Replace, + Merge, +} + +/// Validate a replay ticket and mutate its bounded EventStore as one +/// linearizable operation. The shared lock order is session manager -> stores +/// -> request registry. Session switching cancels replay tickets while holding +/// the manager lock, so either publication wins before the switch or it cannot +/// write after the switch. +fn apply_prewarm_store_if_current( + state: &EventStoreState, + session_id: &str, + episode_id: u64, + request_token: u64, + apply: impl FnOnce(&mut EventStore) -> Result, +) -> Option { + let mut manager = state + .session_manager + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let mut stores = state + .stores + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if !registry.prewarm.get(session_id).is_some_and(|current| { + current.active && current.episode_id == episode_id && current.request_token == request_token + }) { + return None; + } + manager.register(session_id); + Some(apply(stores.entry(session_id.to_string()).or_default())) +} + +fn apply_prewarm_replay_window_if_current( + state: &EventStoreState, + session_id: &str, + episode_id: u64, + request_token: u64, + events: &[SessionEvent], + publish: ReplayWindowPublish, +) -> bool { + apply_prewarm_store_if_current(state, session_id, episode_id, request_token, |store| { + match publish { + ReplayWindowPublish::Replace => store.set_external_replay_window(events.to_vec()), + ReplayWindowPublish::Merge => { + store.merge_external_replay_window_events(events.to_vec()) + } + } + }) + .is_some() +} + +pub(super) fn apply_prewarm_window_if_current( + state: &EventStoreState, + session_id: &str, + episode_id: u64, + request_token: u64, + events: &[SessionEvent], +) -> bool { + apply_prewarm_replay_window_if_current( + state, + session_id, + episode_id, + request_token, + events, + ReplayWindowPublish::Replace, + ) +} + +pub(super) fn apply_foreground_window_if_current( + state: &EventStoreState, + session_id: &str, + episode_id: u64, + request_token: u64, + generation: &str, + events: &[SessionEvent], + publish: ReplayWindowPublish, +) -> bool { + let mut manager = state + .session_manager + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let mut stores = state + .stores + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let Some(current) = registry.foreground.get_mut(session_id) else { + return false; + }; + if current.episode_id != episode_id || current.request_token != request_token { + return false; + } + if matches!(publish, ReplayWindowPublish::Merge) + && current.published_generation.as_deref() != Some(generation) + { + return false; + } + + manager.register(session_id); + let store = stores.entry(session_id.to_string()).or_default(); + match publish { + ReplayWindowPublish::Replace => { + store.set_external_replay_window(events.to_vec()); + current.published_generation = Some(generation.to_string()); + } + ReplayWindowPublish::Merge => store.merge_external_replay_window_events(events.to_vec()), + } + true +} + +pub(super) fn apply_foreground_delta_if_current( + state: &EventStoreState, + session_id: &str, + episode_id: u64, + request_token: u64, + delta: &ExternalReplayDelta, +) -> Option { + let mut manager = state + .session_manager + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let mut stores = state + .stores + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let current = registry.foreground.get_mut(session_id)?; + if current.episode_id != episode_id || current.request_token != request_token { + return None; + } + if !delta.reset_required + && current.published_generation.as_deref() != Some(delta.cursor.generation.as_str()) + { + return None; + } + + manager.register(session_id); + let result = + apply_external_replay_delta(stores.entry(session_id.to_string()).or_default(), delta); + if delta.reset_required { + current.published_generation = Some(delta.cursor.generation.clone()); + } + Some(result) +} + +pub(super) fn begin_replay_request( + session_id: &str, + episode_id: u64, + allow_activate: bool, +) -> Result { + let now = Instant::now(); + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + // A foreground entry is the identity of the renderer's currently open + // replay episode, not a short-lived request cache. Expiring it while the + // user reads an unchanged session makes the next scroll/poll fail with + // "foreground episode is not open". Watcher resources have their own TTL, + // explicit close/switch removes this entry, and the registry remains + // byte-bounded by MAX_REPLAY_REQUEST_STATES plus LRU eviction. + if let Some(current) = registry.foreground.get(session_id) { + if current.episode_id != episode_id && (!allow_activate || episode_id < current.episode_id) + { + return Err(format!( + "stale external replay foreground episode {episode_id}; current episode is {}", + current.episode_id + )); + } + } else if !allow_activate { + return Err("external replay foreground episode is not open".to_string()); + } + let published_generation = registry + .foreground + .get(session_id) + .filter(|current| current.episode_id == episode_id) + .and_then(|current| current.published_generation.clone()); + reserve_request_slot(&mut registry.foreground, session_id); + let request_token = next_request_token(); + registry.foreground.insert( + session_id.to_string(), + ReplayRequestState { + request_token, + episode_id, + published_generation, + touched_at: now, + }, + ); + Ok(request_token) +} + +pub(super) fn begin_validated_foreground_request( + source_id: &str, + session_id: &str, + episode_id: u64, + allow_activate: bool, +) -> Result { + validate_primary_replay_target_identity(source_id, session_id)?; + begin_replay_request(session_id, episode_id, allow_activate) +} + +pub(super) fn is_current_replay_request( + session_id: &str, + episode_id: u64, + request_token: u64, +) -> bool { + replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .foreground + .get(session_id) + .is_some_and(|current| { + current.episode_id == episode_id && current.request_token == request_token + }) +} + +pub(super) fn is_current_replay_episode(session_id: &str, episode_id: u64) -> bool { + replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .foreground + .get(session_id) + .is_some_and(|current| current.episode_id == episode_id) +} + +pub(super) fn release_replay_watch_if_stale_episode(session_id: &str, episode_id: u64) { + if !is_current_replay_episode(session_id, episode_id) { + external_replay_watcher::release_session_if_episode(session_id, episode_id); + } +} + +pub(super) fn release_session_runtime_if_episode(session_id: &str, episode_id: u64) { + let released = { + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if registry + .foreground + .get(session_id) + .is_some_and(|current| current.episode_id == episode_id) + { + registry.foreground.remove(session_id); + if let Some(prewarm) = registry.prewarm.get_mut(session_id) { + prewarm.active = false; + prewarm.touched_at = Instant::now(); + } + true + } else { + false + } + }; + if released { + external_replay_watcher::release_session_if_episode(session_id, episode_id); + } +} + +/// Invalidate pending external replay delivery and release its foreground +/// watcher. Native SDE sessions never create either entry, so calling this +/// from the shared session lifecycle is a strict no-op for native behavior. +pub(in crate::agent_sessions::event_pipeline::commands) fn release_session_runtime( + session_id: &str, +) { + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + registry.foreground.remove(session_id); + if let Some(prewarm) = registry.prewarm.get_mut(session_id) { + prewarm.active = false; + prewarm.touched_at = Instant::now(); + } + drop(registry); + external_replay_watcher::release_session(session_id); +} + +#[cfg(test)] +pub(super) fn has_foreground_request(session_id: &str) -> bool { + replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .foreground + .contains_key(session_id) +} + +#[cfg(test)] +pub(super) fn has_prewarm_request(session_id: &str) -> bool { + replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .prewarm + .contains_key(session_id) +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(super) struct ReplayApplyResult { + pub(super) upserted: u64, + pub(super) removed: u64, + pub(super) changed: bool, +} + +/// External-only authoritative apply path. Native SDE Agent never calls this +/// helper and retains its existing EventStore set/merge semantics. +pub(super) fn apply_external_replay_delta( + store: &mut EventStore, + delta: &ExternalReplayDelta, +) -> ReplayApplyResult { + if delta.reset_required { + store.set_external_replay_window(delta.events.clone()); + return ReplayApplyResult { + upserted: delta.events.len() as u64, + removed: 0, + changed: true, + }; + } + let mut result = ReplayApplyResult::default(); + for event in delta.events.iter().cloned() { + let unchanged = store + .get_by_id(&event.id) + .is_some_and(|existing| session_events_equal(existing, &event)); + if !unchanged { + store.upsert(event); + result.upserted += 1; + } + } + result.removed = store.remove_by_ids(&delta.removed_event_ids) as u64; + result.changed = result.upserted > 0 || result.removed > 0; + result +} + +#[cfg(test)] +mod registry_policy_tests { + use super::*; + + #[test] + fn prewarm_request_slot_policy_prunes_expired_entries_and_evicts_the_lru() { + let now = Instant::now(); + let mut entries = HashMap::new(); + entries.insert( + "expired".to_string(), + ReplayRequestState { + request_token: 1, + episode_id: 1, + published_generation: None, + touched_at: now - REPLAY_REQUEST_STATE_TTL, + }, + ); + prune_expired_request_states(&mut entries, now); + assert!(entries.is_empty()); + + for index in 0..MAX_REPLAY_REQUEST_STATES { + entries.insert( + format!("session-{index}"), + ReplayRequestState { + request_token: index as u64, + episode_id: 1, + published_generation: None, + touched_at: now - Duration::from_millis(index as u64), + }, + ); + } + reserve_request_slot(&mut entries, "incoming"); + assert_eq!(entries.len(), MAX_REPLAY_REQUEST_STATES - 1); + assert!(!entries.contains_key(&format!("session-{}", MAX_REPLAY_REQUEST_STATES - 1))); + } + + #[test] + fn foreground_episode_survives_idle_ttl_until_explicit_release() { + let session_id = "foreground-idle-ttl-regression"; + let episode_id = u64::MAX - 1; + { + let mut registry = replay_request_registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + registry.foreground.insert( + session_id.to_string(), + ReplayRequestState { + request_token: 1, + episode_id, + published_generation: Some("generation-1".to_string()), + touched_at: Instant::now() - REPLAY_REQUEST_STATE_TTL, + }, + ); + } + + assert!(begin_replay_request(session_id, episode_id, false).is_ok()); + assert!(has_foreground_request(session_id)); + + release_session_runtime_if_episode(session_id, episode_id); + assert!(!has_foreground_request(session_id)); + } +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/shell.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/shell.rs new file mode 100644 index 000000000..b18aee441 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/shell.rs @@ -0,0 +1,1153 @@ +use super::*; + +pub(super) async fn persist_shell_replays_for_delivery( + source_id: &str, + session_id: &str, + generation: &str, + revision: u64, + events: &mut Vec, +) -> Result<(), String> { + if !events + .iter() + .any(|event| event.ui_canonical == core_types::tool_names::RUN_SHELL) + { + return Ok(()); + } + let source_id = source_id.to_string(); + let session_id = session_id.to_string(); + let generation = generation.to_string(); + let owned = std::mem::take(events); + *events = tokio::task::spawn_blocking(move || { + persist_shell_replays_bounded(&source_id, &session_id, &generation, revision, owned) + }) + .await + .map_err(|error| format!("join external shell replay persistence: {error}"))??; + Ok(()) +} + +pub(super) fn persist_shell_replays_bounded( + source_id: &str, + session_id: &str, + generation: &str, + revision: u64, + mut events: Vec, +) -> Result, String> { + match resolve_target(source_id, session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => { + with_foreground_replay_connection("external Shell replay", |conn| { + let tx = database::db::begin_immediate(conn).map_err(|error| { + format!("begin external Shell manifest transaction: {error}") + })?; + for event in events.iter_mut().filter(|event| { + event.ui_canonical == core_types::tool_names::RUN_SHELL + && event.shell_replay.is_none() + }) { + persist_imported_shell_manifest( + &tx, + event, + source, + &imported_session_id, + generation, + )?; + } + tx.commit() + .map_err(|error| format!("publish external Shell manifests: {error}")) + })?; + validate_imported_shell_snapshot(source, &imported_session_id, generation, revision)?; + } + ResolvedReplayTarget::CollaborationSnapshot => { + // Collaboration generation identifies the snapshot lineage, while + // same-lineage rewrites advance revision. Artifact scopes must + // include both or a same-length UPDATE can reuse stale content. + let artifact_generation = collaboration_shell_artifact_generation(generation, revision); + with_foreground_replay_connection("collaboration Shell replay", |conn| { + let tx = database::db::begin_immediate(conn).map_err(|error| { + format!("begin collaboration Shell manifest transaction: {error}") + })?; + for event in events.iter_mut().filter(|event| { + event.ui_canonical == core_types::tool_names::RUN_SHELL + && event.shell_replay.is_none() + }) { + persist_scoped_shell_manifest( + &tx, + event, + source_id, + session_id, + &artifact_generation, + |payload_ref, offset, max| { + collaboration_snapshot_payload_range_from_conn( + &tx, + session_id, + generation, + payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id), + &payload_ref.field_path, + offset, + max, + ) + .map(|range| range.text.into_bytes()) + }, + )?; + } + tx.commit() + .map_err(|error| format!("publish collaboration Shell manifests: {error}")) + })?; + validate_collaboration_shell_snapshot(session_id, generation, revision)?; + } + ResolvedReplayTarget::ManagedChunkStore => { + with_foreground_replay_connection("managed Shell replay", |conn| { + let tx = database::db::begin_immediate(conn).map_err(|error| { + format!("begin managed Shell manifest transaction: {error}") + })?; + for event in events.iter_mut().filter(|event| { + event.ui_canonical == core_types::tool_names::RUN_SHELL + && event.shell_replay.is_none() + }) { + persist_scoped_shell_manifest( + &tx, + event, + source_id, + session_id, + generation, + |payload_ref, offset, max| { + managed_chunk_payload_range_from_conn( + &tx, + session_id, + payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id), + &payload_ref.field_path, + offset, + max, + ) + .map(|range| range.text.into_bytes()) + }, + )?; + } + tx.commit() + .map_err(|error| format!("publish managed Shell manifests: {error}")) + })?; + validate_managed_chunk_shell_snapshot(session_id, generation, revision)?; + } + ResolvedReplayTarget::NotReady => {} + } + Ok(events) +} + +pub(super) fn collaboration_shell_artifact_generation(generation: &str, revision: u64) -> String { + format!("{generation}-r{revision}") +} + +pub(super) fn validate_imported_shell_snapshot( + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + revision: u64, +) -> Result<(), String> { + with_foreground_replay_connection("imported Shell validation", |conn| { + validate_imported_shell_snapshot_from_conn(conn, source, session_id, generation, revision) + }) +} + +pub(super) fn validate_imported_shell_snapshot_from_conn( + conn: &mut Connection, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + revision: u64, +) -> Result<(), String> { + // Re-observe the provider, not merely ORGII's last compact state. A file + // or WAL can change (including same-size replacement) while the per-event + // artifact transactions run; a local state lookup would miss that race. + let observed = replay::scan_window_after( + conn, + source, + session_id, + i64::MAX, + ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: 1, + }, + )?; + if observed.cursor.generation == generation && observed.cursor.revision == revision { + return Ok(()); + } + Err(format!( + "Imported Shell replay changed while publishing manifests: expected {generation}@{revision}, found {}@{}; retry the bounded replay request", + observed.cursor.generation, observed.cursor.revision + )) +} + +pub(super) fn validate_collaboration_shell_snapshot( + session_id: &str, + generation: &str, + revision: u64, +) -> Result<(), String> { + let conn = database::db::get_connection() + .map_err(|error| format!("open collaboration Shell validation DB: {error}"))?; + let current = collaboration_snapshot_state(&conn, session_id)?; + if current.generation == generation && current.revision == revision { + return Ok(()); + } + Err(format!( + "Collaboration Shell replay changed while publishing manifests: expected {generation}@{revision}, found {}@{}; retry the bounded replay request", + current.generation, current.revision + )) +} + +pub(super) fn validate_managed_chunk_shell_snapshot( + session_id: &str, + generation: &str, + revision: u64, +) -> Result<(), String> { + let conn = database::db::get_connection() + .map_err(|error| format!("open managed Shell validation DB: {error}"))?; + let current = managed_chunk_stream_cursor(&conn, session_id)?; + if current.0 == generation && current.1.max(0) as u64 == revision { + return Ok(()); + } + Err(format!( + "Managed Shell replay changed while publishing manifests: expected {generation}@{revision}, found {}@{}; retry the bounded replay request", + current.0, current.1 + )) +} + +#[derive(Debug, Clone)] +pub(super) struct CanonicalExternalShellSegment { + pub(super) stream: ShellReplayStream, + pub(super) artifact: replay::ReplayPayloadArtifactLocator, + pub(super) preview: String, +} + +pub(super) fn persist_imported_shell_manifest( + tx: &Transaction<'_>, + event: &mut SessionEvent, + source: ImportedHistorySourceId, + imported_session_id: &str, + generation: &str, +) -> Result<(), String> { + if event.ui_canonical != core_types::tool_names::RUN_SHELL || event.shell_replay.is_some() { + return Ok(()); + } + let selected = select_shell_payload_refs(event); + if selected.is_empty() { + let source_session_id = source.source_session_id(imported_session_id)?; + return persist_inline_shell_manifest( + tx, + event, + source.as_str(), + source_session_id, + generation, + ); + } + let mut segments = Vec::with_capacity(selected.len()); + for payload_ref in selected { + let source_event_id = payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id); + let artifact = replay::materialize_payload_artifact( + tx, + source, + imported_session_id, + generation, + source_event_id, + &payload_ref.field_path, + )?; + if artifact.total_bytes != payload_ref.full_size_bytes as u64 { + return Err(format!( + "External Shell payload changed while publishing manifest: expected {} bytes, found {}", + payload_ref.full_size_bytes, artifact.total_bytes + )); + } + segments.push(CanonicalExternalShellSegment { + stream: shell_stream_for_payload_ref(payload_ref), + artifact, + preview: payload_ref.preview.clone(), + }); + } + publish_external_shell_manifest(tx, event, &segments) +} + +pub(super) fn persist_scoped_shell_manifest( + tx: &Transaction<'_>, + event: &mut SessionEvent, + source_id: &str, + source_session_id: &str, + generation: &str, + mut read_range: impl FnMut(&PayloadRef, u64, usize) -> Result, String>, +) -> Result<(), String> { + if event.ui_canonical != core_types::tool_names::RUN_SHELL || event.shell_replay.is_some() { + return Ok(()); + } + let selected = select_shell_payload_refs(event); + if selected.is_empty() { + return persist_inline_shell_manifest(tx, event, source_id, source_session_id, generation); + } + let mut segments = Vec::with_capacity(selected.len()); + for payload_ref in selected { + let source_event_id = payload_ref + .replay_source_event_id + .as_deref() + .unwrap_or(&payload_ref.event_id); + let expected_bytes = payload_ref.full_size_bytes as u64; + let artifact = if let Some(existing) = replay::find_scoped_payload_artifact( + tx, + source_id, + source_session_id, + generation, + source_event_id, + &payload_ref.field_path, + expected_bytes, + )? { + existing + } else { + replay::store_scoped_payload_artifact_streamed( + tx, + source_id, + source_session_id, + generation, + source_event_id, + &payload_ref.field_path, + expected_bytes, + |writer| { + let mut offset = 0_u64; + while offset < expected_bytes { + let requested = (expected_bytes - offset) + .min(SHELL_REPLAY_RANGE_MAX_BYTES as u64) + as usize; + let bytes = read_range(payload_ref, offset, requested)?; + if bytes.is_empty() || bytes.len() > requested { + return Err(format!( + "External Shell payload made invalid progress at byte {offset}" + )); + } + writer + .write_all(&bytes) + .map_err(|error| format!("write external Shell payload: {error}"))?; + offset = offset.saturating_add(bytes.len() as u64); + } + Ok(()) + }, + )? + }; + segments.push(CanonicalExternalShellSegment { + stream: shell_stream_for_payload_ref(payload_ref), + artifact, + preview: payload_ref.preview.clone(), + }); + } + publish_external_shell_manifest(tx, event, &segments) +} + +pub(super) fn persist_inline_shell_manifest( + tx: &Transaction<'_>, + event: &mut SessionEvent, + source_id: &str, + source_session_id: &str, + generation: &str, +) -> Result<(), String> { + if event + .payload_refs + .iter() + .any(|payload_ref| payload_ref.field_path == "result") + { + return Ok(()); + } + let parts = external_shell_inline_segments(event); + if parts.is_empty() { + return Ok(()); + } + let mut segments = Vec::with_capacity(parts.len()); + for (ordinal, part) in parts.into_iter().enumerate() { + let field_path = format!("__shell_inline.{ordinal}"); + let expected_bytes = part.text.len() as u64; + let artifact = if let Some(existing) = replay::find_scoped_payload_artifact( + tx, + source_id, + source_session_id, + generation, + &event.id, + &field_path, + expected_bytes, + )? { + existing + } else { + replay::store_scoped_payload_artifact_streamed( + tx, + source_id, + source_session_id, + generation, + &event.id, + &field_path, + expected_bytes, + |writer| { + writer + .write_all(part.text.as_bytes()) + .map_err(|error| format!("write inline external Shell payload: {error}")) + }, + )? + }; + segments.push(CanonicalExternalShellSegment { + stream: part.stream, + artifact, + preview: utf8_tail_preview(part.text, SHELL_REPLAY_PREVIEW_BYTES), + }); + } + publish_external_shell_manifest(tx, event, &segments) +} + +pub(super) fn shell_stream_for_payload_ref(payload_ref: &PayloadRef) -> ShellReplayStream { + if payload_ref + .field_path + .rsplit('.') + .next() + .is_some_and(|field| field.eq_ignore_ascii_case("stderr")) + { + ShellReplayStream::Stderr + } else { + ShellReplayStream::Stdout + } +} + +pub(super) fn publish_external_shell_manifest( + tx: &Transaction<'_>, + event: &mut SessionEvent, + segments: &[CanonicalExternalShellSegment], +) -> Result<(), String> { + if segments.is_empty() { + return Ok(()); + } + let logical_call_id = event.call_id.as_deref().unwrap_or(&event.id); + let mut identity = Sha256::new(); + identity.update(b"orgii-external-shell-manifest-v1\0"); + let mut total_bytes = 0_u64; + let mut last_sequence = 0_u64; + let mut manifest_rows = Vec::with_capacity(segments.len()); + let mut preview = String::new(); + for (ordinal, segment) in segments.iter().enumerate() { + let stream_tag = match segment.stream { + ShellReplayStream::Stdout => 1_u8, + ShellReplayStream::Stderr => 2_u8, + }; + identity.update((ordinal as u64).to_le_bytes()); + identity.update([stream_tag]); + identity.update(segment.artifact.total_bytes.to_le_bytes()); + identity.update(segment.artifact.content_hash.as_bytes()); + + let output_byte_start = total_bytes; + total_bytes = total_bytes + .checked_add(segment.artifact.total_bytes) + .ok_or_else(|| "External Shell manifest byte count overflow".to_string())?; + let frame_count = segment + .artifact + .total_bytes + .saturating_add(SHELL_REPLAY_FRAME_MAX_BYTES as u64 - 1) + / SHELL_REPLAY_FRAME_MAX_BYTES as u64; + let first_sequence = last_sequence.saturating_add(1); + last_sequence = last_sequence + .checked_add(frame_count) + .ok_or_else(|| "External Shell manifest sequence overflow".to_string())?; + manifest_rows.push(( + ordinal as u64, + segment, + output_byte_start, + first_sequence, + frame_count, + )); + + if segment.stream == ShellReplayStream::Stderr { + preview.push_str("[stderr] "); + } + preview.push_str(&segment.preview); + if preview.len() > SHELL_REPLAY_PREVIEW_BYTES * 2 { + preview = utf8_tail_preview(&preview, SHELL_REPLAY_PREVIEW_BYTES); + } + } + preview = utf8_tail_preview(&preview, SHELL_REPLAY_PREVIEW_BYTES); + let identity_hash = identity + .finalize() + .iter() + .map(|byte| format!("{byte:02x}")) + .collect::(); + let call_id = format!("{logical_call_id}-external-{identity_hash}"); + + let existing_identity = tx + .query_row( + "SELECT call_id,identity_hash + FROM imported_replay_shell_manifests + WHERE session_id=?1 AND logical_call_id=?2", + params![event.session_id, logical_call_id], + |row| Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)), + ) + .optional() + .map_err(|error| format!("read existing external Shell identity: {error}"))?; + if let Some((existing_call_id, existing_hash)) = existing_identity.as_ref() { + if existing_hash == &identity_hash { + if existing_call_id != &call_id { + return Err("external Shell manifest identity/call id mismatch".to_string()); + } + tx.execute( + "UPDATE imported_replay_shell_manifests + SET accessed_at=strftime('%Y-%m-%dT%H:%M:%fZ','now') + WHERE session_id=?1 AND call_id=?2 + AND accessed_at <= strftime('%Y-%m-%dT%H:%M:%fZ','now','-60 seconds')", + params![event.session_id, call_id], + ) + .map_err(|error| format!("touch unchanged external Shell manifest: {error}"))?; + set_external_shell_state(event, call_id, total_bytes, last_sequence, preview); + return Ok(()); + } + } + + let old_scopes = { + let mut statement = tx + .prepare( + "SELECT DISTINCT segment.source,segment.source_session_id,segment.generation + FROM imported_replay_shell_segments AS segment + JOIN imported_replay_shell_manifests AS manifest + ON manifest.session_id=segment.session_id AND manifest.call_id=segment.call_id + WHERE manifest.session_id=?1 AND manifest.logical_call_id=?2", + ) + .map_err(|error| format!("prepare old Shell manifest scopes: {error}"))?; + let rows = statement + .query_map(params![event.session_id, logical_call_id], |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, String>(2)?, + )) + }) + .map_err(|error| format!("query old Shell manifest scopes: {error}"))?; + rows.collect::>>() + .map_err(|error| format!("read old Shell manifest scopes: {error}"))? + }; + // sessions.db and orgtrack-cli connections do not globally enable + // foreign_keys, so ON DELETE CASCADE is documentation rather than a + // cleanup guarantee. Delete the old references explicitly before their + // manifest; otherwise they retain obsolete content-addressed BLOBs. + if let Some((old_call_id, _)) = existing_identity.as_ref() { + tx.execute( + "DELETE FROM imported_replay_shell_segments + WHERE session_id=?1 AND call_id=?2", + params![event.session_id, old_call_id], + ) + .map_err(|error| format!("delete replaced external Shell segments: {error}"))?; + } + tx.execute( + "DELETE FROM imported_replay_shell_manifests + WHERE session_id=?1 AND logical_call_id=?2", + params![event.session_id, logical_call_id], + ) + .map_err(|error| format!("replace external Shell manifest: {error}"))?; + tx.execute( + "INSERT INTO imported_replay_shell_manifests( + session_id,logical_call_id,call_id,identity_hash,total_bytes,last_sequence, + terminal_preview,completed_at,accessed_at + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8, + strftime('%Y-%m-%dT%H:%M:%fZ','now'))", + params![ + event.session_id, + logical_call_id, + call_id, + identity_hash, + i64::try_from(total_bytes) + .map_err(|_| "External Shell manifest exceeds SQLite INTEGER".to_string())?, + i64::try_from(last_sequence) + .map_err(|_| "External Shell sequence exceeds SQLite INTEGER".to_string())?, + preview, + event.created_at, + ], + ) + .map_err(|error| format!("insert external Shell manifest: {error}"))?; + for (ordinal, segment, output_byte_start, first_sequence, frame_count) in manifest_rows { + tx.execute( + "INSERT INTO imported_replay_shell_segments( + session_id,call_id,ordinal,stream,source,source_session_id,generation, + content_hash,output_byte_start,total_bytes,first_sequence,frame_count + ) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12)", + params![ + event.session_id, + call_id, + i64::try_from(ordinal) + .map_err(|_| "External Shell segment ordinal overflow".to_string())?, + segment.stream.as_wire_str(), + segment.artifact.source_id, + segment.artifact.source_session_id, + segment.artifact.generation, + segment.artifact.content_hash, + i64::try_from(output_byte_start) + .map_err(|_| "External Shell output offset overflow".to_string())?, + i64::try_from(segment.artifact.total_bytes) + .map_err(|_| "External Shell segment size overflow".to_string())?, + i64::try_from(first_sequence) + .map_err(|_| "External Shell first sequence overflow".to_string())?, + i64::try_from(frame_count) + .map_err(|_| "External Shell frame count overflow".to_string())?, + ], + ) + .map_err(|error| format!("insert external Shell segment: {error}"))?; + } + + let mut cleanup_scopes = old_scopes.into_iter().collect::>(); + cleanup_scopes.extend(segments.iter().map(|segment| { + ( + segment.artifact.source_id.clone(), + segment.artifact.source_session_id.clone(), + segment.artifact.generation.clone(), + ) + })); + for (source, source_session_id, generation) in cleanup_scopes { + delete_unreferenced_payload_artifacts(tx, &source, &source_session_id, &generation)?; + } + + set_external_shell_state(event, call_id, total_bytes, last_sequence, preview); + Ok(()) +} + +pub(super) fn set_external_shell_state( + event: &mut SessionEvent, + call_id: String, + total_bytes: u64, + last_sequence: u64, + terminal_preview: String, +) { + event.shell_replay = Some(ShellReplayState { + replay_ref: ShellReplayRef { + session_id: event.session_id.clone(), + call_id, + format_version: SHELL_REPLAY_FORMAT_VERSION, + }, + bookmark: ShellReplayBookmark { + visible_through_sequence: last_sequence, + visible_bytes: total_bytes, + }, + terminal_preview, + status: ShellReplayStatus::Complete, + error: None, + completed_at: Some(event.created_at.clone()), + }); +} + +pub(super) const DELETE_UNREFERENCED_PAYLOAD_ARTIFACTS_SQL: &str = + "DELETE FROM imported_replay_payload_artifacts AS artifact + WHERE artifact.source=?1 AND artifact.source_session_id=?2 AND artifact.generation=?3 + AND artifact.content_hash NOT IN( + SELECT ref.content_hash FROM imported_replay_payload_artifact_refs AS ref + WHERE ref.source=?1 + AND ref.source_session_id=?2 + AND ref.generation=?3 + ) + AND artifact.content_hash NOT IN( + SELECT shell.content_hash FROM imported_replay_shell_segments AS shell + WHERE shell.source=?1 + AND shell.source_session_id=?2 + AND shell.generation=?3 + )"; + +pub(super) fn delete_unreferenced_payload_artifacts( + tx: &Transaction<'_>, + source: &str, + source_session_id: &str, + generation: &str, +) -> Result<(), String> { + tx.execute( + DELETE_UNREFERENCED_PAYLOAD_ARTIFACTS_SQL, + params![source, source_session_id, generation], + ) + .map(|_| ()) + .map_err(|error| format!("delete unreferenced external Shell payload: {error}")) +} + +#[derive(Debug)] +pub(super) struct ExternalShellManifestSegment { + stream: ShellReplayStream, + artifact_row_id: i64, + output_byte_start: u64, + total_bytes: u64, + first_sequence: u64, + frame_count: u64, +} + +/// Read an external-CLI Shell manifest when one exists, otherwise preserve +/// the native SDE Agent's #425 `.slog` command unchanged. An invalid external +/// manifest is an error: silently falling through could surface a stale native +/// replay with the same logical call id. +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +#[tauri::command(rename_all = "camelCase")] +pub async fn shell_replay_read_range( + session_id: String, + call_id: String, + visible_through_sequence: u64, + visible_bytes: u64, + offset_bytes: u64, + limit_bytes: u64, +) -> Result { + // Native #425 replay is a high-frequency range path. Its call ids never + // carry the content-addressed external suffix, so bypass both the extra + // task and the replay-cache connection entirely. + if !is_external_shell_manifest_call_id(&call_id) { + return agent_core::tools::impls::coding::exec::shell_replay::shell_replay_read_range( + session_id, + call_id, + visible_through_sequence, + visible_bytes, + offset_bytes, + limit_bytes, + ) + .await; + } + let external_session_id = session_id.clone(); + let external_call_id = call_id.clone(); + #[cfg(test)] + EXTERNAL_SHELL_MANIFEST_DB_PROBES.fetch_add(1, Ordering::SeqCst); + let external = tokio::task::spawn_blocking(move || { + let conn = database::db::get_connection() + .map_err(|error| format!("open external Shell replay DB: {error}"))?; + read_external_shell_manifest_range( + &conn, + &external_session_id, + &external_call_id, + visible_through_sequence, + visible_bytes, + offset_bytes, + limit_bytes, + ) + }) + .await + .map_err(|error| format!("join external Shell range read: {error}"))??; + if let Some(range) = external { + return Ok(range); + } + agent_core::tools::impls::coding::exec::shell_replay::shell_replay_read_range( + session_id, + call_id, + visible_through_sequence, + visible_bytes, + offset_bytes, + limit_bytes, + ) + .await +} + +pub(super) fn is_external_shell_manifest_call_id(call_id: &str) -> bool { + let Some((_, digest)) = call_id.rsplit_once("-external-") else { + return false; + }; + digest.len() == 64 + && digest + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +pub(super) fn read_external_shell_manifest_range( + conn: &Connection, + session_id: &str, + call_id: &str, + visible_through_sequence: u64, + visible_bytes: u64, + offset_bytes: u64, + limit_bytes: u64, +) -> Result, String> { + let manifest_table_exists = conn + .query_row( + "SELECT EXISTS( + SELECT 1 FROM sqlite_master + WHERE type='table' AND name='imported_replay_shell_manifests' + )", + [], + |row| row.get::<_, bool>(0), + ) + .map_err(|error| format!("inspect external Shell replay schema: {error}"))?; + if !manifest_table_exists { + return Ok(None); + } + let manifest = conn + .query_row( + "SELECT total_bytes,last_sequence + FROM imported_replay_shell_manifests + WHERE session_id=?1 AND call_id=?2", + params![session_id, call_id], + |row| Ok((row.get::<_, i64>(0)?, row.get::<_, i64>(1)?)), + ) + .optional() + .map_err(|error| format!("read external Shell manifest: {error}"))?; + let Some((manifest_total, manifest_last_sequence)) = manifest else { + return Ok(None); + }; + let manifest_total = nonnegative_sqlite_u64(manifest_total, "manifest total_bytes")?; + let manifest_last_sequence = + nonnegative_sqlite_u64(manifest_last_sequence, "manifest last_sequence")?; + + // Acquire cache liveness before opening any artifact BLOB. Cache pruning + // performs selection and deletion under one IMMEDIATE transaction, so + // this conditional write and prune have a clear lock order: either this + // read protects the manifest first, or it observes that prune removed it. + // Failed/corrupt read attempts may retain a small entry for one TTL; that + // is preferable to deleting a payload while a reader is opening it. + conn.execute( + "UPDATE imported_replay_shell_manifests + SET accessed_at=strftime('%Y-%m-%dT%H:%M:%fZ','now') + WHERE session_id=?1 AND call_id=?2 + AND accessed_at <= strftime('%Y-%m-%dT%H:%M:%fZ','now','-60 seconds')", + params![session_id, call_id], + ) + .map_err(|error| format!("touch external Shell manifest: {error}"))?; + + let mut statement = conn + .prepare( + "SELECT segment.ordinal,segment.stream,segment.output_byte_start, + segment.total_bytes,segment.first_sequence,segment.frame_count, + artifact.rowid,LENGTH(artifact.payload) + FROM imported_replay_shell_segments AS segment + LEFT JOIN imported_replay_payload_artifacts AS artifact + ON artifact.source=segment.source + AND artifact.source_session_id=segment.source_session_id + AND artifact.generation=segment.generation + AND artifact.content_hash=segment.content_hash + WHERE segment.session_id=?1 AND segment.call_id=?2 + ORDER BY segment.ordinal ASC", + ) + .map_err(|error| format!("prepare external Shell segments: {error}"))?; + let rows = statement + .query_map(params![session_id, call_id], |row| { + Ok(( + row.get::<_, i64>(0)?, + row.get::<_, String>(1)?, + row.get::<_, i64>(2)?, + row.get::<_, i64>(3)?, + row.get::<_, i64>(4)?, + row.get::<_, i64>(5)?, + row.get::<_, Option>(6)?, + row.get::<_, Option>(7)?, + )) + }) + .map_err(|error| format!("query external Shell segments: {error}"))?; + let raw_segments = rows + .collect::>>() + .map_err(|error| format!("read external Shell segments: {error}"))?; + drop(statement); + + let mut segments = Vec::with_capacity(raw_segments.len()); + let mut expected_output_start = 0_u64; + let mut expected_first_sequence = 1_u64; + for (expected_ordinal, row) in raw_segments.into_iter().enumerate() { + let ( + ordinal, + stream, + output_byte_start, + total_bytes, + first_sequence, + frame_count, + artifact_row_id, + artifact_bytes, + ) = row; + let ordinal = nonnegative_sqlite_u64(ordinal, "segment ordinal")?; + if ordinal != expected_ordinal as u64 { + return Err(format!( + "invalid external Shell segment ordinal {ordinal}; expected {expected_ordinal}" + )); + } + let stream = match stream.as_str() { + "stdout" => ShellReplayStream::Stdout, + "stderr" => ShellReplayStream::Stderr, + other => return Err(format!("invalid external Shell stream {other:?}")), + }; + let output_byte_start = + nonnegative_sqlite_u64(output_byte_start, "segment output_byte_start")?; + let total_bytes = nonnegative_sqlite_u64(total_bytes, "segment total_bytes")?; + let first_sequence = nonnegative_sqlite_u64(first_sequence, "segment first_sequence")?; + let frame_count = nonnegative_sqlite_u64(frame_count, "segment frame_count")?; + let expected_frame_count = total_bytes + .saturating_add(SHELL_REPLAY_FRAME_MAX_BYTES as u64 - 1) + / SHELL_REPLAY_FRAME_MAX_BYTES as u64; + if output_byte_start != expected_output_start + || first_sequence != expected_first_sequence + || frame_count != expected_frame_count + { + return Err(format!( + "invalid external Shell segment layout at ordinal {ordinal}" + )); + } + let artifact_row_id = artifact_row_id.ok_or_else(|| { + format!("external Shell payload artifact missing at ordinal {ordinal}") + })?; + let artifact_bytes = nonnegative_sqlite_u64( + artifact_bytes.ok_or_else(|| { + format!("external Shell payload length missing at ordinal {ordinal}") + })?, + "artifact payload length", + )?; + if artifact_bytes != total_bytes { + return Err(format!( + "external Shell payload length mismatch at ordinal {ordinal}: expected {total_bytes}, found {artifact_bytes}" + )); + } + segments.push(ExternalShellManifestSegment { + stream, + artifact_row_id, + output_byte_start, + total_bytes, + first_sequence, + frame_count, + }); + expected_output_start = expected_output_start + .checked_add(total_bytes) + .ok_or_else(|| "external Shell output byte count overflow".to_string())?; + expected_first_sequence = expected_first_sequence + .checked_add(frame_count) + .ok_or_else(|| "external Shell sequence overflow".to_string())?; + } + if expected_output_start != manifest_total + || expected_first_sequence.saturating_sub(1) != manifest_last_sequence + { + return Err("external Shell manifest summary does not match its segments".to_string()); + } + + let visible_sequence = visible_through_sequence.min(manifest_last_sequence); + let visible_end = visible_bytes.min(manifest_total); + let start = offset_bytes.min(visible_end); + let limit = limit_bytes.min(SHELL_REPLAY_RANGE_MAX_BYTES as u64).max(1); + let range = if start >= visible_end || visible_sequence == 0 { + ShellReplayRange { + frames: Vec::new(), + next_offset_bytes: start, + eof: true, + } + } else { + read_external_shell_frames(conn, &segments, visible_sequence, visible_end, start, limit)? + }; + + Ok(Some(range)) +} + +pub(super) fn read_external_shell_frames( + conn: &Connection, + segments: &[ExternalShellManifestSegment], + visible_sequence: u64, + visible_end: u64, + start: u64, + limit: u64, +) -> Result { + let tail_request = start.saturating_add(limit) >= visible_end; + let mut frames = Vec::new(); + let mut next_offset = start; + let mut response_bytes = 0_u64; + let mut rendered_response_bytes = 0_usize; + 'segments: for segment in segments { + let segment_end = segment + .output_byte_start + .checked_add(segment.total_bytes) + .ok_or_else(|| "external Shell segment end overflow".to_string())?; + if segment_end <= start || segment.output_byte_start >= visible_end { + continue; + } + let blob = conn + .blob_open( + DatabaseName::Main, + "imported_replay_payload_artifacts", + "payload", + segment.artifact_row_id, + true, + ) + .map_err(|error| format!("open external Shell payload BLOB: {error}"))?; + if blob.len() as u64 != segment.total_bytes { + return Err("external Shell payload changed after manifest validation".to_string()); + } + let local_start = start.saturating_sub(segment.output_byte_start); + let mut frame_index = (local_start / SHELL_REPLAY_FRAME_MAX_BYTES as u64) + .saturating_sub(1) + .min(segment.frame_count.saturating_sub(1)); + while frame_index < segment.frame_count { + let sequence = segment + .first_sequence + .checked_add(frame_index) + .ok_or_else(|| "external Shell frame sequence overflow".to_string())?; + if sequence > visible_sequence { + break 'segments; + } + let candidate_start = frame_index + .checked_mul(SHELL_REPLAY_FRAME_MAX_BYTES as u64) + .ok_or_else(|| "external Shell frame offset overflow".to_string())?; + let candidate_end = frame_index + .saturating_add(1) + .saturating_mul(SHELL_REPLAY_FRAME_MAX_BYTES as u64) + .min(segment.total_bytes); + let local_frame_start = + external_shell_utf8_boundary(&blob, candidate_start, segment.total_bytes)?; + let local_frame_end = + external_shell_utf8_boundary(&blob, candidate_end, segment.total_bytes)?; + if local_frame_end <= local_frame_start { + return Err("external Shell UTF-8 frame made no progress".to_string()); + } + let frame_start = segment + .output_byte_start + .checked_add(local_frame_start) + .ok_or_else(|| "external Shell frame start overflow".to_string())?; + let frame_end = segment + .output_byte_start + .checked_add(local_frame_end) + .ok_or_else(|| "external Shell frame end overflow".to_string())?; + frame_index = frame_index.saturating_add(1); + if frame_end <= start { + continue; + } + if frame_start >= visible_end || frame_end > visible_end { + break 'segments; + } + if tail_request + && frame_start < start + && frame_end < visible_end + && visible_end.saturating_sub(frame_start) > limit + { + continue; + } + let frame_bytes = frame_end.saturating_sub(frame_start); + if !frames.is_empty() && response_bytes.saturating_add(frame_bytes) > limit { + break 'segments; + } + let frame_len = usize::try_from(local_frame_end - local_frame_start) + .map_err(|_| "external Shell frame exceeds address space".to_string())?; + if frame_len > SHELL_REPLAY_FRAME_MAX_BYTES + 3 { + return Err("external Shell UTF-8 frame exceeds its hard bound".to_string()); + } + let mut payload = vec![0_u8; frame_len]; + blob.read_at_exact( + &mut payload, + usize::try_from(local_frame_start).map_err(|_| { + "external Shell payload offset exceeds address space".to_string() + })?, + ) + .map_err(|error| format!("read external Shell payload BLOB: {error}"))?; + let text = String::from_utf8(payload) + .map_err(|_| "external Shell payload is not valid UTF-8".to_string())?; + if !frames.is_empty() + && rendered_response_bytes.saturating_add(text.len()) > SHELL_REPLAY_RANGE_MAX_BYTES + { + break 'segments; + } + rendered_response_bytes = rendered_response_bytes.saturating_add(text.len()); + response_bytes = response_bytes.saturating_add(frame_bytes); + next_offset = frame_end; + frames.push(ShellReplayFrame { + sequence, + stream: segment.stream.as_wire_str().to_string(), + byte_start: frame_start, + byte_end: frame_end, + text, + }); + if next_offset >= visible_end || response_bytes >= limit { + break; + } + } + if next_offset >= visible_end || response_bytes >= limit { + break; + } + } + Ok(ShellReplayRange { + frames, + next_offset_bytes: next_offset, + eof: next_offset >= visible_end, + }) +} + +pub(super) fn external_shell_utf8_boundary( + blob: &rusqlite::blob::Blob<'_>, + candidate: u64, + total_bytes: u64, +) -> Result { + if candidate == 0 || candidate >= total_bytes { + return Ok(candidate.min(total_bytes)); + } + let mut boundary = candidate; + let mut byte = [0_u8; 1]; + blob.read_at_exact( + &mut byte, + usize::try_from(boundary) + .map_err(|_| "external Shell UTF-8 boundary exceeds address space".to_string())?, + ) + .map_err(|error| format!("read external Shell UTF-8 boundary: {error}"))?; + if byte[0] & 0b1100_0000 != 0b1000_0000 { + return Ok(boundary); + } + for _ in 0..3 { + boundary = boundary + .checked_sub(1) + .ok_or_else(|| "invalid external Shell UTF-8 prefix".to_string())?; + blob.read_at_exact( + &mut byte, + usize::try_from(boundary) + .map_err(|_| "external Shell UTF-8 boundary exceeds address space".to_string())?, + ) + .map_err(|error| format!("read external Shell UTF-8 boundary: {error}"))?; + if byte[0] & 0b1100_0000 != 0b1000_0000 { + return Ok(boundary); + } + } + Err("external Shell payload has an invalid UTF-8 boundary".to_string()) +} + +pub(super) fn nonnegative_sqlite_u64(value: i64, label: &str) -> Result { + u64::try_from(value).map_err(|_| format!("invalid negative external Shell {label}: {value}")) +} + +pub(super) fn select_shell_payload_refs(event: &SessionEvent) -> Vec<&PayloadRef> { + let result_refs = event + .payload_refs + .iter() + .filter(|payload_ref| payload_ref.field_path.starts_with("result.")) + .collect::>(); + let suffix = |payload_ref: &PayloadRef, names: &[&str]| { + payload_ref + .field_path + .rsplit('.') + .next() + .is_some_and(|field| names.iter().any(|name| field.eq_ignore_ascii_case(name))) + }; + if let Some(interleaved) = result_refs + .iter() + .find(|payload_ref| suffix(payload_ref, &["interleavedOutput", "aggregated_output"])) + { + return vec![*interleaved]; + } + let mut split_streams = result_refs + .iter() + .filter(|payload_ref| suffix(payload_ref, &["stdout", "stderr"])) + .copied() + .collect::>(); + if !split_streams.is_empty() { + split_streams.sort_by_key(|payload_ref| { + usize::from( + payload_ref + .field_path + .rsplit('.') + .next() + .is_some_and(|field| field.eq_ignore_ascii_case("stderr")), + ) + }); + return split_streams; + } + for name in ["output", "observation", "content"] { + if let Some(payload_ref) = result_refs + .iter() + .find(|payload_ref| suffix(payload_ref, &[name])) + { + return vec![*payload_ref]; + } + } + Vec::new() +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/target.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/target.rs new file mode 100644 index 000000000..9ecf84aa6 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/target.rs @@ -0,0 +1,324 @@ +use super::*; + +pub(super) fn open_foreground_imported_window( + source: ImportedHistorySourceId, + session_id: &str, + limits: ReplayLimits, +) -> Result { + let limits = replay_storage_limits_with_normalization_headroom(limits); + // A visible open is authoritative: synchronize the provider before + // returning the newest compact window. Use an independent connection so + // this does not queue behind the process-wide multi-provider catalog + // mutex; SQLite's IMMEDIATE transaction remains the cross-process writer + // boundary. + with_foreground_replay_connection("replay index open", |conn| { + replay::open_window(conn, source, session_id, limits) + }) +} + +pub(super) fn read_foreground_imported_window( + source: ImportedHistorySourceId, + session_id: &str, + before_sequence: Option, + turn_id: Option<&str>, + turn_index: Option, + limits: ReplayLimits, +) -> Result { + let limits = replay_storage_limits_with_normalization_headroom(limits); + let cached = { + let mut conn = database::db::get_connection() + .map_err(|error| format!("open cached replay page DB: {error}"))?; + if let Some(turn_id) = turn_id { + replay::read_cached_turn_window(&mut conn, source, session_id, turn_id, limits) + } else if let Some(turn_index) = turn_index { + replay::read_cached_turn_window_at_index( + &mut conn, source, session_id, turn_index, limits, + ) + } else { + replay::read_cached_window(&conn, source, session_id, before_sequence, limits) + }? + }; + if let Some(window) = cached { + return Ok(window); + } + with_sessions_replay_writer("replay index", |conn| { + if let Some(turn_id) = turn_id { + replay::read_turn_window(conn, source, session_id, turn_id, limits) + } else if let Some(turn_index) = turn_index { + replay::read_turn_window_at_index(conn, source, session_id, turn_index, limits) + } else { + replay::read_window(conn, source, session_id, before_sequence, limits) + } + }) +} + +pub(super) fn load_replay_query_window( + source_id: &str, + session_id: &str, + before_sequence: Option, + turn_id: Option<&str>, + turn_index: Option, + limits: ReplayLimits, +) -> Result { + let locator_count = usize::from(before_sequence.is_some()) + + usize::from(turn_id.is_some()) + + usize::from(turn_index.is_some()); + if locator_count > 1 { + return Err("beforeSequence, turnId and turnIndex are mutually exclusive".to_string()); + } + match resolve_secondary_consumer_target(source_id, session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => with_sessions_replay_writer("replay query index", |conn| { + let limits = replay_storage_limits_with_normalization_headroom(limits); + if let Some(turn_id) = turn_id { + replay::read_turn_window(conn, source, &imported_session_id, turn_id, limits) + } else if let Some(turn_index) = turn_index { + replay::read_turn_window_at_index( + conn, + source, + &imported_session_id, + turn_index, + limits, + ) + } else { + replay::read_window(conn, source, &imported_session_id, before_sequence, limits) + } + .map(ResolvedReplayWindow::Imported) + }), + ResolvedReplayTarget::CollaborationSnapshot => { + let conn = database::db::get_connection() + .map_err(|err| format!("open collaboration replay query DB: {err}"))?; + collaboration_snapshot_read_window_from_conn( + &conn, + session_id, + before_sequence, + turn_id, + turn_index, + limits, + ) + .map(ResolvedReplayWindow::CollaborationSnapshot) + } + ResolvedReplayTarget::ManagedChunkStore => managed_chunk_read_window( + session_id, + before_sequence, + turn_id, + turn_index, + replay_storage_limits_with_normalization_headroom(limits), + ) + .map(ResolvedReplayWindow::ManagedChunks), + ResolvedReplayTarget::NotReady => Ok(ResolvedReplayWindow::NotReady), + } +} + +pub(super) fn resolve_target( + source_id: &str, + session_id: &str, +) -> Result { + if session_id.starts_with(COLLABORATION_SNAPSHOT_SESSION_PREFIX) { + if source_id != COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID { + return Err(format!( + "Collaboration snapshot replay requires sourceId={COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID}" + )); + } + validate_collaboration_snapshot_session_id(session_id)?; + return Ok(ResolvedReplayTarget::CollaborationSnapshot); + } + if session_id.starts_with("cliagent-") { + let session = crate::agent_sessions::cli::persistence::get_session(session_id) + .map_err(|err| format!("load managed CLI replay target: {err}"))? + .ok_or_else(|| format!("Managed CLI session not found: {session_id}"))?; + if session.transcript_source + == crate::agent_sessions::cli::native_transcript::TRANSCRIPT_SOURCE_NATIVE + { + let Some((binding, cli_session_id)) = + crate::agent_sessions::cli::native_transcript::native_store_key_for_managed_session( + session_id, + ) + else { + return Ok(ResolvedReplayTarget::NotReady); + }; + if source_id != MANAGED_CLI_REPLAY_TARGET_ID && source_id != binding.source { + return Err(format!( + "Managed replay source mismatch: requested {source_id}, bound {}", + binding.source + )); + } + return Ok(ResolvedReplayTarget::Imported { + source: ImportedHistorySourceId::parse(binding.source)?, + imported_session_id: binding.imported_session_id(&cli_session_id), + }); + } + if source_id != MANAGED_CLI_REPLAY_TARGET_ID { + return Err(format!( + "Readerless managed CLI sessions require sourceId={MANAGED_CLI_REPLAY_TARGET_ID}" + )); + } + return Ok(ResolvedReplayTarget::ManagedChunkStore); + } + + let source = ImportedHistorySourceId::parse(source_id)?; + source.validate_session_id(session_id)?; + Ok(ResolvedReplayTarget::Imported { + source, + imported_session_id: session_id.to_string(), + }) +} + +/// Validate only identities admitted to the primary bounded-replay registry. +/// This intentionally excludes snapshot-backed native `agentsession-*` forks, +/// whose compact index is available solely to read-only secondary consumers. +pub(super) fn validate_primary_replay_target_identity( + source_id: &str, + session_id: &str, +) -> Result<(), String> { + if session_id.starts_with(COLLABORATION_SNAPSHOT_SESSION_PREFIX) { + if source_id != COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID { + return Err(format!( + "Collaboration snapshot replay requires sourceId={COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID}" + )); + } + return validate_collaboration_snapshot_session_id(session_id); + } + if session_id.starts_with("cliagent-") { + return (source_id == MANAGED_CLI_REPLAY_TARGET_ID) + .then_some(()) + .ok_or_else(|| { + format!("Managed prewarm requires sourceId={MANAGED_CLI_REPLAY_TARGET_ID}") + }); + } + let source = ImportedHistorySourceId::parse(source_id)?; + source.validate_session_id(session_id) +} + +/// Resolve only the read-only/background consumers that are allowed to reuse +/// a Cloud fork's inherited snapshot index. Foreground open/poll/read/release +/// continue to call `resolve_target`, so a native Agent session can never +/// enter replay execution or acquire a replay watcher through this path. +pub(super) fn resolve_secondary_consumer_target( + source_id: &str, + session_id: &str, +) -> Result { + if session_id.starts_with(COLLABORATION_SNAPSHOT_FORK_PREFIX) { + if source_id != COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID { + return Err(format!( + "Snapshot-backed native fork secondary replay requires sourceId={COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID}" + )); + } + if session_id.len() <= COLLABORATION_SNAPSHOT_FORK_PREFIX.len() + || session_id.contains(['/', '\\']) + { + return Err("Invalid snapshot-backed native fork session id".to_string()); + } + return Ok(ResolvedReplayTarget::CollaborationSnapshot); + } + resolve_target(source_id, session_id) +} + +pub(super) fn ensure_replay_watch( + app: &AppHandle, + source_id: &str, + session_id: &str, + episode_id: u64, + generation: Option<&str>, +) -> bool { + match resolve_replay_watch_paths(source_id, session_id) { + Ok(paths) => acquire_replay_watch_set( + paths, + |path| { + external_replay_watcher::acquire( + app, source_id, session_id, episode_id, generation, path, + ) + }, + || external_replay_watcher::release_session_if_episode(session_id, episode_id), + ), + Err(error) => { + // Watchers are an optimization. A failed lookup must preserve the + // typed `watcherAvailable=false` polling fallback, not fail replay. + external_replay_watcher::release_session_if_episode(session_id, episode_id); + log::debug!("[external-replay] watcher paths unavailable: {error}"); + false + } + } +} + +pub(super) fn acquire_replay_watch_set( + mut paths: Vec, + mut acquire: impl FnMut(&PathBuf) -> bool, + release: impl FnOnce(), +) -> bool { + paths.sort(); + paths.dedup(); + if paths.is_empty() || !paths.iter().all(&mut acquire) { + // All-or-nothing: advertising a healthy primary watcher while a + // storage-specific sidecar is unwatched would suppress the renderer's + // visible 5-second fallback and hide changes for up to 60 seconds. + release(); + return false; + } + true +} + +pub(super) fn resolve_replay_watch_paths( + source_id: &str, + session_id: &str, +) -> Result, String> { + match resolve_target(source_id, session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => { + let conn = database::db::get_connection() + .map_err(|error| format!("open replay watcher index DB: {error}"))?; + replay::watch_paths(&conn, source, &imported_session_id) + } + ResolvedReplayTarget::CollaborationSnapshot => Ok(vec![database::db::get_db_path()]), + ResolvedReplayTarget::ManagedChunkStore => Ok(vec![database::db::get_db_path()]), + ResolvedReplayTarget::NotReady => Ok(Vec::new()), + } +} + +#[cfg(test)] +mod replay_watch_set_tests { + use super::*; + + #[test] + fn multi_path_acquire_deduplicates_and_succeeds_only_when_every_path_is_watched() { + let calls = std::cell::RefCell::new(Vec::new()); + let released = std::cell::Cell::new(false); + let ok = acquire_replay_watch_set( + vec![ + PathBuf::from("/tmp/qoder-transcript"), + PathBuf::from("/tmp/qoder-logs"), + PathBuf::from("/tmp/qoder-logs"), + ], + |path| { + calls.borrow_mut().push(path.clone()); + true + }, + || released.set(true), + ); + assert!(ok); + assert_eq!(calls.borrow().len(), 2); + assert!(!released.get()); + } + + #[test] + fn partial_multi_path_failure_releases_the_session_and_forces_poll_fallback() { + let calls = std::cell::Cell::new(0_usize); + let released = std::cell::Cell::new(false); + let ok = acquire_replay_watch_set( + vec![PathBuf::from("/tmp/a"), PathBuf::from("/tmp/b")], + |_| { + let next = calls.get().saturating_add(1); + calls.set(next); + next == 1 + }, + || released.set(true), + ); + assert!(!ok); + assert_eq!(calls.get(), 2); + assert!(released.get()); + } +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/cloud.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/cloud.rs new file mode 100644 index 000000000..bccad184b --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/cloud.rs @@ -0,0 +1,521 @@ +use super::*; + +#[test] +fn cloud_spool_files_live_under_a_private_runtime_directory() { + let temp = tempfile::tempdir().expect("tempdir"); + let root = temp.path().join("replay-cloud-spools"); + ensure_private_cloud_spool_root(&root).expect("private spool root"); + let (final_path, partial_path) = cloud_spool_paths(&root, "private-fixture"); + create_private_cloud_spool_file(&partial_path).expect("private spool file"); + + assert!(final_path.starts_with(&root)); + assert!(partial_path.starts_with(&root)); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + + assert_eq!( + fs::metadata(&root) + .expect("root metadata") + .permissions() + .mode() + & 0o777, + 0o700 + ); + assert_eq!( + fs::metadata(&partial_path) + .expect("file metadata") + .permissions() + .mode() + & 0o777, + 0o600 + ); + } +} + +#[test] +fn cloud_spool_batch_is_byte_bounded_and_prefix_addressable() { + let token = format!("test-{}", uuid::Uuid::new_v4()); + let path = std::env::temp_dir().join(format!("orgii-cloud-spool-{token}.sqlite")); + let conn = rusqlite::Connection::open(&path).expect("spool db"); + conn.execute_batch( + "CREATE TABLE events ( + event_index INTEGER PRIMARY KEY, + event_hash TEXT NOT NULL, + frozen_chain_hash TEXT NOT NULL + ); + CREATE TABLE frozen_segments ( + segment_index INTEGER PRIMARY KEY, + event_index INTEGER NOT NULL, + payload_gz TEXT NOT NULL, + event_count INTEGER NOT NULL, + segment_hash TEXT NOT NULL, + wire_bytes INTEGER NOT NULL + ); + CREATE TABLE tail_segment ( + singleton INTEGER PRIMARY KEY, + payload_gz TEXT NOT NULL, + event_count INTEGER NOT NULL, + segment_hash TEXT NOT NULL, + wire_bytes INTEGER NOT NULL + );", + ) + .expect("spool schema"); + let events = [ + event("a", "cliagent-test", "one"), + event("b", "cliagent-test", "two"), + event("c", "cliagent-test", "three"), + ]; + let mut first_size = 0_usize; + for (index, event) in events.iter().enumerate() { + let (segment, _) = encode_cloud_frozen_event(event, &mut |_, _| { + panic!("compact event has no deferred payload") + }) + .expect("encode event segment"); + if index == 0 { + first_size = segment.wire_bytes as usize; + } + conn.execute( + "INSERT INTO events VALUES (?1, ?2, ?3)", + rusqlite::params![ + index as i64, + format!("event-{index}"), + format!("chain-{index}"), + ], + ) + .expect("insert event"); + conn.execute( + "INSERT INTO frozen_segments VALUES (?1, ?2, ?3, ?4, ?5, ?6)", + rusqlite::params![ + index as i64, + index as i64, + segment.payload_gz, + segment.event_count as i64, + segment.segment_hash, + segment.wire_bytes as i64, + ], + ) + .expect("insert frozen segment"); + } + conn.execute( + "INSERT INTO tail_segment VALUES (1, 'tail-payload', 1, 'tail-hash', 12)", + [], + ) + .expect("insert tail segment"); + drop(conn); + let manifest = ExternalReplayCloudManifest { + token: token.clone(), + generation: "g1".to_string(), + total_count: 3, + frozen_event_count: 3, + tail_event_count: 0, + frozen_chain_hash: "chain-2".to_string(), + tail_hash: None, + }; + cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .insert( + token.clone(), + CloudSpoolEntry { + path: path.clone(), + manifest, + last_used: Instant::now(), + lease_count: 1, + owner_released: false, + }, + ); + + let batch = + read_cloud_spool_batch(&token, 0, 3, None, Some(first_size + 1)).expect("bounded batch"); + assert_eq!(batch.segments.len(), 1); + assert_eq!(batch.next_event_index, 1); + assert!(!batch.eof); + assert!(batch.serialized_bytes <= (first_size + 1) as u64); + let prefix = cloud_spool_prefix_hash(&token, 2).expect("prefix"); + assert_eq!(prefix.frozen_chain_hash, "chain-1"); + let empty = read_cloud_spool_batch(&token, 3, 3, None, None).expect("empty frozen range"); + assert!(empty.segments.is_empty()); + assert_eq!(empty.next_event_index, 3); + assert!(empty.eof); + + release_cloud_spool(&token).expect("release spool"); + assert!(!path.exists()); +} + +#[test] +fn cloud_spool_release_waits_for_in_flight_read_lease() { + let token = format!("test-lease-{}", uuid::Uuid::new_v4()); + let path = std::env::temp_dir().join(format!("orgii-cloud-spool-{token}.sqlite")); + fs::write(&path, b"leased").expect("leased spool file"); + cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .insert( + token.clone(), + CloudSpoolEntry { + path: path.clone(), + manifest: ExternalReplayCloudManifest { + token: token.clone(), + generation: "g-lease".to_string(), + total_count: 0, + frozen_event_count: 0, + tail_event_count: 0, + frozen_chain_hash: sha256_hex(b""), + tail_hash: None, + }, + last_used: Instant::now(), + lease_count: 1, + owner_released: false, + }, + ); + + let read_lease = acquire_cloud_spool_read(&token).expect("acquire read lease"); + release_cloud_spool(&token).expect("release owner lease"); + assert!(path.exists(), "active read must keep the spool file alive"); + assert!(acquire_cloud_spool_read(&token).is_err()); + + drop(read_lease); + assert!(!path.exists(), "last reader removes the released spool"); +} + +#[test] +fn nine_live_cloud_spools_are_not_lru_evicted() { + let mut entries = Vec::new(); + for index in 0..9 { + let token = format!("test-nine-{index}-{}", uuid::Uuid::new_v4()); + let path = std::env::temp_dir().join(format!("orgii-cloud-spool-{token}.sqlite")); + fs::write(&path, b"live").expect("live spool file"); + cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .insert( + token.clone(), + CloudSpoolEntry { + path: path.clone(), + manifest: ExternalReplayCloudManifest { + token: token.clone(), + generation: format!("g-{index}"), + total_count: 0, + frozen_event_count: 0, + tail_event_count: 0, + frozen_chain_hash: sha256_hex(b""), + tail_hash: None, + }, + last_used: Instant::now(), + lease_count: 1, + owner_released: false, + }, + ); + entries.push((token, path)); + } + + cleanup_cloud_spools(); + + for (token, path) in &entries { + assert!(path.exists(), "live spool {token} was evicted"); + drop(acquire_cloud_spool_read(token).expect("live token remains readable")); + } + for (token, path) in entries { + release_cloud_spool(&token).expect("release live spool"); + assert!(!path.exists()); + } +} + +#[test] +fn cloud_spool_physical_cursor_advances_across_zero_event_v2_parts() { + let token = format!("test-v2-{}", uuid::Uuid::new_v4()); + let path = std::env::temp_dir().join(format!("orgii-cloud-spool-{token}.sqlite")); + let conn = rusqlite::Connection::open(&path).expect("V2 spool db"); + conn.execute_batch( + "CREATE TABLE events ( + event_index INTEGER PRIMARY KEY, + event_hash TEXT NOT NULL, + frozen_chain_hash TEXT NOT NULL + ); + CREATE TABLE frozen_segments ( + segment_index INTEGER PRIMARY KEY, + event_index INTEGER NOT NULL, + payload_gz TEXT NOT NULL, + event_count INTEGER NOT NULL, + segment_hash TEXT NOT NULL, + wire_bytes INTEGER NOT NULL + ); + CREATE TABLE tail_segment ( + singleton INTEGER PRIMARY KEY, + payload_gz TEXT NOT NULL, + event_count INTEGER NOT NULL, + segment_hash TEXT NOT NULL, + wire_bytes INTEGER NOT NULL + );", + ) + .expect("V2 spool schema"); + let attachment_id = sha256_hex(b"event-v2"); + let attachment_hash = sha256_hex(b"abcdef"); + let first = encode_cloud_attachment_frame( + &CloudAttachmentFrameHeader { + kind: "event".to_string(), + attachment_id: attachment_id.clone(), + part_index: 0, + chunk_offset: 0, + chunk_bytes: 3, + final_part: false, + event_bytes: None, + attachment_hash: None, + }, + b"abc", + 0, + ) + .expect("first V2 row"); + let final_segment = encode_cloud_attachment_frame( + &CloudAttachmentFrameHeader { + kind: "event".to_string(), + attachment_id: attachment_id.clone(), + part_index: 1, + chunk_offset: 3, + chunk_bytes: 3, + final_part: true, + event_bytes: Some(6), + attachment_hash: Some(attachment_hash.clone()), + }, + b"def", + 1, + ) + .expect("final V2 row"); + for (segment_index, segment) in [first.clone(), final_segment].into_iter().enumerate() { + conn.execute( + "INSERT INTO frozen_segments VALUES (?1, 0, ?2, ?3, ?4, ?5)", + rusqlite::params![ + segment_index as i64, + segment.payload_gz, + segment.event_count as i64, + segment.segment_hash, + segment.wire_bytes as i64, + ], + ) + .expect("insert V2 physical row"); + } + conn.execute("INSERT INTO events VALUES (0, ?1, ?1)", [&attachment_hash]) + .expect("insert V2 logical event"); + drop(conn); + cloud_spools() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .insert( + token.clone(), + CloudSpoolEntry { + path: path.clone(), + manifest: ExternalReplayCloudManifest { + token: token.clone(), + generation: "g-v2".to_string(), + total_count: 1, + frozen_event_count: 1, + tail_event_count: 0, + frozen_chain_hash: attachment_hash, + tail_hash: None, + }, + last_used: Instant::now(), + lease_count: 1, + owner_released: false, + }, + ); + + let first_batch = read_cloud_spool_batch(&token, 0, 1, None, Some(first.wire_bytes as usize)) + .expect("first V2 physical batch"); + assert_eq!(first_batch.segments.len(), 1); + assert_eq!(first_batch.next_event_index, 0); + assert_eq!(first_batch.next_segment_index, 1); + assert!(!first_batch.eof); + let final_batch = + read_cloud_spool_batch(&token, 0, 1, Some(1), None).expect("final V2 physical batch"); + assert_eq!(final_batch.next_event_index, 1); + assert_eq!(final_batch.next_segment_index, 2); + assert!(final_batch.eof); + + release_cloud_spool(&token).expect("release V2 spool"); +} + +#[test] +fn ten_mib_single_event_is_streamed_into_one_bounded_lossless_cloud_wire() { + let total = 10 * 1024 * 1024; + let mut replay_event = event("large", "cliagent-test", "preview"); + replay_event.result = serde_json::json!({"content":"preview"}); + replay_event.display_text = "preview".to_string(); + replay_event.payload_refs = vec![PayloadRef { + event_id: replay_event.id.clone(), + field_path: "result.content".to_string(), + preview: "preview".to_string(), + full_size_bytes: total, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some("codex_app".to_string()), + replay_generation: Some("g1".to_string()), + replay_source_event_id: Some("source-large".to_string()), + }]; + let mut largest_range = 0_usize; + let (segment, _) = encode_cloud_frozen_event(&replay_event, &mut |_, offset| { + let start = offset as usize; + let bytes = total.saturating_sub(start).min(EXPORT_PAYLOAD_RANGE_BYTES); + largest_range = largest_range.max(bytes); + Ok(ReplayPayloadRange { + event_id: "source-large".to_string(), + field_path: "result.content".to_string(), + offset, + text: "x".repeat(bytes), + next_offset: offset.saturating_add(bytes as u64), + eof: start.saturating_add(bytes) >= total, + total_bytes: total as u64, + }) + }) + .expect("compressible large event wire"); + assert!(largest_range <= EXPORT_PAYLOAD_RANGE_BYTES); + assert!(segment.wire_bytes <= CLOUD_SEGMENT_WIRE_MAX_BYTES as u64); + + let compressed = BASE64_STANDARD + .decode(segment.payload_gz) + .expect("base64 segment"); + let mut decoded = String::new(); + GzDecoder::new(compressed.as_slice()) + .read_to_string(&mut decoded) + .expect("gzip segment"); + let value: serde_json::Value = serde_json::from_str(&decoded).expect("segment JSON"); + assert_eq!( + value[0]["result"]["content"] + .as_str() + .expect("full result") + .len(), + total + ); +} + +#[test] +fn replay_attachment_v2_frame_matches_the_published_golden_hash() { + let attachment_id = sha256_hex(b"golden"); + let attachment_hash = sha256_hex(b"abc"); + let header = CloudAttachmentFrameHeader { + kind: "event".to_string(), + attachment_id, + part_index: 0, + chunk_offset: 0, + chunk_bytes: 3, + final_part: true, + event_bytes: Some(3), + attachment_hash: Some(attachment_hash), + }; + let segment = encode_cloud_attachment_frame(&header, b"abc", 1).expect("golden frame"); + assert_eq!( + segment.segment_hash, + "1cf7b415e8558ddb0d72bcf9212ff381c9a57bfd719628824a61e4a67bcf3126" + ); +} + +#[test] +fn ten_mib_high_entropy_event_round_trips_through_bounded_v2_rows() { + let total = 10 * 1024 * 1024; + let mut replay_event = event("random", "cliagent-test", "preview"); + replay_event.result = serde_json::json!({"content":"preview"}); + replay_event.payload_refs = vec![PayloadRef { + event_id: replay_event.id.clone(), + field_path: "result.content".to_string(), + preview: "preview".to_string(), + full_size_bytes: total, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some("codex_app".to_string()), + replay_generation: Some("g1".to_string()), + replay_source_event_id: Some("source-random".to_string()), + }]; + let mut largest_range = 0_usize; + let mut segments = Vec::new(); + let event_hash = encode_cloud_event_segments( + &replay_event, + &mut |_, offset| { + let start = offset as usize; + let bytes = total.saturating_sub(start).min(EXPORT_PAYLOAD_RANGE_BYTES); + largest_range = largest_range.max(bytes); + let text = (start..start + bytes) + .map(|index| { + let mut value = index as u64 + 0x9e37_79b9_7f4a_7c15; + value = (value ^ (value >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9); + value = (value ^ (value >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb); + let alphabet = + b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; + alphabet[(value ^ (value >> 31)) as usize % alphabet.len()] as char + }) + .collect::(); + Ok(ReplayPayloadRange { + event_id: "source-random".to_string(), + field_path: "result.content".to_string(), + offset, + text, + next_offset: offset.saturating_add(bytes as u64), + eof: start.saturating_add(bytes) >= total, + total_bytes: total as u64, + }) + }, + &mut |segment| { + segments.push(segment); + Ok(()) + }, + ) + .expect("V2 attachment rows"); + + assert!(largest_range <= EXPORT_PAYLOAD_RANGE_BYTES); + assert!(segments.len() > 2); + assert_eq!( + segments + .iter() + .map(|segment| segment.event_count) + .sum::(), + 1 + ); + let mut hydrated_event = Vec::new(); + for (part_index, segment) in segments.iter().enumerate() { + assert!(segment.wire_bytes <= CLOUD_SEGMENT_WIRE_MAX_BYTES as u64); + let compressed = BASE64_STANDARD + .decode(&segment.payload_gz) + .expect("base64 V2 frame"); + let mut frame = Vec::new(); + GzDecoder::new(compressed.as_slice()) + .read_to_end(&mut frame) + .expect("gzip V2 frame"); + assert_eq!(sha256_hex(&frame), segment.segment_hash); + assert!(frame.starts_with(CLOUD_ATTACHMENT_V2_MAGIC)); + let header_offset = CLOUD_ATTACHMENT_V2_MAGIC.len(); + let header_len = u32::from_be_bytes( + frame[header_offset..header_offset + 4] + .try_into() + .expect("V2 header length"), + ) as usize; + let payload_offset = header_offset + 4 + header_len; + let header: serde_json::Value = + serde_json::from_slice(&frame[header_offset + 4..payload_offset]) + .expect("V2 header JSON"); + assert_eq!(header["partIndex"].as_u64(), Some(part_index as u64)); + assert_eq!( + header["chunkOffset"].as_u64(), + Some(hydrated_event.len() as u64) + ); + let final_part = part_index + 1 == segments.len(); + assert_eq!(header["finalPart"].as_bool(), Some(final_part)); + assert_eq!(segment.event_count, u64::from(final_part)); + hydrated_event.extend_from_slice(&frame[payload_offset..]); + if final_part { + assert_eq!( + header["eventBytes"].as_u64(), + Some(hydrated_event.len() as u64) + ); + assert_eq!(header["attachmentHash"].as_str(), Some(event_hash.as_str())); + } + } + assert_eq!(sha256_hex(&hydrated_event), event_hash); + let value: serde_json::Value = + serde_json::from_slice(&hydrated_event).expect("hydrated event JSON"); + assert_eq!( + value["result"]["content"] + .as_str() + .expect("full V2 result") + .len(), + total + ); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/collaboration.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/collaboration.rs new file mode 100644 index 000000000..a3b6e688a --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/collaboration.rs @@ -0,0 +1,632 @@ +use super::*; + +fn collaboration_snapshot_test_schema(conn: &rusqlite::Connection) { + conn.execute_batch( + "CREATE TABLE events ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + event_type TEXT NOT NULL, + function_name TEXT, + thread_id TEXT, + args_json TEXT NOT NULL DEFAULT '{}', + result_json TEXT NOT NULL DEFAULT '{}', + content TEXT NOT NULL DEFAULT '', + created_at TEXT NOT NULL, + meta_json TEXT, + history_sequence INTEGER + ); + CREATE INDEX idx_test_events_session_sequence + ON events(session_id,history_sequence);", + ) + .expect("collaboration snapshot schema"); +} + +fn collaboration_snapshot_meta(source: &str, display_text: &str) -> String { + serde_json::to_string(&serde_json::json!({ + "source": source, + "displayText": display_text, + "displayStatus": "completed", + "displayVariant": "message", + "activityStatus": "processed", + "uiCanonical": if source == "user" { "user_message" } else { "assistant_message" }, + })) + .expect("snapshot event metadata") +} + +#[test] +fn collaboration_snapshot_is_special_and_never_matches_native_agent_ids() { + assert!(matches!( + resolve_target( + COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID, + "imported-session-test" + ), + Ok(ResolvedReplayTarget::CollaborationSnapshot) + )); + assert!(resolve_target(COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID, "sdeagent-native").is_err()); + assert!(resolve_target("codex_app", "imported-session-test").is_err()); + assert_eq!(ImportedHistorySourceId::ALL.len(), 15); +} + +#[test] +fn collaboration_snapshot_window_is_bounded_ranges_ten_mib_and_polls_true_deltas() { + let mut conn = rusqlite::Connection::open_in_memory().expect("snapshot replay DB"); + collaboration_snapshot_test_schema(&conn); + let session_id = "imported-session-bounded"; + let large_content = format!("{}END", "snapshot-output\n".repeat(700_000)); + let large_result = serde_json::to_string(&serde_json::json!({ + "content": large_content, + "status": "done", + })) + .expect("large snapshot result"); + { + let tx = conn.transaction().expect("snapshot insert transaction"); + let mut insert = tx + .prepare( + "INSERT INTO events( + id,session_id,event_type,function_name,thread_id,args_json, + result_json,content,created_at,meta_json,history_sequence + ) VALUES(?1,?2,?3,?4,NULL,'{}',?5,'',?6,?7,?8)", + ) + .expect("snapshot insert statement"); + for sequence in 0..205_i64 { + let user = sequence == 0; + let event_id = format!("snapshot-{sequence}"); + let result = if sequence == 204 { + large_result.as_str() + } else { + "{\"content\":\"small\"}" + }; + insert + .execute(rusqlite::params![ + event_id, + session_id, + if user { "user_message" } else { "assistant" }, + if user { + "user_message" + } else { + "assistant_message" + }, + result, + format!("2026-07-22T00:{:02}:{:02}Z", sequence / 60, sequence % 60), + collaboration_snapshot_meta( + if user { "user" } else { "assistant" }, + if user { "start" } else { "answer" }, + ), + sequence, + ]) + .expect("insert snapshot row"); + } + drop(insert); + tx.commit().expect("commit snapshot rows"); + } + + let limits = ReplayLimits { + max_turns: 1, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, + }; + let window = + collaboration_snapshot_read_window_from_conn(&conn, session_id, None, None, None, limits) + .expect("bounded collaboration window"); + assert_eq!(window.events.len(), 200); + assert_eq!(window.total_event_count, 205); + assert!(window.has_older); + assert!(window.stats.ipc_bytes < 4 * 1024 * 1024); + assert_eq!( + window.events.first().map(|event| event.id.as_str()), + Some("snapshot-5") + ); + assert_eq!( + window.events.last().map(|event| event.id.as_str()), + Some("snapshot-204") + ); + assert_eq!(window.window_start_sequence, Some(5)); + let older = collaboration_snapshot_read_window_from_conn( + &conn, + session_id, + window.window_start_sequence, + None, + None, + limits, + ) + .expect("read remaining collaboration turn prefix"); + assert_eq!( + older + .events + .iter() + .map(|event| event.id.as_str()) + .collect::>(), + vec![ + "snapshot-0", + "snapshot-1", + "snapshot-2", + "snapshot-3", + "snapshot-4", + ] + ); + assert_eq!(older.window_start_sequence, Some(0)); + assert!(!older.has_older); + + let exact = collaboration_snapshot_read_window_from_conn( + &conn, + session_id, + None, + None, + Some(0), + limits, + ) + .expect("read exact collaboration turn"); + assert_eq!(exact.events.len(), 200); + assert_eq!( + exact.events.first().map(|event| event.id.as_str()), + Some("snapshot-0") + ); + assert_eq!( + exact.events.get(1).map(|event| event.id.as_str()), + Some("snapshot-6") + ); + assert_eq!( + exact.events.last().map(|event| event.id.as_str()), + Some("snapshot-204") + ); + assert_eq!(exact.window_start_sequence, Some(6)); + assert!(exact.has_older); + assert_eq!(exact.turn_headers.len(), 1); + assert_eq!(exact.turn_headers[0].event_count, 205); + + let large = window + .events + .iter() + .find(|event| event.id == "snapshot-204") + .expect("large event remains in latest window"); + let payload = large + .payload_refs + .iter() + .find(|reference| reference.field_path == "result") + .expect("large result is deferred at the canonical root"); + assert!(payload.full_size_bytes > 10 * 1024 * 1024); + + let mut rebuilt = String::new(); + let mut offset = 0_u64; + loop { + let range = collaboration_snapshot_payload_range_from_conn( + &conn, + session_id, + &window.cursor.generation, + "snapshot-204", + "result", + offset, + replay::HARD_MAX_PAYLOAD_RANGE_BYTES, + ) + .expect("bounded collaboration payload range"); + assert!(range.text.len() <= replay::HARD_MAX_PAYLOAD_RANGE_BYTES); + rebuilt.push_str(&range.text); + if range.eof { + break; + } + assert!(range.next_offset > offset); + offset = range.next_offset; + } + assert_eq!(rebuilt, large_result); + + let unchanged = + collaboration_snapshot_poll_delta_from_conn(&conn, session_id, &window.cursor, limits) + .expect("unchanged collaboration poll"); + assert!(unchanged.events.is_empty()); + assert_eq!(unchanged.stats.parsed_rows, 0); + assert!(!unchanged.reset_required); + + conn.execute( + "INSERT INTO events( + id,session_id,event_type,function_name,args_json,result_json, + content,created_at,meta_json,history_sequence + ) VALUES( + 'snapshot-205',?1,'assistant','assistant_message','{}', + '{\"content\":\"appended\"}','','2026-07-22T00:03:25Z',?2,205 + )", + rusqlite::params![ + session_id, + collaboration_snapshot_meta("assistant", "appended") + ], + ) + .expect("append collaboration event"); + let delta = + collaboration_snapshot_poll_delta_from_conn(&conn, session_id, &window.cursor, limits) + .expect("collaboration append delta"); + assert_eq!(delta.events.len(), 1); + assert_eq!(delta.events[0].id, "snapshot-205"); + assert_eq!(delta.cursor.generation, window.cursor.generation); + assert!(!delta.reset_required); + + conn.execute( + "UPDATE events SET result_json='{\"content\":\"rewritten\"}' + WHERE id='snapshot-205'", + [], + ) + .expect("rewrite collaboration event"); + let reset = + collaboration_snapshot_poll_delta_from_conn(&conn, session_id, &delta.cursor, limits) + .expect("collaboration rewrite reset"); + assert!(reset.reset_required); + assert_ne!(reset.cursor.generation, delta.cursor.generation); + + conn.execute("DELETE FROM events WHERE session_id=?1", [session_id]) + .expect("bulk-delete collaboration snapshot"); + let deleted = collaboration_snapshot_state(&conn, session_id) + .expect("refresh state once after bulk delete"); + assert_eq!(deleted.event_count, 0); + assert_eq!(deleted.max_sequence, -1); +} + +#[test] +fn collaboration_exact_small_turn_keeps_the_anchor_as_its_continuation_boundary() { + let conn = rusqlite::Connection::open_in_memory().expect("snapshot replay DB"); + collaboration_snapshot_test_schema(&conn); + let session_id = "imported-session-small-exact"; + for (sequence, source) in [(0_i64, "user"), (1_i64, "assistant")] { + conn.execute( + "INSERT INTO events( + id,session_id,event_type,function_name,args_json,result_json, + content,created_at,meta_json,history_sequence + ) VALUES(?1,?2,?3,?4,'{}','{}','',?5,?6,?7)", + rusqlite::params![ + format!("small-{sequence}"), + session_id, + if source == "user" { + "user_message" + } else { + "assistant" + }, + if source == "user" { + "user_message" + } else { + "assistant_message" + }, + format!("2026-07-22T00:00:0{sequence}Z"), + collaboration_snapshot_meta(source, source), + sequence, + ], + ) + .expect("insert small collaboration event"); + } + + let window = collaboration_snapshot_read_window_from_conn( + &conn, + session_id, + None, + None, + Some(0), + ReplayLimits::default(), + ) + .expect("read exact small collaboration turn"); + assert_eq!( + window + .events + .iter() + .map(|event| event.id.as_str()) + .collect::>(), + vec!["small-0", "small-1"] + ); + assert_eq!(window.window_start_sequence, Some(0)); + assert!(!window.has_older); + assert_eq!(window.turn_headers[0].event_count, 2); +} + +#[test] +fn snapshot_backed_native_fork_secondary_replay_keeps_native_execution_isolated() { + let mut conn = rusqlite::Connection::open_in_memory().expect("native fork replay DB"); + session_persistence::init_session_tables(&conn).expect("native session schema"); + crate::agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::install_snapshot_schema_for_test(&mut conn) + .expect("collaboration snapshot schema"); + let session_id = "agentsession-snapshot-secondary"; + let inherited_count = 1_000_i64; + { + let tx = conn + .transaction() + .expect("seed inherited snapshot transaction"); + let mut insert_event = tx + .prepare( + "INSERT INTO events( + id,session_id,event_type,function_name,thread_id,args_json, + result_json,content,created_at,meta_json,history_sequence + ) VALUES(?1,?2,?3,?4,NULL,?5,?6,'',?7,?8,?9)", + ) + .expect("prepare inherited event insert"); + let mut insert_map = tx + .prepare( + "INSERT INTO collaboration_snapshot_event_map( + session_id,event_id,original_id,physical_seq,event_index, + logical_index,is_tail + ) VALUES(?1,?2,?3,?4,0,?5,0)", + ) + .expect("prepare inherited map insert"); + for sequence in 0..inherited_count { + let event_id = format!("{session_id}~inherited-{sequence}"); + let user = sequence == 0; + insert_event + .execute(rusqlite::params![ + event_id, + session_id, + if user { "user_message" } else { "assistant" }, + if user { + "user_message" + } else { + "assistant_message" + }, + if user { + "{\"content\":\"inherited question\"}" + } else { + "{}" + }, + if user { + "{}" + } else { + "{\"content\":\"inherited answer\"}" + }, + format!("2026-07-22T00:{:02}:{:02}Z", sequence / 60, sequence % 60), + collaboration_snapshot_meta( + if user { "user" } else { "assistant" }, + if user { + "inherited question" + } else { + "inherited answer" + }, + ), + sequence, + ]) + .expect("insert inherited event"); + insert_map + .execute(rusqlite::params![ + session_id, + event_id, + format!("inherited-{sequence}"), + sequence + 1, + sequence, + ]) + .expect("insert inherited map row"); + } + drop(insert_map); + drop(insert_event); + tx.execute( + "INSERT INTO sessions(session_id,event_count,cached_at) + VALUES(?1,?2,0)", + rusqlite::params![session_id, inherited_count], + ) + .expect("insert native fork session row"); + tx.execute( + "INSERT INTO collaboration_snapshot_ingest_state( + session_id,epoch,frozen_seq,event_count,frozen_event_count, + tail_hash,updated_at + ) VALUES(?1,7,?2,?2,?2,NULL,0)", + rusqlite::params![session_id, inherited_count], + ) + .expect("insert native fork snapshot cursor"); + tx.commit().expect("commit inherited snapshot"); + } + + assert!(resolve_target(COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID, session_id).is_err()); + assert!(matches!( + resolve_secondary_consumer_target(COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID, session_id), + Ok(ResolvedReplayTarget::CollaborationSnapshot) + )); + let replay_state_table: i64 = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master + WHERE type='table' AND name='collaboration_replay_state')", + [], + |row| row.get(0), + ) + .expect("inspect imported replay accounting table"); + assert_eq!(replay_state_table, 0); + let frontier_plan = conn + .prepare( + "EXPLAIN QUERY PLAN SELECT history_sequence FROM events + WHERE session_id=?1 AND history_sequence IS NOT NULL + ORDER BY history_sequence DESC LIMIT 1", + ) + .expect("prepare native fork frontier plan") + .query_map([session_id], |row| row.get::<_, String>(3)) + .expect("query native fork frontier plan") + .collect::>>() + .expect("collect native fork frontier plan"); + assert!(frontier_plan + .iter() + .any(|detail| detail.contains("idx_events_session_sequence"))); + + let limits = ReplayLimits { + max_turns: 1, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, + }; + let inherited_window = + collaboration_snapshot_read_window_from_conn(&conn, session_id, None, None, None, limits) + .expect("read bounded inherited window"); + assert_eq!(inherited_window.events.len(), 200); + assert_eq!(inherited_window.total_event_count, inherited_count as u64); + assert_eq!( + inherited_window.cursor.through_sequence, + inherited_count - 1 + ); + + let suffix_user_id = format!("{session_id}~native-user"); + let suffix_assistant_id = format!("{session_id}~native-assistant"); + let large_result = serde_json::to_string(&serde_json::json!({ + "content": format!("{}END", "native-suffix-output\n".repeat(550_000)), + "status": "done", + })) + .expect("large native suffix result"); + { + let tx = conn + .transaction() + .expect("append native suffix transaction"); + tx.execute( + "INSERT INTO events( + id,session_id,event_type,function_name,args_json,result_json, + content,created_at,meta_json,history_sequence + ) VALUES(?1,?2,'user_message','user_message',?3,'{}','',?4,?5,?6)", + rusqlite::params![ + suffix_user_id, + session_id, + "{\"content\":\"native suffix question\"}", + "2026-07-22T01:00:00Z", + collaboration_snapshot_meta("user", "native suffix question"), + inherited_count, + ], + ) + .expect("insert native suffix user"); + tx.execute( + "INSERT INTO events( + id,session_id,event_type,function_name,args_json,result_json, + content,created_at,meta_json,history_sequence + ) VALUES(?1,?2,'assistant','assistant_message','{}',?3,'',?4,?5,?6)", + rusqlite::params![ + suffix_assistant_id, + session_id, + large_result, + "2026-07-22T01:00:01Z", + collaboration_snapshot_meta("assistant", "native suffix answer"), + inherited_count + 1, + ], + ) + .expect("insert native suffix assistant"); + tx.execute( + "UPDATE sessions SET event_count=?2 WHERE session_id=?1", + rusqlite::params![session_id, inherited_count + 2], + ) + .expect("publish native suffix count"); + tx.commit().expect("commit native suffix"); + } + + let appended = collaboration_snapshot_poll_delta_from_conn( + &conn, + session_id, + &inherited_window.cursor, + limits, + ) + .expect("poll native suffix delta"); + assert!(!appended.reset_required); + assert_eq!( + appended.cursor.generation, + inherited_window.cursor.generation + ); + assert!(appended.cursor.revision > inherited_window.cursor.revision); + assert_eq!( + appended + .events + .iter() + .map(|event| event.id.as_str()) + .collect::>(), + vec![suffix_user_id.as_str(), suffix_assistant_id.as_str()] + ); + + let suffix_turn = collaboration_snapshot_read_window_from_conn( + &conn, + session_id, + None, + Some(&suffix_user_id), + None, + limits, + ) + .expect("address native suffix turn"); + assert_eq!(suffix_turn.events.len(), 2); + assert_eq!(suffix_turn.events[1].id, suffix_assistant_id); + let payload = suffix_turn.events[1] + .payload_refs + .iter() + .find(|reference| reference.field_path == "result") + .expect("native suffix large result is deferred"); + assert!(payload.full_size_bytes > 10 * 1024 * 1024); + + let mut expected_hash = Sha256::new(); + expected_hash.update(large_result.as_bytes()); + let expected_hash = format!("{:x}", expected_hash.finalize()); + let mut actual_hash = Sha256::new(); + let mut offset = 0_u64; + loop { + let range = collaboration_snapshot_payload_range_from_conn( + &conn, + session_id, + &appended.cursor.generation, + &suffix_assistant_id, + "result", + offset, + replay::HARD_MAX_PAYLOAD_RANGE_BYTES, + ) + .expect("read native suffix payload range"); + assert!(range.text.len() <= replay::HARD_MAX_PAYLOAD_RANGE_BYTES); + actual_hash.update(range.text.as_bytes()); + if range.eof { + break; + } + assert!(range.next_offset > offset); + offset = range.next_offset; + } + assert_eq!(format!("{:x}", actual_hash.finalize()), expected_hash); + + let batch_limits = ReplayLimits { + max_turns: 10, + max_events: 17, + max_ipc_bytes: 4 * 1024 * 1024, + }; + let mut lower = -1_i64; + let mut ordered_ids = Vec::new(); + loop { + let batch = query_collaboration_snapshot_events( + &conn, + session_id, + &appended.cursor.generation, + lower, + i64::MAX, + batch_limits, + false, + ) + .expect("stream native fork event batch"); + assert!(batch.len() <= 17); + if batch.is_empty() { + break; + } + lower = batch.last().expect("non-empty batch").0; + ordered_ids.extend(batch.into_iter().map(|(_, event)| event.id)); + } + assert_eq!(ordered_ids.len(), (inherited_count + 2) as usize); + assert_eq!( + ordered_ids.first(), + Some(&format!("{session_id}~inherited-0")) + ); + assert_eq!( + &ordered_ids[ordered_ids.len() - 2..], + &[suffix_user_id.clone(), suffix_assistant_id.clone()] + ); + + conn.execute( + "UPDATE events SET result_json='{\"content\":\"rewritten\"}' WHERE id=?1", + [&suffix_assistant_id], + ) + .expect("rewrite native suffix event"); + let rewritten = + collaboration_snapshot_poll_delta_from_conn(&conn, session_id, &appended.cursor, limits) + .expect("poll rewritten native suffix"); + assert!(rewritten.reset_required); + assert_eq!(rewritten.cursor.generation, appended.cursor.generation); + assert!(rewritten.cursor.revision > appended.cursor.revision); + + conn.execute("DELETE FROM events WHERE id=?1", [&suffix_assistant_id]) + .expect("delete native suffix event"); + conn.execute( + "UPDATE sessions SET event_count=?2 WHERE session_id=?1", + rusqlite::params![session_id, inherited_count + 1], + ) + .expect("publish native suffix delete count"); + let delete_delta = + collaboration_snapshot_poll_delta_from_conn(&conn, session_id, &rewritten.cursor, limits) + .expect("poll deleted native suffix"); + assert!(delete_delta.reset_required); + assert_eq!(delete_delta.cursor.generation, rewritten.cursor.generation); + let deleted = crate::agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_secondary_state( + &conn, session_id, + ) + .expect("read deleted native suffix state") + .expect("native fork remains snapshot-backed after suffix delete"); + assert_eq!(deleted.event_count, (inherited_count + 1) as u64); + assert_eq!(deleted.max_sequence, inherited_count); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/managed_chunks.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/managed_chunks.rs new file mode 100644 index 000000000..c76a7a25b --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/managed_chunks.rs @@ -0,0 +1,786 @@ +use super::*; + +fn managed_replay_test_schema(conn: &rusqlite::Connection) { + conn.execute_batch( + "CREATE TABLE code_session_chunks ( + chunk_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + action_type TEXT NOT NULL, + function TEXT NOT NULL, + args_json TEXT NOT NULL, + result_json TEXT NOT NULL, + thread_id TEXT, + process_id TEXT, + sequence INTEGER NOT NULL, + created_at TEXT NOT NULL + ); + CREATE TABLE code_session_history_mutations ( + session_id TEXT PRIMARY KEY, + epoch INTEGER NOT NULL + );", + ) + .expect("managed replay schema"); +} + +fn insert_managed_replay_chunk( + conn: &rusqlite::Connection, + session_id: &str, + chunk_id: &str, + sequence: i64, + function: &str, +) { + let action_type = match function { + "user_message" => "user", + "assistant_message" => "assistant", + _ => "tool_call", + }; + conn.execute( + "INSERT INTO code_session_chunks( + chunk_id,session_id,action_type,function,args_json,result_json, + thread_id,process_id,sequence,created_at + ) VALUES(?1,?2,?3,?4,'{}','{}',NULL,NULL,?5,?6)", + rusqlite::params![ + chunk_id, + session_id, + action_type, + function, + sequence, + format!("2026-07-22T00:00:{sequence:02}Z") + ], + ) + .expect("insert managed replay chunk"); +} + +fn managed_replay_limits(max_turns: usize) -> ReplayLimits { + ReplayLimits { + max_turns, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, + } +} + +#[test] +fn managed_chunk_replacement_resets_old_window_but_append_stays_delta() { + let _sandbox = test_helpers::test_env::sandbox(); + let conn = database::db::get_connection().expect("managed replay test DB"); + crate::agent_sessions::cli::init_cli_agent_tables(&conn).expect("managed CLI schema"); + for session_id in ["cliagent-epoch-old", "cliagent-epoch-new"] { + conn.execute( + "INSERT INTO code_sessions(session_id,created_at,updated_at) + VALUES(?1,'2026-07-23T00:00:00Z','2026-07-23T00:00:00Z')", + [session_id], + ) + .expect("managed replay session"); + } + drop(conn); + + let make_chunk = |chunk_id: &str, session_id: &str, output: &str| { + let mut chunk = ActivityChunk::new(session_id, "tool_result", "run_command_line"); + chunk.chunk_id = chunk_id.to_string(); + chunk.args = serde_json::json!({"command":"printf replay"}); + chunk.result = serde_json::json!({"output":output}); + chunk.created_at = "2026-07-23T00:00:00Z".to_string(); + chunk + }; + let original = make_chunk("epoch-shell", "cliagent-epoch-old", "AAAA"); + crate::agent_sessions::cli::persistence::insert_chunk(&original, 0) + .expect("initial managed append"); + let opened = managed_chunk_open_window("cliagent-epoch-old", managed_replay_limits(1)) + .expect("initial managed window"); + assert_eq!(opened.cursor.generation, "chunks-0"); + + let appended = make_chunk("epoch-append", "cliagent-epoch-old", "tail"); + crate::agent_sessions::cli::persistence::insert_chunk(&appended, 1) + .expect("managed append delta"); + let append_delta = managed_chunk_poll_delta( + "cliagent-epoch-old", + &opened.cursor, + managed_replay_limits(1), + ) + .expect("managed append poll"); + assert!(!append_delta.reset_required); + assert_eq!(append_delta.cursor.generation, opened.cursor.generation); + assert_eq!(append_delta.chunks.len(), 1); + + crate::agent_sessions::cli::persistence::insert_chunk(&original, 0) + .expect("idempotent managed replace"); + let unchanged = managed_chunk_poll_delta( + "cliagent-epoch-old", + &append_delta.cursor, + managed_replay_limits(1), + ) + .expect("idempotent managed poll"); + assert!(!unchanged.reset_required); + assert!(unchanged.chunks.is_empty()); + + let changed = make_chunk("epoch-shell", "cliagent-epoch-old", "BBBB"); + crate::agent_sessions::cli::persistence::insert_chunk(&changed, 0) + .expect("same-length managed replacement"); + let reset = managed_chunk_poll_delta( + "cliagent-epoch-old", + &append_delta.cursor, + managed_replay_limits(1), + ) + .expect("managed replacement reset"); + assert!(reset.reset_required); + assert_eq!(reset.cursor.generation, "chunks-1"); + + let moved = make_chunk("epoch-shell", "cliagent-epoch-new", "BBBB"); + crate::agent_sessions::cli::persistence::insert_chunk(&moved, 0) + .expect("cross-session managed replacement"); + let old_reset = managed_chunk_poll_delta( + "cliagent-epoch-old", + &reset.cursor, + managed_replay_limits(1), + ) + .expect("old session reset after move"); + assert!(old_reset.reset_required); + assert_eq!(old_reset.cursor.generation, "chunks-2"); + let new_window = managed_chunk_open_window("cliagent-epoch-new", managed_replay_limits(1)) + .expect("new session window after move"); + assert_eq!(new_window.cursor.generation, "chunks-1"); + assert!(new_window + .chunks + .iter() + .any(|chunk| chunk.chunk.chunk_id == "epoch-shell")); +} + +#[test] +fn readerless_managed_cli_pages_compact_turn_headers_without_duplicates() { + let conn = rusqlite::Connection::open_in_memory().expect("managed replay DB"); + managed_replay_test_schema(&conn); + conn.execute( + "INSERT INTO code_session_history_mutations VALUES(?1, 3)", + ["cliagent-turns"], + ) + .expect("managed replay generation"); + for (chunk_id, sequence, function) in [ + ("u0", 0, "user_message"), + ("a0", 1, "assistant_message"), + ("u1", 2, "user_message"), + ("t1", 3, "run_command_line"), + ("a1", 4, "assistant_message"), + ("u2", 5, "user_message"), + ("a2", 6, "assistant_message"), + ] { + insert_managed_replay_chunk(&conn, "cliagent-turns", chunk_id, sequence, function); + } + + let latest = managed_chunk_read_window_from_conn( + &conn, + "cliagent-turns", + None, + None, + None, + managed_replay_limits(1), + ) + .expect("latest managed turn"); + assert_eq!(latest.cursor.generation, "chunks-3"); + assert_eq!(latest.total_turn_count, 3); + assert_eq!(latest.total_event_count, 7); + assert!(latest.has_older); + assert_eq!(latest.turn_headers.len(), 1); + assert_eq!(latest.turn_headers[0].turn_id, "u2"); + assert_eq!(latest.turn_headers[0].turn_index, 2); + assert_eq!(latest.turn_headers[0].start_sequence, 5); + assert_eq!(latest.turn_headers[0].end_sequence, Some(6)); + assert_eq!(latest.turn_headers[0].event_count, 2); + assert_eq!( + latest + .chunks + .iter() + .map(|chunk| (chunk.sequence, chunk.turn_index)) + .collect::>(), + vec![(5, 2), (6, 2)] + ); + + let middle = managed_chunk_read_window_from_conn( + &conn, + "cliagent-turns", + None, + None, + Some(1), + managed_replay_limits(1), + ) + .expect("middle managed turn by index"); + assert_eq!(middle.turn_headers[0].turn_id, "u1"); + assert_eq!( + middle + .chunks + .iter() + .map(|chunk| (chunk.sequence, chunk.turn_index)) + .collect::>(), + vec![(2, 1), (3, 1), (4, 1)] + ); + + let oldest = managed_chunk_read_window_from_conn( + &conn, + "cliagent-turns", + None, + Some("u0"), + None, + managed_replay_limits(1), + ) + .expect("oldest managed turn by id"); + assert!(!oldest.has_older); + assert_eq!(oldest.turn_headers[0].turn_index, 0); + assert_eq!( + oldest + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(), + vec![0, 1] + ); + + let prior = managed_chunk_read_window_from_conn( + &conn, + "cliagent-turns", + Some(latest.turn_headers[0].start_sequence), + None, + None, + managed_replay_limits(1), + ) + .expect("managed turn before latest"); + assert_eq!(prior.turn_headers[0].turn_id, "u1"); + assert_eq!(prior.cursor.generation, latest.cursor.generation); + assert_eq!(prior.cursor.revision, latest.cursor.revision); + assert_ne!( + prior.cursor.through_sequence, + latest.cursor.through_sequence + ); + assert_eq!( + prior + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(), + vec![2, 3, 4] + ); + assert!(prior.chunks.iter().all(|chunk| !latest + .chunks + .iter() + .any(|item| item.sequence == chunk.sequence))); + + let latest_two = managed_chunk_read_window_from_conn( + &conn, + "cliagent-turns", + None, + None, + None, + managed_replay_limits(2), + ) + .expect("latest two managed turns"); + assert_eq!( + latest_two + .turn_headers + .iter() + .map(|header| (&header.turn_id, header.turn_index)) + .collect::>(), + vec![(&"u1".to_string(), 1), (&"u2".to_string(), 2)] + ); + assert_eq!( + latest_two + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(), + vec![2, 3, 4, 5, 6] + ); + + assert!(managed_chunk_read_window_from_conn( + &conn, + "cliagent-turns", + None, + Some("a1"), + None, + managed_replay_limits(1), + ) + .expect_err("non-anchor turn id must fail") + .contains("no longer available")); + assert!(managed_chunk_read_window_from_conn( + &conn, + "cliagent-turns", + None, + None, + Some(3), + managed_replay_limits(1), + ) + .expect_err("stale turn index must fail") + .contains("no longer available")); +} + +#[test] +fn readerless_large_single_turn_pages_from_actual_window_start_without_gaps() { + let conn = rusqlite::Connection::open_in_memory().expect("managed replay DB"); + managed_replay_test_schema(&conn); + let session_id = "cliagent-large-single-turn"; + conn.execute( + "INSERT INTO code_session_history_mutations VALUES(?1, 11)", + [session_id], + ) + .expect("managed replay generation"); + insert_managed_replay_chunk(&conn, session_id, "user-0", 0, "user_message"); + for sequence in 1..=450_i64 { + insert_managed_replay_chunk( + &conn, + session_id, + &format!("assistant-{sequence}"), + sequence, + "assistant_message", + ); + } + let filtered_page = normalize_window( + managed_chunk_read_window_from_conn( + &conn, + session_id, + None, + None, + None, + managed_replay_limits(1), + ) + .expect("filtered large-turn page"), + session_id, + ); + assert!(filtered_page.events.is_empty()); + assert_eq!(filtered_page.window_start_sequence, Some(251)); + assert!(filtered_page.has_older); + conn.execute( + "UPDATE code_session_chunks + SET result_json=json_object('content',chunk_id) + WHERE session_id=?1", + [session_id], + ) + .expect("make large-turn chunks visible to normalization"); + + let latest_chunks = managed_chunk_read_window_from_conn( + &conn, + session_id, + None, + None, + None, + managed_replay_limits(1), + ) + .expect("latest large-turn page"); + let latest_sequences = latest_chunks + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(); + assert_eq!(latest_sequences.len(), 200); + assert_eq!(latest_sequences.first(), Some(&251)); + assert_eq!(latest_sequences.last(), Some(&450)); + let latest = normalize_window(latest_chunks, session_id); + assert_eq!(latest.window_start_sequence, Some(251)); + assert_eq!(latest.turn_headers[0].start_sequence, 0); + assert!(latest.has_older); + + let middle_chunks = managed_chunk_read_window_from_conn( + &conn, + session_id, + latest.window_start_sequence, + None, + None, + managed_replay_limits(1), + ) + .expect("middle large-turn page"); + let middle_sequences = middle_chunks + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(); + assert_eq!(middle_sequences.len(), 200); + assert_eq!(middle_sequences.first(), Some(&51)); + assert_eq!(middle_sequences.last(), Some(&250)); + let middle = normalize_window(middle_chunks, session_id); + assert_eq!(middle.window_start_sequence, Some(51)); + assert_eq!(middle.turn_headers[0].start_sequence, 0); + assert!(middle.has_older); + assert_eq!(middle.cursor.generation, latest.cursor.generation); + assert_eq!(middle.cursor.revision, latest.cursor.revision); + + let oldest_chunks = managed_chunk_read_window_from_conn( + &conn, + session_id, + middle.window_start_sequence, + None, + None, + managed_replay_limits(1), + ) + .expect("oldest large-turn page"); + let oldest_sequences = oldest_chunks + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(); + assert_eq!(oldest_sequences.len(), 51); + assert_eq!(oldest_sequences.first(), Some(&0)); + assert_eq!(oldest_sequences.last(), Some(&50)); + let oldest = normalize_window(oldest_chunks, session_id); + assert_eq!(oldest.window_start_sequence, Some(0)); + assert!(!oldest.has_older); + + let mut sequences = oldest_sequences + .into_iter() + .chain(middle_sequences) + .chain(latest_sequences) + .collect::>(); + sequences.sort_unstable(); + sequences.dedup(); + assert_eq!(sequences, (0..=450_i64).collect::>()); +} + +#[test] +fn readerless_exact_turn_anchors_only_when_the_bounded_tail_has_a_gap() { + let conn = rusqlite::Connection::open_in_memory().expect("managed replay DB"); + managed_replay_test_schema(&conn); + let large_session_id = "cliagent-large-exact-turn"; + conn.execute( + "INSERT INTO code_session_history_mutations VALUES(?1, 11)", + [large_session_id], + ) + .expect("managed replay generation"); + insert_managed_replay_chunk(&conn, large_session_id, "user-0", 0, "user_message"); + for sequence in 1..=450_i64 { + insert_managed_replay_chunk( + &conn, + large_session_id, + &format!("assistant-{sequence}"), + sequence, + "assistant_message", + ); + } + conn.execute( + "UPDATE code_session_chunks + SET result_json=json_object('content',chunk_id) + WHERE session_id=?1", + [large_session_id], + ) + .expect("make large exact turn visible"); + + let large = managed_chunk_read_window_from_conn( + &conn, + large_session_id, + None, + None, + Some(0), + managed_replay_limits(10), + ) + .expect("read exact large managed turn"); + assert_eq!(large.chunks.len(), 200); + assert_eq!(large.chunks.first().map(|chunk| chunk.sequence), Some(0)); + assert_eq!(large.chunks.get(1).map(|chunk| chunk.sequence), Some(252)); + assert_eq!(large.chunks.last().map(|chunk| chunk.sequence), Some(450)); + assert_eq!(large.window_start_sequence, Some(252)); + assert!(large.has_older); + assert_eq!(large.turn_headers.len(), 1); + assert_eq!(large.turn_headers[0].event_count, 451); + + let small_session_id = "cliagent-small-exact-turn"; + conn.execute( + "INSERT INTO code_session_history_mutations VALUES(?1, 12)", + [small_session_id], + ) + .expect("small managed replay generation"); + insert_managed_replay_chunk(&conn, small_session_id, "small-user", 0, "user_message"); + insert_managed_replay_chunk( + &conn, + small_session_id, + "small-assistant", + 1, + "assistant_message", + ); + conn.execute( + "UPDATE code_session_chunks + SET result_json=json_object('content',chunk_id) + WHERE session_id=?1", + [small_session_id], + ) + .expect("make small exact turn visible"); + + let small = managed_chunk_read_window_from_conn( + &conn, + small_session_id, + None, + None, + Some(0), + managed_replay_limits(10), + ) + .expect("read exact small managed turn"); + assert_eq!( + small + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(), + vec![0, 1] + ); + assert_eq!(small.window_start_sequence, Some(0)); + assert!(!small.has_older); +} + +#[test] +fn readerless_managed_cli_uses_one_fallback_turn_without_user_rows() { + let conn = rusqlite::Connection::open_in_memory().expect("managed replay DB"); + managed_replay_test_schema(&conn); + insert_managed_replay_chunk(&conn, "cliagent-fallback", "a0", 10, "assistant_message"); + insert_managed_replay_chunk(&conn, "cliagent-fallback", "t0", 11, "run_command_line"); + + let window = managed_chunk_read_window_from_conn( + &conn, + "cliagent-fallback", + None, + None, + None, + managed_replay_limits(1), + ) + .expect("managed fallback turn"); + assert_eq!(window.total_turn_count, 1); + assert_eq!(window.turn_headers[0].turn_id, "a0"); + assert_eq!(window.turn_headers[0].start_sequence, 10); + assert_eq!(window.turn_headers[0].end_sequence, Some(11)); + assert_eq!(window.turn_headers[0].event_count, 2); + assert_eq!( + window + .chunks + .iter() + .map(|chunk| chunk.sequence) + .collect::>(), + vec![10, 11] + ); +} + +#[test] +fn readerless_managed_cli_compacts_ten_mib_row_and_ranges_without_full_column_reads() { + let conn = rusqlite::Connection::open_in_memory().expect("managed replay DB"); + conn.execute_batch( + "CREATE TABLE code_session_chunks ( + chunk_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + action_type TEXT NOT NULL, + function TEXT NOT NULL, + args_json TEXT NOT NULL, + result_json TEXT NOT NULL, + thread_id TEXT, + process_id TEXT, + sequence INTEGER NOT NULL, + created_at TEXT NOT NULL + ); + CREATE TABLE code_session_history_mutations ( + session_id TEXT PRIMARY KEY, + epoch INTEGER NOT NULL + ); + INSERT INTO code_session_history_mutations VALUES('cliagent-large', 7);", + ) + .expect("managed chunks schema"); + let full = format!("{}END", "中🙂shell-output\n".repeat(550_000)); + let expected_payload_hash = sha256_hex(full.as_bytes()); + let result_json = + serde_json::to_string(&serde_json::json!({"output":full})).expect("large managed result"); + assert!(result_json.len() > 10 * 1024 * 1024); + conn.execute( + "INSERT INTO code_session_chunks VALUES( + 'chunk-large','cliagent-large','tool_call','run_command_line', + '{\"command\":\"printf test\"}',?1,NULL,NULL,0,'2026-07-22T00:00:00Z' + )", + [result_json], + ) + .expect("insert managed chunk"); + + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.store(0, Ordering::Relaxed); + let chunks = query_managed_chunks( + &conn, + "cliagent-large", + "sequence < ?2", + i64::MAX, + None, + ReplayLimits { + max_turns: 1, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, + }, + true, + ) + .expect("bounded managed open row"); + assert_eq!(chunks.len(), 1); + assert!(serde_json::to_vec(&chunks[0].chunk).unwrap().len() < 64 * 1024); + assert_eq!(chunks[0].payloads.len(), 1); + assert_eq!(chunks[0].payloads[0].field_path, "result.output"); + assert!( + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.load(Ordering::Relaxed) + <= MANAGED_CHUNK_INLINE_JSON_MAX_BYTES, + "open must not copy a source-sized SQLite JSON column into Rust" + ); + assert!( + chunks[0] + .chunk + .result + .get("output") + .and_then(serde_json::Value::as_str) + .is_some_and(|preview| preview.len() < 64 * 1024 && preview.is_char_boundary(0)), + "projected preview must remain bounded valid UTF-8" + ); + + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.store(0, Ordering::Relaxed); + let unchanged = query_managed_chunks( + &conn, + "cliagent-large", + "sequence > ?2", + 0, + None, + ReplayLimits { + max_turns: 1, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, + }, + false, + ) + .expect("unchanged managed poll"); + assert!(unchanged.is_empty()); + assert_eq!( + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.load(Ordering::Relaxed), + 0, + "unchanged poll must not fetch any JSON field" + ); + + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.store(0, Ordering::Relaxed); + let mut rebuilt_hash = Sha256::new(); + let mut offset = 0_u64; + let mut calls = 0_usize; + loop { + let range = managed_chunk_payload_range_from_conn( + &conn, + "cliagent-large", + "chunk-large", + "result.output", + offset, + 64 * 1024, + ) + .expect("managed payload range"); + assert!(range.text.len() <= 64 * 1024); + rebuilt_hash.update(range.text.as_bytes()); + calls += 1; + if range.eof { + break; + } + assert!(range.next_offset > offset); + offset = range.next_offset; + } + assert!(calls > 100); + assert_eq!( + format!("{:x}", rebuilt_hash.finalize()), + expected_payload_hash + ); + assert!( + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.load(Ordering::Relaxed) <= 64 * 1024 + 4, + "payload-range must only fetch the requested slice plus UTF-8 boundary bytes" + ); + + #[derive(Default)] + struct HashingSink { + bytes: u64, + hash: Sha256, + } + + impl std::io::Write for HashingSink { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.bytes = self.bytes.saturating_add(bytes.len() as u64); + self.hash.update(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.store(0, Ordering::Relaxed); + let mut export_sink = HashingSink::default(); + let export_generation = stream_managed_chunk_replay_events_from_conn( + &conn, + "cliagent-large", + "testing managed export", + |event, read_payload| write_hydrated_event_json(&mut export_sink, event, read_payload), + ) + .expect("stream managed export"); + assert_eq!(export_generation, "chunks-7"); + assert!(export_sink.bytes > full.len() as u64); + assert!( + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.load(Ordering::Relaxed) + <= EXPORT_PAYLOAD_RANGE_BYTES + MANAGED_CHUNK_UTF8_BOUNDARY_BYTES, + "streamed export must not fetch a source-sized SQLite JSON column" + ); + + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.store(0, Ordering::Relaxed); + let mut cloud_hash = Sha256::new(); + let mut cloud_ranges = 0_usize; + let cloud_generation = stream_managed_chunk_replay_events_from_conn( + &conn, + "cliagent-large", + "testing managed Cloud spool", + |event, read_payload| { + let payload_ref = event + .payload_refs + .iter() + .find(|payload_ref| payload_ref.field_path == "result.output") + .ok_or_else(|| "managed Cloud payload locator is missing".to_string())?; + let mut offset = 0_u64; + loop { + let range = read_payload(payload_ref, offset)?; + cloud_hash.update(range.text.as_bytes()); + cloud_ranges = cloud_ranges.saturating_add(1); + if range.eof { + break; + } + if range.next_offset <= offset { + return Err("managed Cloud payload cursor did not advance".to_string()); + } + offset = range.next_offset; + } + Ok(()) + }, + ) + .expect("stream managed Cloud spool"); + assert_eq!(cloud_generation, "chunks-7"); + assert!(cloud_ranges > 20); + assert_eq!( + format!("{:x}", cloud_hash.finalize()), + expected_payload_hash + ); + assert!( + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.load(Ordering::Relaxed) + <= EXPORT_PAYLOAD_RANGE_BYTES + MANAGED_CHUNK_UTF8_BOUNDARY_BYTES, + "Cloud spool must share the bounded database/range path" + ); + + let invalid = "x".repeat(128 * 1024); + conn.execute( + "INSERT INTO code_session_chunks VALUES( + 'chunk-invalid','cliagent-large','tool_call','tool', + '{}',?1,NULL,NULL,1,'2026-07-22T00:00:01Z' + )", + [invalid], + ) + .expect("insert oversized non-JSON managed chunk"); + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.store(0, Ordering::Relaxed); + let error = query_managed_chunks( + &conn, + "cliagent-large", + "sequence > ?2", + 0, + None, + ReplayLimits { + max_turns: 1, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, + }, + false, + ) + .expect_err("oversized non-JSON must preserve fail-closed semantics"); + assert!(error.contains("invalid JSON")); + assert!( + MANAGED_CHUNK_MAX_DB_JSON_FIELD_BYTES.load(Ordering::Relaxed) + <= replay::NORMAL_PAYLOAD_PREVIEW_BYTES + MANAGED_CHUNK_UTF8_BOUNDARY_BYTES, + "invalid oversized JSON must fail without copying the full column" + ); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/mod.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/mod.rs new file mode 100644 index 000000000..5d49470f9 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/mod.rs @@ -0,0 +1,27 @@ +use std::io::Read; + +use flate2::read::GzDecoder; + +use super::*; + +fn event(id: &str, session_id: &str, content: &str) -> SessionEvent { + ingestion::normalize_single( + &RawActivityChunk { + chunk_id: Some(id.to_string()), + session_id: Some(session_id.to_string()), + action_type: Some("assistant".to_string()), + function: Some("assistant".to_string()), + result: Some(serde_json::json!({"content":content})), + created_at: Some("2026-07-22T00:00:00Z".to_string()), + ..RawActivityChunk::default() + }, + session_id, + ) +} + +mod cloud; +mod collaboration; +mod managed_chunks; +mod shell; +mod window_export; +mod wire_budget; diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/shell.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/shell.rs new file mode 100644 index 000000000..898389326 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/shell.rs @@ -0,0 +1,859 @@ +use super::*; + +fn external_shell_payload_event( + id: &str, + session_id: &str, + call_id: &str, + source_event_id: &str, + full_size_bytes: usize, +) -> SessionEvent { + let mut event = ingestion::normalize_single( + &RawActivityChunk { + chunk_id: Some(id.to_string()), + session_id: Some(session_id.to_string()), + action_type: Some("tool_call".to_string()), + function: Some("run_command_line".to_string()), + result: Some(serde_json::json!({"output":"bounded preview"})), + created_at: Some("2026-07-22T00:00:00Z".to_string()), + ..RawActivityChunk::default() + }, + session_id, + ); + event.ui_canonical = core_types::tool_names::RUN_SHELL.to_string(); + event.call_id = Some(call_id.to_string()); + event.payload_refs = vec![PayloadRef { + event_id: event.id.clone(), + field_path: "result.output".to_string(), + preview: "bounded preview".to_string(), + full_size_bytes, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some(MANAGED_CLI_REPLAY_TARGET_ID.to_string()), + replay_generation: Some("test-generation".to_string()), + replay_source_event_id: Some(source_event_id.to_string()), + }]; + event +} + +fn ten_mib_utf8_shell_payload() -> String { + const TARGET: usize = 10 * 1024 * 1024; + let pattern = "你🙂 shell stdout/stderr boundary\n"; + let mut payload = pattern.repeat(TARGET / pattern.len() + 1); + let mut boundary = TARGET; + while !payload.is_char_boundary(boundary) { + boundary -= 1; + } + payload.truncate(boundary); + payload.extend(std::iter::repeat_n('x', TARGET - boundary)); + assert_eq!(payload.len(), TARGET); + payload +} + +fn bounded_utf8_payload_bytes(text: &str, offset: u64, max_bytes: usize) -> Vec { + let start = offset as usize; + assert!(text.is_char_boundary(start)); + let mut end = start.saturating_add(max_bytes).min(text.len()); + while end > start && !text.is_char_boundary(end) { + end -= 1; + } + text.as_bytes()[start..end].to_vec() +} + +#[test] +fn foreground_replay_connection_does_not_wait_for_catalog_writer_mutex() { + use std::sync::mpsc; + use std::time::Duration; + + let (release_tx, release_rx) = mpsc::sync_channel::<()>(1); + let (locked_tx, locked_rx) = mpsc::sync_channel(1); + let catalog_writer = std::thread::spawn(move || { + let _writer = database::db::sessions_writer_guard(); + locked_tx.send(()).expect("report catalog writer lock"); + release_rx.recv().expect("release catalog writer lock"); + }); + locked_rx + .recv_timeout(Duration::from_secs(2)) + .expect("catalog writer must acquire the mutex"); + + let (probe_tx, probe_rx) = mpsc::sync_channel(1); + let foreground = std::thread::spawn(move || { + let result = with_foreground_replay_connection("lock probe", |conn| { + let tx = database::db::begin_immediate(conn) + .map_err(|error| format!("begin foreground lock probe: {error}"))?; + tx.rollback() + .map_err(|error| format!("rollback foreground lock probe: {error}"))?; + Ok(1_i64) + }); + probe_tx.send(result).expect("report foreground probe"); + }); + let probe = probe_rx.recv_timeout(Duration::from_secs(2)); + release_tx + .send(()) + .expect("release simulated catalog writer"); + catalog_writer.join().expect("catalog writer thread"); + foreground.join().expect("foreground probe thread"); + + assert_eq!( + probe.expect("foreground Shell path must bypass the catalog mutex"), + Ok(1) + ); +} + +fn read_complete_external_shell( + conn: &Connection, + session_id: &str, + call_id: &str, + total_bytes: u64, + last_sequence: u64, +) -> String { + let mut output = String::new(); + let mut offset = 0_u64; + loop { + let range = read_external_shell_manifest_range( + conn, + session_id, + call_id, + last_sequence, + total_bytes, + offset, + SHELL_REPLAY_RANGE_MAX_BYTES as u64, + ) + .expect("read external Shell range") + .expect("external Shell manifest"); + assert!(range + .frames + .iter() + .all(|frame| frame.text.len() <= SHELL_REPLAY_FRAME_MAX_BYTES + 3)); + for frame in range.frames { + output.push_str(&frame.text); + } + if range.eof { + break; + } + assert!(range.next_offset_bytes > offset); + offset = range.next_offset_bytes; + } + output +} + +#[test] +fn native_shell_call_id_performs_zero_external_database_probes() { + assert!(!is_external_shell_manifest_call_id("native-shell-call")); + assert!(!is_external_shell_manifest_call_id( + "looks-external-but-short-external-deadbeef" + )); + assert!(is_external_shell_manifest_call_id(&format!( + "managed-call-external-{}", + "a".repeat(64) + ))); + + let before = EXTERNAL_SHELL_MANIFEST_DB_PROBES.load(Ordering::SeqCst); + let result = tokio::runtime::Runtime::new() + .expect("native Shell range runtime") + .block_on(shell_replay_read_range( + "native-shell-probe-session".to_string(), + "native-shell-probe-call".to_string(), + 1, + 1, + 0, + 1, + )); + assert!( + result.is_err(), + "the synthetic native replay does not exist" + ); + assert_eq!( + EXTERNAL_SHELL_MANIFEST_DB_PROBES.load(Ordering::SeqCst), + before, + "native #425 range reads must bypass the external DB/task path" + ); +} + +#[test] +fn absent_external_schema_is_an_explicit_native_fallback() { + let conn = Connection::open_in_memory().expect("native-only replay DB"); + conn.execute_batch( + "CREATE TABLE shell_replays( + session_id TEXT NOT NULL, + call_id TEXT NOT NULL, + PRIMARY KEY(session_id,call_id) + );", + ) + .expect("native-only Shell schema marker"); + let call_id = format!("legacy-live-external-{}", "b".repeat(64)); + let external = + read_external_shell_manifest_range(&conn, "native-only-session", &call_id, 1, 1, 0, 1) + .expect("missing external table is not corruption"); + assert!(external.is_none()); +} + +#[test] +fn imported_shell_final_guard_reobserves_same_size_provider_rewrite() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let directory = tempfile::tempdir().expect("imported Shell provider fixture"); + let source_path = directory.path().join("codex-session.jsonl"); + let initial = concat!( + "{\"timestamp\":\"2026-07-22T00:00:00Z\",\"type\":\"event_msg\",", + "\"payload\":{\"type\":\"user_message\",\"message\":\"question\"}}\n", + "{\"timestamp\":\"2026-07-22T00:00:01Z\",\"type\":\"event_msg\",", + "\"payload\":{\"type\":\"agent_message\",\"message\":\"answer-AAAA\"}}\n", + ); + let replacement = initial.replace("answer-AAAA", "answer-BBBB"); + assert_eq!(initial.len(), replacement.len()); + fs::write(&source_path, initial).expect("write initial Codex transcript"); + + let source = ImportedHistorySourceId::CodexApp; + let session_id = "codexapp-shell-provider-race"; + let source_session_id = source + .source_session_id(session_id) + .expect("Codex source session id"); + let mut cache = Connection::open_in_memory().expect("imported Shell replay cache"); + SqliteRecordStore::init_tables(&cache).expect("replay tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source cache tables"); + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,?2,?3,?4)", + params![ + source.as_str(), + source_session_id, + session_id, + source_path.to_string_lossy() + ], + ) + .expect("bind Codex provider transcript"); + let opened = replay::open_window(&mut cache, source, session_id, ReplayLimits::default()) + .expect("open initial Codex replay"); + validate_imported_shell_snapshot_from_conn( + &mut cache, + source, + session_id, + &opened.cursor.generation, + opened.cursor.revision, + ) + .expect("unchanged provider remains valid"); + + // Keep the physical size identical so this specifically proves the + // final guard observes provider identity/content rather than trusting + // the previously published compact state or payload length. + std::thread::sleep(Duration::from_millis(2)); + fs::write(&source_path, replacement).expect("rewrite Codex transcript in place"); + let error = validate_imported_shell_snapshot_from_conn( + &mut cache, + source, + session_id, + &opened.cursor.generation, + opened.cursor.revision, + ) + .expect_err("same-size provider rewrite must invalidate Shell delivery"); + assert!(error.contains("changed while publishing manifests")); + assert!(error.contains(&opened.cursor.generation)); +} + +#[test] +fn collaboration_revision_is_part_of_the_shell_artifact_epoch() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let mut conn = Connection::open_in_memory().expect("collaboration Shell cache"); + SqliteRecordStore::init_tables(&conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("cache schema"); + let session_id = "agentsession-collaboration-shell"; + let generation = "collaboration-secondary-v1"; + let first_payload = format!("BEGIN{}END", "A".repeat(96 * 1024)); + let second_payload = format!("BEGIN{}END", "B".repeat(96 * 1024)); + assert_eq!(first_payload.len(), second_payload.len()); + + let mut first = external_shell_payload_event( + "collaboration-shell-event", + session_id, + "collaboration-shell-call", + "collaboration-shell-event", + first_payload.len(), + ); + let first_epoch = collaboration_shell_artifact_generation(generation, 41); + { + let tx = conn.transaction().expect("first collaboration revision"); + persist_scoped_shell_manifest( + &tx, + &mut first, + COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID, + session_id, + &first_epoch, + |_, offset, max_bytes| { + Ok(bounded_utf8_payload_bytes( + &first_payload, + offset, + max_bytes, + )) + }, + ) + .expect("first collaboration Shell manifest"); + tx.commit().expect("commit first collaboration revision"); + } + let first_state = first.shell_replay.expect("first collaboration state"); + let first_hash = conn + .query_row( + "SELECT content_hash FROM imported_replay_shell_segments + WHERE session_id=?1", + [session_id], + |row| row.get::<_, String>(0), + ) + .expect("first collaboration Shell hash"); + + // The collaboration generation is intentionally unchanged; only the + // cursor revision advances after an in-line snapshot rewrite. + let second_epoch = collaboration_shell_artifact_generation(generation, 42); + assert_ne!(first_epoch, second_epoch); + let mut second = external_shell_payload_event( + "collaboration-shell-event", + session_id, + "collaboration-shell-call", + "collaboration-shell-event", + second_payload.len(), + ); + let mut second_reads = 0_usize; + { + let tx = conn + .transaction() + .expect("rewritten collaboration revision"); + persist_scoped_shell_manifest( + &tx, + &mut second, + COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID, + session_id, + &second_epoch, + |_, offset, max_bytes| { + second_reads += 1; + Ok(bounded_utf8_payload_bytes( + &second_payload, + offset, + max_bytes, + )) + }, + ) + .expect("rewritten collaboration Shell manifest"); + tx.commit() + .expect("commit rewritten collaboration revision"); + } + assert!( + second_reads > 0, + "new revision must not hit the old artifact" + ); + let second_state = second.shell_replay.expect("second collaboration state"); + let (second_hash, stored_epoch): (String, String) = conn + .query_row( + "SELECT content_hash,generation FROM imported_replay_shell_segments + WHERE session_id=?1", + [session_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("rewritten collaboration Shell locator"); + assert_ne!(first_hash, second_hash); + assert_ne!( + first_state.replay_ref.call_id, + second_state.replay_ref.call_id + ); + assert_eq!(stored_epoch, second_epoch); + let restored = read_complete_external_shell( + &conn, + session_id, + &second_state.replay_ref.call_id, + second_state.bookmark.visible_bytes, + second_state.bookmark.visible_through_sequence, + ); + assert_eq!(restored, second_payload); +} + +#[test] +fn shell_events_commit_independently_and_retry_after_later_failure() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let mut conn = Connection::open_in_memory().expect("per-event Shell cache"); + SqliteRecordStore::init_tables(&conn).expect("replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("cache schema"); + let session_id = "cliagent-per-event-shell"; + let payload = "bounded-event-payload".repeat(4_096); + let mut first = + external_shell_payload_event("event-a", session_id, "call-a", "payload-a", payload.len()); + { + let tx = conn.transaction().expect("first event transaction"); + persist_scoped_shell_manifest( + &tx, + &mut first, + MANAGED_CLI_REPLAY_TARGET_ID, + session_id, + "chunks-1", + |_, offset, max_bytes| Ok(bounded_utf8_payload_bytes(&payload, offset, max_bytes)), + ) + .expect("first event manifest"); + tx.commit().expect("commit first event manifest"); + } + + let mut second = + external_shell_payload_event("event-b", session_id, "call-b", "payload-b", payload.len()); + { + let tx = conn.transaction().expect("second event transaction"); + let error = persist_scoped_shell_manifest( + &tx, + &mut second, + MANAGED_CLI_REPLAY_TARGET_ID, + session_id, + "chunks-1", + |_, _, _| Ok(Vec::new()), + ) + .expect_err("second event source fails before commit"); + assert!(error.contains("invalid progress")); + tx.rollback().expect("rollback failed second event"); + } + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_manifests", + [], + |row| row.get::<_, i64>(0), + ) + .expect("first manifest survives later failure"), + 1 + ); + assert!(read_external_shell_manifest_range( + &conn, + session_id, + &first + .shell_replay + .as_ref() + .expect("first replay state") + .replay_ref + .call_id, + u64::MAX, + u64::MAX, + 0, + SHELL_REPLAY_RANGE_MAX_BYTES as u64, + ) + .expect("first manifest remains readable") + .is_some()); + + let tx = conn.transaction().expect("second event retry transaction"); + persist_scoped_shell_manifest( + &tx, + &mut second, + MANAGED_CLI_REPLAY_TARGET_ID, + session_id, + "chunks-1", + |_, offset, max_bytes| Ok(bounded_utf8_payload_bytes(&payload, offset, max_bytes)), + ) + .expect("retry second event manifest"); + tx.commit().expect("commit retried second event"); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_manifests", + [], + |row| row.get::<_, i64>(0), + ) + .expect("both per-event manifests"), + 2 + ); +} + +#[test] +fn external_shell_payload_is_one_canonical_body_and_updates_atomically() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let directory = tempfile::tempdir().expect("external Shell cache directory"); + let database_path = directory.path().join("replay-cache.sqlite"); + let mut conn = Connection::open(&database_path).expect("external Shell cache DB"); + SqliteRecordStore::init_tables(&conn).expect("external Shell replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache schema"); + + let session_id = "cliagent-shared-shell"; + let source_event_id = "shared-ten-mib-payload"; + let payload = ten_mib_utf8_shell_payload(); + let mut range_reads = 0_usize; + let mut replay_states = Vec::new(); + { + let tx = conn.transaction().expect("publish shared Shell manifests"); + for index in 0..50 { + let mut event = external_shell_payload_event( + &format!("shell-event-{index}"), + session_id, + &format!("shell-call-{index}"), + source_event_id, + payload.len(), + ); + persist_scoped_shell_manifest( + &tx, + &mut event, + MANAGED_CLI_REPLAY_TARGET_ID, + session_id, + "generation-a", + |_, offset, max_bytes| { + range_reads += 1; + Ok(bounded_utf8_payload_bytes(&payload, offset, max_bytes)) + }, + ) + .expect("publish shared Shell manifest"); + replay_states.push(event.shell_replay.expect("external Shell replay state")); + } + tx.commit().expect("commit shared Shell manifests"); + } + assert!(range_reads > 1, "the first 10 MiB body is range streamed"); + assert!(range_reads < 50, "later calls reuse the immutable artifact"); + + let (artifact_count, artifact_bytes): (i64, i64) = conn + .query_row( + "SELECT COUNT(*),COALESCE(SUM(LENGTH(payload)),0) + FROM imported_replay_payload_artifacts", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("physical artifact accounting"); + assert_eq!(artifact_count, 1); + assert_eq!(artifact_bytes as usize, payload.len()); + let original_content_hash = conn + .query_row( + "SELECT content_hash FROM imported_replay_payload_artifacts + WHERE generation='generation-a'", + [], + |row| row.get::<_, String>(0), + ) + .expect("original canonical content hash"); + let segment_count = conn + .query_row( + "SELECT COUNT(*) FROM imported_replay_shell_segments", + [], + |row| row.get::<_, i64>(0), + ) + .expect("Shell manifest references"); + assert_eq!(segment_count, 50, "50 calls reference one physical body"); + let artifact_reference_count = conn + .query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifact_refs", + [], + |row| row.get::<_, i64>(0), + ) + .expect("canonical source references"); + assert_eq!(artifact_reference_count, 1); + let native_slog_table_count = conn + .query_row( + "SELECT COUNT(*) FROM sqlite_master + WHERE type='table' AND name='shell_replays'", + [], + |row| row.get::<_, i64>(0), + ) + .expect("native .slog table lookup"); + assert_eq!( + native_slog_table_count, 0, + "external path creates no native `.slog`" + ); + let database_bytes = std::fs::metadata(&database_path) + .expect("external Shell cache file") + .len(); + assert!( + // SQLite may retain the temporary content-key pages on its + // freelist until a later vacuum, so allow that one transient body + // in addition to the one live BLOB. Fifty live copies would be + // roughly 500 MiB and fail this bound by a wide margin. + database_bytes < (payload.len() as u64) * 3, + "50 manifests must not make 50 physical 10 MiB bodies: {database_bytes} bytes" + ); + + let first_state = &replay_states[0]; + let restored = read_complete_external_shell( + &conn, + session_id, + &first_state.replay_ref.call_id, + first_state.bookmark.visible_bytes, + first_state.bookmark.visible_through_sequence, + ); + assert_eq!( + sha256_hex(restored.as_bytes()), + sha256_hex(payload.as_bytes()) + ); + + // Delivering the same immutable epoch again must neither read provider + // ranges nor rewrite the unchanged manifest/segments. + let mut unchanged = external_shell_payload_event( + "shell-event-0", + session_id, + "shell-call-0", + source_event_id, + payload.len(), + ); + let unchanged_call_id = { + let tx = conn.transaction().expect("unchanged Shell delivery"); + let mut repeated_reads = 0_usize; + persist_scoped_shell_manifest( + &tx, + &mut unchanged, + MANAGED_CLI_REPLAY_TARGET_ID, + session_id, + "generation-a", + |_, _, _| { + repeated_reads += 1; + Ok(Vec::new()) + }, + ) + .expect("reuse unchanged Shell manifest"); + assert_eq!(repeated_reads, 0); + tx.commit().expect("commit unchanged delivery"); + unchanged + .shell_replay + .as_ref() + .expect("unchanged replay state") + .replay_ref + .call_id + .clone() + }; + assert_eq!(unchanged_call_id, first_state.replay_ref.call_id); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_segments", + [], + |row| row.get::<_, i64>(0), + ) + .expect("unchanged segment count"), + 50 + ); + + // A new immutable generation with the same byte length must be read + // and hashed again; length alone can never select the old body. + let mut changed_payload = payload.clone(); + changed_payload.replace_range(0.."你".len(), "界"); + assert_eq!(changed_payload.len(), payload.len()); + let mut changed = external_shell_payload_event( + "shell-event-0", + session_id, + "shell-call-0", + source_event_id, + changed_payload.len(), + ); + let mut changed_reads = 0_usize; + { + let tx = conn.transaction().expect("changed Shell generation"); + persist_scoped_shell_manifest( + &tx, + &mut changed, + MANAGED_CLI_REPLAY_TARGET_ID, + session_id, + "generation-b", + |_, offset, max_bytes| { + changed_reads += 1; + Ok(bounded_utf8_payload_bytes( + &changed_payload, + offset, + max_bytes, + )) + }, + ) + .expect("publish same-length changed Shell body"); + tx.commit().expect("commit changed Shell generation"); + } + assert!(changed_reads > 1); + let changed_state = changed.shell_replay.expect("changed replay state"); + assert_ne!(changed_state.replay_ref.call_id, unchanged_call_id); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_segments + WHERE session_id=?1 AND call_id=?2", + params![session_id, unchanged_call_id], + |row| row.get::<_, i64>(0), + ) + .expect("obsolete call segment count"), + 0, + "replacement must explicitly remove old segments with foreign_keys OFF" + ); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_segments + WHERE content_hash=?1", + [&original_content_hash], + |row| row.get::<_, i64>(0), + ) + .expect("remaining shared-body references"), + 49, + "the other 49 calls still share and retain the original body" + ); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_segments", + [], + |row| row.get::<_, i64>(0), + ) + .expect("replacement segment count"), + 50 + ); + let changed_restored = read_complete_external_shell( + &conn, + session_id, + &changed_state.replay_ref.call_id, + changed_state.bookmark.visible_bytes, + changed_state.bookmark.visible_through_sequence, + ); + assert_eq!( + sha256_hex(changed_restored.as_bytes()), + sha256_hex(changed_payload.as_bytes()) + ); + assert_ne!( + sha256_hex(payload.as_bytes()), + sha256_hex(changed_payload.as_bytes()) + ); + + // Simulate a failed/crashed replacement: artifact, refs, manifest and + // segment publication share one transaction, so rollback must leave + // the last committed canonical body readable. + let mut rolled_back_payload = changed_payload.clone(); + rolled_back_payload.replace_range(0.."界".len(), "中"); + let mut rolled_back = external_shell_payload_event( + "shell-event-0", + session_id, + "shell-call-0", + source_event_id, + rolled_back_payload.len(), + ); + { + let tx = conn.transaction().expect("failed Shell replacement"); + persist_scoped_shell_manifest( + &tx, + &mut rolled_back, + MANAGED_CLI_REPLAY_TARGET_ID, + session_id, + "generation-c", + |_, offset, max_bytes| { + Ok(bounded_utf8_payload_bytes( + &rolled_back_payload, + offset, + max_bytes, + )) + }, + ) + .expect("stage failed Shell replacement"); + tx.rollback().expect("rollback failed Shell replacement"); + } + let committed_call_id = conn + .query_row( + "SELECT call_id FROM imported_replay_shell_manifests + WHERE session_id=?1 AND logical_call_id='shell-call-0'", + [session_id], + |row| row.get::<_, String>(0), + ) + .expect("committed manifest after rollback"); + assert_eq!(committed_call_id, changed_state.replay_ref.call_id); + let after_rollback = read_complete_external_shell( + &conn, + session_id, + &committed_call_id, + changed_state.bookmark.visible_bytes, + changed_state.bookmark.visible_through_sequence, + ); + assert_eq!( + sha256_hex(after_rollback.as_bytes()), + sha256_hex(changed_payload.as_bytes()) + ); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_payload_artifacts + WHERE generation='generation-c'", + [], + |row| row.get::<_, i64>(0), + ) + .expect("rolled-back artifact count"), + 0 + ); + + // Repeated identity replacements stay at one segment for this + // logical call even though this connection deliberately leaves + // SQLite foreign_keys at its default OFF. + let changed_artifact = conn + .query_row( + "SELECT source,source_session_id,generation,content_hash,LENGTH(payload) + FROM imported_replay_payload_artifacts + WHERE generation='generation-b'", + [], + |row| { + Ok(replay::ReplayPayloadArtifactLocator { + source_id: row.get(0)?, + source_session_id: row.get(1)?, + generation: row.get(2)?, + content_hash: row.get(3)?, + total_bytes: row.get::<_, i64>(4)?.max(0) as u64, + }) + }, + ) + .expect("changed canonical artifact"); + for iteration in 0..6 { + let mut replacement = external_shell_payload_event( + "shell-event-0", + session_id, + "shell-call-0", + source_event_id, + changed_payload.len(), + ); + let tx = conn.transaction().expect("repeat manifest replacement"); + publish_external_shell_manifest( + &tx, + &mut replacement, + &[CanonicalExternalShellSegment { + stream: if iteration % 2 == 0 { + ShellReplayStream::Stderr + } else { + ShellReplayStream::Stdout + }, + artifact: changed_artifact.clone(), + preview: "changed preview".to_string(), + }], + ) + .expect("repeat external Shell replacement"); + tx.commit().expect("commit repeat manifest replacement"); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) FROM imported_replay_shell_segments", + [], + |row| row.get::<_, i64>(0), + ) + .expect("non-staircase segment count"), + 50 + ); + assert_eq!( + conn.query_row( + "SELECT COUNT(*) + FROM imported_replay_shell_segments AS segment + JOIN imported_replay_shell_manifests AS manifest + ON manifest.session_id=segment.session_id + AND manifest.call_id=segment.call_id + WHERE manifest.session_id=?1 + AND manifest.logical_call_id='shell-call-0'", + [session_id], + |row| row.get::<_, i64>(0), + ) + .expect("single live logical-call segment"), + 1 + ); + } +} + +#[test] +fn external_shell_artifact_cleanup_does_not_use_correlated_reference_scans() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let conn = Connection::open_in_memory().expect("open cleanup query-plan DB"); + SqliteRecordStore::init_tables(&conn).expect("initialize replay schema"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("initialize source cache schema"); + + let mut statement = conn + .prepare(&format!( + "EXPLAIN QUERY PLAN {DELETE_UNREFERENCED_PAYLOAD_ARTIFACTS_SQL}" + )) + .expect("prepare cleanup query plan"); + let details = statement + .query_map(params!["codex_app", "session", "generation"], |row| { + row.get::<_, String>(3) + }) + .expect("query cleanup plan") + .collect::>>() + .expect("read cleanup query plan"); + + assert!( + details.iter().all(|detail| !detail.contains("CORRELATED")), + "cleanup must build each referenced-hash set once instead of rescanning it per artifact: {details:?}" + ); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/window_export.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/window_export.rs new file mode 100644 index 000000000..69a9b7ca6 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/window_export.rs @@ -0,0 +1,1199 @@ +use super::super::request_guard::{ + apply_external_replay_delta, begin_prewarm_request, begin_replay_request, + has_foreground_request, has_prewarm_request, is_current_replay_episode, ReplayApplyResult, +}; +use super::*; + +fn delta(events: Vec, removed: Vec, reset: bool) -> ExternalReplayDelta { + ExternalReplayDelta { + cursor: ReplayCursor { + source_id: "codex_app".to_string(), + session_id: "codexapp-test".to_string(), + generation: "g1".to_string(), + revision: 1, + through_sequence: 0, + }, + events, + removed_event_ids: removed, + reset_required: reset, + stats: ReplayStats::default(), + watcher_available: false, + } +} + +fn handoff_chunk( + sequence: i64, + action_type: &str, + function: &str, + content: &str, +) -> ReplayIndexedChunk { + let mut chunk = ActivityChunk::new("codexapp-handoff", action_type, function); + chunk.chunk_id = format!("handoff-{sequence}"); + chunk.result = serde_json::json!({"content":content}); + ReplayIndexedChunk { + sequence, + turn_index: sequence, + chunk, + payloads: Vec::new(), + } +} + +fn handoff_page( + generation: &str, + chunks: Vec, + has_older: bool, +) -> ResolvedReplayWindow { + let through_sequence = chunks.last().map_or(-1, |chunk| chunk.sequence); + let window_start_sequence = chunks.first().map(|chunk| chunk.sequence); + let ipc_bytes = compact_handoff_page_bytes(&chunks) as u64; + ResolvedReplayWindow::Imported(ReplayChunkWindow { + cursor: ReplayCursor { + source_id: "codex_app".to_string(), + session_id: "codexapp-handoff".to_string(), + generation: generation.to_string(), + revision: 1, + through_sequence, + }, + total_turn_count: chunks.len() as u64, + total_event_count: chunks.len() as u64, + chunks, + window_start_sequence, + turn_headers: Vec::new(), + has_older, + stats: ReplayStats { + ipc_bytes, + ..ReplayStats::default() + }, + }) +} + +fn handoff_turn_page( + generation: &str, + turn_index: i64, + total_turns: u64, + mut chunks: Vec, + has_older: bool, +) -> ResolvedReplayWindow { + for chunk in &mut chunks { + chunk.turn_index = turn_index; + } + let start_sequence = chunks.first().map_or(0, |chunk| chunk.sequence); + let end_sequence = chunks.last().map(|chunk| chunk.sequence); + let mut page = handoff_page(generation, chunks, has_older); + if let ResolvedReplayWindow::Imported(window) = &mut page { + window.total_turn_count = total_turns; + window.turn_headers = vec![ReplayTurnHeader { + turn_id: format!("turn-{turn_index}"), + turn_index, + start_sequence, + end_sequence, + started_at: "2026-07-22T00:00:00Z".to_string(), + ended_at: Some("2026-07-22T00:00:01Z".to_string()), + event_count: window.chunks.len() as u64, + }]; + } + page +} + +#[test] +fn stream_cursor_guard_rejects_same_generation_new_revision() { + let current = ReplayCursor { + source_id: "opencode".to_string(), + session_id: "opencode-test".to_string(), + generation: "g1".to_string(), + revision: 8, + through_sequence: 200, + }; + let error = validate_stream_replay_cursor("g1", 7, ¤t, "testing") + .expect_err("same-generation revision changes must reset the stream"); + assert!(error.contains("g1@7")); + assert!(error.contains("g1@8")); + + let error = validate_managed_chunk_stream_cursor( + "chunks-3", + 200, + &("chunks-3".to_string(), 201), + "testing managed replay", + ) + .expect_err("same-generation managed appends must reset the stream"); + assert!(error.contains("chunks-3@200")); + assert!(error.contains("chunks-3@201")); +} + +#[test] +fn export_and_cloud_prepare_three_lazy_kv_turns_before_strict_streaming() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let directory = tempfile::tempdir().expect("lazy stream fixture"); + let source_path = directory.path().join(format!("{}.db", source.as_str())); + let source_conn = Connection::open(&source_path).expect("KV source DB"); + source_conn + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE cursorDiskKV(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("KV source schema"); + let headers = (0..6) + .map(|index| { + serde_json::json!({ + "bubbleId":format!("b{index}"), + "type":if index % 2 == 0 { 1 } else { 2 }, + }) + }) + .collect::>(); + source_conn + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES('composerData:c1',?1)", + [serde_json::json!({ + "composerId":"c1", + "createdAt":1, + "lastUpdatedAt":6, + "fullConversationHeadersOnly":headers, + }) + .to_string()], + ) + .expect("composer row"); + for index in 0..6 { + source_conn + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2)", + rusqlite::params![ + format!("bubbleId:c1:b{index}"), + serde_json::json!({ + "bubbleId":format!("b{index}"), + "type":if index % 2 == 0 { 1 } else { 2 }, + "createdAt":format!("2026-07-22T00:00:{index:02}Z"), + "text":format!("message {index}"), + }) + .to_string(), + ], + ) + .expect("bubble row"); + } + + let mut cache = Connection::open_in_memory().expect("replay cache"); + SqliteRecordStore::init_tables(&cache).expect("replay tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source cache tables"); + let session_id = format!("{}c1", source.descriptor().session_prefix); + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,'c1',?2,?3)", + rusqlite::params![source.as_str(), session_id, source_path.to_string_lossy()], + ) + .expect("cache source path"); + let limits = ReplayLimits { + max_turns: 1, + max_events: 1, + max_ipc_bytes: STREAM_BATCH_MAX_BYTES, + }; + let prepared = prepare_stream_replay_snapshot(&mut cache, source, &session_id, limits) + .expect("prepare export/cloud snapshot"); + let mut after_sequence = -1_i64; + let mut sequences = Vec::new(); + loop { + let scan = replay::scan_window_after_generation( + &mut cache, + source, + &session_id, + &prepared.generation, + prepared.revision, + after_sequence, + limits, + ) + .expect("strict export/cloud scan"); + sequences.extend(scan.chunks.iter().map(|chunk| chunk.sequence)); + after_sequence = scan.cursor.through_sequence; + if !scan.has_more { + break; + } + } + assert_eq!(sequences, vec![0, 1, 2, 3, 4, 5], "{}", source.as_str()); + } +} + +#[test] +fn failed_export_preserves_destination_and_removes_unique_partial() { + let directory = std::env::temp_dir().join(format!( + "orgii-replay-export-atomic-{}", + uuid::Uuid::new_v4() + )); + fs::create_dir_all(&directory).expect("export test directory"); + let destination = directory.join("history.json"); + fs::write(&destination, b"previous-valid-export").expect("old destination"); + + let error = stream_replay_export_to_path( + "not-a-replay-source", + "not-a-session", + &destination, + ReplayExportFormat::Json, + None, + ) + .expect_err("invalid source must fail after opening the temporary file"); + + assert!(!error.is_empty()); + assert_eq!( + fs::read(&destination).expect("preserved destination"), + b"previous-valid-export" + ); + let partials = fs::read_dir(&directory) + .expect("export directory") + .filter_map(Result::ok) + .filter(|entry| { + let name = entry.file_name(); + let name = name.to_string_lossy(); + name.starts_with(".history.json.orgii-") && name.ends_with(".part") + }) + .count(); + assert_eq!(partials, 0, "failed export left a partial file"); + + fs::remove_file(destination).expect("remove destination"); + fs::remove_dir(directory).expect("remove export test directory"); +} + +#[test] +fn export_refuses_to_create_an_unapproved_parent_directory() { + let directory = tempfile::tempdir().expect("export parent fixture"); + let missing_parent = directory.path().join("renderer-chosen").join("nested"); + let destination = missing_parent.join("history.json"); + + let error = stream_replay_export_to_path( + "not-a-replay-source", + "not-a-session", + &destination, + ReplayExportFormat::Json, + None, + ) + .expect_err("missing parent must be rejected before export begins"); + + assert!(error.contains("open replay export directory")); + assert!(!missing_parent.exists()); +} + +#[cfg(unix)] +#[test] +fn export_refuses_a_symbolic_link_destination() { + use std::os::unix::fs::symlink; + + let directory = tempfile::tempdir().expect("export symlink fixture"); + let protected = directory.path().join("protected.txt"); + fs::write(&protected, b"protected").expect("protected target"); + let destination = directory.path().join("history.json"); + symlink(&protected, &destination).expect("destination symlink"); + + let error = stream_replay_export_to_path( + "not-a-replay-source", + "not-a-session", + &destination, + ReplayExportFormat::Json, + None, + ) + .expect_err("symlink destination must be rejected"); + + assert!(error.contains("symbolic-link")); + assert_eq!( + fs::read(&protected).expect("protected target"), + b"protected" + ); +} + +#[test] +fn export_file_name_suggestion_never_preserves_renderer_directories() { + let file_name = super::super::export::sanitize_export_file_name( + Some("../../outside/raw-session.json"), + "codexapp-safe", + ReplayExportFormat::Json, + ); + assert_eq!(file_name, "raw-session.json"); + assert!(!file_name.contains('/')); + assert!(!file_name.contains('\\')); +} + +#[test] +fn not_ready_is_typed_and_non_destructive() { + let window = not_ready_window(MANAGED_CLI_REPLAY_TARGET_ID, "cliagent-test"); + assert!(window.stats.not_ready); + assert!(window.events.is_empty()); + assert_eq!(window.cursor.generation, "pending"); +} + +#[test] +fn fork_handoff_keeps_existing_semantics_skips_reasoning_and_caps_utf16_text() { + let user = handoff_chunk(0, "user_message", "unknown", "fix the sync"); + let thinking = handoff_chunk(1, "reasoning", "thinking", "private chain of thought"); + let mut tool = handoff_chunk(2, "tool_call", "read_file", "old file"); + tool.chunk.args = serde_json::json!({"content":"src/sync.ts"}); + tool.chunk.result = serde_json::json!({"output":"old file"}); + let assistant = handoff_chunk(3, "assistant", "assistant", "I found the issue"); + + assert_eq!( + handoff_item_from_chunk(&user.chunk, "Claude App").as_deref(), + Some("User: fix the sync") + ); + assert!(handoff_item_from_chunk(&thinking.chunk, "Claude App").is_none()); + assert_eq!( + handoff_item_from_chunk(&tool.chunk, "Claude App").as_deref(), + Some( + "[Imported Claude App action]\nTool: read_file\nInput: src/sync.ts\nResult at that time: old file" + ) + ); + assert_eq!( + handoff_item_from_chunk(&assistant.chunk, "Claude App").as_deref(), + Some("Assistant: I found the issue") + ); + + let huge = handoff_chunk(4, "raw", "user_message", &"🙂".repeat(2_000)); + let bounded = handoff_item_from_chunk(&huge.chunk, "Codex App").expect("bounded user item"); + assert!(bounded.encode_utf16().count() <= EXTERNAL_REPLAY_HANDOFF_MAX_TEXT_UTF16); + assert!(bounded.ends_with('…')); +} + +#[test] +fn fork_handoff_pages_backwards_with_one_generation_and_no_runtime_side_effects() { + let session_id = "codexapp-fork-handoff-pure"; + release_session_runtime(session_id); + let mut calls = 0_usize; + let handoff = collect_external_replay_handoff("Codex App", |before, turn, limits| { + calls += 1; + assert!(limits.max_ipc_bytes <= EXTERNAL_REPLAY_HANDOFF_SCAN_BYTES); + match calls { + 1 => { + assert_eq!(before, None); + assert_eq!(turn, None); + // Cursor/Windsurf cold indexes can report `hasOlder=false` + // because only the latest body is hydrated. The compact + // turn header still proves that turn 0 must be requested. + Ok(handoff_turn_page( + "generation-1", + 1, + 2, + vec![handoff_chunk(50, "reasoning", "thinking", "private")], + false, + )) + } + 2 => { + assert_eq!(before, None); + assert_eq!(turn, Some(0)); + Ok(handoff_turn_page( + "generation-1", + 0, + 2, + vec![ + handoff_chunk(10, "raw", "user_message", "usable older ask"), + handoff_chunk(11, "assistant", "assistant", "older answer"), + ], + false, + )) + } + _ => panic!("handoff requested an unnecessary page"), + } + }) + .expect("paged handoff"); + assert_eq!( + handoff.items, + vec![ + "User: usable older ask".to_string(), + "Assistant: older answer".to_string() + ] + ); + assert_eq!(handoff.generation, "generation-1"); + assert_eq!(handoff.scanned_events, 3); + assert!(handoff.scanned_bytes <= EXTERNAL_REPLAY_HANDOFF_SCAN_BYTES as u64); + assert!(!has_foreground_request(session_id)); + assert!(!external_replay_watcher::is_available(session_id)); +} + +#[test] +fn fork_handoff_rejects_cross_page_generation_mixing() { + let mut calls = 0_usize; + let error = collect_external_replay_handoff("Codex App", |_before, _turn, _limits| { + calls += 1; + if calls == 1 { + Ok(handoff_page( + "generation-a", + vec![handoff_chunk(50, "reasoning", "thinking", "private")], + true, + )) + } else { + Ok(handoff_page( + "generation-b", + vec![handoff_chunk(10, "raw", "user_message", "older")], + false, + )) + } + }) + .expect_err("mixed replay generations must fail closed"); + assert!(error.contains("changed generation")); + assert!(error.contains("retry")); +} + +#[test] +fn fork_handoff_rejects_cross_page_revision_mixing() { + let mut calls = 0_usize; + let error = collect_external_replay_handoff("Codex App", |_before, _turn, _limits| { + calls += 1; + if calls == 1 { + Ok(handoff_page( + "generation-a", + vec![handoff_chunk(50, "reasoning", "thinking", "private")], + true, + )) + } else { + let mut page = handoff_page( + "generation-a", + vec![handoff_chunk(10, "raw", "user_message", "older")], + false, + ); + if let ResolvedReplayWindow::Imported(window) = &mut page { + window.cursor.revision = 2; + } + Ok(page) + } + }) + .expect_err("mixed replay revisions must fail closed"); + assert!(error.contains("changed revision")); + assert!(error.contains("retry")); +} + +#[test] +fn fork_handoff_returns_only_the_last_eighty_usable_items() { + let chunks = (0..100) + .map(|sequence| handoff_chunk(sequence, "raw", "user_message", &format!("item-{sequence}"))) + .collect::>(); + let handoff = collect_external_replay_handoff("Codex App", |_before, _turn, _limits| { + Ok(handoff_page("generation-1", chunks.clone(), false)) + }) + .expect("bounded handoff"); + assert_eq!(handoff.items.len(), EXTERNAL_REPLAY_HANDOFF_MAX_ITEMS); + assert_eq!( + handoff.items.first().map(String::as_str), + Some("User: item-20") + ); + assert_eq!( + handoff.items.last().map(String::as_str), + Some("User: item-99") + ); + assert_eq!(handoff.scanned_events, 100); + assert!(handoff.scanned_bytes <= EXTERNAL_REPLAY_HANDOFF_SCAN_BYTES as u64); +} + +#[test] +fn same_id_and_content_is_a_true_no_op() { + let existing = event("same", "codexapp-test", "unchanged"); + let mut store = EventStore::new(); + store.set(vec![existing.clone()]); + let applied = + apply_external_replay_delta(&mut store, &delta(vec![existing], Vec::new(), false)); + assert_eq!(applied, ReplayApplyResult::default()); +} + +#[test] +fn removals_are_applied_to_the_display_session_store() { + let mut store = EventStore::new(); + store.set(vec![event("remove-me", "cliagent-test", "old")]); + let applied = apply_external_replay_delta( + &mut store, + &delta(Vec::new(), vec!["remove-me".to_string()], false), + ); + assert_eq!(applied.removed, 1); + assert!(store.get_by_id("remove-me").is_none()); +} + +#[test] +fn generation_reset_replaces_with_bounded_canonical_window() { + let mut store = EventStore::new(); + store.set(vec![event("ephemeral", "cliagent-test", "old")]); + let canonical = event("canonical", "cliagent-test", "new"); + let applied = + apply_external_replay_delta(&mut store, &delta(vec![canonical], Vec::new(), true)); + assert!(applied.changed); + assert!(store.get_by_id("ephemeral").is_none()); + assert_eq!( + store + .get_by_id("canonical") + .expect("canonical event") + .session_id, + "cliagent-test" + ); + assert_eq!( + store.hydration_mode(), + crate::agent_sessions::event_pipeline::store::HydrationMode::RoundWindow, + "a generation reset is still a bounded replay window, not a fully hydrated transcript" + ); +} + +#[test] +fn newer_request_token_invalidates_late_results_without_cross_session_leakage() { + let session_a = "cliagent-request-token-a"; + let session_b = "cliagent-request-token-b"; + let first_a = begin_replay_request(session_a, 10, true).expect("open A"); + let first_b = begin_replay_request(session_b, 20, true).expect("open B"); + + assert!(is_current_replay_request(session_a, 10, first_a)); + assert!(is_current_replay_request(session_b, 20, first_b)); + + let second_a = begin_replay_request(session_a, 10, false).expect("poll A"); + assert!(!is_current_replay_request(session_a, 10, first_a)); + assert!(is_current_replay_request(session_a, 10, second_a)); + assert!(is_current_replay_episode(session_a, 10)); + assert!(is_current_replay_request(session_b, 20, first_b)); +} + +#[test] +fn release_then_reopen_cannot_resurrect_an_a_to_b_to_a_result() { + let session_a = "codexapp-request-release-a"; + let stale_a = begin_replay_request(session_a, 100, true).expect("first A"); + release_session_runtime_if_episode(session_a, 100); + assert!(!is_current_replay_request(session_a, 100, stale_a)); + assert!(!is_current_replay_episode(session_a, 100)); + + let reopened_a = begin_replay_request(session_a, 101, true).expect("reopen A"); + assert_ne!(reopened_a, stale_a); + assert!(is_current_replay_episode(session_a, 101)); + // A delayed cleanup from the first episode cannot release A2. + release_session_runtime_if_episode(session_a, 100); + assert!(!is_current_replay_request(session_a, 100, stale_a)); + assert!(is_current_replay_request(session_a, 101, reopened_a)); + release_session_runtime(session_a); +} + +#[test] +fn prewarm_episode_is_independent_and_release_rejects_late_a_completion() { + let session_a = "codexapp-prewarm-episode-a"; + let first = begin_prewarm_request(session_a, 40).expect("first prewarm"); + assert!(is_current_prewarm_request(session_a, 40, first)); + assert!(!has_foreground_request(session_a)); + + let current = begin_prewarm_request(session_a, 41).expect("newer prewarm"); + assert!(!is_current_prewarm_request(session_a, 40, first)); + assert!(is_current_prewarm_request(session_a, 41, current)); + assert!(begin_prewarm_request(session_a, 40).is_err()); + + release_session_runtime(session_a); + assert!(!is_current_prewarm_request(session_a, 41, current)); + assert!(begin_prewarm_request(session_a, 41).is_err()); + let reopened = begin_prewarm_request(session_a, 42).expect("reopened prewarm"); + assert_ne!(reopened, first); + assert!(is_current_prewarm_request(session_a, 42, reopened)); + release_session_runtime(session_a); +} + +#[test] +fn foreground_release_cancels_current_prewarm_without_touching_native_state() { + let external_session = "codexapp-prewarm-release"; + let foreground = begin_replay_request(external_session, 500, true).expect("foreground"); + let prewarm = begin_prewarm_request(external_session, 12).expect("prewarm"); + assert!(is_current_replay_request(external_session, 500, foreground)); + assert!(is_current_prewarm_request(external_session, 12, prewarm)); + release_session_runtime_if_episode(external_session, 500); + assert!(!is_current_prewarm_request(external_session, 12, prewarm)); + + let native_session = "sdeagent-native-prewarm-boundary"; + assert!( + begin_validated_prewarm_request("codex_app", native_session, 1).is_err(), + "identity validation must happen before prewarm registration" + ); + assert!( + begin_validated_foreground_request("codex_app", native_session, 1, true).is_err(), + "identity validation must happen before foreground registration" + ); + assert!(!has_foreground_request(native_session)); + assert!(!has_prewarm_request(native_session)); +} + +#[test] +fn cancelled_prewarm_cannot_commit_or_reopen_the_same_episode() { + let session_id = "codexapp-prewarm-atomic-commit"; + let state = EventStoreState::new(); + let stale_request_token = begin_prewarm_request(session_id, 70).expect("first prewarm"); + cancel_prewarm_requests(session_id); + + assert!(!apply_prewarm_window_if_current( + &state, + session_id, + 70, + stale_request_token, + &[event("stale", session_id, "must not publish")], + )); + assert!(state + .with_store_opt(session_id, |store| store.events().len()) + .is_none()); + assert!(begin_prewarm_request(session_id, 70).is_err()); + + let current_request_token = begin_prewarm_request(session_id, 71).expect("next visit"); + assert!(apply_prewarm_window_if_current( + &state, + session_id, + 71, + current_request_token, + &[event("current", session_id, "publish")], + )); + assert_eq!( + state.with_store_opt(session_id, |store| store.events().len()), + Some(1) + ); + release_session_runtime(session_id); +} + +#[test] +fn foreground_window_publish_is_linearized_with_its_request_ticket() { + let session_id = "codexapp-foreground-atomic-commit"; + let state = EventStoreState::new(); + let stale_request_token = begin_replay_request(session_id, 80, true).expect("first open"); + let current_request_token = + begin_replay_request(session_id, 80, false).expect("newer foreground request"); + + assert!(!apply_foreground_window_if_current( + &state, + session_id, + 80, + stale_request_token, + "generation-a", + &[event("stale", session_id, "must not publish")], + ReplayWindowPublish::Replace, + )); + assert!(state + .with_store_opt(session_id, |store| store.events().len()) + .is_none()); + + assert!(apply_foreground_window_if_current( + &state, + session_id, + 80, + current_request_token, + "generation-a", + &[event("current", session_id, "publish")], + ReplayWindowPublish::Replace, + )); + assert_eq!( + state.with_store_opt(session_id, |store| store.events().len()), + Some(1) + ); + release_session_runtime(session_id); +} + +#[test] +fn foreground_merge_rejects_a_different_source_generation_before_store_apply() { + let session_id = "codexapp-foreground-generation-guard"; + let state = EventStoreState::new(); + let open_request = begin_replay_request(session_id, 81, true).expect("open request"); + assert!(apply_foreground_window_if_current( + &state, + session_id, + 81, + open_request, + "generation-a", + &[event("current", session_id, "generation a")], + ReplayWindowPublish::Replace, + )); + + let read_request = begin_replay_request(session_id, 81, false).expect("read request"); + assert!(!apply_foreground_window_if_current( + &state, + session_id, + 81, + read_request, + "generation-b", + &[event("replacement", session_id, "must not merge")], + ReplayWindowPublish::Merge, + )); + assert_eq!( + state.with_store_opt(session_id, |store| { + store + .events() + .iter() + .map(|event| event.id.clone()) + .collect::>() + }), + Some(vec!["current".to_string()]) + ); + release_session_runtime(session_id); +} + +#[test] +fn foreground_same_generation_merge_and_reset_delta_advance_the_guard() { + let session_id = "codexapp-foreground-generation-reset"; + let state = EventStoreState::new(); + let open_request = begin_replay_request(session_id, 82, true).expect("open request"); + assert!(apply_foreground_window_if_current( + &state, + session_id, + 82, + open_request, + "generation-a", + &[event("open", session_id, "generation a")], + ReplayWindowPublish::Replace, + )); + + let merge_request = begin_replay_request(session_id, 82, false).expect("merge request"); + assert!(apply_foreground_window_if_current( + &state, + session_id, + 82, + merge_request, + "generation-a", + &[event("older", session_id, "same generation")], + ReplayWindowPublish::Merge, + )); + + let stale_delta_request = + begin_replay_request(session_id, 82, false).expect("stale delta request"); + let mut stale_delta = delta( + vec![event("stale", session_id, "wrong generation")], + Vec::new(), + false, + ); + stale_delta.cursor.generation = "generation-b".to_string(); + assert!(apply_foreground_delta_if_current( + &state, + session_id, + 82, + stale_delta_request, + &stale_delta, + ) + .is_none()); + + let reset_request = begin_replay_request(session_id, 82, false).expect("reset request"); + let mut reset_delta = delta( + vec![event("reset", session_id, "generation b")], + Vec::new(), + true, + ); + reset_delta.cursor.generation = "generation-b".to_string(); + assert!( + apply_foreground_delta_if_current(&state, session_id, 82, reset_request, &reset_delta,) + .is_some() + ); + + let post_reset_request = + begin_replay_request(session_id, 82, false).expect("post-reset request"); + assert!(apply_foreground_window_if_current( + &state, + session_id, + 82, + post_reset_request, + "generation-b", + &[event("post-reset", session_id, "same new generation")], + ReplayWindowPublish::Merge, + )); + assert_eq!( + state.with_store_opt(session_id, |store| { + let mut ids = store + .events() + .iter() + .map(|event| event.id.clone()) + .collect::>(); + ids.sort(); + ids + }), + Some(vec!["post-reset".to_string(), "reset".to_string()]) + ); + release_session_runtime(session_id); +} + +#[test] +fn direct_turn_selection_merges_with_the_bounded_latest_window() { + let session_id = "codexapp-foreground-navigation-mode"; + let state = EventStoreState::new(); + let open_request = begin_replay_request(session_id, 83, true).expect("open request"); + assert!(apply_foreground_window_if_current( + &state, + session_id, + 83, + open_request, + "generation-a", + &[event("latest", session_id, "latest window")], + ReplayWindowPublish::Replace, + )); + + let older_request = begin_replay_request(session_id, 83, false).expect("older request"); + let older_publish = replay_window_publish_for_locator(Some(200), None, None); + assert_eq!(older_publish, ReplayWindowPublish::Merge); + assert!(apply_foreground_window_if_current( + &state, + session_id, + 83, + older_request, + "generation-a", + &[event("older", session_id, "continuous prepend")], + older_publish, + )); + + let direct_request = begin_replay_request(session_id, 83, false).expect("direct request"); + let direct_publish = replay_window_publish_for_locator(None, None, Some(0)); + assert_eq!(direct_publish, ReplayWindowPublish::Merge); + let direct_events = [event("direct", session_id, "random access page")]; + assert!(apply_foreground_window_if_current( + &state, + session_id, + 83, + direct_request, + "generation-a", + &direct_events, + direct_publish, + )); + state + .cap_external_replay_store_preserving_window( + session_id, + super::super::super::BOUNDED_REPLAY_STORE_MAX_BYTES, + &direct_events, + ) + .expect("cap random-access merge"); + assert_eq!( + state.with_store_opt(session_id, |store| { + let mut ids = store + .events() + .iter() + .map(|event| event.id.clone()) + .collect::>(); + ids.sort(); + ids + }), + Some(vec![ + "direct".to_string(), + "latest".to_string(), + "older".to_string(), + ]) + ); + assert_eq!( + replay_window_publish_for_locator(None, None, None), + ReplayWindowPublish::Replace, + "locator-free canonical reads remain authoritative replacements" + ); + release_session_runtime(session_id); +} + +#[test] +fn authoritative_replacement_cap_preserves_the_provider_user_anchor() { + let session_id = "codexapp-replacement-anchor-cap"; + let state = EventStoreState::new(); + let request_token = begin_replay_request(session_id, 84, true).expect("open request"); + let mut anchor = event("turn-anchor", session_id, "selected provider prompt"); + anchor.source = EventSource::User; + let mut events = vec![anchor]; + for index in 0..4 { + events.push(event( + &format!("turn-body-{index}"), + session_id, + &"x".repeat(3 * 1024 * 1024), + )); + } + + assert!(apply_foreground_window_if_current( + &state, + session_id, + 84, + request_token, + "generation-a", + &events, + ReplayWindowPublish::Replace, + )); + let bytes = + cap_replaced_replay_store(&state, session_id, &events).expect("cap replacement window"); + + assert!(bytes <= super::super::super::BOUNDED_REPLAY_STORE_MAX_BYTES); + assert_eq!( + state.with_store_opt(session_id, |store| store.get_by_id("turn-anchor").is_some()), + Some(true), + "a selected large Round must retain its provider user anchor" + ); + assert_eq!( + state.with_store_opt(session_id, |store| store.get_by_id("turn-body-3").is_some()), + Some(true), + "the ordinary newest suffix must remain available beside the anchor" + ); + release_session_runtime(session_id); +} + +#[test] +fn poll_cap_is_a_true_no_op_without_changes_and_preserves_the_visible_anchor_on_delta() { + let session_id = "codexapp-poll-anchor-cap"; + let state = EventStoreState::new(); + let request_token = begin_replay_request(session_id, 85, true).expect("open request"); + let mut anchor = event( + "visible-turn-anchor", + session_id, + "selected provider prompt", + ); + anchor.source = EventSource::User; + let mut events = vec![anchor]; + for index in 0..4 { + events.push(event( + &format!("visible-turn-body-{index}"), + session_id, + &"y".repeat(3 * 1024 * 1024), + )); + } + assert!(apply_foreground_window_if_current( + &state, + session_id, + 85, + request_token, + "generation-a", + &events, + ReplayWindowPublish::Replace, + )); + cap_replaced_replay_store(&state, session_id, &events).expect("cap selected Round"); + + let before_no_change = state + .with_store_opt(session_id, |store| { + store + .events() + .iter() + .map(|event| event.id.clone()) + .collect::>() + }) + .expect("resident store"); + cap_polled_replay_store( + &state, + session_id, + &delta(Vec::new(), Vec::new(), false), + false, + ) + .expect("no-change poll"); + assert_eq!( + state.with_store_opt(session_id, |store| { + store + .events() + .iter() + .map(|event| event.id.clone()) + .collect::>() + }), + Some(before_no_change), + "an unchanged poll must not recut or otherwise mutate EventStore" + ); + + let mut newest = event( + "newest-provider-delta", + session_id, + &"z".repeat(3 * 1024 * 1024), + ); + newest.created_at = "2026-07-22T00:01:00Z".to_string(); + state.with_store_mut(session_id, |store| { + store.merge_round_window_events(vec![newest.clone()]); + }); + cap_polled_replay_store( + &state, + session_id, + &delta(vec![newest], Vec::new(), false), + true, + ) + .expect("changed poll cap"); + + assert_eq!( + state.with_store_opt(session_id, |store| { + ( + store.get_by_id("visible-turn-anchor").is_some(), + store.get_by_id("newest-provider-delta").is_some(), + ) + }), + Some((true, true)), + "a live delta must keep both the selected Round anchor and newest provider tail" + ); + release_session_runtime(session_id); +} + +#[test] +fn stale_query_generation_or_revision_cannot_be_applied() { + validate_query_apply_version("generation-b", 8, "generation-b", 8) + .expect("current query is accepted"); + assert!(validate_query_apply_version("generation-a", 7, "generation-b", 8).is_err()); + assert!(validate_query_apply_version("generation-b", 7, "generation-b", 8).is_err()); +} + +#[test] +fn native_session_release_does_not_create_replay_runtime_state() { + let native_session = "osagent-native-replay-boundary"; + release_session_runtime(native_session); + assert!(!has_foreground_request(native_session)); + assert!(!external_replay_watcher::is_available(native_session)); +} + +#[test] +fn stable_json_matches_the_frontend_sorted_key_contract() { + let mut bytes = Vec::new(); + write_stable_json( + &mut bytes, + &serde_json::json!({"z": [3, {"b": true, "a": null}], "a": 1}), + ) + .expect("stable json"); + assert_eq!( + String::from_utf8(bytes).expect("utf8"), + r#"{"a":1,"z":[3,{"a":null,"b":true}]}"# + ); +} + +#[test] +fn hashing_writer_reports_exact_stream_hash_and_byte_count() { + let mut writer = HashingWriter::new(Vec::new()); + writer.write_all(b"bounded ").expect("first write"); + writer.write_all(b"export").expect("second write"); + let (bytes, hash, output) = writer.finish(); + assert_eq!(bytes, 14); + assert_eq!(output, b"bounded export"); + assert_eq!(hash, sha256_hex(b"bounded export")); +} + +#[test] +fn hydrated_export_splices_large_json_strings_in_bounded_ranges() { + let preview = "small preview"; + let mut replay_event = event("large", "codexapp-test", preview); + replay_event.payload_refs = vec![PayloadRef { + event_id: replay_event.id.clone(), + field_path: "result.content".to_string(), + preview: preview.to_string(), + full_size_bytes: 0, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some("codex_app".to_string()), + replay_generation: Some("g1".to_string()), + replay_source_event_id: Some("large".to_string()), + }]; + let full = format!("{}END", "quote=\" slash=\\ newline=\n 中🙂 ".repeat(40_000)); + replay_event.payload_refs[0].full_size_bytes = full.len(); + let mut output = Vec::new(); + let mut calls = 0_usize; + write_hydrated_event_json(&mut output, &replay_event, &mut |payload_ref, offset| { + assert_eq!(payload_ref.field_path, "result.content"); + calls += 1; + let start = offset as usize; + let mut end = start.saturating_add(7 * 1024).min(full.len()); + while end > start && !full.is_char_boundary(end) { + end -= 1; + } + Ok(ReplayPayloadRange { + event_id: payload_ref.event_id.clone(), + field_path: payload_ref.field_path.clone(), + offset, + next_offset: end as u64, + eof: end == full.len(), + total_bytes: full.len() as u64, + text: full[start..end].to_string(), + }) + }) + .expect("stream hydrated event"); + + let decoded: serde_json::Value = + serde_json::from_slice(&output).expect("valid streamed event JSON"); + assert_eq!( + decoded.pointer("/result/content").and_then(|v| v.as_str()), + Some(full.as_str()) + ); + assert_eq!( + decoded.get("displayText").and_then(|v| v.as_str()), + Some(full.as_str()) + ); + assert!(decoded.get("payloadRefs").is_none()); + assert!( + calls > 2, + "field and display copies must both use range reads" + ); +} + +#[test] +fn hydrated_export_restores_large_payload_inside_an_array() { + let preview = "array preview"; + let mut replay_event = event("array", "managed-session", preview); + replay_event.args = serde_json::json!({"items":[preview, "kept"]}); + replay_event.display_text = preview.to_string(); + replay_event.payload_refs = vec![PayloadRef { + event_id: replay_event.id.clone(), + field_path: "args.items.0".to_string(), + preview: preview.to_string(), + full_size_bytes: 0, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some(MANAGED_CLI_REPLAY_TARGET_ID.to_string()), + replay_generation: Some("legacy-generation".to_string()), + replay_source_event_id: Some("array".to_string()), + }]; + let mut full = "array-payload-中🙂".repeat(600_000); + while full.len() < 10 * 1024 * 1024 { + full.push_str("tail-🙂"); + } + replay_event.payload_refs[0].full_size_bytes = full.len(); + + let mut output = Vec::new(); + let mut calls = 0_usize; + write_hydrated_event_json(&mut output, &replay_event, &mut |payload_ref, offset| { + calls += 1; + let start = offset as usize; + let mut end = start.saturating_add(256 * 1024).min(full.len()); + while end > start && !full.is_char_boundary(end) { + end -= 1; + } + Ok(ReplayPayloadRange { + event_id: payload_ref.event_id.clone(), + field_path: payload_ref.field_path.clone(), + offset, + next_offset: end as u64, + eof: end == full.len(), + total_bytes: full.len() as u64, + text: full[start..end].to_string(), + }) + }) + .expect("stream array payload"); + + let decoded: serde_json::Value = + serde_json::from_slice(&output).expect("valid streamed array event"); + let restored = decoded + .pointer("/args/items/0") + .and_then(serde_json::Value::as_str) + .expect("restored array payload"); + assert_eq!(sha256_hex(restored.as_bytes()), sha256_hex(full.as_bytes())); + assert_eq!( + decoded + .get("displayText") + .and_then(serde_json::Value::as_str), + Some(full.as_str()) + ); + assert!(calls > 2, "large array payload must be range streamed"); +} + +#[test] +fn hydrated_export_replaces_root_json_payload_without_quoting_it() { + let mut replay_event = event("args", "codexapp-test", "tool"); + replay_event.args = serde_json::json!({"payloadPreview":"truncated"}); + replay_event.payload_refs = vec![PayloadRef { + event_id: replay_event.id.clone(), + field_path: "args".to_string(), + preview: replay_event.args.to_string(), + full_size_bytes: 30, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::JsonValue), + replay_source_id: Some("claude_code".to_string()), + replay_generation: Some("g1".to_string()), + replay_source_event_id: Some("args".to_string()), + }]; + let full = r#"{"command":"printf \"hello\"","path":"src/lib.rs"}"#; + let mut output = Vec::new(); + write_hydrated_event_json(&mut output, &replay_event, &mut |payload_ref, offset| { + assert_eq!(offset, 0); + Ok(ReplayPayloadRange { + event_id: payload_ref.event_id.clone(), + field_path: payload_ref.field_path.clone(), + offset: 0, + next_offset: full.len() as u64, + eof: true, + total_bytes: full.len() as u64, + text: full.to_string(), + }) + }) + .expect("stream root args"); + let decoded: serde_json::Value = serde_json::from_slice(&output).expect("valid event JSON"); + assert_eq!( + decoded.pointer("/args/path").and_then(|v| v.as_str()), + Some("src/lib.rs") + ); + + replace_event_payload(&mut replay_event, "args", full.to_string()); + assert_eq!( + replay_event.args.get("path").and_then(|v| v.as_str()), + Some("src/lib.rs") + ); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/wire_budget.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/wire_budget.rs new file mode 100644 index 000000000..ba1db8009 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/tests/wire_budget.rs @@ -0,0 +1,160 @@ +use super::*; + +fn delta(events: Vec) -> ExternalReplayDelta { + ExternalReplayDelta { + cursor: ReplayCursor { + source_id: "codex_app".to_string(), + session_id: "codexapp-test".to_string(), + generation: "g1".to_string(), + revision: 1, + through_sequence: 0, + }, + events, + removed_event_ids: Vec::new(), + reset_required: false, + stats: ReplayStats::default(), + watcher_available: false, + } +} + +fn max_preview_replay_events(count: usize) -> Vec { + (0..count) + .map(|index| { + let mut row = event( + &format!("event-{index}"), + "codexapp-test", + &"x".repeat(replay::NORMAL_PAYLOAD_PREVIEW_BYTES), + ); + row.payload_refs.push(PayloadRef { + event_id: row.id.clone(), + field_path: "result.content".to_string(), + preview: String::new(), + full_size_bytes: 10 * 1024 * 1024, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some("codex_app".to_string()), + replay_generation: Some("g1".to_string()), + replay_source_event_id: Some(row.id.clone()), + }); + row + }) + .collect() +} + +#[test] +fn final_wire_budget_counts_normalized_payload_refs_and_fails_closed() { + let mut replay_event = event("payload", "codexapp-test", "preview"); + replay_event.payload_refs.push(PayloadRef { + event_id: replay_event.id.clone(), + field_path: "result.content".to_string(), + preview: "x".repeat(70 * 1024), + full_size_bytes: 10 * 1024 * 1024, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some("codex_app".to_string()), + replay_generation: Some("g1".to_string()), + replay_source_event_id: Some("payload".to_string()), + }); + let cursor = ReplayCursor { + source_id: "codex_app".to_string(), + session_id: "codexapp-test".to_string(), + generation: "g1".to_string(), + revision: 7, + through_sequence: 7, + }; + let mut window = ExternalReplayWindow { + cursor: cursor.clone(), + events: vec![replay_event], + window_start_sequence: Some(7), + turn_headers: Vec::new(), + total_turn_count: 1, + total_event_count: 1, + has_older: false, + stats: ReplayStats::default(), + watcher_available: false, + }; + let error = finalize_window_wire_budget(&mut window, 64 * 1024) + .expect_err("payload ref must count toward wire bytes"); + assert!(error.contains("serialized bytes")); + assert_eq!( + window.cursor, cursor, + "failed wire check never advances cursor" + ); +} + +#[test] +fn compact_storage_reads_reserve_normalization_wire_headroom() { + let requested = ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: replay::HARD_MAX_EVENTS, + max_ipc_bytes: replay::HARD_MAX_IPC_BYTES, + }; + let storage = replay_storage_limits_with_normalization_headroom(requested); + + assert_eq!(storage.max_turns, requested.max_turns); + assert_eq!(storage.max_events, requested.max_events); + assert_eq!( + storage.max_ipc_bytes, + replay::HARD_MAX_IPC_BYTES / 2, + "raw compact rows must stop before renderer-only fields and payload refs can overflow IPC" + ); +} + +#[test] +fn two_hundred_max_preview_events_stay_under_the_hard_wire_cap() { + let mut window = ExternalReplayWindow { + cursor: ReplayCursor { + source_id: "codex_app".to_string(), + session_id: "codexapp-test".to_string(), + generation: "g1".to_string(), + revision: 200, + through_sequence: 199, + }, + events: max_preview_replay_events(200), + window_start_sequence: Some(0), + turn_headers: Vec::new(), + total_turn_count: 1, + total_event_count: 200, + has_older: false, + stats: ReplayStats::default(), + watcher_available: false, + }; + finalize_window_wire_budget(&mut window, replay::HARD_MAX_IPC_BYTES) + .expect("200 normal previews fit hard cap"); + assert!(window.stats.ipc_bytes <= replay::HARD_MAX_IPC_BYTES as u64); +} + +#[test] +fn final_delta_wire_budget_counts_payload_refs_and_preserves_the_cursor_on_failure() { + let mut replay_event = event("payload", "codexapp-test", "preview"); + replay_event.payload_refs.push(PayloadRef { + event_id: replay_event.id.clone(), + field_path: "result.content".to_string(), + preview: "x".repeat(70 * 1024), + full_size_bytes: 10 * 1024 * 1024, + truncated: true, + replay_encoding: Some(PayloadRefEncoding::Utf8Text), + replay_source_id: Some("codex_app".to_string()), + replay_generation: Some("g1".to_string()), + replay_source_event_id: Some("payload".to_string()), + }); + let mut response = delta(vec![replay_event]); + let cursor = response.cursor.clone(); + let error = finalize_delta_wire_budget(&mut response, 64 * 1024) + .expect_err("delta payload ref must count toward wire bytes"); + assert!(error.contains("serialized bytes")); + assert_eq!( + response.cursor, cursor, + "failed delta wire check never advances the caller-visible cursor" + ); +} + +#[test] +fn two_hundred_max_preview_delta_events_stay_under_the_hard_wire_cap() { + let mut response = delta(max_preview_replay_events(200)); + response.cursor.revision = 200; + response.cursor.through_sequence = 199; + finalize_delta_wire_budget(&mut response, replay::HARD_MAX_IPC_BYTES) + .expect("200 normal delta previews fit hard cap"); + assert!(response.stats.ipc_bytes <= replay::HARD_MAX_IPC_BYTES as u64); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/window.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/window.rs new file mode 100644 index 000000000..41818f9ac --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/window.rs @@ -0,0 +1,806 @@ +use super::*; + +#[tauri::command] +pub async fn external_replay_open_window( + app: AppHandle, + state: State<'_, EventStoreState>, + source_id: String, + session_id: String, + episode_id: u64, + limits: Option, +) -> Result { + // Reject invalid source/session pairs before they can consume a slot in + // the bounded foreground request registry. + let request_token = + begin_validated_foreground_request(&source_id, &session_id, episode_id, true)?; + let requested_limits = limits.unwrap_or_default().bounded(); + let display_session_id = session_id.clone(); + let requested_source_id = source_id.clone(); + let watcher_app = app.clone(); + let watcher_source_id = source_id.clone(); + let watcher_session_id = session_id.clone(); + let window = tokio::task::spawn_blocking(move || { + // Register before reading the initial snapshot so an append/rewrite in + // the indexing window cannot fall into a watcher-registration gap. + // Generation is filled in after the bounded open commits. Re-check + // the request immediately around acquisition: release can race this + // blocking task before it starts, and stale A1 work must not recreate + // a watcher after A→B→A has moved to a newer episode. + if is_current_replay_request(&watcher_session_id, episode_id, request_token) { + ensure_replay_watch( + &watcher_app, + &watcher_source_id, + &watcher_session_id, + episode_id, + None, + ); + if !is_current_replay_request(&watcher_session_id, episode_id, request_token) { + release_replay_watch_if_stale_episode(&watcher_session_id, episode_id); + } + } + let target = resolve_target(&source_id, &session_id)?; + match target { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => open_foreground_imported_window(source, &imported_session_id, requested_limits) + .map(ResolvedReplayWindow::Imported), + ResolvedReplayTarget::CollaborationSnapshot => { + let conn = database::db::get_connection() + .map_err(|err| format!("open collaboration replay DB: {err}"))?; + collaboration_snapshot_read_window_from_conn( + &conn, + &session_id, + None, + None, + None, + requested_limits, + ) + .map(ResolvedReplayWindow::CollaborationSnapshot) + } + ResolvedReplayTarget::ManagedChunkStore => managed_chunk_open_window( + &session_id, + replay_storage_limits_with_normalization_headroom(requested_limits), + ) + .map(ResolvedReplayWindow::ManagedChunks), + ResolvedReplayTarget::NotReady => Ok(ResolvedReplayWindow::NotReady), + } + }) + .await + .map_err(|err| format!("join replay open task: {err}"))??; + + match window { + ResolvedReplayWindow::NotReady => { + let mut response = not_ready_window(&requested_source_id, &display_session_id); + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + Ok(response) + } + ResolvedReplayWindow::Imported(window) => { + let mut response = normalize_window(window, &display_session_id); + remap_cursor( + &mut response.cursor, + &requested_source_id, + &display_session_id, + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + let generation = response.cursor.generation.clone(); + let revision = response.cursor.revision; + persist_shell_replays_for_delivery( + &requested_source_id, + &display_session_id, + &generation, + revision, + &mut response.events, + ) + .await?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + // Canonical open is authoritative for external/managed history. + // Keep a synthesized managed user bubble only until the native + // transcript provides its real user row. + let has_real_user = response + .events + .iter() + .any(|event| event.source == EventSource::User); + let synthetic = if has_real_user { + None + } else { + state + .with_store_opt(&display_session_id, |store| { + store + .events() + .iter() + .find(|event| { + event.source == EventSource::User + && event.id.contains("synthesized") + }) + .cloned() + }) + .flatten() + }; + if let Some(synthetic) = synthetic { + response.events.insert(0, synthetic); + } + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + response.watcher_available = ensure_replay_watch( + &app, + &requested_source_id, + &display_session_id, + episode_id, + Some(&response.cursor.generation), + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + } + // `false` is one byte larger than `true`; watcher attachment can + // therefore never invalidate the already-checked wire budget. + refresh_window_wire_bytes(&mut response)?; + if !apply_foreground_window_if_current( + &state, + &display_session_id, + episode_id, + request_token, + &response.cursor.generation, + &response.events, + ReplayWindowPublish::Replace, + ) { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + } + cap_replaced_replay_store(&state, &display_session_id, &response.events)?; + schedule_replay_cache_prune(); + schedule_notify(&app, &state, &display_session_id); + Ok(response) + } + ResolvedReplayWindow::CollaborationSnapshot(mut response) => { + remap_cursor( + &mut response.cursor, + &requested_source_id, + &display_session_id, + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + let generation = response.cursor.generation.clone(); + let revision = response.cursor.revision; + persist_shell_replays_for_delivery( + &requested_source_id, + &display_session_id, + &generation, + revision, + &mut response.events, + ) + .await?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + response.watcher_available = ensure_replay_watch( + &app, + &requested_source_id, + &display_session_id, + episode_id, + Some(&response.cursor.generation), + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + } + refresh_window_wire_bytes(&mut response)?; + if !apply_foreground_window_if_current( + &state, + &display_session_id, + episode_id, + request_token, + &response.cursor.generation, + &response.events, + ReplayWindowPublish::Replace, + ) { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + } + cap_replaced_replay_store(&state, &display_session_id, &response.events)?; + schedule_replay_cache_prune(); + schedule_notify(&app, &state, &display_session_id); + Ok(response) + } + ResolvedReplayWindow::ManagedChunks(window) => { + let mut response = normalize_window(window, &display_session_id); + remap_cursor( + &mut response.cursor, + &requested_source_id, + &display_session_id, + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + let generation = response.cursor.generation.clone(); + let revision = response.cursor.revision; + persist_shell_replays_for_delivery( + &requested_source_id, + &display_session_id, + &generation, + revision, + &mut response.events, + ) + .await?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + response.watcher_available = ensure_replay_watch( + &app, + &requested_source_id, + &display_session_id, + episode_id, + Some(&response.cursor.generation), + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + } + refresh_window_wire_bytes(&mut response)?; + if !apply_foreground_window_if_current( + &state, + &display_session_id, + episode_id, + request_token, + &response.cursor.generation, + &response.events, + ReplayWindowPublish::Replace, + ) { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + } + cap_replaced_replay_store(&state, &display_session_id, &response.events)?; + schedule_replay_cache_prune(); + schedule_notify(&app, &state, &display_session_id); + Ok(response) + } + } +} + +#[tauri::command] +pub async fn external_replay_poll_delta( + app: AppHandle, + state: State<'_, EventStoreState>, + source_id: String, + session_id: String, + episode_id: u64, + cursor: ReplayCursor, + limits: Option, +) -> Result { + validate_display_cursor(&source_id, &session_id, &cursor)?; + let request_token = + begin_validated_foreground_request(&source_id, &session_id, episode_id, false)?; + let requested_limits = limits.unwrap_or_default().bounded(); + let display_session_id = session_id.clone(); + let requested_source_id = source_id.clone(); + let delta = + tokio::task::spawn_blocking(move || match resolve_target(&source_id, &session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => { + let mut underlying_cursor = cursor; + underlying_cursor.source_id = source.as_str().to_string(); + underlying_cursor.session_id = imported_session_id.clone(); + with_foreground_replay_connection("replay index poll", |conn| { + replay::poll_delta( + conn, + source, + &imported_session_id, + &underlying_cursor, + replay_storage_limits_with_normalization_headroom(requested_limits), + ) + .map(ResolvedReplayDelta::Imported) + }) + } + ResolvedReplayTarget::CollaborationSnapshot => { + let conn = database::db::get_connection() + .map_err(|err| format!("open collaboration replay DB: {err}"))?; + collaboration_snapshot_poll_delta_from_conn( + &conn, + &session_id, + &cursor, + requested_limits, + ) + .map(ResolvedReplayDelta::CollaborationSnapshot) + } + ResolvedReplayTarget::ManagedChunkStore => managed_chunk_poll_delta( + &session_id, + &cursor, + replay_storage_limits_with_normalization_headroom(requested_limits), + ) + .map(ResolvedReplayDelta::ManagedChunks), + ResolvedReplayTarget::NotReady => Ok(ResolvedReplayDelta::NotReady), + }) + .await + .map_err(|err| format!("join replay poll task: {err}"))??; + + if matches!(delta, ResolvedReplayDelta::NotReady) { + let mut response = not_ready_delta(&requested_source_id, &display_session_id); + finalize_delta_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + return Ok(response); + } + let mut response = match delta { + ResolvedReplayDelta::Imported(delta) | ResolvedReplayDelta::ManagedChunks(delta) => { + normalize_delta(delta, &display_session_id) + } + ResolvedReplayDelta::CollaborationSnapshot(delta) => delta, + ResolvedReplayDelta::NotReady => unreachable!(), + }; + remap_cursor( + &mut response.cursor, + &requested_source_id, + &display_session_id, + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + let generation = response.cursor.generation.clone(); + let revision = response.cursor.revision; + persist_shell_replays_for_delivery( + &requested_source_id, + &display_session_id, + &generation, + revision, + &mut response.events, + ) + .await?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + // Preflight with the largest possible stats and the longer `false` + // watcher value. No EventStore mutation or cursor delivery happens if + // the final normalized wire response exceeds the caller's hard budget. + response.stats.upserted_events = response.events.len() as u64; + response.stats.removed_events = response.removed_event_ids.len() as u64; + finalize_delta_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + response.watcher_available = ensure_replay_watch( + &app, + &requested_source_id, + &display_session_id, + episode_id, + Some(&response.cursor.generation), + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + } + let Some(applied) = apply_foreground_delta_if_current( + &state, + &display_session_id, + episode_id, + request_token, + &response, + ) else { + release_replay_watch_if_stale_episode(&display_session_id, episode_id); + return Ok(response); + }; + response.stats.upserted_events = applied.upserted; + response.stats.removed_events = applied.removed; + cap_polled_replay_store(&state, &display_session_id, &response, applied.changed)?; + if applied.changed { + schedule_notify(&app, &state, &display_session_id); + } + // Actual no-op filtering can only reduce the decimal stats width, and + // `watcherAvailable=true` is shorter than the preflight `false` value. + refresh_delta_wire_bytes(&mut response)?; + if response.reset_required + || response.stats.parsed_bytes > 0 + || response.stats.parsed_rows > 0 + || response.stats.upserted_events > 0 + || response.stats.removed_events > 0 + { + schedule_replay_cache_prune(); + } + Ok(response) +} + +pub(super) fn replay_window_publish_for_locator( + before_sequence: Option, + turn_id: Option<&str>, + turn_index: Option, +) -> ReplayWindowPublish { + if before_sequence.is_some() || turn_id.is_some() || turn_index.is_some() { + // Older continuation and exact-turn reads join the existing bounded + // foreground window. The byte cap pins the requested rows and keeps + // the newest suffix, so random access remains bounded while a user + // can still scroll back toward the latest resident Round. + ReplayWindowPublish::Merge + } else { + // A locator-free read is an authoritative canonical window. + ReplayWindowPublish::Replace + } +} + +/// Cap an authoritative replacement without discarding its rendering anchor. +/// +/// Exact-turn reads deliberately return the first provider user event plus a +/// bounded newest tail. EventStore hydration may make that compact response +/// larger than its resident byte budget. A newest-suffix-only cap would then +/// remove the user event and leave a visibly populated but structurally +/// detached Round. Retain that one lightweight anchor while the existing cap +/// fills the remaining budget from its neighbour and newest suffix. +pub(super) fn cap_replaced_replay_store( + state: &EventStoreState, + session_id: &str, + events: &[SessionEvent], +) -> Result { + let anchor = events + .iter() + .find(|event| event.source == EventSource::User) + .or_else(|| events.first()); + match anchor { + Some(anchor) => state.cap_external_replay_store_preserving_window( + session_id, + super::super::BOUNDED_REPLAY_STORE_MAX_BYTES, + std::slice::from_ref(anchor), + ), + None => state + .cap_external_replay_store(session_id, super::super::BOUNDED_REPLAY_STORE_MAX_BYTES), + } +} + +/// Apply resident-memory policy only when a poll actually changed the store. +/// +/// A no-change poll must be a true EventStore no-op. When a live delta does +/// arrive, its cursor still follows the newest provider tail while the user +/// may be reading an older random-access Round, so retain the current user +/// anchor instead of silently replacing the visible ownership boundary. +pub(super) fn cap_polled_replay_store( + state: &EventStoreState, + session_id: &str, + delta: &ExternalReplayDelta, + changed: bool, +) -> Result<(), String> { + if !changed { + return Ok(()); + } + if delta.reset_required { + cap_replaced_replay_store(state, session_id, &delta.events)?; + } else { + state.cap_external_replay_store_preserving_current_user_anchor( + session_id, + super::super::BOUNDED_REPLAY_STORE_MAX_BYTES, + )?; + } + Ok(()) +} + +#[tauri::command] +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +pub async fn external_replay_read_window( + app: AppHandle, + state: State<'_, EventStoreState>, + source_id: String, + session_id: String, + episode_id: u64, + before_sequence: Option, + turn_id: Option, + turn_index: Option, + limits: Option, +) -> Result { + let locator_count = usize::from(before_sequence.is_some()) + + usize::from(turn_id.is_some()) + + usize::from(turn_index.is_some()); + if locator_count > 1 { + return Err("beforeSequence, turnId and turnIndex are mutually exclusive".to_string()); + } + let request_token = + begin_validated_foreground_request(&source_id, &session_id, episode_id, false)?; + let publish = + replay_window_publish_for_locator(before_sequence, turn_id.as_deref(), turn_index); + let requested_limits = limits.unwrap_or_default().bounded(); + let display_session_id = session_id.clone(); + let requested_source_id = source_id.clone(); + let window = + tokio::task::spawn_blocking(move || match resolve_target(&source_id, &session_id)? { + ResolvedReplayTarget::Imported { + source, + imported_session_id, + } => read_foreground_imported_window( + source, + &imported_session_id, + before_sequence, + turn_id.as_deref(), + turn_index, + requested_limits, + ) + .map(ResolvedReplayWindow::Imported), + ResolvedReplayTarget::CollaborationSnapshot => { + let conn = database::db::get_connection() + .map_err(|err| format!("open collaboration replay DB: {err}"))?; + collaboration_snapshot_read_window_from_conn( + &conn, + &session_id, + before_sequence, + turn_id.as_deref(), + turn_index, + requested_limits, + ) + .map(ResolvedReplayWindow::CollaborationSnapshot) + } + ResolvedReplayTarget::ManagedChunkStore => managed_chunk_read_window( + &session_id, + before_sequence, + turn_id.as_deref(), + turn_index, + replay_storage_limits_with_normalization_headroom(requested_limits), + ) + .map(ResolvedReplayWindow::ManagedChunks), + ResolvedReplayTarget::NotReady => Ok(ResolvedReplayWindow::NotReady), + }) + .await + .map_err(|err| format!("join replay window task: {err}"))??; + if matches!(window, ResolvedReplayWindow::NotReady) { + let mut response = not_ready_window(&requested_source_id, &display_session_id); + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + return Ok(response); + } + let mut response = match window { + ResolvedReplayWindow::Imported(window) | ResolvedReplayWindow::ManagedChunks(window) => { + normalize_window(window, &display_session_id) + } + ResolvedReplayWindow::CollaborationSnapshot(window) => window, + ResolvedReplayWindow::NotReady => unreachable!(), + }; + remap_cursor( + &mut response.cursor, + &requested_source_id, + &display_session_id, + ); + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + let generation = response.cursor.generation.clone(); + let revision = response.cursor.revision; + persist_shell_replays_for_delivery( + &requested_source_id, + &display_session_id, + &generation, + revision, + &mut response.events, + ) + .await?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + response.watcher_available = external_replay_watcher::is_available(&display_session_id); + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + if !is_current_replay_request(&display_session_id, episode_id, request_token) { + return Ok(response); + } + if !apply_foreground_window_if_current( + &state, + &display_session_id, + episode_id, + request_token, + &response.cursor.generation, + &response.events, + publish, + ) { + return Ok(response); + } + match publish { + ReplayWindowPublish::Replace => { + cap_replaced_replay_store(&state, &display_session_id, &response.events)?; + } + ReplayWindowPublish::Merge => { + state.cap_external_replay_store_preserving_window( + &display_session_id, + super::super::BOUNDED_REPLAY_STORE_MAX_BYTES, + &response.events, + )?; + } + } + if !response.events.is_empty() { + schedule_notify(&app, &state, &display_session_id); + } + schedule_replay_cache_prune(); + Ok(response) +} + +/// Side-effect-free bounded replay query for hover cards, export previews, +/// raw transcript virtualization and other read-only consumers. It may advance +/// the persistent compact source index, but it never acquires a foreground +/// watcher, touches EventStore, schedules `es:changed`, or participates in a +/// delivery request token. +#[tauri::command] +#[allow( + clippy::too_many_arguments, + reason = "Tauri wire and replay storage boundaries keep stable fields explicit" +)] +pub async fn external_replay_query_window( + source_id: String, + session_id: String, + before_sequence: Option, + turn_id: Option, + turn_index: Option, + limits: Option, +) -> Result { + let requested_limits = limits.unwrap_or_default().bounded(); + let display_session_id = session_id.clone(); + let requested_source_id = source_id.clone(); + let window = tokio::task::spawn_blocking(move || { + load_replay_query_window( + &source_id, + &session_id, + before_sequence, + turn_id.as_deref(), + turn_index, + requested_limits, + ) + }) + .await + .map_err(|err| format!("join pure replay query task: {err}"))??; + + let mut response = match window { + ResolvedReplayWindow::Imported(window) | ResolvedReplayWindow::ManagedChunks(window) => { + normalize_window(window, &display_session_id) + } + ResolvedReplayWindow::CollaborationSnapshot(window) => window, + ResolvedReplayWindow::NotReady => { + not_ready_window(&requested_source_id, &display_session_id) + } + }; + remap_cursor( + &mut response.cursor, + &requested_source_id, + &display_session_id, + ); + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + if !response.stats.not_ready { + schedule_replay_cache_prune(); + } + Ok(response) +} + +/// Prewarm one bounded external-history window and publish it directly into +/// EventStore. Source parsing, normalization, Shell replay persistence and the +/// authoritative replace all remain in Rust; the renderer receives the window +/// only once and never sends its `SessionEvent[]` back over IPC. +/// +/// Prewarm episodes are deliberately independent from foreground watcher +/// episodes. A session switch/close clears both registries, while a newer +/// prewarm episode invalidates any late completion from an earlier A visit. +#[tauri::command] +pub async fn external_replay_prewarm_window( + app: AppHandle, + state: State<'_, EventStoreState>, + source_id: String, + session_id: String, + episode_id: u64, + limits: Option, +) -> Result { + // This cheap identity check runs before registering an episode, so a + // native SDE session can neither call replay nor leave retained guard + // state, even if a caller spoofs an external source id. + let request_token = begin_validated_prewarm_request(&source_id, &session_id, episode_id)?; + let requested_limits = limits.unwrap_or_default().bounded(); + let display_session_id = session_id.clone(); + let requested_source_id = source_id.clone(); + let window = tokio::task::spawn_blocking(move || { + load_replay_query_window(&source_id, &session_id, None, None, None, requested_limits) + }) + .await + .map_err(|err| format!("join replay prewarm task: {err}"))??; + + let mut response = match window { + ResolvedReplayWindow::Imported(window) | ResolvedReplayWindow::ManagedChunks(window) => { + normalize_window(window, &display_session_id) + } + ResolvedReplayWindow::CollaborationSnapshot(window) => window, + ResolvedReplayWindow::NotReady => { + not_ready_window(&requested_source_id, &display_session_id) + } + }; + remap_cursor( + &mut response.cursor, + &requested_source_id, + &display_session_id, + ); + + if response.stats.not_ready { + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + return Ok(response); + } + if !is_current_prewarm_request(&display_session_id, episode_id, request_token) { + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + return Ok(response); + } + + let generation = response.cursor.generation.clone(); + let revision = response.cursor.revision; + persist_shell_replays_for_delivery( + &requested_source_id, + &display_session_id, + &generation, + revision, + &mut response.events, + ) + .await?; + if !is_current_prewarm_request(&display_session_id, episode_id, request_token) { + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + return Ok(response); + } + + // Prewarming is demand-driven and never owns a foreground watcher. + response.watcher_available = false; + finalize_window_wire_budget(&mut response, requested_limits.max_ipc_bytes)?; + if !apply_prewarm_window_if_current( + &state, + &display_session_id, + episode_id, + request_token, + &response.events, + ) { + return Ok(response); + } + cap_replaced_replay_store(&state, &display_session_id, &response.events)?; + if is_current_prewarm_request(&display_session_id, episode_id, request_token) { + schedule_notify(&app, &state, &display_session_id); + } + schedule_replay_cache_prune(); + Ok(response) +} + +pub(super) fn validate_query_apply_version( + expected_generation: &str, + expected_revision: u64, + current_generation: &str, + current_revision: u64, +) -> Result<(), String> { + if current_generation != expected_generation || current_revision != expected_revision { + return Err(format!( + "stale external replay query {expected_generation}@{expected_revision}; current compact index is {current_generation}@{current_revision}" + )); + } + Ok(()) +} + +#[tauri::command] +pub async fn external_replay_release( + source_id: String, + session_id: String, + episode_id: u64, +) -> Result<(), String> { + // Validate identity so an accidental native-SDE call cannot release an + // unrelated external foreground lease with a colliding session id. + if session_id.starts_with(COLLABORATION_SNAPSHOT_SESSION_PREFIX) { + if source_id != COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID { + return Err(format!( + "collaboration snapshot replay release requires sourceId={COLLABORATION_SNAPSHOT_REPLAY_TARGET_ID}" + )); + } + validate_collaboration_snapshot_session_id(&session_id)?; + } else if session_id.starts_with("cliagent-") { + if source_id != MANAGED_CLI_REPLAY_TARGET_ID { + return Err(format!( + "managed replay release requires sourceId={MANAGED_CLI_REPLAY_TARGET_ID}" + )); + } + } else { + let source = ImportedHistorySourceId::parse(&source_id)?; + source.validate_session_id(&session_id)?; + } + release_session_runtime_if_episode(&session_id, episode_id); + Ok(()) +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/wire_budget.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/wire_budget.rs new file mode 100644 index 000000000..418070877 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay/wire_budget.rs @@ -0,0 +1,295 @@ +use super::*; + +use crate::agent_sessions::event_pipeline::json_size::serialized_json_bytes; + +const NORMALIZED_REPLAY_STORAGE_BUDGET_DIVISOR: usize = 2; + +/// Reserve room for the `ActivityChunk -> SessionEvent` projection. +/// +/// Compact index rows do not include the renderer-only EventStore fields or +/// the payload-ref previews added during normalization. Budgeting raw rows +/// all the way to the public IPC ceiling therefore lets a valid compact page +/// grow past that ceiling after projection (the real Issue 272 transcript +/// reproduced a 4.8 MiB response from a 4 MiB raw request). Keep the public +/// hard limit unchanged and stop the storage read earlier instead. +pub(super) fn replay_storage_limits_with_normalization_headroom( + limits: ReplayLimits, +) -> ReplayLimits { + ReplayLimits { + max_ipc_bytes: limits + .max_ipc_bytes + .div_ceil(NORMALIZED_REPLAY_STORAGE_BUDGET_DIVISOR) + .max(1), + ..limits + } +} + +pub(super) fn normalize_window( + window: ReplayChunkWindow, + session_id: &str, +) -> ExternalReplayWindow { + let window_start_sequence = window.window_start_sequence; + let (events, ipc_bytes) = normalize_indexed_chunks( + window.chunks, + session_id, + &window.cursor.source_id, + &window.cursor.generation, + ); + let mut stats = window.stats; + stats.normalized_events = events.len() as u64; + stats.ipc_bytes = ipc_bytes; + ExternalReplayWindow { + cursor: window.cursor, + events, + window_start_sequence, + turn_headers: window.turn_headers, + total_turn_count: window.total_turn_count, + total_event_count: window.total_event_count, + has_older: window.has_older, + stats, + watcher_available: false, + } +} + +pub(super) fn normalize_delta(delta: ReplayChunkDelta, session_id: &str) -> ExternalReplayDelta { + let (events, ipc_bytes) = normalize_indexed_chunks( + delta.chunks, + session_id, + &delta.cursor.source_id, + &delta.cursor.generation, + ); + let mut stats = delta.stats; + stats.normalized_events = events.len() as u64; + stats.ipc_bytes = ipc_bytes; + ExternalReplayDelta { + cursor: delta.cursor, + events, + removed_event_ids: delta.removed_event_ids, + reset_required: delta.reset_required, + stats, + watcher_available: false, + } +} + +pub(super) fn refresh_window_wire_bytes( + response: &mut ExternalReplayWindow, +) -> Result { + let mut candidate = 0_u64; + for _ in 0..8 { + response.stats.ipc_bytes = candidate; + let measured = serialized_json_bytes(response) + .map_err(|error| format!("serialize bounded replay window: {error}"))? + as u64; + if measured == candidate { + return Ok(measured as usize); + } + candidate = measured; + } + response.stats.ipc_bytes = candidate; + let measured = serialized_json_bytes(response) + .map_err(|error| format!("serialize bounded replay window: {error}"))? + as u64; + response.stats.ipc_bytes = measured; + Ok(measured as usize) +} + +pub(super) fn refresh_delta_wire_bytes( + response: &mut ExternalReplayDelta, +) -> Result { + let mut candidate = 0_u64; + for _ in 0..8 { + response.stats.ipc_bytes = candidate; + let measured = serialized_json_bytes(response) + .map_err(|error| format!("serialize bounded replay delta: {error}"))? + as u64; + if measured == candidate { + return Ok(measured as usize); + } + candidate = measured; + } + response.stats.ipc_bytes = candidate; + let measured = serialized_json_bytes(response) + .map_err(|error| format!("serialize bounded replay delta: {error}"))? + as u64; + response.stats.ipc_bytes = measured; + Ok(measured as usize) +} + +pub(super) fn finalize_window_wire_budget( + response: &mut ExternalReplayWindow, + max_ipc_bytes: usize, +) -> Result<(), String> { + let wire_bytes = refresh_window_wire_bytes(response)?; + if wire_bytes > max_ipc_bytes { + return Err(format!( + "Bounded replay window requires {wire_bytes} serialized bytes after normalization; limit is {max_ipc_bytes}. Reduce maxEvents/maxTurns or read payloads by range" + )); + } + Ok(()) +} + +pub(super) fn finalize_delta_wire_budget( + response: &mut ExternalReplayDelta, + max_ipc_bytes: usize, +) -> Result<(), String> { + let wire_bytes = refresh_delta_wire_bytes(response)?; + if wire_bytes > max_ipc_bytes { + return Err(format!( + "Bounded replay delta requires {wire_bytes} serialized bytes after normalization; limit is {max_ipc_bytes}. Retry with a smaller event window" + )); + } + Ok(()) +} + +pub(super) fn normalize_indexed_chunks( + indexed: Vec, + session_id: &str, + replay_source_id: &str, + replay_generation: &str, +) -> (Vec, u64) { + let mut payloads = HashMap::with_capacity(indexed.len()); + let mut raw = Vec::with_capacity(indexed.len()); + for indexed in indexed { + let source_event_id = indexed.chunk.chunk_id.clone(); + payloads.insert(source_event_id.clone(), (source_event_id, indexed.payloads)); + raw.push(activity_to_raw(indexed.chunk)); + } + let mut events = ingestion::ingest_raw_chunks(&raw, session_id).events; + for event in &mut events { + let source_payloads = event + .chunk_id + .as_ref() + .and_then(|chunk_id| payloads.get(chunk_id)) + .or_else(|| payloads.get(&event.id)); + let Some((source_event_id, descriptors)) = source_payloads else { + continue; + }; + for descriptor in descriptors { + event.payload_refs.push(PayloadRef { + event_id: event.id.clone(), + field_path: descriptor.field_path.clone(), + preview: json_field_preview(event, &descriptor.field_path), + full_size_bytes: descriptor.total_bytes.min(usize::MAX as u64) as usize, + truncated: true, + replay_encoding: Some(match descriptor.resolved_encoding() { + ReplayPayloadEncoding::JsonValue => PayloadRefEncoding::JsonValue, + ReplayPayloadEncoding::Utf8Text => PayloadRefEncoding::Utf8Text, + ReplayPayloadEncoding::LegacyPathInferred => { + unreachable!("resolved replay payload encoding cannot remain legacy") + } + }), + replay_source_id: Some(replay_source_id.to_string()), + replay_generation: Some(replay_generation.to_string()), + replay_source_event_id: Some(source_event_id.clone()), + }); + } + } + let ipc_bytes = serialized_json_bytes(&events).unwrap_or(0) as u64; + (events, ipc_bytes) +} + +pub(super) fn activity_to_raw(chunk: ActivityChunk) -> RawActivityChunk { + RawActivityChunk { + chunk_id: Some(chunk.chunk_id), + session_id: Some(chunk.session_id), + action_type: Some(chunk.action_type), + function: Some(chunk.function), + args: Some(chunk.args), + result: Some(chunk.result), + created_at: Some(chunk.created_at), + thread_id: chunk.thread_id, + process_id: chunk.process_id, + call_id: None, + } +} + +pub(super) fn json_field_preview(event: &SessionEvent, field_path: &str) -> String { + let (root, path) = field_path.split_once('.').unwrap_or((field_path, "")); + let mut value = match root { + "args" => &event.args, + "result" => &event.result, + _ => return String::new(), + }; + for segment in path.split('.').filter(|segment| !segment.is_empty()) { + value = match value { + serde_json::Value::Object(object) => match object.get(segment) { + Some(next) => next, + None => return String::new(), + }, + serde_json::Value::Array(array) => { + let Ok(index) = segment.parse::() else { + return String::new(); + }; + match array.get(index) { + Some(next) => next, + None => return String::new(), + } + } + _ => return String::new(), + }; + } + value + .as_str() + .map(str::to_string) + .unwrap_or_else(|| value.to_string()) +} + +pub(super) fn remap_cursor(cursor: &mut ReplayCursor, source_id: &str, session_id: &str) { + cursor.source_id = source_id.to_string(); + cursor.session_id = session_id.to_string(); +} + +pub(super) fn validate_display_cursor( + source_id: &str, + session_id: &str, + cursor: &ReplayCursor, +) -> Result<(), String> { + if cursor.source_id != source_id || cursor.session_id != session_id { + return Err("Replay cursor belongs to another display session".to_string()); + } + Ok(()) +} + +pub(super) fn not_ready_window(source_id: &str, session_id: &str) -> ExternalReplayWindow { + ExternalReplayWindow { + cursor: ReplayCursor { + source_id: source_id.to_string(), + session_id: session_id.to_string(), + generation: "pending".to_string(), + revision: 0, + through_sequence: -1, + }, + events: Vec::new(), + window_start_sequence: None, + turn_headers: Vec::new(), + total_turn_count: 0, + total_event_count: 0, + has_older: false, + stats: ReplayStats { + not_ready: true, + ..ReplayStats::default() + }, + watcher_available: false, + } +} + +pub(super) fn not_ready_delta(source_id: &str, session_id: &str) -> ExternalReplayDelta { + ExternalReplayDelta { + cursor: not_ready_window(source_id, session_id).cursor, + events: Vec::new(), + removed_event_ids: Vec::new(), + reset_required: false, + stats: ReplayStats { + not_ready: true, + ..ReplayStats::default() + }, + watcher_available: false, + } +} + +pub(super) fn session_events_equal(left: &SessionEvent, right: &SessionEvent) -> bool { + serde_json::to_vec(left).ok() == serde_json::to_vec(right).ok() +} + +pub(super) const REPLAY_REQUEST_STATE_TTL: Duration = Duration::from_secs(5 * 60); +pub(super) const MAX_REPLAY_REQUEST_STATES: usize = 64; diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay_cache.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay_cache.rs new file mode 100644 index 000000000..ee5bd2565 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay_cache.rs @@ -0,0 +1,136 @@ +//! Throttled maintenance for ORGII-owned imported replay indexes. +//! +//! Cache eviction is deliberately outside the provider adapters. The provider +//! transcript remains the source of truth, while this process-local gate keeps +//! maintenance isolated per ORGII database and off the five-second delta path. + +use std::collections::HashMap; +use std::path::PathBuf; +use std::sync::{Mutex, OnceLock}; +use std::time::{Duration, Instant}; + +use orgtrack_core::sources::imported_history::replay::{self, ReplayCachePolicy}; +use rusqlite::Connection; + +const REPLAY_CACHE_PRUNE_INTERVAL: Duration = Duration::from_secs(60); +const REPLAY_CACHE_PRUNE_IDENTITIES: usize = 16; + +#[derive(Default)] +struct ReplayCachePruneGate { + last_started: HashMap, +} + +impl ReplayCachePruneGate { + fn claim(&mut self, identity: PathBuf, now: Instant) -> bool { + if self + .last_started + .get(&identity) + .is_some_and(|last| now.saturating_duration_since(*last) < REPLAY_CACHE_PRUNE_INTERVAL) + { + return false; + } + + // A desktop process normally sees one identity. Bound the registry as + // well so test/dev home switching cannot create a process-lifetime map. + if !self.last_started.contains_key(&identity) + && self.last_started.len() >= REPLAY_CACHE_PRUNE_IDENTITIES + { + if let Some(oldest) = self + .last_started + .iter() + .min_by_key(|(_, started)| **started) + .map(|(path, _)| path.clone()) + { + self.last_started.remove(&oldest); + } + } + self.last_started.insert(identity, now); + true + } +} + +fn prune_gate() -> &'static Mutex { + static GATE: OnceLock> = OnceLock::new(); + GATE.get_or_init(|| Mutex::new(ReplayCachePruneGate::default())) +} + +/// Run at most one cache prune per minute for this ORGII database identity. +/// +/// Maintenance runs on its own blocking worker and is best-effort: failure +/// must not turn an otherwise valid replay operation into a user-visible +/// error, and the claimed interval prevents a corrupt cache row from causing +/// a hot retry loop. +fn prune_replay_cache(conn: &mut Connection) { + match replay::prune_cache(conn, ReplayCachePolicy::default()) { + Ok(report) if report.evicted_entries > 0 => log::info!( + "[ExternalReplay] pruned {} cache entries ({} bytes; {} bytes remain)", + report.evicted_entries, + report.evicted_bytes, + report.after_bytes + ), + Ok(_) => {} + Err(error) => log::warn!("[ExternalReplay] replay cache prune failed: {error}"), + } +} + +/// Queue maintenance after a successful replay operation. The command does +/// not wait for cache accounting/eviction, and concurrent callers collapse at +/// the identity-aware gate before doing any SQLite scan. +pub(super) fn schedule_replay_cache_prune() { + // Claim before spawning or opening SQLite. Unchanged five-second polls + // therefore do no worker dispatch and no database work between the + // identity-scoped maintenance intervals. + let identity = database::db::get_db_path(); + let claimed = prune_gate() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .claim(identity, Instant::now()); + if !claimed { + return; + } + + let _prune_task = tokio::task::spawn_blocking(|| { + let mut conn = match database::db::get_connection() { + Ok(conn) => conn, + Err(error) => { + log::warn!("[ExternalReplay] open replay cache for prune failed: {error}"); + return; + } + }; + prune_replay_cache(&mut conn); + }); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn prune_gate_is_throttled_and_isolated_by_runtime_identity() { + let now = Instant::now(); + let mut gate = ReplayCachePruneGate::default(); + let first = PathBuf::from("/orgii-home-a/sessions.db"); + let second = PathBuf::from("/orgii-home-b/sessions.db"); + + assert!(gate.claim(first.clone(), now)); + assert!(!gate.claim(first.clone(), now + Duration::from_secs(59))); + assert!(gate.claim(second, now + Duration::from_secs(59))); + assert!(gate.claim(first, now + REPLAY_CACHE_PRUNE_INTERVAL)); + } + + #[test] + fn prune_gate_registry_is_bounded() { + let now = Instant::now(); + let mut gate = ReplayCachePruneGate::default(); + for index in 0..(REPLAY_CACHE_PRUNE_IDENTITIES + 5) { + assert!(gate.claim( + PathBuf::from(format!("/orgii-home-{index}/sessions.db")), + now + Duration::from_secs(index as u64), + )); + } + assert_eq!(gate.last_started.len(), REPLAY_CACHE_PRUNE_IDENTITIES); + assert!(!gate + .last_started + .contains_key(&PathBuf::from("/orgii-home-0/sessions.db"))); + } +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay_watcher.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay_watcher.rs new file mode 100644 index 000000000..ca7551324 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/external_replay_watcher.rs @@ -0,0 +1,777 @@ +//! Process-local, source-deduplicated watcher leases for bounded external replay. +//! +//! A lease exists only while an external replay session is in the foreground. +//! The registry is keyed by the ORGII runtime database plus the canonical +//! physical source path, so two sessions stored in one SQLite database share a +//! single native watcher without leaking state across isolated ORGII homes. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex, OnceLock}; +use std::time::{Duration, Instant}; + +use notify::{Config, Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher}; +use orgtrack_core::sources::imported_history::replay::{ReplayInvalidated, INVALIDATED_EVENT_NAME}; +use tauri::{AppHandle, Emitter}; + +const INVALIDATION_DEBOUNCE: Duration = Duration::from_millis(150); +const LEASE_TTL: Duration = Duration::from_secs(3 * 60); +const MAX_WATCHED_SOURCES: usize = 16; +const MAX_LEASES_PER_SOURCE: usize = 32; +const MAX_REGISTRY_ESTIMATED_BYTES: usize = 256 * 1024; +const BASE_ENTRY_ESTIMATED_BYTES: usize = 4 * 1024; +static WATCH_PRUNE_TASK_RUNNING: AtomicBool = AtomicBool::new(false); + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +struct WatchKey { + runtime_identity: PathBuf, + physical_source: PathBuf, +} + +#[derive(Debug, Clone)] +struct WatchLease { + source_id: String, + /// Renderer foreground episode. Public session ids can repeat during an + /// A→B→A switch, so the id alone cannot distinguish stale work. + episode_id: u64, + generation: Option, + touched_at: Instant, +} + +#[derive(Debug, Clone)] +struct WatchFilter { + physical_source: PathBuf, + source_is_directory: bool, +} + +impl WatchFilter { + fn from_source_path(source_path: &Path) -> Result<(Self, PathBuf, RecursiveMode), String> { + let physical_source = normalize_existing_path(source_path)?; + let source_is_directory = physical_source.is_dir(); + if source_is_directory { + return Ok(( + Self { + physical_source: physical_source.clone(), + source_is_directory: true, + }, + physical_source, + RecursiveMode::Recursive, + )); + } + + // Watch the parent so atomic replacement, rotation, and SQLite WAL + // creation are visible even when the sidecar did not exist at open. + // SHM is reader-lock coordination, not logical history; watching it + // lets our own SQLite reads wake the replay loop indefinitely. + let parent = physical_source + .parent() + .filter(|parent| parent.parent().is_some()) + .map(Path::to_path_buf) + .unwrap_or_else(|| physical_source.clone()); + Ok(( + Self { + physical_source, + source_is_directory: false, + }, + parent, + RecursiveMode::NonRecursive, + )) + } + + fn touches(&self, event: &Event) -> bool { + if matches!(event.kind, EventKind::Access(_)) { + return false; + } + event.paths.iter().any(|path| self.touches_path(path)) + } + + fn touches_path(&self, path: &Path) -> bool { + let path = normalize_event_path(path); + if self.source_is_directory { + return path.starts_with(&self.physical_source); + } + if path == self.physical_source { + return true; + } + let source = self.physical_source.to_string_lossy(); + path == format!("{source}-wal") + } +} + +struct SharedWatchState { + filter: WatchFilter, + leases: HashMap, + notification_pending: bool, + healthy: bool, + last_used: Instant, +} + +struct WatchEntry { + // Dropping the native watcher is the stop operation. + _watcher: RecommendedWatcher, + state: Arc>, + estimated_bytes: usize, +} + +#[derive(Default)] +struct WatchRegistry { + entries: HashMap, + estimated_bytes: usize, +} + +fn registry() -> &'static Mutex { + static REGISTRY: OnceLock> = OnceLock::new(); + REGISTRY.get_or_init(|| Mutex::new(WatchRegistry::default())) +} + +fn schedule_registry_prune() { + if WATCH_PRUNE_TASK_RUNNING + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + return; + } + tauri::async_runtime::spawn(async { + loop { + let next_delay = { + let now = Instant::now(); + let mut registry = registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + prune_registry(&mut registry, now); + next_registry_prune_delay(®istry, now) + }; + if let Some(next_delay) = next_delay { + tokio::time::sleep(next_delay).await; + continue; + } + + WATCH_PRUNE_TASK_RUNNING.store(false, Ordering::Release); + let needs_restart = !registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .entries + .is_empty(); + if !needs_restart { + break; + } + if WATCH_PRUNE_TASK_RUNNING + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + break; + } + } + }); +} + +fn next_registry_prune_delay(registry: &WatchRegistry, now: Instant) -> Option { + registry + .entries + .values() + .filter_map(|entry| { + let state = entry + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + next_watch_state_prune_delay(&state, now) + }) + .min() +} + +fn next_watch_state_prune_delay(state: &SharedWatchState, now: Instant) -> Option { + state + .leases + .values() + .map(|lease| lease.touched_at) + .chain(std::iter::once(state.last_used)) + .map(|touched_at| { + LEASE_TTL.saturating_sub(now.checked_duration_since(touched_at).unwrap_or_default()) + }) + .min() +} + +/// Acquire or refresh the foreground lease for one display session. +/// +/// `false` is a supported outcome: the renderer keeps its visible-only 5s +/// delta fallback until a later open/poll successfully attaches a watcher. +pub(super) fn acquire( + app: &AppHandle, + source_id: &str, + session_id: &str, + episode_id: u64, + generation: Option<&str>, + source_path: &Path, +) -> bool { + let (filter, watch_root, recursive_mode) = match WatchFilter::from_source_path(source_path) { + Ok(target) => target, + Err(error) => { + log::warn!("[external-replay] cannot normalize watcher source: {error}"); + return false; + } + }; + let key = WatchKey { + runtime_identity: runtime_identity(), + physical_source: filter.physical_source.clone(), + }; + let now = Instant::now(); + let mut registry = registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + prune_registry(&mut registry, now); + + let current_registry_bytes = registry.estimated_bytes; + let updated_registry_bytes = if let Some(entry) = registry.entries.get_mut(&key) { + let mut state = entry + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if !accepts_episode(state.leases.get(session_id), episode_id) { + return false; + } + if !state.healthy { + None + } else { + if !state.leases.contains_key(session_id) && state.leases.len() >= MAX_LEASES_PER_SOURCE + { + return false; + } + let old_lease_bytes = state + .leases + .get(session_id) + .map_or(0, |lease| estimate_lease_bytes(session_id, lease)); + let new_lease = WatchLease { + source_id: source_id.to_string(), + episode_id, + generation: generation.map(str::to_string), + touched_at: now, + }; + let new_lease_bytes = estimate_lease_bytes(session_id, &new_lease); + let updated_registry_bytes = current_registry_bytes + .saturating_sub(old_lease_bytes) + .saturating_add(new_lease_bytes); + if updated_registry_bytes > MAX_REGISTRY_ESTIMATED_BYTES { + return false; + } + state.leases.insert(session_id.to_string(), new_lease); + state.last_used = now; + entry.estimated_bytes = entry + .estimated_bytes + .saturating_sub(old_lease_bytes) + .saturating_add(new_lease_bytes); + Some(updated_registry_bytes) + } + } else { + None + }; + if let Some(updated_registry_bytes) = updated_registry_bytes { + registry.estimated_bytes = updated_registry_bytes; + drop(registry); + schedule_registry_prune(); + return true; + } + + // Retire an unhealthy entry before attempting a fresh native watcher. + if let Some(entry) = registry.entries.remove(&key) { + registry.estimated_bytes = registry + .estimated_bytes + .saturating_sub(entry.estimated_bytes); + drop(entry); + } + let initial_lease = WatchLease { + source_id: source_id.to_string(), + episode_id, + generation: generation.map(str::to_string), + touched_at: now, + }; + let estimated_bytes = + estimate_entry_bytes(&key).saturating_add(estimate_lease_bytes(session_id, &initial_lease)); + if estimated_bytes > MAX_REGISTRY_ESTIMATED_BYTES + || !evict_lru_until_capacity(&mut registry, estimated_bytes) + { + return false; + } + + let shared = Arc::new(Mutex::new(SharedWatchState { + filter, + leases: HashMap::from([(session_id.to_string(), initial_lease)]), + notification_pending: false, + healthy: true, + last_used: now, + })); + let callback_state = Arc::clone(&shared); + let callback_app = app.clone(); + let mut watcher = match RecommendedWatcher::new( + move |result| handle_native_event(result, &callback_state, &callback_app), + Config::default(), + ) { + Ok(watcher) => watcher, + Err(error) => { + log::warn!("[external-replay] cannot create native watcher: {error}"); + return false; + } + }; + if let Err(error) = watcher.watch(&watch_root, recursive_mode) { + log::warn!( + "[external-replay] cannot watch {}: {error}", + watch_root.display() + ); + return false; + } + + registry.estimated_bytes = registry.estimated_bytes.saturating_add(estimated_bytes); + registry.entries.insert( + key, + WatchEntry { + _watcher: watcher, + state: shared, + estimated_bytes, + }, + ); + drop(registry); + schedule_registry_prune(); + true +} + +/// Return current health without creating a watcher. A safety poll uses this +/// to downgrade to the short fallback after an asynchronous watcher error. +pub(super) fn is_available(session_id: &str) -> bool { + let now = Instant::now(); + let runtime_identity = runtime_identity(); + let mut registry = registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + prune_registry(&mut registry, now); + all_live_watch_entries_healthy(registry.entries.iter().filter_map(|(key, entry)| { + if key.runtime_identity != runtime_identity { + return None; + } + let state = entry + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let holds_live_lease = state + .leases + .get(session_id) + .is_some_and(|lease| lease_is_live(lease, now)); + Some((holds_live_lease, state.healthy)) + })) +} + +/// A replay source can own several physical watchers (for example Qoder's +/// transcript, shared launch logs, edit snapshots and spill output). The +/// renderer may suppress its 5-second fallback only while every watcher that +/// still holds this session's live lease is healthy. Entries from another +/// session, another runtime, or an expired lease do not participate. +fn all_live_watch_entries_healthy(entries: impl IntoIterator) -> bool { + let mut found_live_lease = false; + for (holds_live_lease, healthy) in entries { + if !holds_live_lease { + continue; + } + found_live_lease = true; + if !healthy { + return false; + } + } + found_live_lease +} + +fn lease_is_live(lease: &WatchLease, now: Instant) -> bool { + now.checked_duration_since(lease.touched_at) + .unwrap_or_default() + < LEASE_TTL +} + +/// Stop foreground delivery for one session. The native watcher is dropped as +/// soon as the last session sharing that physical source releases its lease. +pub(super) fn release_session(session_id: &str) { + release_session_matching_episode(session_id, None); +} + +/// Stop only the watcher lease owned by one renderer episode. This is used by +/// delayed request cleanup: an A1 completion must never tear down a newer A2 +/// lease that happens to share the same public session id. +pub(super) fn release_session_if_episode(session_id: &str, episode_id: u64) { + release_session_matching_episode(session_id, Some(episode_id)); +} + +fn release_session_matching_episode(session_id: &str, episode_id: Option) { + let runtime_identity = runtime_identity(); + let mut registry = registry() + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let mut empty_keys = Vec::new(); + let mut removed_lease_bytes = 0_usize; + for (key, entry) in &mut registry.entries { + if key.runtime_identity != runtime_identity { + continue; + } + let mut state = entry + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if let Some(lease) = take_matching_lease(&mut state.leases, session_id, episode_id) { + let lease_bytes = estimate_lease_bytes(session_id, &lease); + entry.estimated_bytes = entry.estimated_bytes.saturating_sub(lease_bytes); + removed_lease_bytes = removed_lease_bytes.saturating_add(lease_bytes); + } + if state.leases.is_empty() { + empty_keys.push(key.clone()); + } + } + registry.estimated_bytes = registry.estimated_bytes.saturating_sub(removed_lease_bytes); + let removed = empty_keys + .into_iter() + .filter_map(|key| registry.entries.remove(&key)) + .collect::>(); + for entry in &removed { + registry.estimated_bytes = registry + .estimated_bytes + .saturating_sub(entry.estimated_bytes); + } + drop(registry); + drop(removed); +} + +fn accepts_episode(current: Option<&WatchLease>, incoming_episode_id: u64) -> bool { + current.is_none_or(|current| current.episode_id <= incoming_episode_id) +} + +fn take_matching_lease( + leases: &mut HashMap, + session_id: &str, + episode_id: Option, +) -> Option { + let matches = leases + .get(session_id) + .is_some_and(|lease| episode_id.is_none_or(|episode_id| lease.episode_id == episode_id)); + if !matches { + return None; + } + leases.remove(session_id) +} + +fn handle_native_event( + result: notify::Result, + shared: &Arc>, + app: &AppHandle, +) { + let now = Instant::now(); + let should_schedule = { + let mut state = shared + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let relevant = match result { + Ok(event) => state.filter.touches(&event), + Err(error) => { + state.healthy = false; + log::warn!("[external-replay] watcher failed; using polling fallback: {error}"); + true + } + }; + let has_live_lease = state.leases.values().any(|lease| lease_is_live(lease, now)); + if !relevant || !has_live_lease || state.notification_pending { + false + } else { + state.notification_pending = true; + state.last_used = now; + true + } + }; + if !should_schedule { + return; + } + + let state = Arc::clone(shared); + let app = app.clone(); + tauri::async_runtime::spawn(async move { + tokio::time::sleep(INVALIDATION_DEBOUNCE).await; + let invalidations = { + let now = Instant::now(); + let mut state = state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + state.notification_pending = false; + state + .leases + .iter() + .filter(|(_, lease)| lease_is_live(lease, now)) + .map(|(session_id, lease)| ReplayInvalidated { + session_id: session_id.clone(), + source_id: lease.source_id.clone(), + generation: lease.generation.clone(), + }) + .collect::>() + }; + for invalidation in invalidations { + if let Err(error) = app.emit(INVALIDATED_EVENT_NAME, invalidation) { + log::debug!("[external-replay] invalidation listener unavailable: {error}"); + } + } + }); +} + +fn prune_registry(registry: &mut WatchRegistry, now: Instant) { + let mut expired = Vec::new(); + let mut removed_lease_bytes = 0_usize; + for (key, entry) in &mut registry.entries { + let mut state = entry + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let expired_sessions = state + .leases + .iter() + .filter(|(_, lease)| !lease_is_live(lease, now)) + .map(|(session_id, _)| session_id.clone()) + .collect::>(); + for session_id in expired_sessions { + if let Some(lease) = state.leases.remove(&session_id) { + let lease_bytes = estimate_lease_bytes(&session_id, &lease); + entry.estimated_bytes = entry.estimated_bytes.saturating_sub(lease_bytes); + removed_lease_bytes = removed_lease_bytes.saturating_add(lease_bytes); + } + } + if state.leases.is_empty() && now.duration_since(state.last_used) >= LEASE_TTL { + expired.push(key.clone()); + } + } + registry.estimated_bytes = registry.estimated_bytes.saturating_sub(removed_lease_bytes); + for key in expired { + if let Some(entry) = registry.entries.remove(&key) { + registry.estimated_bytes = registry + .estimated_bytes + .saturating_sub(entry.estimated_bytes); + } + } +} + +/// Capacity pressure degrades the least-recently-used source back to its +/// renderer safety poll. In normal operation there is one foreground lease; +/// this path bounds pathological multi-window/multi-source cases. +fn evict_lru_until_capacity(registry: &mut WatchRegistry, incoming_bytes: usize) -> bool { + while registry.entries.len() >= MAX_WATCHED_SOURCES + || registry.estimated_bytes.saturating_add(incoming_bytes) > MAX_REGISTRY_ESTIMATED_BYTES + { + let lru_key = registry + .entries + .iter() + .min_by_key(|(_, entry)| { + entry + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .last_used + }) + .map(|(key, _)| key.clone()); + let Some(lru_key) = lru_key else { + return false; + }; + if let Some(entry) = registry.entries.remove(&lru_key) { + registry.estimated_bytes = registry + .estimated_bytes + .saturating_sub(entry.estimated_bytes); + } + } + true +} + +fn estimate_entry_bytes(key: &WatchKey) -> usize { + BASE_ENTRY_ESTIMATED_BYTES + .saturating_add(key.runtime_identity.as_os_str().len()) + .saturating_add(key.physical_source.as_os_str().len()) +} + +fn estimate_lease_bytes(session_id: &str, lease: &WatchLease) -> usize { + std::mem::size_of::() + .saturating_add(session_id.len()) + .saturating_add(lease.source_id.len()) + .saturating_add(lease.generation.as_ref().map_or(0, String::len)) +} + +fn runtime_identity() -> PathBuf { + normalize_event_path(&database::db::get_db_path()) +} + +fn normalize_existing_path(path: &Path) -> Result { + path.canonicalize() + .map_err(|error| format!("{}: {error}", path.display())) +} + +fn normalize_event_path(path: &Path) -> PathBuf { + path.canonicalize().unwrap_or_else(|_| { + if path.is_absolute() { + path.to_path_buf() + } else { + std::env::current_dir().unwrap_or_default().join(path) + } + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use notify::event::{DataChange, ModifyKind}; + + fn modified(path: impl Into) -> Event { + Event { + kind: EventKind::Modify(ModifyKind::Data(DataChange::Content)), + paths: vec![path.into()], + attrs: Default::default(), + } + } + + fn lease(episode_id: u64) -> WatchLease { + WatchLease { + source_id: "codex_app".to_string(), + episode_id, + generation: Some("generation-1".to_string()), + touched_at: Instant::now(), + } + } + + #[test] + fn sqlite_filter_includes_wal_but_ignores_shm_lock_churn() { + let filter = WatchFilter { + physical_source: PathBuf::from("/tmp/history.db"), + source_is_directory: false, + }; + assert!(filter.touches(&modified("/tmp/history.db"))); + assert!(filter.touches(&modified("/tmp/history.db-wal"))); + assert!(!filter.touches(&modified("/tmp/history.db-shm"))); + assert!(!filter.touches(&modified("/tmp/another.db-wal"))); + } + + #[test] + fn file_filter_ignores_read_access_and_unrelated_atomic_rewrites() { + let filter = WatchFilter { + physical_source: PathBuf::from("/tmp/transcript.jsonl"), + source_is_directory: false, + }; + let mut access = modified("/tmp/transcript.jsonl"); + access.kind = EventKind::Access(notify::event::AccessKind::Any); + assert!(!filter.touches(&access)); + assert!(!filter.touches(&modified("/tmp/transcript.jsonl.tmp"))); + } + + #[test] + fn directory_filter_covers_manifest_children_only() { + let filter = WatchFilter { + physical_source: PathBuf::from("/tmp/manifest-root"), + source_is_directory: true, + }; + assert!(filter.touches(&modified("/tmp/manifest-root/blobs/one"))); + assert!(!filter.touches(&modified("/tmp/manifest-root-copy/one"))); + } + + #[test] + fn availability_requires_every_live_session_watch_to_be_healthy() { + assert!(all_live_watch_entries_healthy([ + (true, true), + (true, true), + (false, false), + ])); + assert!(!all_live_watch_entries_healthy([ + (true, true), + (true, false), + (false, true), + ])); + assert!(!all_live_watch_entries_healthy([ + (false, true), + (false, false), + ])); + } + + #[test] + fn stale_episode_cannot_replace_or_release_a_reopened_session_lease() { + let mut leases = HashMap::from([("codexapp-a".to_string(), lease(102))]); + + assert!(!accepts_episode(leases.get("codexapp-a"), 100)); + assert!(take_matching_lease(&mut leases, "codexapp-a", Some(100)).is_none()); + assert_eq!( + leases.get("codexapp-a").map(|lease| lease.episode_id), + Some(102) + ); + + assert!(take_matching_lease(&mut leases, "codexapp-a", Some(102)).is_some()); + assert!(!leases.contains_key("codexapp-a")); + } + + #[test] + fn physical_source_key_single_flights_sessions_but_partitions_runtimes() { + let first = WatchKey { + runtime_identity: PathBuf::from("/orgii-home-a/sessions.db"), + physical_source: PathBuf::from("/cli/history.db"), + }; + let same_process_and_source = WatchKey { + runtime_identity: PathBuf::from("/orgii-home-a/sessions.db"), + physical_source: PathBuf::from("/cli/history.db"), + }; + let other_runtime = WatchKey { + runtime_identity: PathBuf::from("/orgii-home-b/sessions.db"), + physical_source: PathBuf::from("/cli/history.db"), + }; + + assert_eq!(first, same_process_and_source); + assert_ne!(first, other_runtime); + let mut entries = HashMap::new(); + entries.insert(first, "session-a"); + entries.insert(same_process_and_source, "session-b"); + entries.insert(other_runtime, "other-instance"); + assert_eq!(entries.len(), 2); + } + + #[test] + fn watcher_lease_expires_at_the_three_minute_boundary() { + let now = Instant::now(); + let mut active = lease(1); + active.touched_at = now - LEASE_TTL + Duration::from_nanos(1); + assert!(lease_is_live(&active, now)); + + active.touched_at = now - LEASE_TTL; + assert!(!lease_is_live(&active, now)); + } + + #[test] + fn watcher_prune_schedule_tracks_the_nearest_lease_expiry() { + let now = Instant::now(); + let mut older = lease(1); + older.touched_at = now - LEASE_TTL + Duration::from_secs(2); + let state = SharedWatchState { + filter: WatchFilter { + physical_source: PathBuf::from("/tmp/transcript.jsonl"), + source_is_directory: false, + }, + leases: HashMap::from([ + ("codexapp-older".to_string(), older), + ("codexapp-newer".to_string(), lease(2)), + ]), + notification_pending: false, + healthy: true, + last_used: now, + }; + + assert_eq!( + next_watch_state_prune_delay(&state, now), + Some(Duration::from_secs(2)) + ); + } + + #[test] + fn watcher_registry_limits_are_byte_and_count_bounded() { + const { + assert!(MAX_WATCHED_SOURCES > 0); + assert!(MAX_LEASES_PER_SOURCE > 0); + assert!(MAX_REGISTRY_ESTIMATED_BYTES >= BASE_ENTRY_ESTIMATED_BYTES); + } + let key = WatchKey { + runtime_identity: PathBuf::from("/orgii-home/sessions.db"), + physical_source: PathBuf::from("/cli/transcript.jsonl"), + }; + let estimated = estimate_entry_bytes(&key) + .saturating_add(estimate_lease_bytes("codexapp-session", &lease(1))); + assert!(estimated < MAX_REGISTRY_ESTIMATED_BYTES); + } +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/ingestion.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/ingestion.rs index 04198db02..c972d5b0c 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/ingestion.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/ingestion.rs @@ -20,7 +20,9 @@ pub async fn es_ingest_chunks( chunks: Vec, ) -> Result { let result = process_chunks_with_external_replays(chunks, session_id.clone()).await?; + let incoming_bytes = state.validate_bounded_replay_input(&session_id, &result.events)?; state.with_store_mut(&session_id, |store| store.append(result.events.clone())); + state.account_bounded_replay_write(&session_id, incoming_bytes)?; schedule_notify(&app, &state, &session_id); Ok(result) } diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/mod.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/mod.rs index fa7678d47..41ca5a4ec 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/mod.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/mod.rs @@ -1,30 +1,30 @@ -//! Tauri commands for the Rust EventStore +//! Tauri commands for the Rust EventStore. //! -//! Thin wrappers around per-session `EventStore` instances and derived computations. -//! Each command acquires the `Mutex`, resolves the target session (explicit -//! `session_id` argument or the active session), performs the operation, and -//! returns. -//! -//! Notification scheduling (100ms batched `es:changed` events) is handled by -//! a background tokio task spawned at app startup. Each snapshot is tagged with -//! the `sessionId` it describes so the frontend can route to per-session -//! listeners. +//! The public command paths remain in this facade while implementation, +//! notification, persistence and bounded-replay concerns live in focused +//! submodules. mod analytics; mod batch_update; mod cache_bridge; +pub mod collaboration_snapshot_ingest; pub(crate) mod event_conversion; +pub mod external_replay; +mod external_replay_cache; +mod external_replay_watcher; mod extractors; mod history; mod ingestion; mod notify; mod pagination; mod push_events; +mod replay_cloud_wire; mod runtime_artifacts; mod search; mod session_manager; mod snapshot; mod state; +mod state_bounded_replay; mod store_commands; mod turn_window; mod write_retry; @@ -49,273 +49,49 @@ pub(crate) fn prepare_loaded_events( event_conversion::backfill_tool_inputs_from_messages(session_id, &mut events); event_conversion::backfill_subagent_links(session_id, &mut events); backfill_provider_subagent_prompts(&mut events); - // Old terminal `.txt` files have no Snapshot watermarks. Import the body - // once into the append-only replay artifact and attach only the mutable - // final shell state; the migration deliberately never seeds historical - // event bookmarks, so an early playback cursor cannot see future output. agent_core::tools::impls::coding::exec::legacy_replay::hydrate_legacy_shell_replays( &mut events, ); - // Compaction boundaries live only in `agent_messages` (never in the - // event cache) — merge them in so the chat shows the compacted marker. event_conversion::merge_compact_boundary_events(session_id, &mut events); events } #[cfg(test)] -mod streaming_snapshot_delta_tests { - use super::notify::build_streaming_snapshot_delta; - use super::*; - use crate::agent_sessions::event_pipeline::store::EventStore; - - fn test_event(id: &str, created_at: &str) -> SessionEvent { - serde_json::from_value(serde_json::json!({ - "id": id, - "chunk_id": null, - "sessionId": "streaming-delta-test", - "createdAt": created_at, - "functionName": "assistant_message", - "uiCanonical": "message", - "actionType": "assistant", - "args": {}, - "result": { "content": id }, - "source": "assistant", - "displayText": id, - "displayStatus": "running", - "displayVariant": "message", - "activityStatus": "agent" - })) - .expect("valid test event") - } - - #[test] - fn streaming_delta_compacts_only_changed_events_and_tracks_positions() { - let mut store = EventStore::new(); - let baseline = (0..100) - .map(|index| { - test_event( - &format!("event-{index:03}"), - &format!("2026-07-22T00:00:{index:02}.000Z"), - ) - }) - .collect(); - store.set(baseline); - store.mark_full_snapshot_emitted(); - store.set_streaming(true); - - store.upsert(test_event("event-050", "2026-07-22T00:00:50.000Z")); - store.append(vec![test_event("event-100", "2026-07-22T00:01:40.000Z")]); - - let delta = build_streaming_snapshot_delta(&mut store); - assert!(delta.incremental_orders); - assert!(delta.streaming); - assert_eq!(delta.upserts.len(), 2); - assert_eq!(delta.memberships.len(), 2); - assert_eq!(delta.memberships[0].id, "event-050"); - assert_eq!(delta.memberships[0].event_index, 50); - assert_eq!(delta.memberships[1].id, "event-100"); - assert_eq!(delta.memberships[1].event_index, 100); - assert!(delta.event_ids.is_empty()); - assert!(delta.chat_event_ids.is_empty()); - assert!(delta.sorted_simulator_event_ids.is_empty()); - - let no_op = build_streaming_snapshot_delta(&mut store); - assert_eq!(no_op.base_version, delta.version); - assert!(no_op.upserts.is_empty()); - assert!(no_op.memberships.is_empty()); - } - - #[test] - fn round_window_reorder_requires_a_new_full_baseline() { - let mut store = EventStore::new(); - store.set(vec![test_event("event-newer", "2026-07-22T00:01:00.000Z")]); - store.mark_full_snapshot_emitted(); - store.set_streaming(true); - - store - .merge_round_window_events(vec![test_event("event-older", "2026-07-22T00:00:00.000Z")]); - - assert!(store.should_emit_full_snapshot()); - } -} +#[path = "tests/streaming_snapshot_delta.rs"] +mod streaming_snapshot_delta_tests; #[cfg(test)] -mod runtime_artifact_tests { - use super::push_events::collect_post_merge_persistable_events; - use super::*; - use core_types::extracted::ExtractedEditData; - use orgtrack_core::canonical::{AgentMetadata, SOURCE_ORGII_RUST_AGENTS}; - use orgtrack_core::edit_extraction::{artifacts_from_extracted_edit, EditArtifactContext}; - use orgtrack_core::repo_sync::paths::record_id; - use std::collections::HashSet; - - fn test_event(id: &str, call_id: Option<&str>) -> SessionEvent { - serde_json::from_value(serde_json::json!({ - "id": id, - "chunk_id": null, - "sessionId": "test-session", - "createdAt": "2026-07-19T00:00:00Z", - "functionName": "run_shell", - "uiCanonical": "run_shell", - "actionType": "tool_call", - "args": {}, - "result": {}, - "source": "assistant", - "displayText": id, - "displayStatus": "running", - "displayVariant": "tool_call", - "activityStatus": "agent", - "callId": call_id - })) - .unwrap() - } - - #[test] - fn post_merge_persistence_preserves_timeline_order() { - let events = vec![ - test_event("event-c", Some("call-c")), - test_event("event-a", Some("call-a")), - test_event("event-b", Some("call-b")), - ]; - let incoming_ids = HashSet::from(["event-b".to_string(), "event-c".to_string()]); - let result_call_ids = HashSet::from(["call-a".to_string()]); - - let selected = - collect_post_merge_persistable_events(&events, &incoming_ids, &result_call_ids); +#[path = "tests/runtime_artifact.rs"] +mod runtime_artifact_tests; - assert_eq!( - selected - .into_iter() - .map(|event| event.id) - .collect::>(), - vec!["event-c", "event-a", "event-b"] - ); - } - - #[test] - fn runtime_projection_uses_backfill_record_id_shape() { - let edit = ExtractedEditData { - file_path: "src/main.rs".to_string(), - file_name: "main.rs".to_string(), - language: "rust".to_string(), - content: None, - line_count: None, - old_content: Some("fn main() {}\n".to_string()), - new_content: Some("fn main() { println!(\"hi\"); }\n".to_string()), - diff: None, - old_start_line: Some(1), - new_start_line: Some(1), - lines_added: Some(1), - lines_removed: Some(1), - is_deleted: false, - apply_patch_segments: Vec::new(), - }; - let context = EditArtifactContext { - source: SOURCE_ORGII_RUST_AGENTS.to_string(), - source_session_id: Some("sdeagent-1".to_string()), - session_id: "sdeagent-1".to_string(), - source_event_id: Some("tool-call-1".to_string()), - turn_id: Some("turn-1".to_string()), - sequence_index: 7, - timestamp: Some("2026-06-17T00:00:00Z".to_string()), - workspace_path: Some("/tmp/repo".to_string()), - metadata: AgentMetadata::default(), - }; - - let artifacts = artifacts_from_extracted_edit(&context, &edit); - - assert_eq!(artifacts.edits.len(), 1); - assert_eq!(artifacts.chunks.len(), 1); - assert_eq!( - artifacts.edits[0].record_id, - record_id(&[ - "edit", - SOURCE_ORGII_RUST_AGENTS, - "sdeagent-1", - "tool-call-1", - "7", - "0", - "src/main.rs", - ]) - ); - assert_eq!( - artifacts.chunks[0].record_id, - record_id(&[ - "diff_chunk", - SOURCE_ORGII_RUST_AGENTS, - "sdeagent-1", - "tool-call-1", - "7", - "0", - "src/main.rs", - ]) - ); - } -} - -// ============================================================================ -// Re-exports -// -// Re-export all Tauri commands from submodules. Using `pub use *` ensures the -// `#[tauri::command]` macro-generated `__cmd__` functions are also exported. -// ============================================================================ - -// Managed state pub use state::EventStoreState; +const BOUNDED_REPLAY_STORE_MAX_BYTES: usize = state_bounded_replay::BOUNDED_REPLAY_STORE_MAX_BYTES; -// Notification helpers pub(crate) use notify::schedule_notify; - -// SQLite write-through with retry pub(crate) use write_retry::save_events_retry; use write_retry::CRITICAL_WRITE_MAX_RETRIES; pub(super) use write_retry::{persist_events_with_retry, BULK_WRITE_MAX_RETRIES}; -// Runtime orgtrack artifact persistence use runtime_artifacts::persist_runtime_orgtrack_records_async; pub(crate) use runtime_artifacts::runtime_artifact_session_record; -// Push-events write path pub use push_events::{ push_events_to_session, update_spawning_tool_args_with_persist, update_tool_args_by_call_id_with_persist, }; -// Store commands -pub use store_commands::*; - -// Session manager commands -pub use session_manager::*; - -// Snapshot commands -pub use snapshot::*; - -// Cache bridge commands -pub use cache_bridge::*; - -// Event conversion helpers (CachedEvent <-> SessionEvent, dedup, backfill, filtering) -pub use event_conversion::*; - -// Turn window commands -pub use turn_window::*; - -// Analytics commands +// Re-export command macros from every implementation module so existing +// `generate_handler!` paths and wire names remain unchanged. pub use analytics::*; - -// Pagination commands -pub use pagination::*; - -// Batch update commands pub use batch_update::*; - -// Ingestion commands -pub use ingestion::*; - -// Extractor commands +pub use cache_bridge::*; +pub use event_conversion::*; pub use extractors::*; - -// Search commands -pub use search::*; - -// History commands pub use history::*; +pub use ingestion::*; +pub use pagination::*; +pub use search::*; +pub use session_manager::*; +pub use snapshot::*; +pub use store_commands::*; +pub use turn_window::*; diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/pagination.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/pagination.rs index 983a4ad9e..ed30da735 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/pagination.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/pagination.rs @@ -8,6 +8,7 @@ use tauri::State; use crate::agent_sessions::event_pipeline::pagination::{ self, EventFilters, FunctionUsageCount, PaginatedEvents, PaginationRequest, }; +use crate::agent_sessions::event_pipeline::session_providers; use crate::agent_sessions::event_pipeline::types::SessionEvent; use session_persistence as sqlite_cache; @@ -35,6 +36,7 @@ pub async fn es_paginate_cached_events( session_id: String, request: PaginationRequest, ) -> Result { + session_providers::reject_bounded_replay_full_load(&session_id)?; let cached = tokio::task::spawn_blocking(move || sqlite_cache::load_events(&session_id)) .await .map_err(|e| e.to_string())? diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/replay_cloud_wire.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/replay_cloud_wire.rs new file mode 100644 index 000000000..0e9cbdc9b --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/replay_cloud_wire.rs @@ -0,0 +1,248 @@ +//! Authoritative physical wire contract for bounded replay Cloud rows. +//! +//! The external replay spool writes this format and collaboration snapshot +//! ingest reads it. Keeping the framing, field layout and budgets here avoids +//! encoder/decoder drift across the two command subsystems. + +use std::io::Write; + +use orgtrack_core::sources::imported_history::replay; +use serde::{Deserialize, Serialize}; + +pub(crate) const CLOUD_SEGMENT_WIRE_MAX_BYTES: usize = replay::HARD_MAX_PAYLOAD_RANGE_BYTES; +/// One already-published Attachment-V1 physical row may predate the bounded +/// wire contract. New writers remain capped by `CLOUD_SEGMENT_WIRE_MAX_BYTES`; +/// ingest grants this larger ceiling to at most one candidate row per page +/// and verifies after decompression that it is actually V1. +pub(crate) const LEGACY_V1_MAX_WIRE_BYTES: usize = 64 * 1024 * 1024; +pub(crate) const CLOUD_PAGE_MAX_BYTES: usize = replay::HARD_MAX_IPC_BYTES; +pub(crate) const CLOUD_PAGE_MAX_SEGMENTS: usize = replay::HARD_MAX_EVENTS; +/// Leaves room for frame metadata, gzip overhead, base64 growth and the +/// containing JSON object under [`CLOUD_SEGMENT_WIRE_MAX_BYTES`]. +pub(crate) const REPLAY_ATTACHMENT_CHUNK_BYTES: usize = 176 * 1024; +pub(crate) const REPLAY_ATTACHMENT_V2_MAGIC: &[u8] = b"ORGII-REPLAY-ATTACHMENT-V2\0"; +pub(crate) const REPLAY_ATTACHMENT_V2_MAX_DECOMPRESSED_BYTES: u64 = 512 * 1024; +pub(crate) const LEGACY_V1_MAX_DECOMPRESSED_BYTES: u64 = 64 * 1024 * 1024; + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +pub(crate) struct ReplayAttachmentV2FrameHeader { + pub(crate) kind: String, + pub(crate) attachment_id: String, + pub(crate) part_index: u64, + pub(crate) chunk_offset: u64, + pub(crate) chunk_bytes: u64, + pub(crate) final_part: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) event_bytes: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) attachment_hash: Option, +} + +#[derive(Debug)] +pub(crate) struct DecodedReplayAttachmentV2Frame<'a> { + pub(crate) header: ReplayAttachmentV2FrameHeader, + pub(crate) chunk: &'a [u8], +} + +fn is_sha256_hex(value: &str) -> bool { + value.len() == 64 && value.bytes().all(|byte| byte.is_ascii_hexdigit()) +} + +fn validate_header( + header: &ReplayAttachmentV2FrameHeader, + actual_chunk_bytes: usize, +) -> Result<(), String> { + if header.kind != "event" || !is_sha256_hex(&header.attachment_id) { + return Err("header kind or attachmentId is invalid".to_string()); + } + if header.chunk_bytes != actual_chunk_bytes as u64 { + return Err("chunk length is inconsistent with chunkBytes".to_string()); + } + match ( + header.final_part, + header.event_bytes, + header.attachment_hash.as_deref(), + ) { + (false, None, None) => Ok(()), + (true, Some(event_bytes), Some(hash)) + if is_sha256_hex(hash) + && header + .chunk_offset + .checked_add(header.chunk_bytes) + .is_some_and(|end| end == event_bytes) => + { + Ok(()) + } + _ => Err("final-part metadata is inconsistent".to_string()), + } +} + +pub(crate) fn write_replay_attachment_v2_frame( + output: &mut impl Write, + header: &ReplayAttachmentV2FrameHeader, + chunk: &[u8], +) -> Result<(), String> { + validate_header(header, chunk.len())?; + let header_json = serde_json::to_vec(header) + .map_err(|error| format!("serialize attachment V2 header: {error}"))?; + let header_len = u32::try_from(header_json.len()) + .map_err(|_| "attachment V2 header exceeds u32".to_string())?; + output + .write_all(REPLAY_ATTACHMENT_V2_MAGIC) + .map_err(|error| format!("write attachment V2 magic: {error}"))?; + output + .write_all(&header_len.to_be_bytes()) + .map_err(|error| format!("write attachment V2 header length: {error}"))?; + output + .write_all(&header_json) + .map_err(|error| format!("write attachment V2 header: {error}"))?; + output + .write_all(chunk) + .map_err(|error| format!("write attachment V2 chunk: {error}")) +} + +pub(crate) fn encode_replay_attachment_v2_frame( + header: &ReplayAttachmentV2FrameHeader, + chunk: &[u8], +) -> Result, String> { + let mut frame = Vec::with_capacity( + REPLAY_ATTACHMENT_V2_MAGIC + .len() + .saturating_add(4) + .saturating_add(256) + .saturating_add(chunk.len()), + ); + write_replay_attachment_v2_frame(&mut frame, header, chunk)?; + Ok(frame) +} + +pub(crate) fn decode_replay_attachment_v2_frame( + bytes: &[u8], +) -> Result, String> { + let prefix_bytes = REPLAY_ATTACHMENT_V2_MAGIC.len().saturating_add(4); + if bytes.len() < prefix_bytes || !bytes.starts_with(REPLAY_ATTACHMENT_V2_MAGIC) { + return Err("magic prefix is invalid".to_string()); + } + let header_len = u32::from_be_bytes( + bytes[REPLAY_ATTACHMENT_V2_MAGIC.len()..prefix_bytes] + .try_into() + .map_err(|_| "header length is truncated".to_string())?, + ) as usize; + let payload_offset = prefix_bytes + .checked_add(header_len) + .ok_or_else(|| "header length overflows the frame".to_string())?; + if header_len == 0 || payload_offset > bytes.len() { + return Err("header is truncated".to_string()); + } + let header = serde_json::from_slice::( + &bytes[prefix_bytes..payload_offset], + ) + .map_err(|error| format!("header JSON is invalid: {error}"))?; + let chunk = &bytes[payload_offset..]; + validate_header(&header, chunk.len())?; + Ok(DecodedReplayAttachmentV2Frame { header, chunk }) +} + +#[cfg(test)] +mod tests { + use sha2::{Digest, Sha256}; + + use super::*; + + #[test] + fn attachment_v2_frame_keeps_the_published_golden_hash() { + let header = ReplayAttachmentV2FrameHeader { + kind: "event".to_string(), + attachment_id: "dd56de4137951d9c92681b03416ec15f886b4482a27e3a517d32f085244cbe5d" + .to_string(), + part_index: 0, + chunk_offset: 0, + chunk_bytes: 3, + final_part: true, + event_bytes: Some(3), + attachment_hash: Some( + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad".to_string(), + ), + }; + let frame = encode_replay_attachment_v2_frame(&header, b"abc").expect("golden frame"); + assert_eq!( + format!("{:x}", Sha256::digest(&frame)), + "1cf7b415e8558ddb0d72bcf9212ff381c9a57bfd719628824a61e4a67bcf3126" + ); + let decoded = decode_replay_attachment_v2_frame(&frame).expect("decode golden frame"); + assert_eq!(decoded.header, header); + assert_eq!(decoded.chunk, b"abc"); + } + + #[test] + fn attachment_v2_decoder_rejects_corrupt_framing_and_metadata() { + let header = ReplayAttachmentV2FrameHeader { + kind: "event".to_string(), + attachment_id: "a".repeat(64), + part_index: 0, + chunk_offset: 0, + chunk_bytes: 3, + final_part: true, + event_bytes: Some(3), + attachment_hash: Some("b".repeat(64)), + }; + let frame = encode_replay_attachment_v2_frame(&header, b"abc").expect("valid frame"); + + let mut bad_magic = frame.clone(); + bad_magic[0] ^= 1; + assert!(decode_replay_attachment_v2_frame(&bad_magic) + .expect_err("bad magic must fail") + .contains("magic")); + + let mut wrong_chunk_bytes = header.clone(); + wrong_chunk_bytes.chunk_bytes = 2; + assert!( + encode_replay_attachment_v2_frame(&wrong_chunk_bytes, b"abc") + .expect_err("wrong chunk length must fail") + .contains("chunk length") + ); + + let mut missing_final_hash = header; + missing_final_hash.attachment_hash = None; + assert!( + encode_replay_attachment_v2_frame(&missing_final_hash, b"abc") + .expect_err("incomplete final metadata must fail") + .contains("final-part metadata") + ); + } + + #[test] + fn shared_budget_manifest_matches_the_rust_wire_contract() { + let manifest: serde_json::Value = serde_json::from_str(include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../src/shared/externalReplayBudgets.json" + ))) + .expect("shared replay budget manifest"); + assert_eq!(manifest["replayMaxTurns"], replay::HARD_MAX_TURNS); + assert_eq!(manifest["replayMaxEvents"], replay::HARD_MAX_EVENTS); + assert_eq!(manifest["replayMaxIpcBytes"], replay::HARD_MAX_IPC_BYTES); + assert_eq!( + manifest["payloadRangeMaxBytes"], + replay::HARD_MAX_PAYLOAD_RANGE_BYTES + ); + assert_eq!( + manifest["shellReplayRangeMaxBytes"], + agent_core::tools::impls::coding::exec::shell_replay::SHELL_REPLAY_RANGE_MAX_BYTES + ); + assert_eq!( + manifest["cloudSegmentMaxBytes"], + CLOUD_SEGMENT_WIRE_MAX_BYTES + ); + assert_eq!( + manifest["cloudLegacyV1SegmentMaxBytes"], + LEGACY_V1_MAX_WIRE_BYTES + ); + assert_eq!(manifest["cloudPageMaxBytes"], CLOUD_PAGE_MAX_BYTES); + assert_eq!(manifest["cloudPageMaxSegments"], CLOUD_PAGE_MAX_SEGMENTS); + assert_eq!( + manifest["cloudAttachmentChunkBytes"], + REPLAY_ATTACHMENT_CHUNK_BYTES + ); + } +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/session_manager.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/session_manager.rs index 171832e70..5ebeb3851 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/session_manager.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/session_manager.rs @@ -10,7 +10,7 @@ use tauri::{AppHandle, State}; use crate::agent_sessions::event_pipeline::types::SessionEvent; -use super::{schedule_notify, EventStoreState}; +use super::{external_replay, schedule_notify, EventStoreState}; /// Set the active session (the default target for commands without an /// explicit `session_id`). Returns true if a store for the session already @@ -23,12 +23,26 @@ pub async fn es_switch_session( state: State<'_, EventStoreState>, session_id: String, ) -> Result { - let (hit, event_count, evicted) = { + let (previous_active, hit, event_count, evicted) = { let mut mgr = state .session_manager .lock() .unwrap_or_else(|e| e.into_inner()); + let previous_active = mgr.active_id().map(str::to_string); + // Serialize old-session prewarm cancellation with its EventStore + // commit. The commit takes the same manager lock before checking its + // independent episode ticket, so a late A request cannot write after + // the active session has switched to B. + if let Some(previous_active) = previous_active + .as_deref() + .filter(|previous_active| *previous_active != session_id) + { + external_replay::cancel_prewarm_requests(previous_active); + } let evicted = mgr.set_active(&session_id); + for evicted_session_id in &evicted { + external_replay::cancel_prewarm_requests(evicted_session_id); + } let event_count = { let stores = state.stores.lock().unwrap_or_else(|e| e.into_inner()); stores @@ -36,12 +50,20 @@ pub async fn es_switch_session( .map(|store| store.events().len()) .unwrap_or(0) }; - (event_count > 0, event_count, evicted) + (previous_active, event_count > 0, event_count, evicted) }; + if let Some(previous_active) = previous_active + .as_deref() + .filter(|previous_active| *previous_active != session_id) + { + external_replay::release_session_runtime(previous_active); + } + if !evicted.is_empty() { let mut stores = state.stores.lock().unwrap_or_else(|e| e.into_inner()); for sid in &evicted { + external_replay::release_session_runtime(sid); stores.remove(sid); } } @@ -82,6 +104,7 @@ pub async fn es_unpin_session( if !evicted.is_empty() { let mut stores = state.stores.lock().unwrap_or_else(|e| e.into_inner()); for sid in &evicted { + external_replay::release_session_runtime(sid); stores.remove(sid); } } @@ -94,6 +117,7 @@ pub async fn es_evict_session( state: State<'_, EventStoreState>, session_id: String, ) -> Result<(), String> { + external_replay::release_session_runtime(&session_id); { let mut mgr = state .session_manager @@ -119,7 +143,9 @@ pub async fn es_buffer_events( session_id: String, events: Vec, ) -> Result<(), String> { + let incoming_bytes = state.validate_bounded_replay_input(&session_id, &events)?; state.with_store_mut(&session_id, |store| store.append(events)); + state.account_bounded_replay_write(&session_id, incoming_bytes)?; schedule_notify(&app, &state, &session_id); Ok(()) } diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/state.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/state.rs index 6e1634465..42650852f 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/state.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/state.rs @@ -5,6 +5,8 @@ //! the batched-notification bookkeeping consumed by [`super::schedule_notify`]. use std::collections::{HashMap, HashSet}; +#[cfg(test)] +use std::sync::atomic::AtomicUsize; use std::sync::Mutex; use crate::agent_sessions::event_pipeline::session_manager::SessionStoreManager; @@ -24,6 +26,8 @@ pub struct EventStoreState { pub session_manager: Mutex, /// Tracks which sessions already have a batched notification pending. pub notify_pending: Mutex>, + #[cfg(test)] + pub(super) bounded_replay_exact_cap_count: AtomicUsize, } impl Default for EventStoreState { @@ -38,6 +42,8 @@ impl EventStoreState { stores: Mutex::new(HashMap::new()), session_manager: Mutex::new(SessionStoreManager::new()), notify_pending: Mutex::new(HashSet::new()), + #[cfg(test)] + bounded_replay_exact_cap_count: AtomicUsize::new(0), } } diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/state_bounded_replay.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/state_bounded_replay.rs new file mode 100644 index 000000000..83b61c03b --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/state_bounded_replay.rs @@ -0,0 +1,321 @@ +//! Memory policy for external and managed CLI replay stores. +//! +//! Native SDE sessions never call these helpers. Keeping the policy beside, +//! rather than inside, the generic state registry makes that boundary visible. + +use std::collections::HashSet; +#[cfg(test)] +use std::sync::atomic::Ordering; + +use crate::agent_sessions::event_pipeline::session_providers; +use crate::agent_sessions::event_pipeline::types::{SessionEvent, SessionEventPatch}; + +use super::{external_replay, EventStoreState}; + +enum ExternalReplayCapPolicy<'a> { + NewestSuffix, + PreservedEventIds(&'a HashSet), + CurrentUserAnchor, +} + +/// Generic writes used by managed/imported CLI sessions must obey the same +/// resident-memory budget as explicit external replay windows. +pub(super) const BOUNDED_REPLAY_STORE_MAX_BYTES: usize = 16 * 1024 * 1024; +/// A renderer-originated write must stay small. Larger bodies belong behind +/// replay/payload locators, not in the resident EventStore. +pub(super) const BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES: usize = 4 * 1024 * 1024; +/// Exact compaction leaves headroom so same-ID streaming updates can be +/// accounted conservatively without rescanning the store for every token. +const BOUNDED_REPLAY_GENERIC_COMPACT_BYTES: usize = 14 * 1024 * 1024; + +impl EventStoreState { + /// Apply the external-replay-only per-store byte cap, publish its actual + /// serialized footprint to the aggregate byte LRU, and remove idle stores + /// evicted by that policy. + pub fn cap_external_replay_store( + &self, + session_id: &str, + max_bytes: usize, + ) -> Result { + self.cap_external_replay_store_inner( + session_id, + max_bytes, + ExternalReplayCapPolicy::NewestSuffix, + ) + } + + /// Apply the external replay byte cap while pinning one foreground window. + /// This is used only after an older-page merge so the returned page remains + /// visible long enough for the renderer snapshot barrier to observe it. + pub fn cap_external_replay_store_preserving_window( + &self, + session_id: &str, + max_bytes: usize, + window: &[SessionEvent], + ) -> Result { + let preserved_event_ids = window + .iter() + .map(|event| event.id.clone()) + .collect::>(); + self.cap_external_replay_store_inner( + session_id, + max_bytes, + ExternalReplayCapPolicy::PreservedEventIds(&preserved_event_ids), + ) + } + + /// Keep the provider user row that owns the currently visible replay + /// body while incorporating a live delta. This is distinct from retaining + /// the delta rows themselves: the poll cursor follows the newest source + /// tail even when the user is reading an older random-access Round. + pub fn cap_external_replay_store_preserving_current_user_anchor( + &self, + session_id: &str, + max_bytes: usize, + ) -> Result { + self.cap_external_replay_store_inner( + session_id, + max_bytes, + ExternalReplayCapPolicy::CurrentUserAnchor, + ) + } + + fn cap_external_replay_store_inner( + &self, + session_id: &str, + max_bytes: usize, + policy: ExternalReplayCapPolicy<'_>, + ) -> Result { + // Match the EventStore write/switch lock order: manager -> stores. + let (bytes, evicted) = { + let mut manager = self + .session_manager + .lock() + .unwrap_or_else(|error| error.into_inner()); + let mut stores = self + .stores + .lock() + .unwrap_or_else(|error| error.into_inner()); + let Some(store) = stores.get_mut(session_id) else { + return Ok(0); + }; + #[cfg(test)] + self.bounded_replay_exact_cap_count + .fetch_add(1, Ordering::Relaxed); + let bytes = match policy { + ExternalReplayCapPolicy::NewestSuffix => { + store.cap_external_replay_bytes(max_bytes)? + } + ExternalReplayCapPolicy::PreservedEventIds(preserved_event_ids) => { + store.cap_external_replay_bytes_preserving(max_bytes, preserved_event_ids)? + } + ExternalReplayCapPolicy::CurrentUserAnchor => { + store.cap_external_replay_bytes_preserving_current_user_anchor(max_bytes)? + } + }; + let evicted = manager.update_estimated_bytes(session_id, bytes); + (bytes, evicted) + }; + self.remove_evicted_replay_stores(evicted); + Ok(bytes) + } + + fn remove_evicted_replay_stores(&self, evicted: Vec) { + if evicted.is_empty() { + return; + } + let mut stores = self + .stores + .lock() + .unwrap_or_else(|error| error.into_inner()); + for evicted_session_id in evicted { + external_replay::release_session_runtime(&evicted_session_id); + stores.remove(&evicted_session_id); + } + } + + /// Reject an input that cannot fit in an otherwise empty bounded store. + pub(crate) fn validate_bounded_replay_input( + &self, + session_id: &str, + events: &[SessionEvent], + ) -> Result { + if !session_providers::uses_bounded_replay(session_id) || events.is_empty() { + return Ok(0); + } + let incoming_bytes = serde_json::to_vec(events) + .map_err(|error| format!("serialize bounded replay write: {error}"))? + .len(); + if incoming_bytes > BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES { + return Err(format!( + "bounded replay write for {session_id} exceeds the {} byte generic input budget; store large bodies behind a payload locator", + BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES + )); + } + Ok(incoming_bytes) + } + + /// Preflight a partial update before it clones payload into target rows. + pub(crate) fn validate_bounded_replay_patch( + &self, + session_id: &str, + ids: &[String], + patch: &SessionEventPatch, + ) -> Result<(), String> { + if !session_providers::uses_bounded_replay(session_id) || ids.is_empty() { + return Ok(()); + } + + let patch_bytes = serde_json::to_vec(patch) + .map_err(|error| format!("serialize bounded replay patch: {error}"))? + .len(); + if patch_bytes > BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES { + return Err(format!( + "bounded replay patch for {session_id} exceeds the {} byte generic input budget", + BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES + )); + } + let unique_ids = ids.iter().map(String::as_str).collect::>(); + self.with_store_opt(session_id, |store| { + let target_count = unique_ids + .iter() + .filter(|id| store.get_by_id(id).is_some()) + .count(); + if patch_bytes.saturating_mul(target_count) + > BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES + { + return Err(format!( + "bounded replay patch for {session_id} exceeds the {} byte amplification budget across {target_count} events", + BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES + )); + } + + for id in &unique_ids { + let Some(existing) = store.get_by_id(id) else { + continue; + }; + let mut projected = existing.clone(); + patch.apply_to(&mut projected); + let projected_bytes = serde_json::to_vec(std::slice::from_ref(&projected)) + .map_err(|error| format!("serialize projected bounded replay event: {error}"))? + .len(); + if projected_bytes > BOUNDED_REPLAY_GENERIC_COMPACT_BYTES { + return Err(format!( + "bounded replay patch would make event {id} exceed the {} byte compacted store budget", + BOUNDED_REPLAY_GENERIC_COMPACT_BYTES + )); + } + } + Ok(()) + }) + .unwrap_or(Ok(())) + } + + /// Bound the command that copies one JSON object into sibling tool rows. + pub(crate) fn validate_bounded_replay_args_merge( + &self, + session_id: &str, + function_names: &[&str], + merge_args: &serde_json::Value, + ) -> Result<(), String> { + if !session_providers::uses_bounded_replay(session_id) { + return Ok(()); + } + let merge_bytes = serde_json::to_vec(merge_args) + .map_err(|error| format!("serialize bounded replay args merge: {error}"))? + .len(); + if merge_bytes > BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES { + return Err(format!( + "bounded replay args merge for {session_id} exceeds the {} byte generic input budget", + BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES + )); + } + + self.with_store_opt(session_id, |store| { + let Some(primary_index) = store.find_last_spawning_tool(function_names) else { + return Ok(()); + }; + let primary = &store.events()[primary_index]; + let target_count = primary.call_id.as_deref().map_or(1, |call_id| { + store + .events() + .iter() + .filter(|event| { + event.action_type == "tool_call" + && event.call_id.as_deref() == Some(call_id) + }) + .count() + }); + if merge_bytes.saturating_mul(target_count) + > BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES + { + return Err(format!( + "bounded replay args merge for {session_id} exceeds the {} byte amplification budget across {target_count} events", + BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES + )); + } + for event in store.events().iter().filter(|event| { + event.id == primary.id + || primary.call_id.as_deref().is_some_and(|call_id| { + event.action_type == "tool_call" + && event.call_id.as_deref() == Some(call_id) + }) + }) { + let existing_bytes = serde_json::to_vec(event) + .map_err(|error| format!("serialize bounded replay merge target: {error}"))? + .len(); + if existing_bytes.saturating_add(merge_bytes) + > BOUNDED_REPLAY_GENERIC_COMPACT_BYTES + { + return Err(format!( + "bounded replay args merge would make event {} exceed the {} byte compacted store budget", + event.id, BOUNDED_REPLAY_GENERIC_COMPACT_BYTES + )); + } + } + Ok(()) + }) + .unwrap_or(Ok(())) + } + + /// Apply the byte cap and refresh accounting after a generic mutation. + pub(crate) fn enforce_bounded_replay_store_policy( + &self, + session_id: &str, + ) -> Result<(), String> { + if !session_providers::uses_bounded_replay(session_id) { + return Ok(()); + } + self.cap_external_replay_store(session_id, BOUNDED_REPLAY_GENERIC_COMPACT_BYTES)?; + Ok(()) + } + + /// Account a generic append/upsert/merge using a conservative upper bound. + pub(crate) fn account_bounded_replay_write( + &self, + session_id: &str, + incoming_upper_bound: usize, + ) -> Result<(), String> { + if !session_providers::uses_bounded_replay(session_id) { + return Ok(()); + } + let (estimated_bytes, evicted) = { + let mut manager = self + .session_manager + .lock() + .unwrap_or_else(|error| error.into_inner()); + manager.add_estimated_bytes(session_id, incoming_upper_bound) + }; + let current_was_evicted = evicted.iter().any(|evicted_id| evicted_id == session_id); + self.remove_evicted_replay_stores(evicted); + if !current_was_evicted && estimated_bytes > BOUNDED_REPLAY_STORE_MAX_BYTES { + self.cap_external_replay_store(session_id, BOUNDED_REPLAY_GENERIC_COMPACT_BYTES)?; + } + Ok(()) + } + + #[cfg(test)] + pub(super) fn bounded_replay_exact_cap_count(&self) -> usize { + self.bounded_replay_exact_cap_count.load(Ordering::Relaxed) + } +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/store_commands.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/store_commands.rs index 55c72d81c..dea3a2b79 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/store_commands.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/store_commands.rs @@ -64,7 +64,9 @@ pub async fn es_set( ) -> Result<(), String> { normalize_events(&mut events); let sid = state.resolve_session_id(session_id)?; + state.validate_bounded_replay_input(&sid, &events)?; state.with_store_mut(&sid, |store| store.set(events)); + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); Ok(()) } @@ -99,6 +101,7 @@ pub async fn es_append( normalize_event_records(event); } let sid = state.resolve_session_id(session_id)?; + let incoming_bytes = state.validate_bounded_replay_input(&sid, &events)?; // Persist user-authored events so the truncate-on-edit path can locate // them by ID. Non-user events appended via es_append are UI-only @@ -134,6 +137,7 @@ pub async fn es_append( .map(session_event_to_cached_event) .collect::>() }); + state.account_bounded_replay_write(&sid, incoming_bytes)?; schedule_notify(&app, &state, &sid); if !user_events.is_empty() { @@ -183,7 +187,9 @@ pub async fn es_upsert( } normalize_event_records(&mut event); let sid = state.resolve_session_id(session_id)?; + let incoming_bytes = state.validate_bounded_replay_input(&sid, std::slice::from_ref(&event))?; state.with_store_mut(&sid, |store| store.upsert(event)); + state.account_bounded_replay_write(&sid, incoming_bytes)?; schedule_notify(&app, &state, &sid); Ok(()) } @@ -198,8 +204,10 @@ pub async fn es_update_by_id( patch: SessionEventPatch, ) -> Result { let sid = state.resolve_session_id(session_id)?; + state.validate_bounded_replay_patch(&sid, std::slice::from_ref(&id), &patch)?; let found = state.with_store_mut(&sid, |store| store.update_by_id(&id, &patch)); if found { + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); } Ok(found) @@ -250,7 +258,9 @@ pub async fn es_merge_events( ) -> Result<(), String> { normalize_events(&mut events); let sid = state.resolve_session_id(session_id)?; + let incoming_bytes = state.validate_bounded_replay_input(&sid, &events)?; state.with_store_mut(&sid, |store| store.merge_events(events)); + state.account_bounded_replay_write(&sid, incoming_bytes)?; schedule_notify(&app, &state, &sid); Ok(()) } @@ -264,7 +274,9 @@ pub async fn es_merge_round_window_events( ) -> Result<(), String> { normalize_events(&mut events); let sid = state.resolve_session_id(session_id)?; + let incoming_bytes = state.validate_bounded_replay_input(&sid, &events)?; state.with_store_mut(&sid, |store| store.merge_round_window_events(events)); + state.account_bounded_replay_write(&sid, incoming_bytes)?; schedule_notify(&app, &state, &sid); Ok(()) } @@ -304,6 +316,7 @@ pub async fn es_clear( ) -> Result<(), String> { let sid = state.resolve_session_id(session_id)?; state.with_store_mut(&sid, |store| store.clear()); + state.enforce_bounded_replay_store_policy(&sid)?; schedule_notify(&app, &state, &sid); Ok(()) } @@ -319,6 +332,7 @@ pub async fn es_truncate_before_id( let sid = state.resolve_session_id(session_id)?; let found = state.with_store_mut(&sid, |store| store.truncate_before_id(&event_id)); if found { + state.enforce_bounded_replay_store_policy(&sid)?; let persist_sid = sid.clone(); let persist_event_id = event_id.clone(); tokio::task::spawn_blocking(move || { diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/tests/runtime_artifact.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/tests/runtime_artifact.rs new file mode 100644 index 000000000..270d1b76d --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/tests/runtime_artifact.rs @@ -0,0 +1,291 @@ +use super::*; +use std::collections::HashSet; + +use core_types::extracted::ExtractedEditData; +use core_types::session_event::SessionEventPatch; +use orgtrack_core::canonical::{AgentMetadata, SOURCE_ORGII_RUST_AGENTS}; +use orgtrack_core::edit_extraction::{artifacts_from_extracted_edit, EditArtifactContext}; +use orgtrack_core::repo_sync::paths::record_id; + +use super::push_events::collect_post_merge_persistable_events; +use super::state_bounded_replay::BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES; + +fn test_event(id: &str, call_id: Option<&str>) -> SessionEvent { + serde_json::from_value(serde_json::json!({ + "id": id, + "chunk_id": null, + "sessionId": "test-session", + "createdAt": "2026-07-19T00:00:00Z", + "functionName": "run_shell", + "uiCanonical": "run_shell", + "actionType": "tool_call", + "args": {}, + "result": {}, + "source": "assistant", + "displayText": id, + "displayStatus": "running", + "displayVariant": "tool_call", + "activityStatus": "agent", + "callId": call_id + })) + .unwrap() +} + +fn replay_policy_event(session_id: &str, id: &str, body_bytes: usize) -> SessionEvent { + let mut event = test_event(id, None); + event.session_id = session_id.to_string(); + event.function_name = "assistant_message".to_string(); + event.ui_canonical = "assistant_message".to_string(); + event.action_type = "raw".to_string(); + event.display_text = "x".repeat(body_bytes); + event +} + +#[test] +fn post_merge_persistence_preserves_timeline_order() { + let events = vec![ + test_event("event-c", Some("call-c")), + test_event("event-a", Some("call-a")), + test_event("event-b", Some("call-b")), + ]; + let incoming_ids = HashSet::from(["event-b".to_string(), "event-c".to_string()]); + let result_call_ids = HashSet::from(["call-a".to_string()]); + + let selected = collect_post_merge_persistable_events(&events, &incoming_ids, &result_call_ids); + + assert_eq!( + selected + .into_iter() + .map(|event| event.id) + .collect::>(), + vec!["event-c", "event-a", "event-b"] + ); +} + +#[test] +fn managed_cli_generic_live_writes_remain_under_resident_budget() { + const EVENT_BODY_BYTES: usize = 256 * 1024; + let session_id = "cliagent-live-budget"; + let state = EventStoreState::new(); + assert!(state + .session_manager + .lock() + .unwrap_or_else(|error| error.into_inner()) + .set_active(session_id) + .is_empty()); + + for index in 0..80 { + let event = replay_policy_event( + session_id, + &format!("managed-live-{index}"), + EVENT_BODY_BYTES, + ); + let incoming_bytes = state + .validate_bounded_replay_input(session_id, std::slice::from_ref(&event)) + .expect("individual live event fits bounded store"); + state.with_store_mut(session_id, |store| store.append(vec![event])); + state + .account_bounded_replay_write(session_id, incoming_bytes) + .expect("live write policy succeeds"); + } + + let (bytes, count) = state + .with_store_opt(session_id, |store| { + ( + serde_json::to_vec(store.events()) + .expect("serialize managed CLI store") + .len(), + store.event_count(), + ) + }) + .expect("managed CLI store exists"); + assert!(bytes <= BOUNDED_REPLAY_STORE_MAX_BYTES); + assert!(count < 80, "old live events must be evicted by bytes"); + assert!(state.bounded_replay_exact_cap_count() > 0); +} + +#[test] +fn hundreds_of_same_id_stream_updates_do_not_rescan_store_per_token() { + const EVENT_BODY_BYTES: usize = 64 * 1024; + const UPDATE_COUNT: usize = 400; + let session_id = "cliagent-same-id-stream"; + let state = EventStoreState::new(); + + for _ in 0..UPDATE_COUNT { + let event = replay_policy_event(session_id, "streaming-message", EVENT_BODY_BYTES); + let incoming_bytes = state + .validate_bounded_replay_input(session_id, std::slice::from_ref(&event)) + .expect("stream update fits input budget"); + state.with_store_mut(session_id, |store| store.upsert(event)); + state + .account_bounded_replay_write(session_id, incoming_bytes) + .expect("amortized stream accounting succeeds"); + } + + let bytes = state + .with_store_opt(session_id, |store| { + assert_eq!(store.event_count(), 1); + serde_json::to_vec(store.events()) + .expect("serialize same-ID stream store") + .len() + }) + .expect("same-ID store exists"); + let exact_scans = state.bounded_replay_exact_cap_count(); + assert!(bytes <= BOUNDED_REPLAY_STORE_MAX_BYTES); + assert!( + exact_scans > 0, + "conservative debt eventually forces a scan" + ); + assert!( + exact_scans < UPDATE_COUNT / 20, + "streaming must not rescan the resident store for every token: {exact_scans} scans" + ); +} + +#[test] +fn native_sde_generic_writes_do_not_enter_external_store_policy() { + const EVENT_BODY_BYTES: usize = 1024 * 1024; + let session_id = "sdeagent-native-budget-control"; + let state = EventStoreState::new(); + + for index in 0..17 { + let event = replay_policy_event( + session_id, + &format!("native-live-{index}"), + EVENT_BODY_BYTES, + ); + let incoming_bytes = state + .validate_bounded_replay_input(session_id, std::slice::from_ref(&event)) + .expect("native preflight is a no-op"); + state.with_store_mut(session_id, |store| store.append(vec![event])); + state + .account_bounded_replay_write(session_id, incoming_bytes) + .expect("native policy is a no-op"); + } + + let (bytes, count) = state + .with_store_opt(session_id, |store| { + ( + serde_json::to_vec(store.events()) + .expect("serialize native store") + .len(), + store.event_count(), + ) + }) + .expect("native store exists"); + assert!(bytes > BOUNDED_REPLAY_STORE_MAX_BYTES); + assert_eq!(count, 17, "native EventStore semantics stay unchanged"); + assert_eq!(state.bounded_replay_exact_cap_count(), 0); +} + +#[test] +fn oversized_managed_cli_write_fails_before_mutating_store() { + let session_id = "cliagent-oversized-live"; + let state = EventStoreState::new(); + let event = replay_policy_event( + session_id, + "oversized-event", + BOUNDED_REPLAY_GENERIC_INPUT_MAX_BYTES, + ); + + let error = state + .validate_bounded_replay_input(session_id, std::slice::from_ref(&event)) + .expect_err("serialization overhead must push the event over budget"); + assert!(error.contains("exceeds")); + assert!(state.with_store_opt(session_id, |_| ()).is_none()); +} + +#[test] +fn repeated_large_managed_cli_patch_fails_before_payload_amplification() { + let session_id = "cliagent-large-patch"; + let state = EventStoreState::new(); + let events = (0..17) + .map(|index| replay_policy_event(session_id, &format!("patch-{index}"), 32)) + .collect::>(); + let ids = events + .iter() + .map(|event| event.id.clone()) + .collect::>(); + state.with_store_mut(session_id, |store| store.append(events)); + let patch = SessionEventPatch { + display_text: Some("p".repeat(1024 * 1024)), + ..SessionEventPatch::default() + }; + + let error = state + .validate_bounded_replay_patch(session_id, &ids, &patch) + .expect_err("repeated patch must be rejected before it is cloned"); + assert!(error.contains("amplification")); + assert!( + state.with_store_opt(session_id, |store| { + store + .events() + .iter() + .all(|event| event.display_text.len() == 32) + }) == Some(true) + ); + + assert!(state + .validate_bounded_replay_patch("sdeagent-native", &ids, &patch) + .is_ok()); +} + +#[test] +fn runtime_projection_uses_backfill_record_id_shape() { + let edit = ExtractedEditData { + file_path: "src/main.rs".to_string(), + file_name: "main.rs".to_string(), + language: "rust".to_string(), + content: None, + line_count: None, + old_content: Some("fn main() {}\n".to_string()), + new_content: Some("fn main() { println!(\"hi\"); }\n".to_string()), + diff: None, + old_start_line: Some(1), + new_start_line: Some(1), + lines_added: Some(1), + lines_removed: Some(1), + is_deleted: false, + apply_patch_segments: Vec::new(), + }; + let context = EditArtifactContext { + source: SOURCE_ORGII_RUST_AGENTS.to_string(), + source_session_id: Some("sdeagent-1".to_string()), + session_id: "sdeagent-1".to_string(), + source_event_id: Some("tool-call-1".to_string()), + turn_id: Some("turn-1".to_string()), + sequence_index: 7, + timestamp: Some("2026-06-17T00:00:00Z".to_string()), + workspace_path: Some("/tmp/repo".to_string()), + metadata: AgentMetadata::default(), + }; + + let artifacts = artifacts_from_extracted_edit(&context, &edit); + + assert_eq!(artifacts.edits.len(), 1); + assert_eq!(artifacts.chunks.len(), 1); + assert_eq!( + artifacts.edits[0].record_id, + record_id(&[ + "edit", + SOURCE_ORGII_RUST_AGENTS, + "sdeagent-1", + "tool-call-1", + "7", + "0", + "src/main.rs", + ]) + ); + assert_eq!( + artifacts.chunks[0].record_id, + record_id(&[ + "diff_chunk", + SOURCE_ORGII_RUST_AGENTS, + "sdeagent-1", + "tool-call-1", + "7", + "0", + "src/main.rs", + ]) + ); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/tests/streaming_snapshot_delta.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/tests/streaming_snapshot_delta.rs new file mode 100644 index 000000000..aa2f3f05b --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/tests/streaming_snapshot_delta.rs @@ -0,0 +1,73 @@ +use super::*; +use crate::agent_sessions::event_pipeline::store::EventStore; + +use super::notify::build_streaming_snapshot_delta; + +fn test_event(id: &str, created_at: &str) -> SessionEvent { + serde_json::from_value(serde_json::json!({ + "id": id, + "chunk_id": null, + "sessionId": "streaming-delta-test", + "createdAt": created_at, + "functionName": "assistant_message", + "uiCanonical": "message", + "actionType": "assistant", + "args": {}, + "result": { "content": id }, + "source": "assistant", + "displayText": id, + "displayStatus": "running", + "displayVariant": "message", + "activityStatus": "agent" + })) + .expect("valid test event") +} + +#[test] +fn streaming_delta_compacts_only_changed_events_and_tracks_positions() { + let mut store = EventStore::new(); + let baseline = (0..100) + .map(|index| { + test_event( + &format!("event-{index:03}"), + &format!("2026-07-22T00:00:{index:02}.000Z"), + ) + }) + .collect(); + store.set(baseline); + store.mark_full_snapshot_emitted(); + store.set_streaming(true); + + store.upsert(test_event("event-050", "2026-07-22T00:00:50.000Z")); + store.append(vec![test_event("event-100", "2026-07-22T00:01:40.000Z")]); + + let delta = build_streaming_snapshot_delta(&mut store); + assert!(delta.incremental_orders); + assert!(delta.streaming); + assert_eq!(delta.upserts.len(), 2); + assert_eq!(delta.memberships.len(), 2); + assert_eq!(delta.memberships[0].id, "event-050"); + assert_eq!(delta.memberships[0].event_index, 50); + assert_eq!(delta.memberships[1].id, "event-100"); + assert_eq!(delta.memberships[1].event_index, 100); + assert!(delta.event_ids.is_empty()); + assert!(delta.chat_event_ids.is_empty()); + assert!(delta.sorted_simulator_event_ids.is_empty()); + + let no_op = build_streaming_snapshot_delta(&mut store); + assert_eq!(no_op.base_version, delta.version); + assert!(no_op.upserts.is_empty()); + assert!(no_op.memberships.is_empty()); +} + +#[test] +fn round_window_reorder_requires_a_new_full_baseline() { + let mut store = EventStore::new(); + store.set(vec![test_event("event-newer", "2026-07-22T00:01:00.000Z")]); + store.mark_full_snapshot_emitted(); + store.set_streaming(true); + + store.merge_round_window_events(vec![test_event("event-older", "2026-07-22T00:00:00.000Z")]); + + assert!(store.should_emit_full_snapshot()); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/commands/turn_window.rs b/src-tauri/src/agent_sessions/event_pipeline/commands/turn_window.rs index a878f8d95..4cd8f8fd0 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/commands/turn_window.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/commands/turn_window.rs @@ -9,6 +9,7 @@ use std::collections::HashSet; use serde::{Deserialize, Serialize}; use tauri::{AppHandle, State}; +use crate::agent_sessions::event_pipeline::session_providers; use crate::agent_sessions::event_pipeline::types::{ ActivityStatus, EventDisplayStatus, EventDisplayVariant, EventSource, SessionEvent, }; @@ -178,6 +179,7 @@ pub async fn cache_load_session_turn_body( session_id: String, turn_id: String, ) -> Result { + session_providers::reject_bounded_replay_full_load(&session_id)?; let sid = session_id.clone(); let tid = turn_id.clone(); let window = @@ -203,6 +205,7 @@ pub(super) async fn load_initial_turn_window_events( session_id: &str, recent_turn_count: Option, ) -> Result { + session_providers::reject_bounded_replay_full_load(session_id)?; let sid = session_id.to_string(); let recent_count = recent_turn_count.unwrap_or(DEFAULT_RECENT_TURN_BODY_COUNT); let window = tokio::task::spawn_blocking(move || { @@ -283,6 +286,7 @@ pub async fn es_unload_turn_body( session_id: String, turn_id: String, ) -> Result { + session_providers::reject_bounded_replay_full_load(&session_id)?; let lookup_sid = session_id.clone(); let lookup_turn_id = turn_id.clone(); let persisted_turn = @@ -316,7 +320,7 @@ pub async fn es_unload_turn_body( #[cfg(test)] mod tests { - use super::normalize_turn_user_preview; + use super::{normalize_turn_user_preview, session_providers}; #[test] fn imported_user_alias_is_removed_from_placeholder_preview() { @@ -326,4 +330,11 @@ mod tests { "native hello" ); } + + #[test] + fn unload_turn_body_rpc_boundary_rejects_bounded_replay_sessions() { + assert!(session_providers::reject_bounded_replay_full_load("cliagent-managed").is_err()); + assert!(session_providers::reject_bounded_replay_full_load("codexapp-fixture").is_err()); + assert!(session_providers::reject_bounded_replay_full_load("sdeagent-native").is_ok()); + } } diff --git a/src-tauri/src/agent_sessions/event_pipeline/json_size.rs b/src-tauri/src/agent_sessions/event_pipeline/json_size.rs new file mode 100644 index 000000000..b41ab50bf --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/json_size.rs @@ -0,0 +1,56 @@ +//! Allocation-free JSON byte measurement for hot replay paths. +//! +//! Replay budgets need the exact serialized size, but they do not need to +//! retain the serialized bytes. Writing into this counter avoids repeatedly +//! allocating multi-megabyte temporary `Vec` buffers while a long external +//! transcript is opened or prepended. + +use std::io::{self, Write}; + +use serde::Serialize; + +#[derive(Default)] +struct CountingWriter { + bytes: usize, +} + +impl Write for CountingWriter { + fn write(&mut self, buffer: &[u8]) -> io::Result { + self.bytes = self + .bytes + .checked_add(buffer.len()) + .ok_or_else(|| io::Error::other("serialized JSON byte count overflow"))?; + Ok(buffer.len()) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } +} + +pub(crate) fn serialized_json_bytes( + value: &T, +) -> Result { + let mut writer = CountingWriter::default(); + serde_json::to_writer(&mut writer, value)?; + Ok(writer.bytes) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn matches_serde_json_vec_length() { + let value = serde_json::json!({ + "text": "bounded replay", + "values": [1, 2, 3], + "enabled": true, + }); + + assert_eq!( + serialized_json_bytes(&value).expect("measure JSON"), + serde_json::to_vec(&value).expect("serialize JSON").len() + ); + } +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/mod.rs b/src-tauri/src/agent_sessions/event_pipeline/mod.rs index 99b6ccd90..81a0e1f81 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/mod.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/mod.rs @@ -27,6 +27,7 @@ pub mod derived; pub mod extractors; pub mod history; pub mod ingestion; +pub(crate) mod json_size; pub mod pagination; pub mod payload_compaction; pub mod search; diff --git a/src-tauri/src/agent_sessions/event_pipeline/payload_compaction.rs b/src-tauri/src/agent_sessions/event_pipeline/payload_compaction.rs index 8571ca967..5fee5835d 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/payload_compaction.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/payload_compaction.rs @@ -100,6 +100,10 @@ fn compact_display_text(event: &mut SessionEvent, refs: &mut Vec) { preview: preview.clone(), full_size_bytes: event.display_text.len(), truncated: true, + replay_encoding: None, + replay_source_id: None, + replay_generation: None, + replay_source_event_id: None, }); event.display_text = preview; } @@ -141,6 +145,10 @@ fn compact_string_value( preview: preview.clone(), full_size_bytes: value.len(), truncated: true, + replay_encoding: None, + replay_source_id: None, + replay_generation: None, + replay_source_event_id: None, }); *value = preview; } diff --git a/src-tauri/src/agent_sessions/event_pipeline/session_manager.rs b/src-tauri/src/agent_sessions/event_pipeline/session_manager.rs index 61875c455..c7302c1ac 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/session_manager.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/session_manager.rs @@ -5,7 +5,8 @@ //! //! - tracks which session is currently *active* (default target when commands //! omit `session_id`), -//! - keeps a pin set so long-running agent sessions are never evicted, +//! - keeps a pin set for long-running agent ownership; rebuildable bounded +//! external stores may still be dropped under aggregate byte pressure, //! - maintains an LRU order of "idle" sessions and enforces max cache size. //! //! All event data lives in the per-session stores. This struct does not touch @@ -18,6 +19,9 @@ use std::collections::{HashMap, HashSet, VecDeque}; const MAX_CACHED_IDLE: usize = 15; /// Total cap across idle + pinned. const MAX_TOTAL_CACHED: usize = 25; +/// Byte budget for stores whose owners publish an estimate (bounded external +/// replay does; native SDE stores intentionally keep their existing policy). +const MAX_TRACKED_CACHED_BYTES: usize = 96 * 1024 * 1024; /// Metadata about a cached session. Events live in /// `EventStoreState::stores[session_id]`; this struct only tracks "when was @@ -25,6 +29,7 @@ const MAX_TOTAL_CACHED: usize = 25; #[derive(Debug, Clone)] struct SessionMeta { touched_at_ms: u64, + estimated_bytes: usize, } /// Session registry + LRU policy engine. @@ -32,7 +37,9 @@ pub struct SessionStoreManager { /// All known sessions (active + idle + pinned). Mirrors the key set of the /// outer stores `HashMap` — kept in sync by the `EventStoreState` helpers. known: HashMap, - /// Pinned sessions are never LRU-evicted (agent currently running). + /// Running-session ownership. Count-based LRU never evicts these entries. + /// Aggregate byte pressure may drop a rebuildable external store while + /// preserving this pin so the next write/reopen restores it as pinned. pinned: HashSet, /// FIFO of unpinned sessions in touched order (front = oldest). lru_order: VecDeque, @@ -84,6 +91,7 @@ impl SessionStoreManager { .and_modify(|m| m.touched_at_ms = now_ms()) .or_insert_with(|| SessionMeta { touched_at_ms: now_ms(), + estimated_bytes: 0, }); if !self.pinned.contains(session_id) && self.active_id.as_deref() != Some(session_id) @@ -93,6 +101,40 @@ impl SessionStoreManager { } } + /// Publish the current bounded-store footprint and enforce the aggregate + /// byte LRU. Native SDE callers never invoke this, so their cache behavior + /// remains unchanged. + pub fn update_estimated_bytes( + &mut self, + session_id: &str, + estimated_bytes: usize, + ) -> Vec { + self.register(session_id); + if let Some(meta) = self.known.get_mut(session_id) { + meta.estimated_bytes = estimated_bytes; + } + self.enforce_limits() + } + + /// Add a conservative upper bound for a generic bounded-replay mutation. + /// Same-ID upserts deliberately over-count; an exact compaction pass resets + /// the estimate when it reaches the per-store threshold. + pub fn add_estimated_bytes( + &mut self, + session_id: &str, + added_bytes: usize, + ) -> (usize, Vec) { + self.register(session_id); + let estimated_bytes = if let Some(meta) = self.known.get_mut(session_id) { + meta.estimated_bytes = meta.estimated_bytes.saturating_add(added_bytes); + meta.estimated_bytes + } else { + 0 + }; + let evicted = self.enforce_limits(); + (estimated_bytes, evicted) + } + /// Pin a session (agent started running). Pinned sessions skip LRU eviction. pub fn pin(&mut self, session_id: &str) { self.register(session_id); @@ -194,8 +236,41 @@ impl SessionStoreManager { break; } } + while self.tracked_bytes() > MAX_TRACKED_CACHED_BYTES { + // Only owners that explicitly published a positive estimate are + // rebuildable bounded-replay stores. Native SDE stores retain the + // legacy estimate of zero and must never become collateral damage + // of the external byte budget. A non-active running external store + // may be removed, but its pin remains as owner state so rebuilding + // it does not accidentally make it count-evictable. + let oldest_rebuildable = self + .known + .iter() + .filter(|(session_id, meta)| { + meta.estimated_bytes > 0 + && self.active_id.as_deref() != Some(session_id.as_str()) + }) + .min_by(|(left_id, left), (right_id, right)| { + left.touched_at_ms + .cmp(&right.touched_at_ms) + .then_with(|| left_id.cmp(right_id)) + }) + .map(|(session_id, _)| session_id.clone()); + let Some(oldest) = oldest_rebuildable else { + break; + }; + self.known.remove(&oldest); + self.remove_lru(&oldest); + evicted.push(oldest); + } evicted } + + fn tracked_bytes(&self) -> usize { + self.known.values().fold(0_usize, |total, meta| { + total.saturating_add(meta.estimated_bytes) + }) + } } fn now_ms() -> u64 { diff --git a/src-tauri/src/agent_sessions/event_pipeline/session_providers.rs b/src-tauri/src/agent_sessions/event_pipeline/session_providers.rs index b9219d122..e7ff3a371 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/session_providers.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/session_providers.rs @@ -1,7 +1,7 @@ use core_types::session::CLI_SESSION_PREFIX; use orgtrack_core::sources::cursor_ide::history::CURSORIDE_SESSION_PREFIX; +use orgtrack_core::sources::imported_history::replay::ImportedHistorySourceId; -use crate::agent_sessions::event_pipeline::types::SessionEvent; use crate::agent_sessions::external_cli_adapter; trait SessionProvider: Send + Sync { @@ -11,10 +11,6 @@ trait SessionProvider: Send + Sync { false } - fn load_history_events(&self, _session_id: &str) -> Result, String> { - Ok(Vec::new()) - } - fn subagent_prompt(&self, _child_session_id: &str) -> Option { None } @@ -42,13 +38,10 @@ impl SessionProvider for CursorIdeProvider { /// Managed CLI sessions (`cliagent-*`). Their transcript of record is never /// the `events` table: chunks-mode sessions replay `code_session_chunks`, /// native-mode sessions replay the CLI's own store via the imported-history -/// loaders (`cli_agent_chunks` routes both). Persisting the in-memory -/// EventStore for them only mirrors ephemeral turn state (frontend synthetic -/// user bubble, streamed placeholders, replay copies) into SQLite, where the -/// rows resurface as duplicate bubbles on the next merge. Live streaming is -/// unaffected — it renders from in-memory snapshots (`es:changed`), and -/// `es_load_from_cache` returning 0 rows simply leaves the chat to the -/// adapter's replay loader, which is the transcript of record. +/// bounded replay bridge. Persisting the in-memory EventStore for them only +/// mirrors ephemeral turn state into SQLite, where rows would resurface as +/// duplicate bubbles on the next merge. Live streaming still renders from +/// in-memory snapshots (`es:changed`). struct ManagedCliProvider; impl SessionProvider for ManagedCliProvider { @@ -63,6 +56,30 @@ impl SessionProvider for ManagedCliProvider { static PROVIDERS: &[&(dyn SessionProvider + Sync)] = &[&CursorIdeProvider, &ManagedCliProvider]; +const COLLABORATION_SNAPSHOT_SESSION_PREFIX: &str = "imported-session-"; + +/// Whether a public session id is backed by the bounded replay pipeline. +/// +/// Full-cache commands must fail closed for these ids. Their SQLite `events` +/// rows are either absent, a compact compatibility window, or an ORGII-owned +/// collaboration snapshot; none is permission to hydrate a whole transcript +/// through the native SDE cache path. +pub(crate) fn uses_bounded_replay(session_id: &str) -> bool { + session_id.starts_with(CLI_SESSION_PREFIX) + || (session_id.starts_with(COLLABORATION_SNAPSHOT_SESSION_PREFIX) + && session_id.len() > COLLABORATION_SNAPSHOT_SESSION_PREFIX.len()) + || ImportedHistorySourceId::from_session_id(session_id).is_some() +} + +pub(crate) fn reject_bounded_replay_full_load(session_id: &str) -> Result<(), String> { + if uses_bounded_replay(session_id) { + return Err(format!( + "Session {session_id} uses bounded external replay; full SQLite hydration is disabled" + )); + } + Ok(()) +} + pub(crate) fn skips_event_cache_save(session_id: &str) -> bool { external_cli_adapter::adapter_for_imported_session(session_id).is_some() || PROVIDERS.iter().any(|provider| { @@ -70,20 +87,6 @@ pub(crate) fn skips_event_cache_save(session_id: &str) -> bool { }) } -pub(crate) fn load_history_events(session_id: &str) -> Result, String> { - if let Some(adapter) = external_cli_adapter::adapter_for_imported_session(session_id) { - return adapter.load_history_events(session_id); - } - - let Some(provider) = PROVIDERS - .iter() - .find(|provider| provider.matches_session(session_id)) - else { - return Ok(Vec::new()); - }; - provider.load_history_events(session_id) -} - pub(crate) fn subagent_prompt(child_session_id: &str) -> Option { if let Some(adapter) = external_cli_adapter::adapter_for_imported_session(child_session_id) { return adapter.resolve_subagent_prompt(child_session_id); @@ -109,3 +112,26 @@ pub(crate) fn imported_parent_session_ids(parent_session_id: &str) -> Result, + ) { + self.set_with_hydration(events, HydrationMode::RoundWindow); + } + pub(super) fn set_with_hydration( &mut self, mut events: Vec, @@ -117,6 +131,25 @@ impl EventStore { self.sort_round_window_events_by_timeline(); } + /// Merge an imported/managed replay page without invalidating the existing + /// renderer baseline. + /// + /// External replay already publishes the page through the ordinary + /// non-streaming snapshot delta, which contains the complete replacement + /// order plus only the changed event bodies. Forcing a full derived + /// snapshot here would deep-clone the entire resident transcript on every + /// continuous-scroll window. Native SDE pagination keeps using + /// `merge_round_window_events` and its established full-baseline behavior. + pub fn merge_external_replay_window_events( + &mut self, + incoming: Vec, + ) { + let loaded_turn_ids = loaded_turn_ids_from_events(&incoming); + self.remove_turn_placeholders_for_turns(&loaded_turn_ids); + self.merge_events_with_hydration(incoming, false); + self.sort_external_replay_events_by_timeline(); + } + pub(super) fn merge_events_with_hydration( &mut self, incoming: Vec, @@ -293,6 +326,14 @@ impl EventStore { } pub(super) fn sort_round_window_events_by_timeline(&mut self) { + self.sort_external_replay_events_by_timeline(); + // Incremental streaming order patches describe only changed ids. + // Native round-window hydration historically publishes a full + // baseline after reordering; preserve that native SDE contract. + self.last_full_snapshot_version = 0; + } + + fn sort_external_replay_events_by_timeline(&mut self) { self.events.sort_by(|left, right| { left.created_at .cmp(&right.created_at) @@ -302,9 +343,5 @@ impl EventStore { .then_with(|| left.id.cmp(&right.id)) }); self.rebuild_indexes(); - // Incremental streaming order patches describe only changed ids. - // A round-window merge may move untouched historical ids as well, so - // the next notification must replace the normalized baseline once. - self.last_full_snapshot_version = 0; } } diff --git a/src-tauri/src/agent_sessions/event_pipeline/store/mod.rs b/src-tauri/src/agent_sessions/event_pipeline/store/mod.rs index d99eb386e..0abfd20db 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/store/mod.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/store/mod.rs @@ -16,6 +16,7 @@ use std::collections::{HashMap, HashSet}; +use crate::agent_sessions::event_pipeline::json_size::serialized_json_bytes; use crate::agent_sessions::event_pipeline::types::{ SessionEvent, ShellReplayState, ShellReplayStatus, }; @@ -556,6 +557,194 @@ impl EventStore { } } + /// External-replay-only byte cap. Repeated bounded older-turn reads must + /// not accumulate up to the generic 8,000-event limit (8,000 shell + /// previews alone can exceed 250 MiB). Native SDE paths never call this. + /// + /// The retained set is a contiguous newest suffix. When the byte cut + /// lands inside an older turn, advance to the next user row so the store + /// drops that partial turn instead of rendering a detached body. + pub fn cap_external_replay_bytes(&mut self, max_bytes: usize) -> Result { + let mut retained_bytes = 2_usize; // `[]` + let mut keep_from = self.events.len(); + for (index, event) in self.events.iter().enumerate().rev() { + let event_bytes = serialized_json_bytes(event) + .map_err(|error| format!("serialize external replay store event: {error}"))?; + let separator = usize::from(keep_from < self.events.len()); + if retained_bytes + .saturating_add(separator) + .saturating_add(event_bytes) + > max_bytes + { + break; + } + retained_bytes = retained_bytes + .saturating_add(separator) + .saturating_add(event_bytes); + keep_from = index; + } + + if keep_from > 0 && keep_from < self.events.len() { + if let Some(next_turn) = self.events[keep_from..].iter().position(|event| { + event.source == crate::agent_sessions::event_pipeline::types::EventSource::User + }) { + keep_from = keep_from.saturating_add(next_turn); + } + } + + if keep_from > 0 { + let removed_ids = self.events[..keep_from] + .iter() + .map(|event| event.id.clone()) + .collect::>(); + self.events.drain(..keep_from); + for event_id in removed_ids { + self.mark_removed(event_id); + } + self.rebuild_indexes(); + } + + serialized_json_bytes(&self.events) + .map_err(|error| format!("serialize capped external replay store: {error}")) + } + + /// Cap an older-page merge without immediately discarding the page that + /// the foreground reader just requested. + /// + /// The requested page is retained first. Up to half of the budget then + /// keeps its immediately newer resident neighbour so a prepend has a + /// stable scroll anchor. The remaining budget retains the ordinary newest + /// suffix. This policy is external-replay-only; native SDE stores never + /// call it. + pub fn cap_external_replay_bytes_preserving( + &mut self, + max_bytes: usize, + preserved_event_ids: &HashSet, + ) -> Result { + if preserved_event_ids.is_empty() { + return self.cap_external_replay_bytes(max_bytes); + } + + let event_bytes = self + .events + .iter() + .map(|event| { + serialized_json_bytes(event) + .map_err(|error| format!("serialize external replay store event: {error}")) + }) + .collect::, _>>()?; + let mut keep = vec![false; self.events.len()]; + let mut retained_bytes = 2_usize; // `[]` + let mut retained_count = 0_usize; + let mut last_preserved_index = None; + + for (index, (event, event_bytes)) in self.events.iter().zip(&event_bytes).enumerate() { + if !preserved_event_ids.contains(&event.id) { + continue; + } + let separator = usize::from(retained_count > 0); + let next_bytes = retained_bytes + .saturating_add(separator) + .saturating_add(*event_bytes); + if next_bytes > max_bytes { + return Err(format!( + "requested external replay window exceeds the {max_bytes} byte resident budget" + )); + } + keep[index] = true; + retained_bytes = next_bytes; + retained_count += 1; + last_preserved_index = Some(index); + } + + let Some(last_preserved_index) = last_preserved_index else { + return self.cap_external_replay_bytes(max_bytes); + }; + + // Keep the immediately newer resident range as the prepend anchor, but + // reserve at least half of the store for the newest live suffix. + let neighbourhood_budget = max_bytes / 2; + for (index, event_bytes) in event_bytes + .iter() + .enumerate() + .skip(last_preserved_index.saturating_add(1)) + { + if keep[index] { + continue; + } + let separator = usize::from(retained_count > 0); + let next_bytes = retained_bytes + .saturating_add(separator) + .saturating_add(*event_bytes); + if next_bytes > neighbourhood_budget { + break; + } + keep[index] = true; + retained_bytes = next_bytes; + retained_count += 1; + } + + // Fill the remainder with the same newest-first priority used by the + // ordinary external replay cap. + for (index, event_bytes) in event_bytes.iter().enumerate().rev() { + if keep[index] { + continue; + } + let separator = usize::from(retained_count > 0); + let next_bytes = retained_bytes + .saturating_add(separator) + .saturating_add(*event_bytes); + if next_bytes > max_bytes { + break; + } + keep[index] = true; + retained_bytes = next_bytes; + retained_count += 1; + } + + let mut index = 0_usize; + let mut removed_ids = Vec::new(); + self.events.retain(|event| { + let retained = keep[index]; + index += 1; + if !retained { + removed_ids.push(event.id.clone()); + } + retained + }); + for event_id in removed_ids { + self.mark_removed(event_id); + } + self.rebuild_indexes(); + + serialized_json_bytes(&self.events) + .map_err(|error| format!("serialize capped external replay store: {error}")) + } + + /// Keep the provider user row that owns the current visible replay body. + /// + /// Live polling follows the newest provider cursor even while the user is + /// reading an older random-access Round. Preserving the first resident + /// user row prevents the subsequent newest-suffix cap from detaching that + /// body. The remaining policy is identical to the ordinary preserving + /// cap, including the hard byte ceiling and newest-tail preference. + pub fn cap_external_replay_bytes_preserving_current_user_anchor( + &mut self, + max_bytes: usize, + ) -> Result { + let Some(anchor_id) = self + .events + .iter() + .find(|event| { + event.source == crate::agent_sessions::event_pipeline::types::EventSource::User + }) + .map(|event| event.id.clone()) + else { + return self.cap_external_replay_bytes(max_bytes); + }; + self.cap_external_replay_bytes_preserving(max_bytes, &HashSet::from([anchor_id])) + } + pub(super) fn stamp_repo(&self, event: &mut SessionEvent) { if event.repo_id.is_none() { event.repo_id = self.repo_id.clone(); diff --git a/src-tauri/src/agent_sessions/event_pipeline/tests/session_manager_tests.rs b/src-tauri/src/agent_sessions/event_pipeline/tests/session_manager_tests.rs index b71de584d..484d96aec 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/tests/session_manager_tests.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/tests/session_manager_tests.rs @@ -73,6 +73,21 @@ fn test_register_is_idempotent() { assert_eq!(mgr.known_count(), 1); } +#[test] +fn native_registration_keeps_the_existing_non_evicting_write_contract() { + let mut mgr = SessionStoreManager::new(); + + // Generic EventStore writes (including native SDE traffic) only register + // the store. External replay opts into enforcement later by publishing + // its bounded byte estimate; adding that policy must not silently change + // when native stores are evicted. + for index in 0..30 { + mgr.register(&format!("native-{index}")); + } + + assert_eq!(mgr.known_count(), 30); +} + #[test] fn test_evict_clears_all_state() { let mut mgr = SessionStoreManager::new(); @@ -133,3 +148,56 @@ fn test_enforce_limits_evicts_oldest_unpinned() { assert!(mgr.has_known("new-active")); assert!(mgr.idle_count() <= 15); } + +#[test] +fn tracked_external_bytes_evict_oldest_idle_store() { + let mut mgr = SessionStoreManager::new(); + let per_session = 25 * 1024 * 1024; + let mut evicted = Vec::new(); + for id in ["external-1", "external-2", "external-3", "external-4"] { + evicted.extend(mgr.update_estimated_bytes(id, per_session)); + } + assert_eq!(evicted, vec!["external-1".to_string()]); + assert!(!mgr.has_known("external-1")); + assert!(mgr.has_known("external-4")); +} + +#[test] +fn tracked_external_bytes_can_drop_pinned_rebuildable_store_but_keep_owner_pin() { + let mut mgr = SessionStoreManager::new(); + let per_session = 16 * 1024 * 1024; + + // Native SDE ownership is deliberately untracked by the external byte + // budget and must remain resident even if it is older than every replay. + mgr.register("sdeagent-native-running"); + mgr.pin("sdeagent-native-running"); + + mgr.set_active("cliagent-active"); + mgr.pin("cliagent-active"); + assert!(mgr + .update_estimated_bytes("cliagent-active", per_session) + .is_empty()); + + let mut evicted = Vec::new(); + for index in 1..=6 { + let session_id = format!("cliagent-pinned-{index}"); + mgr.pin(&session_id); + evicted.extend(mgr.update_estimated_bytes(&session_id, per_session)); + } + + assert_eq!(evicted, vec!["cliagent-pinned-1".to_string()]); + assert!(!mgr.has_known("cliagent-pinned-1")); + assert!( + mgr.is_pinned("cliagent-pinned-1"), + "running ownership survives cache eviction so a rebuild stays pinned" + ); + assert!(mgr.has_known("sdeagent-native-running")); + assert!(mgr.is_pinned("sdeagent-native-running")); + assert!(mgr.has_known("cliagent-active")); + + // A later event/reopen can materialize the evicted store without losing + // its running-owner pin semantics. + mgr.register("cliagent-pinned-1"); + assert!(mgr.has_known("cliagent-pinned-1")); + assert!(mgr.is_pinned("cliagent-pinned-1")); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests.rs b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests.rs index 51cbf7759..738024a41 100644 --- a/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests.rs +++ b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests.rs @@ -1,1540 +1,10 @@ -use crate::agent_sessions::event_pipeline::store::{capture_shell_replay_bookmarks, EventStore}; -use crate::agent_sessions::event_pipeline::types::*; - -fn make_event(id: &str, action_type: &str) -> SessionEvent { - SessionEvent { - id: id.to_string(), - chunk_id: Some(id.to_string()), - session_id: "test-session".to_string(), - created_at: "2026-01-01T00:00:00Z".to_string(), - function_name: "test".to_string(), - ui_canonical: "test".to_string(), - action_type: action_type.to_string(), - args: serde_json::json!({}), - result: serde_json::json!({}), - source: EventSource::Assistant, - display_text: "test".to_string(), - display_status: EventDisplayStatus::Completed, - display_variant: EventDisplayVariant::ToolCall, - activity_status: ActivityStatus::Agent, - thread_id: None, - process_id: None, - call_id: None, - file_path: None, - command: None, - is_delta: None, - repo_id: None, - repo_path: None, - extracted: None, - payload_refs: Vec::new(), - shell_replay: None, - shell_replay_bookmarks: None, - last_extract_at: None, - } -} - -fn make_tool_call(id: &str, call_id: &str) -> SessionEvent { - let mut event = make_event(id, "tool_call"); - event.call_id = Some(call_id.to_string()); - event.display_status = EventDisplayStatus::Running; - event.args = serde_json::json!({ "command": "ls", "streamOutput": "..." }); - event -} - -fn make_tool_result(id: &str, call_id: &str) -> SessionEvent { - let mut event = make_event(id, "tool_result"); - event.call_id = Some(call_id.to_string()); - event.result = serde_json::json!({ "content": "file1.txt\nfile2.txt" }); - event -} - -#[test] -fn test_set_replaces_all() { - let mut store = EventStore::new(); - store.set(vec![ - make_event("a", "tool_call"), - make_event("b", "message"), - ]); - assert_eq!(store.event_count(), 2); - assert_eq!(store.version(), 1); - - store.set(vec![make_event("c", "tool_call")]); - assert_eq!(store.event_count(), 1); - assert_eq!(store.version(), 2); - assert!(store.get_by_id("a").is_none()); - assert!(store.get_by_id("c").is_some()); -} - -#[test] -fn test_append_deduplicates() { - let mut store = EventStore::new(); - store.append(vec![ - make_event("a", "tool_call"), - make_event("b", "message"), - ]); - assert_eq!(store.event_count(), 2); - - store.append(vec![ - make_event("b", "message"), - make_event("c", "tool_call"), - ]); - assert_eq!(store.event_count(), 3); -} - -#[test] -fn test_delta_tracking_records_append_upsert_and_remove() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - store.mark_full_snapshot_emitted(); - - store.append(vec![make_event("b", "tool_call")]); - let (base_version, changed_ids, removed_ids) = store.take_delta_tracking(); - assert_eq!(base_version, 1); - assert_eq!(store.version(), 2); - assert_eq!(changed_ids, vec!["b".to_string()]); - assert!(removed_ids.is_empty()); - - let mut updated = make_event("a", "message"); - updated.display_text = "updated".to_string(); - store.upsert(updated); - assert_eq!(store.remove_by_id_prefix("b"), 1); - - let (base_version, mut changed_ids, removed_ids) = store.take_delta_tracking(); - changed_ids.sort(); - assert_eq!(base_version, 2); - assert_eq!(store.version(), 4); - assert_eq!(changed_ids, vec!["a".to_string()]); - assert_eq!(removed_ids, vec!["b".to_string()]); -} - -#[test] -fn test_update_by_id() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "tool_call")]); - - let patch = SessionEventPatch { - display_status: Some(EventDisplayStatus::Failed), - display_text: Some("error occurred".to_string()), - ..Default::default() - }; - assert!(store.update_by_id("a", &patch)); - assert_eq!( - store.get_by_id("a").unwrap().display_status, - EventDisplayStatus::Failed - ); - assert_eq!(store.get_by_id("a").unwrap().display_text, "error occurred"); - - assert!(!store.update_by_id("nonexistent", &patch)); -} - -#[test] -fn test_upsert() { - let mut store = EventStore::new(); - store.upsert(make_event("a", "tool_call")); - assert_eq!(store.event_count(), 1); - assert_eq!(store.version(), 1); - - let mut updated = make_event("a", "tool_call"); - updated.display_text = "updated text".to_string(); - store.upsert(updated); - assert_eq!(store.event_count(), 1); - assert_eq!(store.version(), 2); - assert_eq!(store.get_by_id("a").unwrap().display_text, "updated text"); - - store.upsert(make_event("b", "message")); - assert_eq!(store.event_count(), 2); -} - -#[test] -fn test_merge_tool_result() { - let mut store = EventStore::new(); - store.set(vec![make_tool_call("tc-1", "call-1")]); - assert_eq!( - store.get_by_id("tc-1").unwrap().display_status, - EventDisplayStatus::Running - ); - - store.merge_events(vec![make_tool_result("tr-1", "call-1")]); - - let merged = store.get_by_id("tc-1").unwrap(); - assert_eq!(merged.display_status, EventDisplayStatus::Completed); - assert_eq!(merged.activity_status, ActivityStatus::Processed); - assert_eq!(merged.result["content"], "file1.txt\nfile2.txt"); - // Args from original tool_call should be preserved (except streamOutput) - assert_eq!(merged.args["command"], "ls"); - assert!(merged.args.get("streamOutput").is_none()); - // tool_result should NOT appear as separate event - assert!(store.get_by_id("tr-1").is_none()); - assert_eq!(store.event_count(), 1); -} - -#[test] -fn test_merge_tool_result_preserves_background_shell_until_exact_exit_callback() { - let mut shell = make_tool_call("tc-background", "call-background"); - shell.function_name = "run_shell".to_string(); - shell.ui_canonical = core_types::tool_names::RUN_SHELL.to_string(); - shell.args = serde_json::json!({ - "command": "sleep 10", - "shellPid": 4242, - "shellProcessStatus": "background" - }); - let mut store = EventStore::new(); - store.set(vec![shell]); - - store.merge_events(vec![make_tool_result("tr-background", "call-background")]); - - let merged = store.get_by_id("tc-background").unwrap(); - assert_eq!(merged.display_status, EventDisplayStatus::Completed); - assert_eq!(merged.args["shellProcessStatus"], "background"); - assert_eq!(merged.args["shellPid"], 4242); -} - -#[test] -fn test_merge_tool_result_preserves_args_and_merges_metadata() { - let mut store = EventStore::new(); - - // Tool call with file_path in args - let mut tool_call = make_tool_call("tc-1", "call-1"); - tool_call.args = - serde_json::json!({ "path": "/src/main.rs", "command": "edit", "streamOutput": "..." }); - tool_call.file_path = Some("/src/main.rs".to_string()); - store.set(vec![tool_call]); - - // Tool result with additional metadata - let mut tool_result = make_tool_result("tr-1", "call-1"); - tool_result.args = serde_json::json!({ "execution_time": 150 }); // Extra metadata - tool_result.command = Some("git diff".to_string()); - store.merge_events(vec![tool_result]); - - let merged = store.get_by_id("tc-1").unwrap(); - // Original args preserved - assert_eq!(merged.args["path"], "/src/main.rs"); - assert_eq!(merged.args["command"], "edit"); - // Extra metadata from result merged in - assert_eq!(merged.args["execution_time"], 150); - // streamOutput removed - assert!(merged.args.get("streamOutput").is_none()); - // file_path preserved from original - assert_eq!(merged.file_path, Some("/src/main.rs".to_string())); - // command propagated from result (original was None) - assert_eq!(merged.command, Some("git diff".to_string())); -} - -#[test] -fn test_merge_updates_existing() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - - let mut updated = make_event("a", "message"); - updated.display_text = "new text".to_string(); - store.merge_events(vec![updated]); - - assert_eq!(store.event_count(), 1); - assert_eq!(store.get_by_id("a").unwrap().display_text, "new text"); -} - -#[test] -fn test_merge_appends_new() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - store.merge_events(vec![make_event("b", "tool_call")]); - assert_eq!(store.event_count(), 2); -} - -#[test] -fn test_cap_at_max_events() { - let mut store = EventStore::new(); - let events: Vec = (0..8010) - .map(|i| make_event(&format!("evt-{}", i), "message")) - .collect(); - store.set(events); - assert_eq!(store.event_count(), 8000); - // Oldest events should have been trimmed - assert!(store.get_by_id("evt-0").is_none()); - assert!(store.get_by_id("evt-10").is_some()); -} - -#[test] -fn test_clear() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - store.clear(); - assert_eq!(store.event_count(), 0); - assert!(store.get_by_id("a").is_none()); -} - -#[test] -fn test_streaming_flag() { - let mut store = EventStore::new(); - assert!(!store.is_streaming()); - store.set_streaming(true); - assert!(store.is_streaming()); - store.set_streaming(false); - assert!(!store.is_streaming()); -} - -// ============================================================================ -// Batch operation tests -// ============================================================================ - -fn make_running_event(id: &str) -> SessionEvent { - let mut event = make_event(id, "message"); - event.display_status = EventDisplayStatus::Running; - event -} - -fn make_task_tool_call(id: &str) -> SessionEvent { - let mut event = make_event(id, "tool_call"); - event.function_name = "task".to_string(); - event.display_status = EventDisplayStatus::Running; - event.args = serde_json::json!({ "description": "explore codebase" }); - event -} - -fn make_shell_tool_call(id: &str) -> SessionEvent { - let mut event = make_event(id, "tool_call"); - event.function_name = "run_shell".to_string(); - event.ui_canonical = "run_shell".to_string(); - event.call_id = Some(format!("call-{id}")); - event.display_status = EventDisplayStatus::Running; - event.args = serde_json::json!({ "command": "ls" }); - event -} - -#[test] -fn test_complete_last_running() { - let mut store = EventStore::new(); - store.set(vec![ - make_event("a", "message"), - make_running_event("b"), - make_running_event("c"), - ]); - let v_before = store.version(); - let result = store.complete_last_running(); - assert_eq!(result, Some("c".to_string())); - assert_eq!( - store.get_by_id("c").unwrap().display_status, - EventDisplayStatus::Completed - ); - assert_eq!( - store.get_by_id("b").unwrap().display_status, - EventDisplayStatus::Running - ); - assert!(store.version() > v_before); -} - -#[test] -fn test_complete_last_running_none() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - let v_before = store.version(); - let result = store.complete_last_running(); - assert!(result.is_none()); - assert_eq!(store.version(), v_before); -} - -fn make_awaiting_user_event(id: &str) -> SessionEvent { - let mut event = make_event(id, "tool_call"); - event.function_name = "ask_user_questions".to_string(); - event.display_status = EventDisplayStatus::AwaitingUser; - event -} - -#[test] -fn test_complete_last_running_skips_awaiting_user() { - // Regression: AskQuestionCard used to disappear because `agent:complete` - // for the surrounding turn called `complete_last_running`, which flipped - // the blocking `ask_user_questions` tool_call to Completed. - // - // With the AwaitingUser phase, only explicit `interaction_finalized` - // (via `merge_events`) is allowed to complete it. - let mut store = EventStore::new(); - store.set(vec![ - make_event("msg", "message"), - make_awaiting_user_event("tool-call-ask"), - ]); - let v_before = store.version(); - let result = store.complete_last_running(); - assert!( - result.is_none(), - "AwaitingUser event must not be treated as Running" - ); - assert_eq!( - store.get_by_id("tool-call-ask").unwrap().display_status, - EventDisplayStatus::AwaitingUser - ); - assert_eq!( - store.version(), - v_before, - "version must not bump when nothing changes" - ); -} - -#[test] -fn test_complete_last_running_picks_running_past_awaiting_user() { - // If a real Running event exists before the AwaitingUser one in insertion - // order (AwaitingUser inserted LAST), `complete_last_running` should skip - // AwaitingUser and land on the Running event behind it. - let mut store = EventStore::new(); - store.set(vec![ - make_running_event("running-thinking"), - make_awaiting_user_event("tool-call-ask"), - ]); - let result = store.complete_last_running(); - assert_eq!(result, Some("running-thinking".to_string())); - assert_eq!( - store.get_by_id("running-thinking").unwrap().display_status, - EventDisplayStatus::Completed - ); - assert_eq!( - store.get_by_id("tool-call-ask").unwrap().display_status, - EventDisplayStatus::AwaitingUser, - "AwaitingUser must remain untouched" - ); -} - -#[test] -fn test_merge_events_transitions_awaiting_user_to_completed() { - // The `interaction_finalized` path emits a tool_result that merges into - // the AwaitingUser tool_call; that merge is the sole legitimate way to - // transition into Completed. - let mut store = EventStore::new(); - let mut call = make_awaiting_user_event("tool-call-ask"); - call.call_id = Some("ask-123".to_string()); - store.set(vec![call]); - - let mut result_event = make_event("tool-result-ask", "tool_result"); - result_event.call_id = Some("ask-123".to_string()); - result_event.result = serde_json::json!({ "answers": ["use_redis"], "status": "answered" }); - - store.merge_events(vec![result_event]); - - let completed = store.get_by_id("tool-call-ask").unwrap(); - assert_eq!( - completed.display_status, - EventDisplayStatus::Completed, - "interaction_finalized must flip AwaitingUser → Completed" - ); -} - -#[test] -fn test_patch_by_ids() { - let mut store = EventStore::new(); - store.set(vec![ - make_running_event("a"), - make_running_event("b"), - make_running_event("c"), - ]); - let patch = SessionEventPatch { - display_status: Some(EventDisplayStatus::Completed), - is_delta: Some(false), - ..Default::default() - }; - let count = store.patch_by_ids(&["a".to_string(), "c".to_string()], &patch); - assert_eq!(count, 2); - assert_eq!( - store.get_by_id("a").unwrap().display_status, - EventDisplayStatus::Completed - ); - assert_eq!( - store.get_by_id("b").unwrap().display_status, - EventDisplayStatus::Running - ); - assert_eq!( - store.get_by_id("c").unwrap().display_status, - EventDisplayStatus::Completed - ); -} - -#[test] -fn test_patch_by_ids_with_missing() { - let mut store = EventStore::new(); - store.set(vec![make_running_event("a")]); - let patch = SessionEventPatch { - display_status: Some(EventDisplayStatus::Completed), - ..Default::default() - }; - let count = store.patch_by_ids(&["a".to_string(), "nonexistent".to_string()], &patch); - assert_eq!(count, 1); -} - -#[test] -fn test_remove_by_id_prefix() { - let mut store = EventStore::new(); - store.set(vec![ - make_event("stream-msg-1", "message"), - make_event("stream-msg-2", "message"), - make_event("normal-1", "tool_call"), - make_event("stream-think-1", "message"), - ]); - let removed = store.remove_by_id_prefix("stream-msg-"); - assert_eq!(removed, 2); - assert_eq!(store.event_count(), 2); - assert!(store.get_by_id("stream-msg-1").is_none()); - assert!(store.get_by_id("normal-1").is_some()); - assert!(store.get_by_id("stream-think-1").is_some()); -} - -#[test] -fn test_remove_by_id_prefix_no_match() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - let v_before = store.version(); - let removed = store.remove_by_id_prefix("nonexistent-"); - assert_eq!(removed, 0); - assert_eq!(store.version(), v_before); -} - -#[test] -fn test_remove_by_ids() { - let mut store = EventStore::new(); - store.set(vec![ - make_event("stream-msg-1", "message"), - make_event("normal-1", "tool_call"), - make_event("stream-think-1", "message"), - ]); - let removed = store.remove_by_ids(&[ - "stream-msg-1".to_string(), - "stream-think-1".to_string(), - "nonexistent".to_string(), - ]); - assert_eq!(removed, 2); - assert!(store.get_by_id("stream-msg-1").is_none()); - assert!(store.get_by_id("stream-think-1").is_none()); - assert!(store.get_by_id("normal-1").is_some()); - - // Removed ids surface in delta tracking so `es:changed` subscribers drop them. - let (_, _, removed_ids) = store.take_delta_tracking(); - assert!(removed_ids.contains(&"stream-msg-1".to_string())); - assert!(removed_ids.contains(&"stream-think-1".to_string())); -} - -#[test] -fn test_remove_by_ids_no_match_keeps_version() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - let v_before = store.version(); - let removed = store.remove_by_ids(&["nonexistent".to_string()]); - assert_eq!(removed, 0); - assert_eq!(store.version(), v_before); -} - -#[test] -fn test_remove_synthetic_user_inputs_keeps_backend_user_input_ids() { - let mut store = EventStore::new(); - let mut synthetic = make_event("user-input-synthetic", "raw"); - synthetic.source = EventSource::User; - synthetic.function_name = "user_message".to_string(); - synthetic.ui_canonical = "user_message".to_string(); - synthetic.result = serde_json::json!({ "syntheticUserInput": true }); - synthetic.chunk_id = None; - - let mut backend = make_event("user-input-cliagent-real", "raw"); - backend.source = EventSource::User; - backend.function_name = "user_message".to_string(); - backend.ui_canonical = "user_message".to_string(); - backend.display_text = "authoritative different text".to_string(); - - store.set(vec![synthetic, backend]); - let removed = store.remove_synthetic_user_inputs(); - - assert_eq!(removed, 1); - assert!(store.get_by_id("user-input-synthetic").is_none()); - assert!(store.get_by_id("user-input-cliagent-real").is_some()); -} - -#[test] -fn test_merge_authoritative_user_message_evicts_matching_synthetic_placeholder() { - let mut store = EventStore::new(); - let mut synthetic = make_event("user-input-synthetic", "raw"); - synthetic.source = EventSource::User; - synthetic.function_name = "user_message".to_string(); - synthetic.ui_canonical = "user_message".to_string(); - synthetic.result = serde_json::json!({ "syntheticUserInput": true }); - synthetic.chunk_id = None; - synthetic.display_text = "hello from user".to_string(); - - store.append(vec![synthetic]); - assert!(store.get_by_id("user-input-synthetic").is_some()); - - let mut backend = make_event("user-input-cliagent-real", "raw"); - backend.source = EventSource::User; - backend.function_name = "user".to_string(); - backend.ui_canonical = "user".to_string(); - backend.display_text = "hello from user".to_string(); - - store.merge_events(vec![backend]); - - assert!(store.get_by_id("user-input-synthetic").is_none()); - assert!(store.get_by_id("user-input-cliagent-real").is_some()); -} - -#[test] -fn test_set_reconciles_persisted_matching_synthetic_placeholder() { - let mut store = EventStore::new(); - let mut synthetic = make_event("user-input-synthetic", "raw"); - synthetic.source = EventSource::User; - synthetic.function_name = "user_message".to_string(); - synthetic.result = serde_json::json!({ "syntheticUserInput": true }); - synthetic.display_text = "persisted duplicate".to_string(); - - let mut backend = make_event("user-input-real", "raw"); - backend.source = EventSource::User; - backend.function_name = "provider_specific_user_event".to_string(); - backend.display_text = "persisted duplicate".to_string(); - - store.set(vec![synthetic, backend]); - - assert!(store.get_by_id("user-input-synthetic").is_none()); - assert!(store.get_by_id("user-input-real").is_some()); -} - -#[test] -fn test_merge_authoritative_message_keeps_legitimate_repeated_user_text() { - let mut store = EventStore::new(); - let mut first = make_event("user-input-first", "raw"); - first.source = EventSource::User; - first.function_name = "user".to_string(); - first.ui_canonical = "user".to_string(); - first.display_text = "repeat me".to_string(); - - let mut second = make_event("user-input-second", "raw"); - second.source = EventSource::User; - second.function_name = "user".to_string(); - second.ui_canonical = "user".to_string(); - second.display_text = "repeat me".to_string(); - - store.append(vec![first]); - store.merge_events(vec![second]); - - assert!(store.get_by_id("user-input-first").is_some()); - assert!(store.get_by_id("user-input-second").is_some()); -} - -#[test] -fn test_merge_authoritative_message_keeps_non_matching_synthetic_text() { - let mut store = EventStore::new(); - let mut synthetic = make_event("user-input-synthetic", "raw"); - synthetic.source = EventSource::User; - synthetic.function_name = "user_message".to_string(); - synthetic.ui_canonical = "user_message".to_string(); - synthetic.result = serde_json::json!({ "syntheticUserInput": true }); - synthetic.display_text = "different pending text".to_string(); - - let mut backend = make_event("user-input-real", "raw"); - backend.source = EventSource::User; - backend.function_name = "provider_specific_user_event".to_string(); - backend.ui_canonical = "user".to_string(); - backend.display_text = "authoritative text".to_string(); - - store.append(vec![synthetic]); - store.merge_events(vec![backend]); - - assert!(store.get_by_id("user-input-synthetic").is_some()); - assert!(store.get_by_id("user-input-real").is_some()); -} - -#[test] -fn test_replace_and_remove() { - let mut store = EventStore::new(); - let mut placeholder = make_event("stream-1", "message"); - placeholder.created_at = "2026-05-22T06:48:20.100Z".to_string(); - let mut normal_event = make_event("normal-1", "tool_call"); - normal_event.created_at = "2026-05-22T06:48:21.000Z".to_string(); - store.set(vec![placeholder, normal_event]); - let mut new_event = make_event("final-1", "message"); - new_event.created_at = "2026-05-22T06:48:30.000Z".to_string(); - - store.replace_and_remove(Some("stream-1"), new_event); - - assert!(store.get_by_id("stream-1").is_none()); - assert!(store.get_by_id("final-1").is_some()); - assert!(store.get_by_id("normal-1").is_some()); - assert_eq!(store.events()[0].id, "final-1"); - assert_eq!(store.events()[0].created_at, "2026-05-22T06:48:20.100Z"); - assert_eq!(store.events()[1].id, "normal-1"); -} - -#[test] -fn test_replace_and_remove_removes_placeholder_when_final_already_exists() { - let mut store = EventStore::new(); - let mut placeholder = make_event("stream-think-ts-1", "message"); - placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); - let mut existing_final = make_event("stream-think-1-final", "message"); - existing_final.created_at = "2026-05-22T07:18:22.000Z".to_string(); - let normal_event = make_event("normal-1", "tool_call"); - store.set(vec![placeholder, existing_final, normal_event]); - - let mut new_event = make_event("stream-think-1-final", "message"); - new_event.created_at = "2026-05-22T07:18:30.000Z".to_string(); - store.replace_and_remove(Some("stream-think-ts-1"), new_event); - - assert!(store.get_by_id("stream-think-ts-1").is_none()); - assert!(store.get_by_id("stream-think-1-final").is_some()); - assert_eq!( - store - .events() - .iter() - .filter(|event| event.id == "stream-think-1-final") - .count(), - 1 - ); - assert_eq!(store.events()[0].id, "stream-think-1-final"); - assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); - assert_eq!(store.events()[1].id, "normal-1"); -} - -#[test] -fn test_replace_and_remove_removes_tail_placeholder_when_final_already_exists() { - let mut store = EventStore::new(); - let mut existing_final = make_event("stream-think-1-final", "message"); - existing_final.created_at = "2026-05-22T07:18:22.000Z".to_string(); - let normal_event = make_event("normal-1", "tool_call"); - let mut placeholder = make_event("stream-think-ts-1", "message"); - placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); - store.set(vec![existing_final, normal_event, placeholder]); - - let mut new_event = make_event("stream-think-1-final", "message"); - new_event.created_at = "2026-05-22T07:18:30.000Z".to_string(); - store.replace_and_remove(Some("stream-think-ts-1"), new_event); - - assert!(store.get_by_id("stream-think-ts-1").is_none()); - assert!(store.get_by_id("stream-think-1-final").is_some()); - assert_eq!(store.event_count(), 2); - assert_eq!(store.events()[0].id, "stream-think-1-final"); - assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); - assert_eq!(store.events()[1].id, "normal-1"); -} - -#[test] -fn test_replace_and_remove_no_remove() { - let mut store = EventStore::new(); - store.set(vec![make_event("a", "message")]); - let new_event = make_event("b", "message"); - store.replace_and_remove(None, new_event); - assert_eq!(store.event_count(), 2); -} - -#[test] -fn test_authoritative_stream_upsert_replaces_matching_ts_placeholder() { - let mut store = EventStore::new(); - let mut placeholder = make_event("stream-think-ts-test-session-100", "llm_thinking"); - placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); - placeholder.display_text = "same thought".to_string(); - let normal_event = make_event("normal-1", "tool_call"); - store.set(vec![placeholder, normal_event]); - - let mut authoritative = make_event("stream-think-test-session-1-final", "llm_thinking"); - authoritative.created_at = "2026-05-22T07:18:30.000Z".to_string(); - authoritative.display_text = "same thought".to_string(); - store.upsert(authoritative); - - assert!(store - .get_by_id("stream-think-ts-test-session-100") - .is_none()); - assert!(store - .get_by_id("stream-think-test-session-1-final") - .is_some()); - assert_eq!(store.event_count(), 2); - assert_eq!(store.events()[0].id, "stream-think-test-session-1-final"); - assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); - assert_eq!(store.events()[1].id, "normal-1"); -} - -#[test] -fn test_authoritative_stream_upsert_removes_placeholder_when_final_already_exists() { - let mut store = EventStore::new(); - let mut placeholder = make_event("stream-think-ts-test-session-100", "llm_thinking"); - placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); - placeholder.display_text = "same thought".to_string(); - let mut existing_final = make_event("stream-think-test-session-1-final", "llm_thinking"); - existing_final.created_at = "2026-05-22T07:18:22.000Z".to_string(); - existing_final.display_text = "same thought".to_string(); - let normal_event = make_event("normal-1", "tool_call"); - store.set(vec![placeholder, existing_final, normal_event]); - - let mut authoritative = make_event("stream-think-test-session-1-final", "llm_thinking"); - authoritative.created_at = "2026-05-22T07:18:30.000Z".to_string(); - authoritative.display_text = "same thought".to_string(); - store.upsert(authoritative); - - assert!(store - .get_by_id("stream-think-ts-test-session-100") - .is_none()); - assert!(store - .get_by_id("stream-think-test-session-1-final") - .is_some()); - assert_eq!( - store - .events() - .iter() - .filter(|event| event.id == "stream-think-test-session-1-final") - .count(), - 1 - ); - assert_eq!(store.event_count(), 2); - assert_eq!(store.events()[0].id, "stream-think-test-session-1-final"); - assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); - assert_eq!(store.events()[1].id, "normal-1"); -} - -#[test] -fn test_authoritative_thinking_upsert_replaces_duplicate_in_current_turn() { - let mut store = EventStore::new(); - let mut user = make_event("user-1", "user_message"); - user.source = EventSource::User; - user.display_variant = EventDisplayVariant::Message; - user.display_text = "first prompt".to_string(); - - let mut first = make_event("stream-think-session-1", "llm_thinking"); - first.display_variant = EventDisplayVariant::Thinking; - first.display_text = "same thought".to_string(); - first.created_at = "2026-05-22T07:18:20.100Z".to_string(); - - store.set(vec![user, first]); - - let mut duplicate = make_event("stream-think-session-2", "llm_thinking"); - duplicate.display_variant = EventDisplayVariant::Thinking; - duplicate.display_text = "same thought".to_string(); - duplicate.created_at = "2026-05-22T07:18:30.000Z".to_string(); - - store.upsert(duplicate); - - assert!(store.get_by_id("stream-think-session-1").is_none()); - assert!(store.get_by_id("stream-think-session-2").is_some()); - assert_eq!(store.event_count(), 2); - assert_eq!(store.events()[1].id, "stream-think-session-2"); - assert_eq!(store.events()[1].created_at, "2026-05-22T07:18:20.100Z"); -} - -#[test] -fn test_authoritative_thinking_upsert_preserves_same_text_across_turns() { - let mut store = EventStore::new(); - let mut user_one = make_event("user-1", "user_message"); - user_one.source = EventSource::User; - user_one.display_variant = EventDisplayVariant::Message; - user_one.display_text = "first prompt".to_string(); - - let mut first = make_event("stream-think-session-1", "llm_thinking"); - first.display_variant = EventDisplayVariant::Thinking; - first.display_text = "same thought".to_string(); - - let mut user_two = make_event("user-2", "user_message"); - user_two.source = EventSource::User; - user_two.display_variant = EventDisplayVariant::Message; - user_two.display_text = "second prompt".to_string(); - - store.set(vec![user_one, first, user_two]); - - let mut repeated_next_turn = make_event("stream-think-session-2", "llm_thinking"); - repeated_next_turn.display_variant = EventDisplayVariant::Thinking; - repeated_next_turn.display_text = "same thought".to_string(); - - store.upsert(repeated_next_turn); - - assert!(store.get_by_id("stream-think-session-1").is_some()); - assert!(store.get_by_id("stream-think-session-2").is_some()); - assert_eq!(store.event_count(), 4); -} - -#[test] -fn test_authoritative_message_upsert_replaces_duplicate_in_current_turn() { - let mut store = EventStore::new(); - let mut user = make_event("user-1", "user_message"); - user.source = EventSource::User; - user.display_variant = EventDisplayVariant::Message; - user.display_text = "first prompt".to_string(); - - let mut first = make_event("stream-msg-session-1", "message"); - first.display_variant = EventDisplayVariant::Message; - first.display_text = "same assistant note".to_string(); - first.created_at = "2026-05-22T07:18:20.100Z".to_string(); - - store.set(vec![user, first]); - - let mut duplicate = make_event("stream-msg-session-2", "message"); - duplicate.display_variant = EventDisplayVariant::Message; - duplicate.display_text = "same assistant note".to_string(); - duplicate.created_at = "2026-05-22T07:18:30.000Z".to_string(); - - store.upsert(duplicate); - - assert!(store.get_by_id("stream-msg-session-1").is_none()); - assert!(store.get_by_id("stream-msg-session-2").is_some()); - assert_eq!(store.event_count(), 2); - assert_eq!(store.events()[1].id, "stream-msg-session-2"); - assert_eq!(store.events()[1].created_at, "2026-05-22T07:18:20.100Z"); -} - -#[test] -fn test_authoritative_message_upsert_preserves_same_text_across_turns() { - let mut store = EventStore::new(); - let mut user_one = make_event("user-1", "user_message"); - user_one.source = EventSource::User; - user_one.display_variant = EventDisplayVariant::Message; - user_one.display_text = "first prompt".to_string(); - - let mut first = make_event("stream-msg-session-1", "message"); - first.display_variant = EventDisplayVariant::Message; - first.display_text = "same assistant note".to_string(); - - let mut user_two = make_event("user-2", "user_message"); - user_two.source = EventSource::User; - user_two.display_variant = EventDisplayVariant::Message; - user_two.display_text = "second prompt".to_string(); - - store.set(vec![user_one, first, user_two]); - - let mut repeated_next_turn = make_event("stream-msg-session-2", "message"); - repeated_next_turn.display_variant = EventDisplayVariant::Message; - repeated_next_turn.display_text = "same assistant note".to_string(); - - store.upsert(repeated_next_turn); - - assert!(store.get_by_id("stream-msg-session-1").is_some()); - assert!(store.get_by_id("stream-msg-session-2").is_some()); - assert_eq!(store.event_count(), 4); -} - -#[test] -fn test_update_spawning_tool_args() { - let mut store = EventStore::new(); - store.set(vec![ - make_event("msg-1", "message"), - make_task_tool_call("task-1"), - ]); - let task_names = &["task"]; - let result = store.update_spawning_tool_args( - task_names, - serde_json::json!({ - "reasoningText": "analyzing code...", - "subActivities": [{"tool": "read", "args": {}}] - }), - ); - assert_eq!(result, Some("task-1".to_string())); - let task = store.get_by_id("task-1").unwrap(); - assert_eq!(task.args["reasoningText"], "analyzing code..."); - assert_eq!(task.args["description"], "explore codebase"); -} - -#[test] -fn test_update_spawning_tool_args_none() { - let mut store = EventStore::new(); - store.set(vec![make_event("msg-1", "message")]); - let task_names = &["task"]; - let result = store.update_spawning_tool_args(task_names, serde_json::json!({"key": "value"})); - assert!(result.is_none()); -} - -#[test] -fn test_update_spawning_tool_args_multi_names() { - let mut store = EventStore::new(); - let mut session_call = make_event("session-1", "tool_call"); - session_call.function_name = "session".to_string(); - session_call.display_status = EventDisplayStatus::Running; - session_call.args = serde_json::json!({ "desc": "test" }); - store.set(vec![make_event("msg-1", "message"), session_call]); - - let names = &["task", "session", "spawn"]; - let result = store.update_spawning_tool_args(names, serde_json::json!({"subActivities": []})); - assert_eq!(result, Some("session-1".to_string())); - let updated = store.get_by_id("session-1").unwrap(); - assert_eq!(updated.args["desc"], "test"); - assert!(updated.args["subActivities"].is_array()); -} - -fn replay_state( - call_id: &str, - sequence: u64, - visible_bytes: u64, - status: ShellReplayStatus, -) -> ShellReplayState { - ShellReplayState { - replay_ref: ShellReplayRef { - session_id: "test-session".to_string(), - call_id: call_id.to_string(), - format_version: 1, - }, - bookmark: ShellReplayBookmark { - visible_through_sequence: sequence, - visible_bytes, - }, - terminal_preview: format!("preview-{sequence}"), - status, - error: None, - completed_at: None, - } -} - -#[test] -fn test_shell_replay_exact_update_is_monotonic_and_seed_is_immutable() { - let mut shell = make_shell_tool_call("shell-1"); - shell.shell_replay_bookmarks = Some(Default::default()); - let mut sibling = shell.clone(); - sibling.id = "shell-1-sibling".to_string(); - let mut store = EventStore::new(); - store.set(vec![shell, sibling]); - - let initial = replay_state("call-shell-1", 1, 100, ShellReplayStatus::Running); - assert_eq!( - store.update_shell_replay_by_call_id("call-shell-1", initial.clone(), true), - Some("shell-1-sibling".to_string()) - ); - let latest = replay_state("call-shell-1", 2, 200, ShellReplayStatus::Running); - store.update_shell_replay_by_call_id("call-shell-1", latest.clone(), false); - store.update_shell_replay_by_call_id("call-shell-1", initial.clone(), true); - - for id in ["shell-1", "shell-1-sibling"] { - let event = store.get_by_id(id).unwrap(); - assert_eq!(event.shell_replay, Some(latest.clone())); - assert_eq!( - event - .shell_replay_bookmarks - .as_ref() - .and_then(|bookmarks| bookmarks.get("call-shell-1")), - Some(&initial) - ); - } -} - -#[test] -fn test_shell_replay_terminal_state_cannot_regress_to_running() { - let mut shell = make_shell_tool_call("shell-1"); - shell.shell_replay_bookmarks = Some(Default::default()); - let mut store = EventStore::new(); - store.set(vec![shell]); - - let mut complete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Complete); - complete.completed_at = Some("2026-01-01T00:01:00Z".to_string()); - store.update_shell_replay_by_call_id("call-shell-1", complete.clone(), true); - store.update_shell_replay_by_call_id( - "call-shell-1", - replay_state("call-shell-1", 4, 400, ShellReplayStatus::Running), - false, - ); - - assert_eq!( - store.get_by_id("shell-1").unwrap().shell_replay, - Some(complete) - ); -} - -#[test] -fn test_shell_replay_complete_can_be_corrected_to_incomplete() { - let mut shell = make_shell_tool_call("shell-1"); - shell.shell_replay_bookmarks = Some(Default::default()); - let mut store = EventStore::new(); - store.set(vec![shell]); - - let mut complete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Complete); - complete.completed_at = Some("2026-01-01T00:01:00Z".to_string()); - store.update_shell_replay_by_call_id("call-shell-1", complete, true); - - let mut incomplete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Incomplete); - incomplete.error = Some("final persistence barrier failed".to_string()); - store.update_shell_replay_by_call_id("call-shell-1", incomplete.clone(), false); - - assert_eq!( - store.get_by_id("shell-1").unwrap().shell_replay, - Some(incomplete) - ); -} - -#[test] -fn test_shell_replay_incomplete_can_correct_an_optimistic_higher_watermark() { - let mut shell = make_shell_tool_call("shell-1"); - shell.shell_replay_bookmarks = Some(Default::default()); - let mut store = EventStore::new(); - store.set(vec![shell]); - - store.update_shell_replay_by_call_id( - "call-shell-1", - replay_state("call-shell-1", 9, 900, ShellReplayStatus::Complete), - true, - ); - let mut recovered = replay_state("call-shell-1", 8, 800, ShellReplayStatus::Incomplete); - recovered.error = Some("torn final frame removed during recovery".to_string()); - store.update_shell_replay_by_call_id("call-shell-1", recovered.clone(), false); - - assert_eq!( - store.get_by_id("shell-1").unwrap().shell_replay, - Some(recovered) - ); -} - -#[test] -fn test_shell_replay_incomplete_cannot_be_overwritten_by_complete() { - let mut shell = make_shell_tool_call("shell-1"); - shell.shell_replay_bookmarks = Some(Default::default()); - let mut store = EventStore::new(); - store.set(vec![shell]); - - let mut incomplete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Incomplete); - incomplete.error = Some("disk full".to_string()); - store.update_shell_replay_by_call_id("call-shell-1", incomplete.clone(), true); - store.update_shell_replay_by_call_id( - "call-shell-1", - replay_state("call-shell-1", 4, 400, ShellReplayStatus::Complete), - false, - ); - - assert_eq!( - store.get_by_id("shell-1").unwrap().shell_replay, - Some(incomplete) - ); -} - -#[test] -fn test_shell_replay_update_requires_exact_session_and_call() { - let shell = make_shell_tool_call("shell-1"); - let mut store = EventStore::new(); - store.set(vec![shell]); - - assert!(store - .update_shell_replay_by_call_id( - "different-call", - replay_state("call-shell-1", 1, 100, ShellReplayStatus::Running), - true, - ) - .is_none()); - - let mut wrong_session = replay_state("call-shell-1", 1, 100, ShellReplayStatus::Running); - wrong_session.replay_ref.session_id = "different-session".to_string(); - assert!(store - .update_shell_replay_by_call_id("call-shell-1", wrong_session, true) - .is_none()); - assert!(store.get_by_id("shell-1").unwrap().shell_replay.is_none()); -} - -#[test] -fn test_same_id_upsert_preserves_first_insert_bookmarks() { - let initial = replay_state("other-call", 5, 500, ShellReplayStatus::Running); - let mut first = make_event("timeline-1", "message"); - first.shell_replay_bookmarks = Some(std::collections::HashMap::from([( - "other-call".to_string(), - initial.clone(), - )])); - let mut store = EventStore::new(); - store.set(vec![first]); - - let future = replay_state("other-call", 99, 9_900, ShellReplayStatus::Complete); - let mut update = make_event("timeline-1", "message"); - update.display_text = "updated".to_string(); - update.shell_replay_bookmarks = Some(std::collections::HashMap::from([( - "other-call".to_string(), - future, - )])); - store.upsert(update); - - let mut merge_update = make_event("timeline-1", "message"); - merge_update.display_text = "merged".to_string(); - merge_update.shell_replay_bookmarks = Some(std::collections::HashMap::from([( - "other-call".to_string(), - replay_state("other-call", 100, 10_000, ShellReplayStatus::Complete), - )])); - store.merge_events(vec![merge_update]); - - assert_eq!( - store - .get_by_id("timeline-1") - .unwrap() - .shell_replay_bookmarks - .as_ref() - .and_then(|bookmarks| bookmarks.get("other-call")), - Some(&initial) - ); -} - -#[test] -fn test_first_insert_bookmark_winner_fills_only_missing_active_calls() { - let first = replay_state("call-a", 1, 100, ShellReplayStatus::Running); - let future = replay_state("call-a", 9, 900, ShellReplayStatus::Running); - let active_b = replay_state("call-b", 2, 200, ShellReplayStatus::Running); - let mut event = make_event("timeline-1", "message"); - event.shell_replay_bookmarks = Some(std::collections::HashMap::from([( - "call-a".to_string(), - first.clone(), - )])); - - capture_shell_replay_bookmarks( - &mut event, - &std::collections::HashMap::from([ - ("call-a".to_string(), future), - ("call-b".to_string(), active_b.clone()), - ]), - ); - - let bookmarks = event.shell_replay_bookmarks.unwrap(); - assert_eq!(bookmarks.get("call-a"), Some(&first)); - assert_eq!(bookmarks.get("call-b"), Some(&active_b)); -} - -#[test] -fn test_live_shell_event_keeps_only_bounded_replay_payload() { - let mut shell = make_shell_tool_call("shell-1"); - shell.args["streamOutput"] = serde_json::Value::String("duplicate".repeat(20_000)); - shell.result = serde_json::json!({ - "content": "duplicate".repeat(20_000), - "observation": "duplicate".repeat(20_000) - }); - let mut state = replay_state("call-shell-1", 1, 80_000, ShellReplayStatus::Running); - state.terminal_preview = "中".repeat(20_000); - shell.shell_replay = Some(state); - let mut store = EventStore::new(); - store.upsert(shell); - - let stored = store.get_by_id("shell-1").unwrap(); - assert!(stored.args.get("streamOutput").is_none()); - assert_eq!(stored.result, serde_json::json!({})); - assert!(stored.shell_replay.as_ref().unwrap().terminal_preview.len() <= 32 * 1024); -} - -#[test] -fn test_live_external_shell_without_replay_never_becomes_an_empty_card() { - let mut shell = make_shell_tool_call("external-shell-no-replay"); - shell.display_status = EventDisplayStatus::Completed; - shell.shell_replay = None; - shell.args["streamOutput"] = serde_json::Value::String(String::new()); - shell.result = serde_json::json!({ - "stdout": format!("{}EXTERNAL-TAIL", "x".repeat(80_000)), - "exit_code": 0 - }); - - let mut store = EventStore::new(); - store.upsert(shell); - - let stored = store.get_by_id("external-shell-no-replay").unwrap(); - assert_eq!(stored.result, serde_json::json!({})); - let replay = stored - .shell_replay - .as_ref() - .expect("bounded external fallback preview"); - assert_eq!(replay.status, ShellReplayStatus::Incomplete); - assert_eq!(replay.bookmark, ShellReplayBookmark::default()); - assert!(replay.terminal_preview.len() <= 32 * 1024); - assert!(replay.terminal_preview.ends_with("EXTERNAL-TAIL")); - assert!(replay - .error - .as_deref() - .is_some_and(|error| error.contains("仅显示有界预览"))); -} - -#[test] -fn test_running_external_shell_without_replay_keeps_bounded_stream_preview() { - let mut shell = make_shell_tool_call("external-shell-running"); - shell.shell_replay = None; - shell.args["streamOutput"] = - serde_json::Value::String(format!("{}RUNNING-TAIL", "x".repeat(80_000))); - - let mut store = EventStore::new(); - store.upsert(shell); - - let stored = store.get_by_id("external-shell-running").unwrap(); - assert!(stored.shell_replay.is_none()); - let preview = stored.args["streamOutput"].as_str().unwrap(); - assert!(preview.len() <= 32 * 1024); - assert!(preview.ends_with("RUNNING-TAIL")); -} - -#[test] -fn test_hydration_converts_legacy_shell_output_to_bounded_incomplete_preview() { - let mut shell = make_shell_tool_call("legacy-shell"); - shell.args["streamOutput"] = serde_json::Value::String("old-stream".repeat(10_000)); - shell.result = serde_json::json!({ - "output": { - "success": { - "stdout": format!("{}TAIL-SENTINEL", "x".repeat(80_000)), - "exitCode": 7 - } - } - }); - shell.shell_replay = None; - shell.shell_replay_bookmarks = None; - - let mut store = EventStore::new(); - store.set(vec![shell]); - - let stored = store.get_by_id("legacy-shell").unwrap(); - assert_eq!(stored.result, serde_json::json!({})); - assert!(stored.args.get("streamOutput").is_none()); - assert_eq!(stored.args["shellExitCode"], 7); - assert!(stored.shell_replay_bookmarks.is_none()); - let replay = stored.shell_replay.as_ref().unwrap(); - assert_eq!(replay.status, ShellReplayStatus::Incomplete); - assert!(replay.completed_at.is_none()); - assert!(replay.terminal_preview.len() <= 32 * 1024); - assert!(replay.terminal_preview.ends_with("TAIL-SENTINEL")); - assert_eq!(replay.bookmark, ShellReplayBookmark::default()); -} - -#[test] -fn test_find_last_spawning_tool() { - let mut store = EventStore::new(); - store.set(vec![ - make_task_tool_call("task-1"), - make_event("msg-1", "message"), - ]); - assert_eq!(store.find_last_spawning_tool(&["task"]), Some(0)); -} - -#[test] -fn test_find_last_spawning_tool_none() { - let mut store = EventStore::new(); - store.set(vec![make_event("msg-1", "message")]); - assert!(store.find_last_spawning_tool(&["task"]).is_none()); -} - -#[test] -fn test_find_last_spawning_tool_stops_at_result() { - let mut store = EventStore::new(); - let mut task_call = make_task_tool_call("task-1"); - task_call.action_type = "tool_call".to_string(); - let mut task_result = make_event("task-r", "tool_result"); - task_result.function_name = "task".to_string(); - store.set(vec![task_call, task_result, make_event("msg-1", "message")]); - assert!(store.find_last_spawning_tool(&["task"]).is_none()); -} - -#[test] -fn test_has_active_spawning_tool() { - let mut store = EventStore::new(); - store.set(vec![make_task_tool_call("task-1")]); - assert!(store.has_active_spawning_tool(&["task"])); - assert!(!store.has_active_spawning_tool(&["session"])); -} - -// ============================================================================ -// cancel_orphan_interactive_events tests -// ============================================================================ - -#[test] -fn test_cancel_orphan_interactive_events_cancels_awaiting_user() { - let mut store = EventStore::new(); - let mut orphan = make_tool_call("ask-1", "call-ask-1"); - orphan.display_status = EventDisplayStatus::AwaitingUser; - store.set(vec![make_event("msg-1", "message"), orphan]); - - let cancelled = store.cancel_orphan_interactive_events(); - - assert_eq!(cancelled, vec!["ask-1".to_string()]); - let event = store.get_by_id("ask-1").unwrap(); - assert_eq!(event.display_status, EventDisplayStatus::Completed); - assert_eq!(event.result["status"], "cancelled"); -} - -#[test] -fn test_cancel_orphan_interactive_events_leaves_running_untouched() { - let mut store = EventStore::new(); - let running = make_tool_call("run-1", "call-run-1"); - store.set(vec![running]); - - let cancelled = store.cancel_orphan_interactive_events(); - - assert!(cancelled.is_empty()); - let event = store.get_by_id("run-1").unwrap(); - assert_eq!(event.display_status, EventDisplayStatus::Running); -} - -#[test] -fn test_cancel_orphan_interactive_events_mixed() { - let mut store = EventStore::new(); - let running = make_tool_call("run-1", "call-run-1"); - let mut awaiting1 = make_tool_call("ask-1", "call-ask-1"); - awaiting1.display_status = EventDisplayStatus::AwaitingUser; - let mut awaiting2 = make_tool_call("ask-2", "call-ask-2"); - awaiting2.display_status = EventDisplayStatus::AwaitingUser; - // A pre-completed event (not AwaitingUser, not Running). - let mut already_done = make_event("done-1", "tool_call"); - already_done.display_status = EventDisplayStatus::Completed; - store.set(vec![running, awaiting1, awaiting2, already_done]); - - let cancelled = store.cancel_orphan_interactive_events(); - - assert_eq!(cancelled.len(), 2); - assert!(cancelled.contains(&"ask-1".to_string())); - assert!(cancelled.contains(&"ask-2".to_string())); - // running stays Running - assert_eq!( - store.get_by_id("run-1").unwrap().display_status, - EventDisplayStatus::Running - ); - // pre-completed stays Completed with original empty result - assert_eq!( - store.get_by_id("done-1").unwrap().display_status, - EventDisplayStatus::Completed - ); - assert!(store - .get_by_id("done-1") - .unwrap() - .result - .as_object() - .unwrap() - .is_empty()); -} - -fn make_user_turn_header(turn_id: &str, created_at: &str) -> SessionEvent { - let mut event = make_event(turn_id, "raw"); - event.function_name = "user_message".to_string(); - event.ui_canonical = "user_message".to_string(); - event.source = EventSource::User; - event.display_variant = EventDisplayVariant::Message; - event.created_at = created_at.to_string(); - event -} - -fn make_turn_placeholder(turn_id: &str, next_turn_id: Option<&str>) -> SessionEvent { - let mut event = make_event(&format!("turn-placeholder-{turn_id}"), "turn_placeholder"); - event.function_name = "turn_placeholder".to_string(); - event.ui_canonical = "turn_placeholder".to_string(); - event.result = serde_json::json!({ - "unloadedTurn": { - "turnId": turn_id, - "bodyEventCount": 2, - "nextTurnId": next_turn_id, - } - }); - event -} - -#[test] -fn test_round_window_hydration_mode() { - let mut store = EventStore::new(); - assert_eq!( - store.hydration_mode(), - crate::agent_sessions::event_pipeline::store::HydrationMode::Full - ); - - store.set_round_window(vec![make_user_turn_header( - "turn-1", - "2026-01-01T00:00:00Z", - )]); - assert_eq!( - store.hydration_mode(), - crate::agent_sessions::event_pipeline::store::HydrationMode::RoundWindow - ); - - store.merge_events(vec![make_event("live-1", "message")]); - assert_eq!( - store.hydration_mode(), - crate::agent_sessions::event_pipeline::store::HydrationMode::LivePartial - ); -} - -#[test] -fn test_empty_round_window_does_not_clobber_existing_events() { - let mut store = EventStore::new(); - store.set_round_window(vec![ - make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), - make_event("turn-1-body-1", "message"), - ]); - assert_eq!(store.events().len(), 2); - - // An empty round window (turn index mid-rebuild) must not wipe the store. - store.set_round_window(Vec::new()); - - assert_eq!(store.events().len(), 2); - assert!(store.get_by_id("turn-1").is_some()); - assert!(store.get_by_id("turn-1-body-1").is_some()); -} - -#[test] -fn test_empty_round_window_on_empty_store_is_noop_set() { - let mut store = EventStore::new(); - // Empty window on an already-empty store stays empty (no panic, no events). - store.set_round_window(Vec::new()); - assert_eq!(store.events().len(), 0); -} - -#[test] -fn test_unload_turn_body_restores_placeholder_and_preserves_headers() { - let mut store = EventStore::new(); - store.set_round_window(vec![ - make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), - make_event("turn-1-body-1", "message"), - make_event("turn-1-body-2", "tool_call"), - make_user_turn_header("turn-2", "2026-01-01T00:01:00Z"), - make_event("turn-2-body-1", "message"), - ]); - - let removed = store.unload_turn_body("turn-1", make_turn_placeholder("turn-1", Some("turn-2"))); - - assert_eq!(removed, 2); - assert!(store.get_by_id("turn-1").is_some()); - assert!(store.get_by_id("turn-placeholder-turn-1").is_some()); - assert!(store.get_by_id("turn-1-body-1").is_none()); - assert!(store.get_by_id("turn-1-body-2").is_none()); - assert!(store.get_by_id("turn-2").is_some()); - assert!(store.get_by_id("turn-2-body-1").is_some()); - assert_eq!( - store.hydration_mode(), - crate::agent_sessions::event_pipeline::store::HydrationMode::RoundWindow - ); -} - -#[test] -fn test_unload_turn_body_preserves_final_reply_as_preview() { - let mut final_reply = make_event("turn-1-final-reply", "assistant"); - final_reply.function_name = "assistant".to_string(); - final_reply.ui_canonical = "agent_message".to_string(); - final_reply.display_variant = EventDisplayVariant::Message; - final_reply.display_text = "Finished the work".to_string(); - - let mut store = EventStore::new(); - store.set_round_window(vec![ - make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), - make_event("turn-1-tool", "tool_call"), - final_reply, - make_user_turn_header("turn-2", "2026-01-01T00:01:00Z"), - ]); - - let removed = store.unload_turn_body("turn-1", make_turn_placeholder("turn-1", Some("turn-2"))); - - assert_eq!(removed, 1); - let preview = store.get_by_id("turn-1-final-reply").unwrap(); - assert_eq!( - preview.args.get("turnPreviewOnly"), - Some(&serde_json::Value::Bool(true)) - ); - assert!(store.get_by_id("turn-1-tool").is_none()); - assert!(store.get_by_id("turn-placeholder-turn-1").is_some()); -} - -#[test] -fn test_merge_round_window_events_removes_loaded_turn_placeholder() { - let mut store = EventStore::new(); - store.set_round_window(vec![ - make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), - make_turn_placeholder("turn-1", Some("turn-2")), - make_user_turn_header("turn-2", "2026-01-01T00:01:00Z"), - ]); - - let mut body_1 = make_event("turn-1-body-1", "message"); - body_1.created_at = "2026-01-01T00:00:20Z".to_string(); - let mut body_2 = make_event("turn-1-body-2", "tool_call"); - body_2.created_at = "2026-01-01T00:00:40Z".to_string(); - - store.merge_round_window_events(vec![ - make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), - body_1, - body_2, - ]); - - assert!(store.get_by_id("turn-placeholder-turn-1").is_none()); - assert!(store.get_by_id("turn-1").is_some()); - assert!(store.get_by_id("turn-1-body-1").is_some()); - assert!(store.get_by_id("turn-1-body-2").is_some()); - assert!(store.get_by_id("turn-2").is_some()); - let event_ids = store - .events() - .iter() - .map(|event| event.id.as_str()) - .collect::>(); - assert_eq!( - event_ids, - vec!["turn-1", "turn-1-body-1", "turn-1-body-2", "turn-2"] - ); - assert_eq!( - store.hydration_mode(), - crate::agent_sessions::event_pipeline::store::HydrationMode::RoundWindow - ); -} +#[path = "store_tests/core_mutations.rs"] +mod core_mutations; +#[path = "store_tests/external_replay_window.rs"] +mod external_replay_window; +#[path = "store_tests/lifecycle.rs"] +mod lifecycle; +#[path = "store_tests/shell_replay.rs"] +mod shell_replay; +#[path = "store_tests/support.rs"] +mod support; diff --git a/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/core_mutations.rs b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/core_mutations.rs new file mode 100644 index 000000000..4e8a4fed8 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/core_mutations.rs @@ -0,0 +1,868 @@ +use crate::agent_sessions::event_pipeline::store::EventStore; +use crate::agent_sessions::event_pipeline::types::*; + +use super::support::*; + +#[test] +fn test_set_replaces_all() { + let mut store = EventStore::new(); + store.set(vec![ + make_event("a", "tool_call"), + make_event("b", "message"), + ]); + assert_eq!(store.event_count(), 2); + assert_eq!(store.version(), 1); + + store.set(vec![make_event("c", "tool_call")]); + assert_eq!(store.event_count(), 1); + assert_eq!(store.version(), 2); + assert!(store.get_by_id("a").is_none()); + assert!(store.get_by_id("c").is_some()); +} + +#[test] +fn test_append_deduplicates() { + let mut store = EventStore::new(); + store.append(vec![ + make_event("a", "tool_call"), + make_event("b", "message"), + ]); + assert_eq!(store.event_count(), 2); + + store.append(vec![ + make_event("b", "message"), + make_event("c", "tool_call"), + ]); + assert_eq!(store.event_count(), 3); +} + +#[test] +fn test_delta_tracking_records_append_upsert_and_remove() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + store.mark_full_snapshot_emitted(); + + store.append(vec![make_event("b", "tool_call")]); + let (base_version, changed_ids, removed_ids) = store.take_delta_tracking(); + assert_eq!(base_version, 1); + assert_eq!(store.version(), 2); + assert_eq!(changed_ids, vec!["b".to_string()]); + assert!(removed_ids.is_empty()); + + let mut updated = make_event("a", "message"); + updated.display_text = "updated".to_string(); + store.upsert(updated); + assert_eq!(store.remove_by_id_prefix("b"), 1); + + let (base_version, mut changed_ids, removed_ids) = store.take_delta_tracking(); + changed_ids.sort(); + assert_eq!(base_version, 2); + assert_eq!(store.version(), 4); + assert_eq!(changed_ids, vec!["a".to_string()]); + assert_eq!(removed_ids, vec!["b".to_string()]); +} + +#[test] +fn test_update_by_id() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "tool_call")]); + + let patch = SessionEventPatch { + display_status: Some(EventDisplayStatus::Failed), + display_text: Some("error occurred".to_string()), + ..Default::default() + }; + assert!(store.update_by_id("a", &patch)); + assert_eq!( + store.get_by_id("a").unwrap().display_status, + EventDisplayStatus::Failed + ); + assert_eq!(store.get_by_id("a").unwrap().display_text, "error occurred"); + + assert!(!store.update_by_id("nonexistent", &patch)); +} + +#[test] +fn test_upsert() { + let mut store = EventStore::new(); + store.upsert(make_event("a", "tool_call")); + assert_eq!(store.event_count(), 1); + assert_eq!(store.version(), 1); + + let mut updated = make_event("a", "tool_call"); + updated.display_text = "updated text".to_string(); + store.upsert(updated); + assert_eq!(store.event_count(), 1); + assert_eq!(store.version(), 2); + assert_eq!(store.get_by_id("a").unwrap().display_text, "updated text"); + + store.upsert(make_event("b", "message")); + assert_eq!(store.event_count(), 2); +} + +#[test] +fn test_merge_tool_result() { + let mut store = EventStore::new(); + store.set(vec![make_tool_call("tc-1", "call-1")]); + assert_eq!( + store.get_by_id("tc-1").unwrap().display_status, + EventDisplayStatus::Running + ); + + store.merge_events(vec![make_tool_result("tr-1", "call-1")]); + + let merged = store.get_by_id("tc-1").unwrap(); + assert_eq!(merged.display_status, EventDisplayStatus::Completed); + assert_eq!(merged.activity_status, ActivityStatus::Processed); + assert_eq!(merged.result["content"], "file1.txt\nfile2.txt"); + // Args from original tool_call should be preserved (except streamOutput) + assert_eq!(merged.args["command"], "ls"); + assert!(merged.args.get("streamOutput").is_none()); + // tool_result should NOT appear as separate event + assert!(store.get_by_id("tr-1").is_none()); + assert_eq!(store.event_count(), 1); +} + +#[test] +fn test_merge_tool_result_preserves_background_shell_until_exact_exit_callback() { + let mut shell = make_tool_call("tc-background", "call-background"); + shell.function_name = "run_shell".to_string(); + shell.ui_canonical = core_types::tool_names::RUN_SHELL.to_string(); + shell.args = serde_json::json!({ + "command": "sleep 10", + "shellPid": 4242, + "shellProcessStatus": "background" + }); + let mut store = EventStore::new(); + store.set(vec![shell]); + + store.merge_events(vec![make_tool_result("tr-background", "call-background")]); + + let merged = store.get_by_id("tc-background").unwrap(); + assert_eq!(merged.display_status, EventDisplayStatus::Completed); + assert_eq!(merged.args["shellProcessStatus"], "background"); + assert_eq!(merged.args["shellPid"], 4242); +} + +#[test] +fn test_merge_tool_result_preserves_args_and_merges_metadata() { + let mut store = EventStore::new(); + + // Tool call with file_path in args + let mut tool_call = make_tool_call("tc-1", "call-1"); + tool_call.args = + serde_json::json!({ "path": "/src/main.rs", "command": "edit", "streamOutput": "..." }); + tool_call.file_path = Some("/src/main.rs".to_string()); + store.set(vec![tool_call]); + + // Tool result with additional metadata + let mut tool_result = make_tool_result("tr-1", "call-1"); + tool_result.args = serde_json::json!({ "execution_time": 150 }); // Extra metadata + tool_result.command = Some("git diff".to_string()); + store.merge_events(vec![tool_result]); + + let merged = store.get_by_id("tc-1").unwrap(); + // Original args preserved + assert_eq!(merged.args["path"], "/src/main.rs"); + assert_eq!(merged.args["command"], "edit"); + // Extra metadata from result merged in + assert_eq!(merged.args["execution_time"], 150); + // streamOutput removed + assert!(merged.args.get("streamOutput").is_none()); + // file_path preserved from original + assert_eq!(merged.file_path, Some("/src/main.rs".to_string())); + // command propagated from result (original was None) + assert_eq!(merged.command, Some("git diff".to_string())); +} + +#[test] +fn test_merge_updates_existing() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + + let mut updated = make_event("a", "message"); + updated.display_text = "new text".to_string(); + store.merge_events(vec![updated]); + + assert_eq!(store.event_count(), 1); + assert_eq!(store.get_by_id("a").unwrap().display_text, "new text"); +} + +#[test] +fn test_merge_appends_new() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + store.merge_events(vec![make_event("b", "tool_call")]); + assert_eq!(store.event_count(), 2); +} + +#[test] +fn test_cap_at_max_events() { + let mut store = EventStore::new(); + let events: Vec = (0..8010) + .map(|i| make_event(&format!("evt-{}", i), "message")) + .collect(); + store.set(events); + assert_eq!(store.event_count(), 8000); + // Oldest events should have been trimmed + assert!(store.get_by_id("evt-0").is_none()); + assert!(store.get_by_id("evt-10").is_some()); +} + +#[test] +fn test_clear() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + store.clear(); + assert_eq!(store.event_count(), 0); + assert!(store.get_by_id("a").is_none()); +} + +#[test] +fn test_streaming_flag() { + let mut store = EventStore::new(); + assert!(!store.is_streaming()); + store.set_streaming(true); + assert!(store.is_streaming()); + store.set_streaming(false); + assert!(!store.is_streaming()); +} + +// ============================================================================ +// Batch operation tests +// ============================================================================ + +#[test] +fn test_complete_last_running() { + let mut store = EventStore::new(); + store.set(vec![ + make_event("a", "message"), + make_running_event("b"), + make_running_event("c"), + ]); + let v_before = store.version(); + let result = store.complete_last_running(); + assert_eq!(result, Some("c".to_string())); + assert_eq!( + store.get_by_id("c").unwrap().display_status, + EventDisplayStatus::Completed + ); + assert_eq!( + store.get_by_id("b").unwrap().display_status, + EventDisplayStatus::Running + ); + assert!(store.version() > v_before); +} + +#[test] +fn test_complete_last_running_none() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + let v_before = store.version(); + let result = store.complete_last_running(); + assert!(result.is_none()); + assert_eq!(store.version(), v_before); +} + +#[test] +fn test_complete_last_running_skips_awaiting_user() { + // Regression: AskQuestionCard used to disappear because `agent:complete` + // for the surrounding turn called `complete_last_running`, which flipped + // the blocking `ask_user_questions` tool_call to Completed. + // + // With the AwaitingUser phase, only explicit `interaction_finalized` + // (via `merge_events`) is allowed to complete it. + let mut store = EventStore::new(); + store.set(vec![ + make_event("msg", "message"), + make_awaiting_user_event("tool-call-ask"), + ]); + let v_before = store.version(); + let result = store.complete_last_running(); + assert!( + result.is_none(), + "AwaitingUser event must not be treated as Running" + ); + assert_eq!( + store.get_by_id("tool-call-ask").unwrap().display_status, + EventDisplayStatus::AwaitingUser + ); + assert_eq!( + store.version(), + v_before, + "version must not bump when nothing changes" + ); +} + +#[test] +fn test_complete_last_running_picks_running_past_awaiting_user() { + // If a real Running event exists before the AwaitingUser one in insertion + // order (AwaitingUser inserted LAST), `complete_last_running` should skip + // AwaitingUser and land on the Running event behind it. + let mut store = EventStore::new(); + store.set(vec![ + make_running_event("running-thinking"), + make_awaiting_user_event("tool-call-ask"), + ]); + let result = store.complete_last_running(); + assert_eq!(result, Some("running-thinking".to_string())); + assert_eq!( + store.get_by_id("running-thinking").unwrap().display_status, + EventDisplayStatus::Completed + ); + assert_eq!( + store.get_by_id("tool-call-ask").unwrap().display_status, + EventDisplayStatus::AwaitingUser, + "AwaitingUser must remain untouched" + ); +} + +#[test] +fn test_merge_events_transitions_awaiting_user_to_completed() { + // The `interaction_finalized` path emits a tool_result that merges into + // the AwaitingUser tool_call; that merge is the sole legitimate way to + // transition into Completed. + let mut store = EventStore::new(); + let mut call = make_awaiting_user_event("tool-call-ask"); + call.call_id = Some("ask-123".to_string()); + store.set(vec![call]); + + let mut result_event = make_event("tool-result-ask", "tool_result"); + result_event.call_id = Some("ask-123".to_string()); + result_event.result = serde_json::json!({ "answers": ["use_redis"], "status": "answered" }); + + store.merge_events(vec![result_event]); + + let completed = store.get_by_id("tool-call-ask").unwrap(); + assert_eq!( + completed.display_status, + EventDisplayStatus::Completed, + "interaction_finalized must flip AwaitingUser → Completed" + ); +} + +#[test] +fn test_patch_by_ids() { + let mut store = EventStore::new(); + store.set(vec![ + make_running_event("a"), + make_running_event("b"), + make_running_event("c"), + ]); + let patch = SessionEventPatch { + display_status: Some(EventDisplayStatus::Completed), + is_delta: Some(false), + ..Default::default() + }; + let count = store.patch_by_ids(&["a".to_string(), "c".to_string()], &patch); + assert_eq!(count, 2); + assert_eq!( + store.get_by_id("a").unwrap().display_status, + EventDisplayStatus::Completed + ); + assert_eq!( + store.get_by_id("b").unwrap().display_status, + EventDisplayStatus::Running + ); + assert_eq!( + store.get_by_id("c").unwrap().display_status, + EventDisplayStatus::Completed + ); +} + +#[test] +fn test_patch_by_ids_with_missing() { + let mut store = EventStore::new(); + store.set(vec![make_running_event("a")]); + let patch = SessionEventPatch { + display_status: Some(EventDisplayStatus::Completed), + ..Default::default() + }; + let count = store.patch_by_ids(&["a".to_string(), "nonexistent".to_string()], &patch); + assert_eq!(count, 1); +} + +#[test] +fn test_remove_by_id_prefix() { + let mut store = EventStore::new(); + store.set(vec![ + make_event("stream-msg-1", "message"), + make_event("stream-msg-2", "message"), + make_event("normal-1", "tool_call"), + make_event("stream-think-1", "message"), + ]); + let removed = store.remove_by_id_prefix("stream-msg-"); + assert_eq!(removed, 2); + assert_eq!(store.event_count(), 2); + assert!(store.get_by_id("stream-msg-1").is_none()); + assert!(store.get_by_id("normal-1").is_some()); + assert!(store.get_by_id("stream-think-1").is_some()); +} + +#[test] +fn test_remove_by_id_prefix_no_match() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + let v_before = store.version(); + let removed = store.remove_by_id_prefix("nonexistent-"); + assert_eq!(removed, 0); + assert_eq!(store.version(), v_before); +} + +#[test] +fn test_remove_by_ids() { + let mut store = EventStore::new(); + store.set(vec![ + make_event("stream-msg-1", "message"), + make_event("normal-1", "tool_call"), + make_event("stream-think-1", "message"), + ]); + let removed = store.remove_by_ids(&[ + "stream-msg-1".to_string(), + "stream-think-1".to_string(), + "nonexistent".to_string(), + ]); + assert_eq!(removed, 2); + assert!(store.get_by_id("stream-msg-1").is_none()); + assert!(store.get_by_id("stream-think-1").is_none()); + assert!(store.get_by_id("normal-1").is_some()); + + // Removed ids surface in delta tracking so `es:changed` subscribers drop them. + let (_, _, removed_ids) = store.take_delta_tracking(); + assert!(removed_ids.contains(&"stream-msg-1".to_string())); + assert!(removed_ids.contains(&"stream-think-1".to_string())); +} + +#[test] +fn test_remove_by_ids_no_match_keeps_version() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + let v_before = store.version(); + let removed = store.remove_by_ids(&["nonexistent".to_string()]); + assert_eq!(removed, 0); + assert_eq!(store.version(), v_before); +} + +#[test] +fn test_remove_synthetic_user_inputs_keeps_backend_user_input_ids() { + let mut store = EventStore::new(); + let mut synthetic = make_event("user-input-synthetic", "raw"); + synthetic.source = EventSource::User; + synthetic.function_name = "user_message".to_string(); + synthetic.ui_canonical = "user_message".to_string(); + synthetic.result = serde_json::json!({ "syntheticUserInput": true }); + synthetic.chunk_id = None; + + let mut backend = make_event("user-input-cliagent-real", "raw"); + backend.source = EventSource::User; + backend.function_name = "user_message".to_string(); + backend.ui_canonical = "user_message".to_string(); + backend.display_text = "authoritative different text".to_string(); + + store.set(vec![synthetic, backend]); + let removed = store.remove_synthetic_user_inputs(); + + assert_eq!(removed, 1); + assert!(store.get_by_id("user-input-synthetic").is_none()); + assert!(store.get_by_id("user-input-cliagent-real").is_some()); +} + +#[test] +fn test_merge_authoritative_user_message_evicts_matching_synthetic_placeholder() { + let mut store = EventStore::new(); + let mut synthetic = make_event("user-input-synthetic", "raw"); + synthetic.source = EventSource::User; + synthetic.function_name = "user_message".to_string(); + synthetic.ui_canonical = "user_message".to_string(); + synthetic.result = serde_json::json!({ "syntheticUserInput": true }); + synthetic.chunk_id = None; + synthetic.display_text = "hello from user".to_string(); + + store.append(vec![synthetic]); + assert!(store.get_by_id("user-input-synthetic").is_some()); + + let mut backend = make_event("user-input-cliagent-real", "raw"); + backend.source = EventSource::User; + backend.function_name = "user".to_string(); + backend.ui_canonical = "user".to_string(); + backend.display_text = "hello from user".to_string(); + + store.merge_events(vec![backend]); + + assert!(store.get_by_id("user-input-synthetic").is_none()); + assert!(store.get_by_id("user-input-cliagent-real").is_some()); +} + +#[test] +fn test_set_reconciles_persisted_matching_synthetic_placeholder() { + let mut store = EventStore::new(); + let mut synthetic = make_event("user-input-synthetic", "raw"); + synthetic.source = EventSource::User; + synthetic.function_name = "user_message".to_string(); + synthetic.result = serde_json::json!({ "syntheticUserInput": true }); + synthetic.display_text = "persisted duplicate".to_string(); + + let mut backend = make_event("user-input-real", "raw"); + backend.source = EventSource::User; + backend.function_name = "provider_specific_user_event".to_string(); + backend.display_text = "persisted duplicate".to_string(); + + store.set(vec![synthetic, backend]); + + assert!(store.get_by_id("user-input-synthetic").is_none()); + assert!(store.get_by_id("user-input-real").is_some()); +} + +#[test] +fn test_merge_authoritative_message_keeps_legitimate_repeated_user_text() { + let mut store = EventStore::new(); + let mut first = make_event("user-input-first", "raw"); + first.source = EventSource::User; + first.function_name = "user".to_string(); + first.ui_canonical = "user".to_string(); + first.display_text = "repeat me".to_string(); + + let mut second = make_event("user-input-second", "raw"); + second.source = EventSource::User; + second.function_name = "user".to_string(); + second.ui_canonical = "user".to_string(); + second.display_text = "repeat me".to_string(); + + store.append(vec![first]); + store.merge_events(vec![second]); + + assert!(store.get_by_id("user-input-first").is_some()); + assert!(store.get_by_id("user-input-second").is_some()); +} + +#[test] +fn test_merge_authoritative_message_keeps_non_matching_synthetic_text() { + let mut store = EventStore::new(); + let mut synthetic = make_event("user-input-synthetic", "raw"); + synthetic.source = EventSource::User; + synthetic.function_name = "user_message".to_string(); + synthetic.ui_canonical = "user_message".to_string(); + synthetic.result = serde_json::json!({ "syntheticUserInput": true }); + synthetic.display_text = "different pending text".to_string(); + + let mut backend = make_event("user-input-real", "raw"); + backend.source = EventSource::User; + backend.function_name = "provider_specific_user_event".to_string(); + backend.ui_canonical = "user".to_string(); + backend.display_text = "authoritative text".to_string(); + + store.append(vec![synthetic]); + store.merge_events(vec![backend]); + + assert!(store.get_by_id("user-input-synthetic").is_some()); + assert!(store.get_by_id("user-input-real").is_some()); +} + +#[test] +fn test_replace_and_remove() { + let mut store = EventStore::new(); + let mut placeholder = make_event("stream-1", "message"); + placeholder.created_at = "2026-05-22T06:48:20.100Z".to_string(); + let mut normal_event = make_event("normal-1", "tool_call"); + normal_event.created_at = "2026-05-22T06:48:21.000Z".to_string(); + store.set(vec![placeholder, normal_event]); + let mut new_event = make_event("final-1", "message"); + new_event.created_at = "2026-05-22T06:48:30.000Z".to_string(); + + store.replace_and_remove(Some("stream-1"), new_event); + + assert!(store.get_by_id("stream-1").is_none()); + assert!(store.get_by_id("final-1").is_some()); + assert!(store.get_by_id("normal-1").is_some()); + assert_eq!(store.events()[0].id, "final-1"); + assert_eq!(store.events()[0].created_at, "2026-05-22T06:48:20.100Z"); + assert_eq!(store.events()[1].id, "normal-1"); +} + +#[test] +fn test_replace_and_remove_removes_placeholder_when_final_already_exists() { + let mut store = EventStore::new(); + let mut placeholder = make_event("stream-think-ts-1", "message"); + placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); + let mut existing_final = make_event("stream-think-1-final", "message"); + existing_final.created_at = "2026-05-22T07:18:22.000Z".to_string(); + let normal_event = make_event("normal-1", "tool_call"); + store.set(vec![placeholder, existing_final, normal_event]); + + let mut new_event = make_event("stream-think-1-final", "message"); + new_event.created_at = "2026-05-22T07:18:30.000Z".to_string(); + store.replace_and_remove(Some("stream-think-ts-1"), new_event); + + assert!(store.get_by_id("stream-think-ts-1").is_none()); + assert!(store.get_by_id("stream-think-1-final").is_some()); + assert_eq!( + store + .events() + .iter() + .filter(|event| event.id == "stream-think-1-final") + .count(), + 1 + ); + assert_eq!(store.events()[0].id, "stream-think-1-final"); + assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); + assert_eq!(store.events()[1].id, "normal-1"); +} + +#[test] +fn test_replace_and_remove_removes_tail_placeholder_when_final_already_exists() { + let mut store = EventStore::new(); + let mut existing_final = make_event("stream-think-1-final", "message"); + existing_final.created_at = "2026-05-22T07:18:22.000Z".to_string(); + let normal_event = make_event("normal-1", "tool_call"); + let mut placeholder = make_event("stream-think-ts-1", "message"); + placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); + store.set(vec![existing_final, normal_event, placeholder]); + + let mut new_event = make_event("stream-think-1-final", "message"); + new_event.created_at = "2026-05-22T07:18:30.000Z".to_string(); + store.replace_and_remove(Some("stream-think-ts-1"), new_event); + + assert!(store.get_by_id("stream-think-ts-1").is_none()); + assert!(store.get_by_id("stream-think-1-final").is_some()); + assert_eq!(store.event_count(), 2); + assert_eq!(store.events()[0].id, "stream-think-1-final"); + assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); + assert_eq!(store.events()[1].id, "normal-1"); +} + +#[test] +fn test_replace_and_remove_no_remove() { + let mut store = EventStore::new(); + store.set(vec![make_event("a", "message")]); + let new_event = make_event("b", "message"); + store.replace_and_remove(None, new_event); + assert_eq!(store.event_count(), 2); +} + +#[test] +fn test_authoritative_stream_upsert_replaces_matching_ts_placeholder() { + let mut store = EventStore::new(); + let mut placeholder = make_event("stream-think-ts-test-session-100", "llm_thinking"); + placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); + placeholder.display_text = "same thought".to_string(); + let normal_event = make_event("normal-1", "tool_call"); + store.set(vec![placeholder, normal_event]); + + let mut authoritative = make_event("stream-think-test-session-1-final", "llm_thinking"); + authoritative.created_at = "2026-05-22T07:18:30.000Z".to_string(); + authoritative.display_text = "same thought".to_string(); + store.upsert(authoritative); + + assert!(store + .get_by_id("stream-think-ts-test-session-100") + .is_none()); + assert!(store + .get_by_id("stream-think-test-session-1-final") + .is_some()); + assert_eq!(store.event_count(), 2); + assert_eq!(store.events()[0].id, "stream-think-test-session-1-final"); + assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); + assert_eq!(store.events()[1].id, "normal-1"); +} + +#[test] +fn test_authoritative_stream_upsert_removes_placeholder_when_final_already_exists() { + let mut store = EventStore::new(); + let mut placeholder = make_event("stream-think-ts-test-session-100", "llm_thinking"); + placeholder.created_at = "2026-05-22T07:18:20.100Z".to_string(); + placeholder.display_text = "same thought".to_string(); + let mut existing_final = make_event("stream-think-test-session-1-final", "llm_thinking"); + existing_final.created_at = "2026-05-22T07:18:22.000Z".to_string(); + existing_final.display_text = "same thought".to_string(); + let normal_event = make_event("normal-1", "tool_call"); + store.set(vec![placeholder, existing_final, normal_event]); + + let mut authoritative = make_event("stream-think-test-session-1-final", "llm_thinking"); + authoritative.created_at = "2026-05-22T07:18:30.000Z".to_string(); + authoritative.display_text = "same thought".to_string(); + store.upsert(authoritative); + + assert!(store + .get_by_id("stream-think-ts-test-session-100") + .is_none()); + assert!(store + .get_by_id("stream-think-test-session-1-final") + .is_some()); + assert_eq!( + store + .events() + .iter() + .filter(|event| event.id == "stream-think-test-session-1-final") + .count(), + 1 + ); + assert_eq!(store.event_count(), 2); + assert_eq!(store.events()[0].id, "stream-think-test-session-1-final"); + assert_eq!(store.events()[0].created_at, "2026-05-22T07:18:20.100Z"); + assert_eq!(store.events()[1].id, "normal-1"); +} + +#[test] +fn test_authoritative_thinking_upsert_replaces_duplicate_in_current_turn() { + let mut store = EventStore::new(); + let mut user = make_event("user-1", "user_message"); + user.source = EventSource::User; + user.display_variant = EventDisplayVariant::Message; + user.display_text = "first prompt".to_string(); + + let mut first = make_event("stream-think-session-1", "llm_thinking"); + first.display_variant = EventDisplayVariant::Thinking; + first.display_text = "same thought".to_string(); + first.created_at = "2026-05-22T07:18:20.100Z".to_string(); + + store.set(vec![user, first]); + + let mut duplicate = make_event("stream-think-session-2", "llm_thinking"); + duplicate.display_variant = EventDisplayVariant::Thinking; + duplicate.display_text = "same thought".to_string(); + duplicate.created_at = "2026-05-22T07:18:30.000Z".to_string(); + + store.upsert(duplicate); + + assert!(store.get_by_id("stream-think-session-1").is_none()); + assert!(store.get_by_id("stream-think-session-2").is_some()); + assert_eq!(store.event_count(), 2); + assert_eq!(store.events()[1].id, "stream-think-session-2"); + assert_eq!(store.events()[1].created_at, "2026-05-22T07:18:20.100Z"); +} + +#[test] +fn test_authoritative_thinking_upsert_preserves_same_text_across_turns() { + let mut store = EventStore::new(); + let mut user_one = make_event("user-1", "user_message"); + user_one.source = EventSource::User; + user_one.display_variant = EventDisplayVariant::Message; + user_one.display_text = "first prompt".to_string(); + + let mut first = make_event("stream-think-session-1", "llm_thinking"); + first.display_variant = EventDisplayVariant::Thinking; + first.display_text = "same thought".to_string(); + + let mut user_two = make_event("user-2", "user_message"); + user_two.source = EventSource::User; + user_two.display_variant = EventDisplayVariant::Message; + user_two.display_text = "second prompt".to_string(); + + store.set(vec![user_one, first, user_two]); + + let mut repeated_next_turn = make_event("stream-think-session-2", "llm_thinking"); + repeated_next_turn.display_variant = EventDisplayVariant::Thinking; + repeated_next_turn.display_text = "same thought".to_string(); + + store.upsert(repeated_next_turn); + + assert!(store.get_by_id("stream-think-session-1").is_some()); + assert!(store.get_by_id("stream-think-session-2").is_some()); + assert_eq!(store.event_count(), 4); +} + +#[test] +fn test_authoritative_message_upsert_replaces_duplicate_in_current_turn() { + let mut store = EventStore::new(); + let mut user = make_event("user-1", "user_message"); + user.source = EventSource::User; + user.display_variant = EventDisplayVariant::Message; + user.display_text = "first prompt".to_string(); + + let mut first = make_event("stream-msg-session-1", "message"); + first.display_variant = EventDisplayVariant::Message; + first.display_text = "same assistant note".to_string(); + first.created_at = "2026-05-22T07:18:20.100Z".to_string(); + + store.set(vec![user, first]); + + let mut duplicate = make_event("stream-msg-session-2", "message"); + duplicate.display_variant = EventDisplayVariant::Message; + duplicate.display_text = "same assistant note".to_string(); + duplicate.created_at = "2026-05-22T07:18:30.000Z".to_string(); + + store.upsert(duplicate); + + assert!(store.get_by_id("stream-msg-session-1").is_none()); + assert!(store.get_by_id("stream-msg-session-2").is_some()); + assert_eq!(store.event_count(), 2); + assert_eq!(store.events()[1].id, "stream-msg-session-2"); + assert_eq!(store.events()[1].created_at, "2026-05-22T07:18:20.100Z"); +} + +#[test] +fn test_authoritative_message_upsert_preserves_same_text_across_turns() { + let mut store = EventStore::new(); + let mut user_one = make_event("user-1", "user_message"); + user_one.source = EventSource::User; + user_one.display_variant = EventDisplayVariant::Message; + user_one.display_text = "first prompt".to_string(); + + let mut first = make_event("stream-msg-session-1", "message"); + first.display_variant = EventDisplayVariant::Message; + first.display_text = "same assistant note".to_string(); + + let mut user_two = make_event("user-2", "user_message"); + user_two.source = EventSource::User; + user_two.display_variant = EventDisplayVariant::Message; + user_two.display_text = "second prompt".to_string(); + + store.set(vec![user_one, first, user_two]); + + let mut repeated_next_turn = make_event("stream-msg-session-2", "message"); + repeated_next_turn.display_variant = EventDisplayVariant::Message; + repeated_next_turn.display_text = "same assistant note".to_string(); + + store.upsert(repeated_next_turn); + + assert!(store.get_by_id("stream-msg-session-1").is_some()); + assert!(store.get_by_id("stream-msg-session-2").is_some()); + assert_eq!(store.event_count(), 4); +} + +#[test] +fn test_update_spawning_tool_args() { + let mut store = EventStore::new(); + store.set(vec![ + make_event("msg-1", "message"), + make_task_tool_call("task-1"), + ]); + let task_names = &["task"]; + let result = store.update_spawning_tool_args( + task_names, + serde_json::json!({ + "reasoningText": "analyzing code...", + "subActivities": [{"tool": "read", "args": {}}] + }), + ); + assert_eq!(result, Some("task-1".to_string())); + let task = store.get_by_id("task-1").unwrap(); + assert_eq!(task.args["reasoningText"], "analyzing code..."); + assert_eq!(task.args["description"], "explore codebase"); +} + +#[test] +fn test_update_spawning_tool_args_none() { + let mut store = EventStore::new(); + store.set(vec![make_event("msg-1", "message")]); + let task_names = &["task"]; + let result = store.update_spawning_tool_args(task_names, serde_json::json!({"key": "value"})); + assert!(result.is_none()); +} + +#[test] +fn test_update_spawning_tool_args_multi_names() { + let mut store = EventStore::new(); + let mut session_call = make_event("session-1", "tool_call"); + session_call.function_name = "session".to_string(); + session_call.display_status = EventDisplayStatus::Running; + session_call.args = serde_json::json!({ "desc": "test" }); + store.set(vec![make_event("msg-1", "message"), session_call]); + + let names = &["task", "session", "spawn"]; + let result = store.update_spawning_tool_args(names, serde_json::json!({"subActivities": []})); + assert_eq!(result, Some("session-1".to_string())); + let updated = store.get_by_id("session-1").unwrap(); + assert_eq!(updated.args["desc"], "test"); + assert!(updated.args["subActivities"].is_array()); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/external_replay_window.rs b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/external_replay_window.rs new file mode 100644 index 000000000..431d8025e --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/external_replay_window.rs @@ -0,0 +1,372 @@ +use crate::agent_sessions::event_pipeline::store::EventStore; +use crate::agent_sessions::event_pipeline::types::*; + +use super::support::*; + +#[test] +fn test_round_window_hydration_mode() { + let mut store = EventStore::new(); + assert_eq!( + store.hydration_mode(), + crate::agent_sessions::event_pipeline::store::HydrationMode::Full + ); + + store.set_round_window(vec![make_user_turn_header( + "turn-1", + "2026-01-01T00:00:00Z", + )]); + assert_eq!( + store.hydration_mode(), + crate::agent_sessions::event_pipeline::store::HydrationMode::RoundWindow + ); + + store.merge_events(vec![make_event("live-1", "message")]); + assert_eq!( + store.hydration_mode(), + crate::agent_sessions::event_pipeline::store::HydrationMode::LivePartial + ); +} + +#[test] +fn test_empty_round_window_does_not_clobber_existing_events() { + let mut store = EventStore::new(); + store.set_round_window(vec![ + make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), + make_event("turn-1-body-1", "message"), + ]); + assert_eq!(store.events().len(), 2); + + // An empty round window (turn index mid-rebuild) must not wipe the store. + store.set_round_window(Vec::new()); + + assert_eq!(store.events().len(), 2); + assert!(store.get_by_id("turn-1").is_some()); + assert!(store.get_by_id("turn-1-body-1").is_some()); +} + +#[test] +fn test_empty_round_window_on_empty_store_is_noop_set() { + let mut store = EventStore::new(); + // Empty window on an already-empty store stays empty (no panic, no events). + store.set_round_window(Vec::new()); + assert_eq!(store.events().len(), 0); +} + +#[test] +fn external_replay_empty_window_authoritatively_clears_stale_history() { + let mut store = EventStore::new(); + store.set_external_replay_window(vec![ + make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), + make_event("turn-1-body-1", "message"), + ]); + assert_eq!(store.events().len(), 2); + + store.set_external_replay_window(Vec::new()); + + assert!(store.events().is_empty()); +} + +#[test] +fn external_replay_byte_cap_keeps_a_newest_whole_turn_suffix() { + let mut events = Vec::new(); + for turn in 0..4 { + events.push(make_user_turn_header( + &format!("turn-{turn}"), + &format!("2026-01-01T00:0{turn}:00Z"), + )); + let mut body = make_event(&format!("turn-{turn}-body"), "assistant"); + body.display_text = format!("turn {turn} {}", "x".repeat(2 * 1024)); + events.push(body); + } + let latest_two_turn_bytes = serde_json::to_vec(&events[4..]) + .expect("serialize expected suffix") + .len(); + let mut store = EventStore::new(); + store.set_round_window(events); + + let actual = store + .cap_external_replay_bytes(latest_two_turn_bytes) + .expect("cap external replay"); + + assert!(actual <= latest_two_turn_bytes); + assert!(store.get_by_id("turn-0").is_none()); + assert!(store.get_by_id("turn-1-body").is_none()); + assert!(store.get_by_id("turn-2").is_some()); + assert!(store.get_by_id("turn-3-body").is_some()); + assert_eq!( + store.events().first().map(|event| event.source.clone()), + Some(EventSource::User), + "the byte cut must not leave a detached partial turn" + ); +} + +#[test] +fn repeated_external_older_turn_merges_remain_byte_bounded() { + const STORE_BUDGET: usize = 32 * 1024; + let mut store = EventStore::new(); + for turn in 0..100 { + let mut body = make_event(&format!("turn-{turn}-body"), "assistant"); + body.display_text = format!("turn {turn} {}", "y".repeat(2 * 1024)); + let created_at = format!("2026-01-01T{:02}:{:02}:00Z", turn / 60, turn % 60); + body.created_at = created_at.clone(); + store.merge_round_window_events(vec![ + make_user_turn_header(&format!("turn-{turn}"), &created_at), + body, + ]); + let bytes = store + .cap_external_replay_bytes(STORE_BUDGET) + .expect("cap merged replay turns"); + assert!(bytes <= STORE_BUDGET); + } + assert!(store.get_by_id("turn-99").is_some()); + assert!(store.get_by_id("turn-99-body").is_some()); + assert!(store.events().len() < 100); +} + +#[test] +fn external_replay_merge_preserves_delta_baseline_while_native_round_merge_resets_it() { + let mut external_store = EventStore::new(); + external_store.set_external_replay_window(vec![make_event("latest", "assistant")]); + external_store.mark_full_snapshot_emitted(); + assert!(!external_store.should_emit_full_snapshot()); + + let mut older = make_event("older", "assistant"); + older.created_at = "2025-12-31T23:59:00Z".to_string(); + external_store.merge_external_replay_window_events(vec![older]); + + assert!( + !external_store.should_emit_full_snapshot(), + "external prepend must use the existing order-replacement delta" + ); + let (base_version, changed_ids, _) = external_store.take_delta_tracking(); + assert!(base_version > 0); + assert_eq!(changed_ids, vec!["older".to_string()]); + assert_eq!( + external_store + .events() + .iter() + .map(|event| event.id.as_str()) + .collect::>(), + vec!["older", "latest"] + ); + + let mut native_store = EventStore::new(); + native_store.set_round_window(vec![make_event("latest", "assistant")]); + native_store.mark_full_snapshot_emitted(); + native_store.merge_round_window_events(vec![make_event("older", "assistant")]); + assert!( + native_store.should_emit_full_snapshot(), + "native SDE round hydration must keep its established baseline reset" + ); +} + +#[test] +fn external_replay_resident_budget_keeps_selected_older_turn_and_prefetch_neighbours() { + const STORE_BUDGET: usize = 16 * 1024 * 1024; + const LARGE_BODY_BYTES: usize = 3 * 1024 * 1024 + 512 * 1024; + + fn large_turn(turn: u32, minute: u32, body_bytes: usize) -> Vec { + let created_at = format!("2026-01-01T00:{minute:02}:00Z"); + let mut body = make_event(&format!("turn-{turn}-body"), "assistant"); + body.created_at = created_at.clone(); + body.display_text = format!("turn {turn} {}", "z".repeat(body_bytes)); + vec![ + make_user_turn_header(&format!("turn-{turn}"), &created_at), + body, + ] + } + + let mut store = EventStore::new(); + store.set_external_replay_window(large_turn(99, 59, LARGE_BODY_BYTES)); + store + .cap_external_replay_bytes(STORE_BUDGET) + .expect("cap latest external turn"); + + // The selected historical page and its two prefetch neighbours are each + // valid <4 MiB replay windows. They must remain visible alongside the + // latest page instead of being discarded by the former 4 MiB suffix cap. + for (turn, minute) in [(10, 10), (9, 9), (11, 11)] { + store.merge_round_window_events(large_turn(turn, minute, LARGE_BODY_BYTES)); + let bytes = store + .cap_external_replay_bytes(STORE_BUDGET) + .expect("cap selected external turn window"); + assert!(bytes <= STORE_BUDGET); + } + + // A normal small live delta must not immediately evict the selected body + // while the user is reading it. + let mut poll_event = make_event("turn-99-poll", "assistant"); + poll_event.created_at = "2026-01-01T00:59:30Z".to_string(); + poll_event.display_text = "p".repeat(256 * 1024); + store.merge_round_window_events(vec![poll_event]); + let bytes = store + .cap_external_replay_bytes(STORE_BUDGET) + .expect("cap live external delta"); + + assert!(bytes <= STORE_BUDGET); + for id in [ + "turn-9", + "turn-9-body", + "turn-10", + "turn-10-body", + "turn-11", + "turn-11-body", + "turn-99", + "turn-99-body", + "turn-99-poll", + ] { + assert!(store.get_by_id(id).is_some(), "{id} must remain resident"); + } +} + +#[test] +fn external_replay_preserving_cap_keeps_each_newly_prepended_page_visible() { + const STORE_BUDGET: usize = 16 * 1024 * 1024; + const PAGE_BODY_BYTES: usize = 3 * 1024 * 1024 + 256 * 1024; + + fn large_turn(turn: u32, minute: u32) -> Vec { + let created_at = format!("2026-01-01T00:{minute:02}:00Z"); + let mut body = make_event(&format!("turn-{turn}-body"), "assistant"); + body.created_at = created_at.clone(); + body.display_text = format!("turn {turn} {}", "w".repeat(PAGE_BODY_BYTES)); + vec![ + make_user_turn_header(&format!("turn-{turn}"), &created_at), + body, + ] + } + + let mut store = EventStore::new(); + store.set_external_replay_window(large_turn(99, 59)); + store + .cap_external_replay_bytes(STORE_BUDGET) + .expect("cap latest external turn"); + + let mut previous_turn = None; + for turn in (4..=10_u32).rev() { + let page = large_turn(turn, turn); + let preserved_ids = page + .iter() + .map(|event| event.id.clone()) + .collect::>(); + store.merge_round_window_events(page); + let bytes = store + .cap_external_replay_bytes_preserving(STORE_BUDGET, &preserved_ids) + .expect("cap prepended external replay page"); + + assert!(bytes <= STORE_BUDGET); + assert!(store.get_by_id(&format!("turn-{turn}")).is_some()); + assert!(store.get_by_id(&format!("turn-{turn}-body")).is_some()); + assert!(store.get_by_id("turn-99").is_some()); + if let Some(previous_turn) = previous_turn { + assert!( + store + .get_by_id(&format!("turn-{previous_turn}-body")) + .is_some(), + "the immediately newer page must remain as the prepend anchor" + ); + } + previous_turn = Some(turn); + } + + let retained_old_pages = (4..=10_u32) + .filter(|turn| store.get_by_id(&format!("turn-{turn}-body")).is_some()) + .count(); + assert!( + retained_old_pages < 7, + "non-adjacent historical pages must still be evicted under the resident cap" + ); +} + +#[test] +fn test_unload_turn_body_restores_placeholder_and_preserves_headers() { + let mut store = EventStore::new(); + store.set_round_window(vec![ + make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), + make_event("turn-1-body-1", "message"), + make_event("turn-1-body-2", "tool_call"), + make_user_turn_header("turn-2", "2026-01-01T00:01:00Z"), + make_event("turn-2-body-1", "message"), + ]); + + let removed = store.unload_turn_body("turn-1", make_turn_placeholder("turn-1", Some("turn-2"))); + + assert_eq!(removed, 2); + assert!(store.get_by_id("turn-1").is_some()); + assert!(store.get_by_id("turn-placeholder-turn-1").is_some()); + assert!(store.get_by_id("turn-1-body-1").is_none()); + assert!(store.get_by_id("turn-1-body-2").is_none()); + assert!(store.get_by_id("turn-2").is_some()); + assert!(store.get_by_id("turn-2-body-1").is_some()); + assert_eq!( + store.hydration_mode(), + crate::agent_sessions::event_pipeline::store::HydrationMode::RoundWindow + ); +} + +#[test] +fn test_unload_turn_body_preserves_final_reply_as_preview() { + let mut final_reply = make_event("turn-1-final-reply", "assistant"); + final_reply.function_name = "assistant".to_string(); + final_reply.ui_canonical = "agent_message".to_string(); + final_reply.display_variant = EventDisplayVariant::Message; + final_reply.display_text = "Finished the work".to_string(); + + let mut store = EventStore::new(); + store.set_round_window(vec![ + make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), + make_event("turn-1-tool", "tool_call"), + final_reply, + make_user_turn_header("turn-2", "2026-01-01T00:01:00Z"), + ]); + + let removed = store.unload_turn_body("turn-1", make_turn_placeholder("turn-1", Some("turn-2"))); + + assert_eq!(removed, 1); + let preview = store.get_by_id("turn-1-final-reply").unwrap(); + assert_eq!( + preview.args.get("turnPreviewOnly"), + Some(&serde_json::Value::Bool(true)) + ); + assert!(store.get_by_id("turn-1-tool").is_none()); + assert!(store.get_by_id("turn-placeholder-turn-1").is_some()); +} + +#[test] +fn test_merge_round_window_events_removes_loaded_turn_placeholder() { + let mut store = EventStore::new(); + store.set_round_window(vec![ + make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), + make_turn_placeholder("turn-1", Some("turn-2")), + make_user_turn_header("turn-2", "2026-01-01T00:01:00Z"), + ]); + + let mut body_1 = make_event("turn-1-body-1", "message"); + body_1.created_at = "2026-01-01T00:00:20Z".to_string(); + let mut body_2 = make_event("turn-1-body-2", "tool_call"); + body_2.created_at = "2026-01-01T00:00:40Z".to_string(); + + store.merge_round_window_events(vec![ + make_user_turn_header("turn-1", "2026-01-01T00:00:00Z"), + body_1, + body_2, + ]); + + assert!(store.get_by_id("turn-placeholder-turn-1").is_none()); + assert!(store.get_by_id("turn-1").is_some()); + assert!(store.get_by_id("turn-1-body-1").is_some()); + assert!(store.get_by_id("turn-1-body-2").is_some()); + assert!(store.get_by_id("turn-2").is_some()); + let event_ids = store + .events() + .iter() + .map(|event| event.id.as_str()) + .collect::>(); + assert_eq!( + event_ids, + vec!["turn-1", "turn-1-body-1", "turn-1-body-2", "turn-2"] + ); + assert_eq!( + store.hydration_mode(), + crate::agent_sessions::event_pipeline::store::HydrationMode::RoundWindow + ); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/lifecycle.rs b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/lifecycle.rs new file mode 100644 index 000000000..de01b71c3 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/lifecycle.rs @@ -0,0 +1,109 @@ +use crate::agent_sessions::event_pipeline::store::EventStore; +use crate::agent_sessions::event_pipeline::types::*; + +use super::support::*; + +#[test] +fn test_find_last_spawning_tool() { + let mut store = EventStore::new(); + store.set(vec![ + make_task_tool_call("task-1"), + make_event("msg-1", "message"), + ]); + assert_eq!(store.find_last_spawning_tool(&["task"]), Some(0)); +} + +#[test] +fn test_find_last_spawning_tool_none() { + let mut store = EventStore::new(); + store.set(vec![make_event("msg-1", "message")]); + assert!(store.find_last_spawning_tool(&["task"]).is_none()); +} + +#[test] +fn test_find_last_spawning_tool_stops_at_result() { + let mut store = EventStore::new(); + let mut task_call = make_task_tool_call("task-1"); + task_call.action_type = "tool_call".to_string(); + let mut task_result = make_event("task-r", "tool_result"); + task_result.function_name = "task".to_string(); + store.set(vec![task_call, task_result, make_event("msg-1", "message")]); + assert!(store.find_last_spawning_tool(&["task"]).is_none()); +} + +#[test] +fn test_has_active_spawning_tool() { + let mut store = EventStore::new(); + store.set(vec![make_task_tool_call("task-1")]); + assert!(store.has_active_spawning_tool(&["task"])); + assert!(!store.has_active_spawning_tool(&["session"])); +} + +// ============================================================================ +// cancel_orphan_interactive_events tests +// ============================================================================ + +#[test] +fn test_cancel_orphan_interactive_events_cancels_awaiting_user() { + let mut store = EventStore::new(); + let mut orphan = make_tool_call("ask-1", "call-ask-1"); + orphan.display_status = EventDisplayStatus::AwaitingUser; + store.set(vec![make_event("msg-1", "message"), orphan]); + + let cancelled = store.cancel_orphan_interactive_events(); + + assert_eq!(cancelled, vec!["ask-1".to_string()]); + let event = store.get_by_id("ask-1").unwrap(); + assert_eq!(event.display_status, EventDisplayStatus::Completed); + assert_eq!(event.result["status"], "cancelled"); +} + +#[test] +fn test_cancel_orphan_interactive_events_leaves_running_untouched() { + let mut store = EventStore::new(); + let running = make_tool_call("run-1", "call-run-1"); + store.set(vec![running]); + + let cancelled = store.cancel_orphan_interactive_events(); + + assert!(cancelled.is_empty()); + let event = store.get_by_id("run-1").unwrap(); + assert_eq!(event.display_status, EventDisplayStatus::Running); +} + +#[test] +fn test_cancel_orphan_interactive_events_mixed() { + let mut store = EventStore::new(); + let running = make_tool_call("run-1", "call-run-1"); + let mut awaiting1 = make_tool_call("ask-1", "call-ask-1"); + awaiting1.display_status = EventDisplayStatus::AwaitingUser; + let mut awaiting2 = make_tool_call("ask-2", "call-ask-2"); + awaiting2.display_status = EventDisplayStatus::AwaitingUser; + // A pre-completed event (not AwaitingUser, not Running). + let mut already_done = make_event("done-1", "tool_call"); + already_done.display_status = EventDisplayStatus::Completed; + store.set(vec![running, awaiting1, awaiting2, already_done]); + + let cancelled = store.cancel_orphan_interactive_events(); + + assert_eq!(cancelled.len(), 2); + assert!(cancelled.contains(&"ask-1".to_string())); + assert!(cancelled.contains(&"ask-2".to_string())); + // running stays Running + assert_eq!( + store.get_by_id("run-1").unwrap().display_status, + EventDisplayStatus::Running + ); + // pre-completed stays Completed with original empty result + assert_eq!( + store.get_by_id("done-1").unwrap().display_status, + EventDisplayStatus::Completed + ); + assert!(store + .get_by_id("done-1") + .unwrap() + .result + .as_object() + .unwrap() + .is_empty()); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/shell_replay.rs b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/shell_replay.rs new file mode 100644 index 000000000..589e4bab3 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/shell_replay.rs @@ -0,0 +1,305 @@ +use crate::agent_sessions::event_pipeline::store::{capture_shell_replay_bookmarks, EventStore}; +use crate::agent_sessions::event_pipeline::types::*; + +use super::support::*; + +#[test] +fn test_shell_replay_exact_update_is_monotonic_and_seed_is_immutable() { + let mut shell = make_shell_tool_call("shell-1"); + shell.shell_replay_bookmarks = Some(Default::default()); + let mut sibling = shell.clone(); + sibling.id = "shell-1-sibling".to_string(); + let mut store = EventStore::new(); + store.set(vec![shell, sibling]); + + let initial = replay_state("call-shell-1", 1, 100, ShellReplayStatus::Running); + assert_eq!( + store.update_shell_replay_by_call_id("call-shell-1", initial.clone(), true), + Some("shell-1-sibling".to_string()) + ); + let latest = replay_state("call-shell-1", 2, 200, ShellReplayStatus::Running); + store.update_shell_replay_by_call_id("call-shell-1", latest.clone(), false); + store.update_shell_replay_by_call_id("call-shell-1", initial.clone(), true); + + for id in ["shell-1", "shell-1-sibling"] { + let event = store.get_by_id(id).unwrap(); + assert_eq!(event.shell_replay, Some(latest.clone())); + assert_eq!( + event + .shell_replay_bookmarks + .as_ref() + .and_then(|bookmarks| bookmarks.get("call-shell-1")), + Some(&initial) + ); + } +} + +#[test] +fn test_shell_replay_terminal_state_cannot_regress_to_running() { + let mut shell = make_shell_tool_call("shell-1"); + shell.shell_replay_bookmarks = Some(Default::default()); + let mut store = EventStore::new(); + store.set(vec![shell]); + + let mut complete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Complete); + complete.completed_at = Some("2026-01-01T00:01:00Z".to_string()); + store.update_shell_replay_by_call_id("call-shell-1", complete.clone(), true); + store.update_shell_replay_by_call_id( + "call-shell-1", + replay_state("call-shell-1", 4, 400, ShellReplayStatus::Running), + false, + ); + + assert_eq!( + store.get_by_id("shell-1").unwrap().shell_replay, + Some(complete) + ); +} + +#[test] +fn test_shell_replay_complete_can_be_corrected_to_incomplete() { + let mut shell = make_shell_tool_call("shell-1"); + shell.shell_replay_bookmarks = Some(Default::default()); + let mut store = EventStore::new(); + store.set(vec![shell]); + + let mut complete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Complete); + complete.completed_at = Some("2026-01-01T00:01:00Z".to_string()); + store.update_shell_replay_by_call_id("call-shell-1", complete, true); + + let mut incomplete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Incomplete); + incomplete.error = Some("final persistence barrier failed".to_string()); + store.update_shell_replay_by_call_id("call-shell-1", incomplete.clone(), false); + + assert_eq!( + store.get_by_id("shell-1").unwrap().shell_replay, + Some(incomplete) + ); +} + +#[test] +fn test_shell_replay_incomplete_can_correct_an_optimistic_higher_watermark() { + let mut shell = make_shell_tool_call("shell-1"); + shell.shell_replay_bookmarks = Some(Default::default()); + let mut store = EventStore::new(); + store.set(vec![shell]); + + store.update_shell_replay_by_call_id( + "call-shell-1", + replay_state("call-shell-1", 9, 900, ShellReplayStatus::Complete), + true, + ); + let mut recovered = replay_state("call-shell-1", 8, 800, ShellReplayStatus::Incomplete); + recovered.error = Some("torn final frame removed during recovery".to_string()); + store.update_shell_replay_by_call_id("call-shell-1", recovered.clone(), false); + + assert_eq!( + store.get_by_id("shell-1").unwrap().shell_replay, + Some(recovered) + ); +} + +#[test] +fn test_shell_replay_incomplete_cannot_be_overwritten_by_complete() { + let mut shell = make_shell_tool_call("shell-1"); + shell.shell_replay_bookmarks = Some(Default::default()); + let mut store = EventStore::new(); + store.set(vec![shell]); + + let mut incomplete = replay_state("call-shell-1", 3, 300, ShellReplayStatus::Incomplete); + incomplete.error = Some("disk full".to_string()); + store.update_shell_replay_by_call_id("call-shell-1", incomplete.clone(), true); + store.update_shell_replay_by_call_id( + "call-shell-1", + replay_state("call-shell-1", 4, 400, ShellReplayStatus::Complete), + false, + ); + + assert_eq!( + store.get_by_id("shell-1").unwrap().shell_replay, + Some(incomplete) + ); +} + +#[test] +fn test_shell_replay_update_requires_exact_session_and_call() { + let shell = make_shell_tool_call("shell-1"); + let mut store = EventStore::new(); + store.set(vec![shell]); + + assert!(store + .update_shell_replay_by_call_id( + "different-call", + replay_state("call-shell-1", 1, 100, ShellReplayStatus::Running), + true, + ) + .is_none()); + + let mut wrong_session = replay_state("call-shell-1", 1, 100, ShellReplayStatus::Running); + wrong_session.replay_ref.session_id = "different-session".to_string(); + assert!(store + .update_shell_replay_by_call_id("call-shell-1", wrong_session, true) + .is_none()); + assert!(store.get_by_id("shell-1").unwrap().shell_replay.is_none()); +} + +#[test] +fn test_same_id_upsert_preserves_first_insert_bookmarks() { + let initial = replay_state("other-call", 5, 500, ShellReplayStatus::Running); + let mut first = make_event("timeline-1", "message"); + first.shell_replay_bookmarks = Some(std::collections::HashMap::from([( + "other-call".to_string(), + initial.clone(), + )])); + let mut store = EventStore::new(); + store.set(vec![first]); + + let future = replay_state("other-call", 99, 9_900, ShellReplayStatus::Complete); + let mut update = make_event("timeline-1", "message"); + update.display_text = "updated".to_string(); + update.shell_replay_bookmarks = Some(std::collections::HashMap::from([( + "other-call".to_string(), + future, + )])); + store.upsert(update); + + let mut merge_update = make_event("timeline-1", "message"); + merge_update.display_text = "merged".to_string(); + merge_update.shell_replay_bookmarks = Some(std::collections::HashMap::from([( + "other-call".to_string(), + replay_state("other-call", 100, 10_000, ShellReplayStatus::Complete), + )])); + store.merge_events(vec![merge_update]); + + assert_eq!( + store + .get_by_id("timeline-1") + .unwrap() + .shell_replay_bookmarks + .as_ref() + .and_then(|bookmarks| bookmarks.get("other-call")), + Some(&initial) + ); +} + +#[test] +fn test_first_insert_bookmark_winner_fills_only_missing_active_calls() { + let first = replay_state("call-a", 1, 100, ShellReplayStatus::Running); + let future = replay_state("call-a", 9, 900, ShellReplayStatus::Running); + let active_b = replay_state("call-b", 2, 200, ShellReplayStatus::Running); + let mut event = make_event("timeline-1", "message"); + event.shell_replay_bookmarks = Some(std::collections::HashMap::from([( + "call-a".to_string(), + first.clone(), + )])); + + capture_shell_replay_bookmarks( + &mut event, + &std::collections::HashMap::from([ + ("call-a".to_string(), future), + ("call-b".to_string(), active_b.clone()), + ]), + ); + + let bookmarks = event.shell_replay_bookmarks.unwrap(); + assert_eq!(bookmarks.get("call-a"), Some(&first)); + assert_eq!(bookmarks.get("call-b"), Some(&active_b)); +} + +#[test] +fn test_live_shell_event_keeps_only_bounded_replay_payload() { + let mut shell = make_shell_tool_call("shell-1"); + shell.args["streamOutput"] = serde_json::Value::String("duplicate".repeat(20_000)); + shell.result = serde_json::json!({ + "content": "duplicate".repeat(20_000), + "observation": "duplicate".repeat(20_000) + }); + let mut state = replay_state("call-shell-1", 1, 80_000, ShellReplayStatus::Running); + state.terminal_preview = "中".repeat(20_000); + shell.shell_replay = Some(state); + let mut store = EventStore::new(); + store.upsert(shell); + + let stored = store.get_by_id("shell-1").unwrap(); + assert!(stored.args.get("streamOutput").is_none()); + assert_eq!(stored.result, serde_json::json!({})); + assert!(stored.shell_replay.as_ref().unwrap().terminal_preview.len() <= 32 * 1024); +} + +#[test] +fn test_live_external_shell_without_replay_never_becomes_an_empty_card() { + let mut shell = make_shell_tool_call("external-shell-no-replay"); + shell.display_status = EventDisplayStatus::Completed; + shell.shell_replay = None; + shell.args["streamOutput"] = serde_json::Value::String(String::new()); + shell.result = serde_json::json!({ + "stdout": format!("{}EXTERNAL-TAIL", "x".repeat(80_000)), + "exit_code": 0 + }); + + let mut store = EventStore::new(); + store.upsert(shell); + + let stored = store.get_by_id("external-shell-no-replay").unwrap(); + assert_eq!(stored.result, serde_json::json!({})); + let replay = stored + .shell_replay + .as_ref() + .expect("bounded external fallback preview"); + assert_eq!(replay.status, ShellReplayStatus::Incomplete); + assert_eq!(replay.bookmark, ShellReplayBookmark::default()); + assert!(replay.terminal_preview.len() <= 32 * 1024); + assert!(replay.terminal_preview.ends_with("EXTERNAL-TAIL")); + assert!(replay + .error + .as_deref() + .is_some_and(|error| error.contains("仅显示有界预览"))); +} + +#[test] +fn test_running_external_shell_without_replay_keeps_bounded_stream_preview() { + let mut shell = make_shell_tool_call("external-shell-running"); + shell.shell_replay = None; + shell.args["streamOutput"] = + serde_json::Value::String(format!("{}RUNNING-TAIL", "x".repeat(80_000))); + + let mut store = EventStore::new(); + store.upsert(shell); + + let stored = store.get_by_id("external-shell-running").unwrap(); + assert!(stored.shell_replay.is_none()); + let preview = stored.args["streamOutput"].as_str().unwrap(); + assert!(preview.len() <= 32 * 1024); + assert!(preview.ends_with("RUNNING-TAIL")); +} + +#[test] +fn test_hydration_converts_legacy_shell_output_to_bounded_incomplete_preview() { + let mut shell = make_shell_tool_call("legacy-shell"); + shell.args["streamOutput"] = serde_json::Value::String("old-stream".repeat(10_000)); + shell.result = serde_json::json!({ + "output": { + "success": { + "stdout": format!("{}TAIL-SENTINEL", "x".repeat(80_000)), + "exitCode": 7 + } + } + }); + shell.shell_replay = None; + shell.shell_replay_bookmarks = None; + + let mut store = EventStore::new(); + store.set(vec![shell]); + + let stored = store.get_by_id("legacy-shell").unwrap(); + assert_eq!(stored.result, serde_json::json!({})); + assert!(stored.args.get("streamOutput").is_none()); + assert_eq!(stored.args["shellExitCode"], 7); + assert!(stored.shell_replay_bookmarks.is_none()); + let replay = stored.shell_replay.as_ref().unwrap(); + assert_eq!(replay.status, ShellReplayStatus::Incomplete); + assert!(replay.completed_at.is_none()); + assert!(replay.terminal_preview.len() <= 32 * 1024); + assert!(replay.terminal_preview.ends_with("TAIL-SENTINEL")); + assert_eq!(replay.bookmark, ShellReplayBookmark::default()); +} diff --git a/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/support.rs b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/support.rs new file mode 100644 index 000000000..8ad242d36 --- /dev/null +++ b/src-tauri/src/agent_sessions/event_pipeline/tests/store_tests/support.rs @@ -0,0 +1,126 @@ +use crate::agent_sessions::event_pipeline::types::*; + +pub(super) fn make_event(id: &str, action_type: &str) -> SessionEvent { + SessionEvent { + id: id.to_string(), + chunk_id: Some(id.to_string()), + session_id: "test-session".to_string(), + created_at: "2026-01-01T00:00:00Z".to_string(), + function_name: "test".to_string(), + ui_canonical: "test".to_string(), + action_type: action_type.to_string(), + args: serde_json::json!({}), + result: serde_json::json!({}), + source: EventSource::Assistant, + display_text: "test".to_string(), + display_status: EventDisplayStatus::Completed, + display_variant: EventDisplayVariant::ToolCall, + activity_status: ActivityStatus::Agent, + thread_id: None, + process_id: None, + call_id: None, + file_path: None, + command: None, + is_delta: None, + repo_id: None, + repo_path: None, + extracted: None, + payload_refs: Vec::new(), + shell_replay: None, + shell_replay_bookmarks: None, + last_extract_at: None, + } +} + +pub(super) fn make_tool_call(id: &str, call_id: &str) -> SessionEvent { + let mut event = make_event(id, "tool_call"); + event.call_id = Some(call_id.to_string()); + event.display_status = EventDisplayStatus::Running; + event.args = serde_json::json!({ "command": "ls", "streamOutput": "..." }); + event +} + +pub(super) fn make_tool_result(id: &str, call_id: &str) -> SessionEvent { + let mut event = make_event(id, "tool_result"); + event.call_id = Some(call_id.to_string()); + event.result = serde_json::json!({ "content": "file1.txt\nfile2.txt" }); + event +} + +pub(super) fn make_running_event(id: &str) -> SessionEvent { + let mut event = make_event(id, "message"); + event.display_status = EventDisplayStatus::Running; + event +} + +pub(super) fn make_task_tool_call(id: &str) -> SessionEvent { + let mut event = make_event(id, "tool_call"); + event.function_name = "task".to_string(); + event.display_status = EventDisplayStatus::Running; + event.args = serde_json::json!({ "description": "explore codebase" }); + event +} + +pub(super) fn make_shell_tool_call(id: &str) -> SessionEvent { + let mut event = make_event(id, "tool_call"); + event.function_name = "run_shell".to_string(); + event.ui_canonical = "run_shell".to_string(); + event.call_id = Some(format!("call-{id}")); + event.display_status = EventDisplayStatus::Running; + event.args = serde_json::json!({ "command": "ls" }); + event +} + +pub(super) fn make_awaiting_user_event(id: &str) -> SessionEvent { + let mut event = make_event(id, "tool_call"); + event.function_name = "ask_user_questions".to_string(); + event.display_status = EventDisplayStatus::AwaitingUser; + event +} + +pub(super) fn replay_state( + call_id: &str, + sequence: u64, + visible_bytes: u64, + status: ShellReplayStatus, +) -> ShellReplayState { + ShellReplayState { + replay_ref: ShellReplayRef { + session_id: "test-session".to_string(), + call_id: call_id.to_string(), + format_version: 1, + }, + bookmark: ShellReplayBookmark { + visible_through_sequence: sequence, + visible_bytes, + }, + terminal_preview: format!("preview-{sequence}"), + status, + error: None, + completed_at: None, + } +} + +pub(super) fn make_user_turn_header(turn_id: &str, created_at: &str) -> SessionEvent { + let mut event = make_event(turn_id, "raw"); + event.function_name = "user_message".to_string(); + event.ui_canonical = "user_message".to_string(); + event.source = EventSource::User; + event.display_variant = EventDisplayVariant::Message; + event.created_at = created_at.to_string(); + event +} + +pub(super) fn make_turn_placeholder(turn_id: &str, next_turn_id: Option<&str>) -> SessionEvent { + let mut event = make_event(&format!("turn-placeholder-{turn_id}"), "turn_placeholder"); + event.function_name = "turn_placeholder".to_string(); + event.ui_canonical = "turn_placeholder".to_string(); + event.result = serde_json::json!({ + "unloadedTurn": { + "turnId": turn_id, + "bodyEventCount": 2, + "nextTurnId": next_turn_id, + } + }); + event +} diff --git a/src-tauri/src/agent_sessions/external_cli_adapter/mod.rs b/src-tauri/src/agent_sessions/external_cli_adapter/mod.rs index d46cc7c2a..a62d85229 100644 --- a/src-tauri/src/agent_sessions/external_cli_adapter/mod.rs +++ b/src-tauri/src/agent_sessions/external_cli_adapter/mod.rs @@ -1,5 +1,3 @@ -use crate::agent_sessions::event_pipeline::types::SessionEvent; - pub mod opencode; pub trait ExternalCliAdapter: Send + Sync { @@ -9,7 +7,6 @@ pub trait ExternalCliAdapter: Send + Sync { fn matches_imported_session(&self, session_id: &str) -> bool; fn imported_session_id_from_native(&self, native_session_id: &str) -> String; fn native_session_id_from_imported<'a>(&self, imported_session_id: &'a str) -> Option<&'a str>; - fn load_history_events(&self, imported_session_id: &str) -> Result, String>; fn resolve_subagent_prompt(&self, child_session_id: &str) -> Option; fn imported_parent_session_id( &self, diff --git a/src-tauri/src/agent_sessions/external_cli_adapter/opencode.rs b/src-tauri/src/agent_sessions/external_cli_adapter/opencode.rs index a9181a9e8..61e750b09 100644 --- a/src-tauri/src/agent_sessions/external_cli_adapter/opencode.rs +++ b/src-tauri/src/agent_sessions/external_cli_adapter/opencode.rs @@ -1,12 +1,10 @@ -use core_types::activity::ActivityChunk; use database::db::get_connection; -use orgtrack_core::sources::opencode::history as opencode_history; +use orgtrack_core::sources::imported_history::replay::{ + self, ImportedHistorySourceId, ReplayLimits, +}; use rusqlite::{Connection, OptionalExtension}; use crate::agent_sessions::event_pipeline::ingestion::prompt_backfill; -use crate::agent_sessions::event_pipeline::types::{ - ActivityStatus, EventDisplayStatus, EventDisplayVariant, EventSource, SessionEvent, -}; use crate::agent_sessions::external_cli_adapter::ExternalCliAdapter; pub const OPENCODE_SOURCE: &str = "opencode"; @@ -18,6 +16,29 @@ pub static OPENCODE_ADAPTER: OpenCodeAdapter = OpenCodeAdapter; pub struct OpenCodeAdapter; impl OpenCodeAdapter { + fn resolve_subagent_prompt_from_replay(&self, child_session_id: &str) -> Option { + let _writer = database::db::sessions_writer_guard(); + let mut conn = get_connection().ok()?; + let batch = replay::scan_window_after( + &mut conn, + ImportedHistorySourceId::OpenCode, + child_session_id, + -1, + ReplayLimits { + max_turns: 1, + max_events: 32, + max_ipc_bytes: 256 * 1024, + }, + ) + .ok()?; + let chunks = batch + .chunks + .into_iter() + .map(|indexed| indexed.chunk) + .collect::>(); + prompt_backfill::prompt_from_history_chunks(&chunks) + } + fn resolve_subagent_prompt_from_conn( &self, conn: &Connection, @@ -98,19 +119,12 @@ impl ExternalCliAdapter for OpenCodeAdapter { imported_session_id.strip_prefix(self.imported_session_prefix()) } - fn load_history_events(&self, imported_session_id: &str) -> Result, String> { - let chunks = opencode_history::load_opencode_history_for_session(imported_session_id)?; - Ok(chunks.iter().map(activity_chunk_to_session_event).collect()) - } - fn resolve_subagent_prompt(&self, child_session_id: &str) -> Option { if !self.matches_imported_session(child_session_id) { return None; } - if let Ok(chunks) = opencode_history::load_opencode_history_for_session(child_session_id) { - if let Some(prompt) = prompt_backfill::prompt_from_history_chunks(&chunks) { - return Some(prompt); - } + if let Some(prompt) = self.resolve_subagent_prompt_from_replay(child_session_id) { + return Some(prompt); } let conn = get_connection().ok()?; self.resolve_subagent_prompt_from_conn(&conn, child_session_id) @@ -129,51 +143,6 @@ impl ExternalCliAdapter for OpenCodeAdapter { } } -fn activity_chunk_to_session_event(chunk: &ActivityChunk) -> SessionEvent { - let function_name = if chunk.function.is_empty() { - chunk.action_type.clone() - } else { - chunk.function.clone() - }; - SessionEvent { - id: if chunk.chunk_id.is_empty() { - uuid::Uuid::new_v4().to_string() - } else { - chunk.chunk_id.clone() - }, - chunk_id: if chunk.chunk_id.is_empty() { - None - } else { - Some(chunk.chunk_id.clone()) - }, - session_id: chunk.session_id.clone(), - created_at: chunk.created_at.clone(), - function_name: function_name.clone(), - ui_canonical: function_name, - action_type: chunk.action_type.clone(), - args: chunk.args.clone(), - result: chunk.result.clone(), - source: EventSource::Assistant, - display_text: format!("{}: {}", chunk.action_type, chunk.function), - display_status: EventDisplayStatus::Completed, - display_variant: EventDisplayVariant::ToolCall, - activity_status: ActivityStatus::Processed, - thread_id: chunk.thread_id.clone(), - process_id: chunk.process_id.clone(), - call_id: None, - file_path: None, - command: None, - is_delta: None, - repo_id: None, - repo_path: None, - extracted: None, - payload_refs: Vec::new(), - shell_replay: None, - shell_replay_bookmarks: None, - last_extract_at: None, - } -} - #[cfg(test)] mod tests { use super::*; diff --git a/src-tauri/src/agent_sessions/session_directory/agent_org_annotations.rs b/src-tauri/src/agent_sessions/session_directory/agent_org_annotations.rs new file mode 100644 index 000000000..56f7d971f --- /dev/null +++ b/src-tauri/src/agent_sessions/session_directory/agent_org_annotations.rs @@ -0,0 +1,85 @@ +//! Bounded Agent Org metadata projection for already-selected sidebar rows. +//! +//! Session listing must never enumerate every historical org run just to +//! decorate a 10–50 row page. This module performs one exact `IN` query for +//! the root session IDs that survived the page query. + +use std::collections::HashMap; + +use agent_core::definitions::orgs::OrgDefinition; +use database::db::get_connection; +use rusqlite::{params_from_iter, types::Value as SqlValue}; + +use super::types::SessionAggregateRecord; + +const AGENT_ORG_ICON_ID: &str = "network"; + +fn agent_org_display_name(org_id: &str, org_snapshot_json: Option<&str>) -> String { + org_snapshot_json + .and_then(|json| serde_json::from_str::(json).ok()) + .map(|org| org.name) + .unwrap_or_else(|| org_id.to_string()) +} + +pub(super) fn annotate_agent_org_root_rows( + sessions: &mut [SessionAggregateRecord], +) -> Result<(), String> { + if sessions.is_empty() { + return Ok(()); + } + let session_ids = sessions + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(); + let placeholders = (1..=session_ids.len()) + .map(|index| format!("?{index}")) + .collect::>() + .join(","); + let sql = format!( + "SELECT root_session_id, org_id, org_snapshot_json + FROM agent_org_runs + WHERE root_session_id IN ({placeholders}) + ORDER BY updated_at DESC" + ); + let conn = + get_connection().map_err(|error| format!("Failed to open Agent Org database: {error}"))?; + let mut statement = conn + .prepare(&sql) + .map_err(|error| format!("Failed to prepare Agent Org sidebar annotation: {error}"))?; + let rows = statement + .query_map( + params_from_iter( + session_ids + .into_iter() + .map(|session_id| SqlValue::from(session_id.to_string())), + ), + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, Option>(2)?, + )) + }, + ) + .map_err(|error| format!("Failed to query Agent Org sidebar annotation: {error}"))?; + + // Preserve the historical list-runs/collect precedence if malformed old + // data contains multiple runs for one root: older rows overwrite newer + // rows because the query is ordered newest first. + let mut annotations = HashMap::with_capacity(sessions.len()); + for row in rows { + let (session_id, org_id, snapshot) = + row.map_err(|error| format!("Failed to read Agent Org sidebar annotation: {error}"))?; + let org_name = agent_org_display_name(&org_id, snapshot.as_deref()); + annotations.insert(session_id, (org_id, org_name)); + } + + for session in sessions { + if let Some((org_id, org_name)) = annotations.get(&session.session_id) { + session.agent_icon_id = Some(AGENT_ORG_ICON_ID.to_string()); + session.agent_org_id = Some(org_id.clone()); + session.agent_org_name = Some(org_name.clone()); + } + } + Ok(()) +} diff --git a/src-tauri/src/agent_sessions/session_directory/aggregation.rs b/src-tauri/src/agent_sessions/session_directory/aggregation.rs index 304c8b520..2934dc0c6 100644 --- a/src-tauri/src/agent_sessions/session_directory/aggregation.rs +++ b/src-tauri/src/agent_sessions/session_directory/aggregation.rs @@ -8,339 +8,41 @@ use std::collections::HashSet; use crate::agent_sessions::cli::persistence as cli_session_persistence; -use agent_core::coordination::agent_org_runs::{AgentOrgRunRecord, AgentOrgRunStore}; -use agent_core::definitions::orgs::OrgDefinition; use agent_core::session::persistence::{self as session_persistence, session_type}; use chrono::DateTime; use core_types::key_source::KeySource; use database::db::get_connection; -use orgtrack_core::sources::claude_code::history as claude_code_history; -use orgtrack_core::sources::cline::history as cline_history; -use orgtrack_core::sources::codex::app as codex_app_history; -use orgtrack_core::sources::cursor_cli::history as cursor_cli_history; -use orgtrack_core::sources::cursor_ide::history as cursor_ide_history; -use orgtrack_core::sources::cursor_ide::history::CursorIdeSessionPage; use orgtrack_core::sources::imported_history::cache as imported_history_cache; -use orgtrack_core::sources::imported_history::metadata::{ - SOURCE_CLAUDE_CODE, SOURCE_CLINE, SOURCE_CODEX_APP, SOURCE_CURSOR_CLI, SOURCE_CURSOR_IDE, - SOURCE_MIMO_CODE, SOURCE_OMP, SOURCE_OPENCODE, SOURCE_QODER, SOURCE_QODER_CLI, SOURCE_TRAE, - SOURCE_WARP, SOURCE_WINDSURF, SOURCE_WORKBUDDY, SOURCE_ZCODE, -}; -use orgtrack_core::sources::imported_history::ImportedHistorySessionPage; -use orgtrack_core::sources::imported_history::IMPORTED_STATUS_COMPLETED; -use orgtrack_core::sources::mimo_code::history as mimo_code_history; -use orgtrack_core::sources::omp::history as omp_history; -use orgtrack_core::sources::opencode::history as opencode_history; -use orgtrack_core::sources::qoder::history as qoder_history; -use orgtrack_core::sources::qoder_cli::history as qoder_cli_history; -use orgtrack_core::sources::trae::history as trae_history; -use orgtrack_core::sources::warp::history as warp_history; -use orgtrack_core::sources::windsurf::history as windsurf_history; -use orgtrack_core::sources::workbuddy as workbuddy_history; -use orgtrack_core::sources::zcode::history as zcode_history; - -const AGENT_ORG_ICON_ID: &str = "network"; +use orgtrack_core::sources::imported_history::catalog as imported_history_catalog; +use orgtrack_core::sources::imported_history::replay::ImportedHistorySourceId; +use super::agent_org_annotations::annotate_agent_org_root_rows; use super::conversion::{ - cli_session_to_aggregate_record, cursor_ide_history_to_aggregate_record, - human_session_to_aggregate_record, imported_history_to_aggregate_record, - os_session_to_aggregate_record, sde_session_to_aggregate_record, AgentMetadataResolver, + cli_session_to_aggregate_record, human_session_to_aggregate_record, + imported_history_to_aggregate_record, os_session_to_aggregate_record, + sde_session_to_aggregate_record, AgentMetadataResolver, }; use super::display::matches_text_query; +use super::sidebar_discovery::bounded_search_or_pinned_page; +use super::sidebar_queries::{self, CliPageRequest, RustAgentGroupPageRequest, SidebarSeekCursor}; +use super::status::decorate_imported_live_status; use super::types::{SessionAggregateRecord, SessionFilter, SessionListResponse}; const IMPORTED_HISTORY_PAGE_SIZE: usize = 500; -enum ExternalHistoryPage { - Imported(ImportedHistorySessionPage), - CursorIde(CursorIdeSessionPage), -} - -struct ExternalHistorySourceLoader { - source: &'static str, - load_page: fn(&mut rusqlite::Connection, usize, usize) -> Result, -} - -fn load_claude_code_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - claude_code_history::list_claude_code_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_codex_app_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - codex_app_history::list_codex_app_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_cursor_ide_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - cursor_ide_history::list_cursor_ide_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::CursorIde) -} - -fn load_cursor_cli_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - cursor_cli_history::list_cursor_cli_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_opencode_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - opencode_history::list_opencode_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_windsurf_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - windsurf_history::list_windsurf_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_workbuddy_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - workbuddy_history::list_workbuddy_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_trae_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - trae_history::list_trae_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_cline_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - cline_history::list_cline_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_warp_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - warp_history::list_warp_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_zcode_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - zcode_history::list_zcode_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_qoder_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - qoder_history::list_qoder_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_mimo_code_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - mimo_code_history::list_mimo_code_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_omp_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - omp_history::list_omp_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -fn load_qoder_cli_external_history_page( - conn: &mut rusqlite::Connection, - limit: usize, - offset: usize, -) -> Result { - qoder_cli_history::list_qoder_cli_history_sessions_paginated(conn, limit, offset) - .map(ExternalHistoryPage::Imported) -} - -const EXTERNAL_HISTORY_SOURCE_LOADERS: &[ExternalHistorySourceLoader] = &[ - ExternalHistorySourceLoader { - source: SOURCE_CLAUDE_CODE, - load_page: load_claude_code_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_CODEX_APP, - load_page: load_codex_app_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_CURSOR_IDE, - load_page: load_cursor_ide_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_CURSOR_CLI, - load_page: load_cursor_cli_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_OPENCODE, - load_page: load_opencode_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_WINDSURF, - load_page: load_windsurf_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_WORKBUDDY, - load_page: load_workbuddy_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_TRAE, - load_page: load_trae_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_CLINE, - load_page: load_cline_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_WARP, - load_page: load_warp_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_ZCODE, - load_page: load_zcode_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_QODER, - load_page: load_qoder_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_MIMO_CODE, - load_page: load_mimo_code_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_OMP, - load_page: load_omp_external_history_page, - }, - ExternalHistorySourceLoader { - source: SOURCE_QODER_CLI, - load_page: load_qoder_cli_external_history_page, - }, -]; - -/// Discover a source's current records and incrementally re-sync its metadata -/// cache, discarding the returned page. This runs the exact sync the -/// sidebar/list path performs (re-parsing every record whose signature changed, -/// e.g. after a parser-version bump), so a manual update can refresh counts and -/// names immediately instead of waiting for a lazy list load. +/// Refresh only the compact imported-history catalog. Transcript hydration is +/// deliberately not part of the directory/listing contract. pub fn resync_external_history_source( conn: &mut rusqlite::Connection, source: &str, -) -> Result { - let loader = EXTERNAL_HISTORY_SOURCE_LOADERS - .iter() - .find(|loader| loader.source == source) - .ok_or_else(|| format!("Unknown external history source: {source}"))?; - let changes_before = conn.total_changes(); - (loader.load_page)(conn, IMPORTED_HISTORY_PAGE_SIZE, 0)?; - Ok(conn.total_changes() > changes_before) -} - -fn append_external_history_page( - records: &mut Vec, - source: &str, - page: ExternalHistoryPage, -) -> usize { - match page { - ExternalHistoryPage::Imported(page) => { - let page_len = page.sessions.len(); - records.extend( - page.sessions - .into_iter() - .map(|row| imported_history_to_aggregate_record(row, source)), - ); - page_len - } - ExternalHistoryPage::CursorIde(page) => { - let page_len = page.sessions.len(); - records.extend( - page.sessions - .into_iter() - .map(|row| cursor_ide_history_to_aggregate_record(row, source)), - ); - page_len - } - } -} - -/// How long after the last transcript write a hook-less CLI still counts as -/// running. Scan cadence (60s focused) bounds how fresh `updated_at` can be, -/// so the effective "running" window is roughly one to two scan ticks. -const IMPORTED_MTIME_ACTIVE_WINDOW_MS: i64 = 60_000; - -/// Live-status decoration for imported rows: a fresh lifecycle-hook state -/// wins; otherwise a transcript updated moments ago flips the row to -/// `running` — the only liveness signal CLIs without any hook surface -/// (aider, goose, cline, warp, ...) can give us. -fn decorate_imported_live_status(records: &mut [SessionAggregateRecord]) { - let now_ms = chrono::Utc::now().timestamp_millis(); - for record in records.iter_mut() { - if let Some((status, _entry)) = - crate::orgtrack::agent_live_status::effective_live_status(&record.session_id) - { - record.status = status.to_string(); - record.is_active = super::status::is_active_status(status); - continue; - } - if record.status == IMPORTED_STATUS_COMPLETED { - let recently_updated = DateTime::parse_from_rfc3339(&record.updated_at) - .map(|updated| { - now_ms - updated.timestamp_millis() < IMPORTED_MTIME_ACTIVE_WINDOW_MS - }) - .unwrap_or(false); - if recently_updated { - record.status = "running".to_string(); - record.is_active = true; - } - } - } +) -> Result<(), String> { + imported_history_catalog::refresh_source(conn, ImportedHistorySourceId::parse(source)?) } fn load_imported_history_sessions( filter: Option<&SessionFilter>, ) -> Result, String> { - let mut conn = + let conn = get_connection().map_err(|err| format!("Failed to open orgtrack cache DB: {err}"))?; let mut records = Vec::new(); let source_filter = filter.and_then(|filter| filter.external_history_source.as_deref()); @@ -394,21 +96,52 @@ fn load_imported_history_sessions( 0 }; - for loader in EXTERNAL_HISTORY_SOURCE_LOADERS { - if source_filter.is_some_and(|source| source != loader.source) { + for source in ImportedHistorySourceId::ALL { + let source_id = source.as_str(); + if source_filter.is_some_and(|expected| expected != source_id) { continue; } - if disabled_sources.contains(loader.source) { + if disabled_sources.contains(source_id) { continue; } - let page = (loader.load_page)(&mut conn, page_limit, page_offset)?; - append_external_history_page(&mut records, loader.source, page); + let page = imported_history_cache::query_imported_session_page_from_conn( + &conn, + source_id, + page_limit, + page_offset, + )?; + records.extend( + page.sessions + .into_iter() + .map(|row| imported_history_to_aggregate_record(row, source_id)), + ); } decorate_imported_live_status(&mut records); Ok(records) } +#[cfg(test)] +fn load_rust_agent_group_page( + group: &str, + limit: usize, + offset: usize, +) -> Result, String> { + super::sidebar_queries::load_scoped_rust_agent_group_page( + super::sidebar_queries::RustAgentGroupPageRequest { + group, + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: None, + limit, + offset, + }, + ) +} + // ============================================================================ // Core Aggregation // ============================================================================ @@ -432,17 +165,25 @@ fn plain_native_page( let Some(filter) = filter else { return Ok(None); }; + if let Some(page) = bounded_search_or_pinned_page(filter)? { + return Ok(Some(page)); + } let SessionFilter { session_ids, category, status, key_source, repo_path, + repo_path_exact, + missing_repo_path, org_id, + org_ids, project_slug, work_item_id, limit, offset, + before_updated_at, + before_session_id, text_query, sort_by, sort_order, @@ -451,17 +192,54 @@ fn plain_native_page( disabled_external_history_sources: _, created_after_ms, created_before_ms, + updated_after_ms, + updated_before_ms, active_only, + pinned_only, // Only meaningful with session_ids, and plain requires session_ids // to be absent, so the flag cannot affect the fast path. include_continuation_superseded: _, } = filter; + if repo_path.is_some() && *missing_repo_path == Some(true) { + return Err("Session page cannot combine repoPath and missingRepoPath".to_string()); + } + if repo_path_exact == &Some(true) && repo_path.is_none() { + return Err("repoPathExact requires repoPath".to_string()); + } + if updated_after_ms + .zip(*updated_before_ms) + .is_some_and(|(start, end)| start >= end) + { + return Err("updatedAfterMs must precede updatedBeforeMs".to_string()); + } + if before_updated_at.is_some() != before_session_id.is_some() { + return Err("beforeUpdatedAt and beforeSessionId must be provided together".to_string()); + } + let before = before_updated_at + .as_deref() + .zip(before_session_id.as_deref()) + .map(|(updated_at, session_id)| SidebarSeekCursor { + updated_at, + session_id, + }); + + let exact_repo_scope = repo_path.is_some() && *repo_path_exact == Some(true); + let missing_repo_scope = *missing_repo_path == Some(true); + let has_group_scope = exact_repo_scope + || missing_repo_scope + || org_ids.is_some() + || updated_after_ms.is_some() + || updated_before_ms.is_some() + || before.is_some(); let plain = session_ids.is_none() && status.is_none() && key_source.is_none() - && repo_path.is_none() + && (repo_path.is_none() || exact_repo_scope) + && repo_path_exact.is_none_or(|exact| exact == exact_repo_scope) + && missing_repo_path.is_none_or(|missing| missing == missing_repo_scope) && org_id.is_none() + && pinned_only.is_none_or(|pinned| !pinned) && project_slug.is_none() && work_item_id.is_none() && text_query.is_none() @@ -474,6 +252,9 @@ fn plain_native_page( if !plain { return Ok(None); } + if has_group_scope && category.is_none() { + return Ok(None); + } let limit = limit.unwrap_or(usize::MAX); let offset = offset.unwrap_or(0); @@ -487,13 +268,24 @@ fn plain_native_page( let mut sessions = match category.as_deref() { Some("cli") => { - let page = cli_session_persistence::list_sessions_page(limit, offset) - .map_err(|err| format!("Failed to load CLI session page: {}", err))?; + let page = sidebar_queries::load_scoped_cli_page(CliPageRequest { + org_ids: org_ids.as_deref(), + repo_path: repo_path.as_deref(), + missing_repo_path: missing_repo_scope, + updated_after_ms: *updated_after_ms, + updated_before_ms: *updated_before_ms, + before, + limit, + offset, + })?; page.into_iter() .map(cli_session_to_aggregate_record) .collect::>() } Some("agent") => { + if has_group_scope { + return Ok(None); + } let sde_filter = agent_core::session::SessionListFilter { type_names: Some(vec![ session_type::CODING.to_string(), @@ -503,7 +295,7 @@ fn plain_native_page( offset: Some(offset), ..Default::default() }; - let page = session_persistence::list_sessions(&sde_filter) + let page = session_persistence::list_root_sessions(&sde_filter) .map_err(|err| format!("Failed to load agent session page: {}", err))?; let mut resolver = AgentMetadataResolver::new(); let mut rows = page @@ -513,32 +305,50 @@ fn plain_native_page( annotate_agent_org_root_rows(&mut rows)?; rows } - Some("os") => { - let os_filter = agent_core::session::SessionListFilter { - type_name: Some(session_type::DESKTOP.to_string()), - limit: Some(limit), - offset: Some(offset), - ..Default::default() - }; - let page = session_persistence::list_sessions(&os_filter) - .map_err(|err| format!("Failed to load OS session page: {}", err))?; + Some(group @ ("sde" | "agent_org" | "os" | "wingman" | "custom")) => { + let page = + sidebar_queries::load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group, + org_ids: org_ids.as_deref(), + repo_path: repo_path.as_deref(), + missing_repo_path: missing_repo_scope, + updated_after_ms: *updated_after_ms, + updated_before_ms: *updated_before_ms, + before, + limit, + offset, + })?; let mut resolver = AgentMetadataResolver::new(); - page.into_iter() - .map(|session| os_session_to_aggregate_record(session, &mut resolver)) - .collect::>() + let mut rows = page + .into_iter() + .map(|session| { + if group == "os" { + os_session_to_aggregate_record(session, &mut resolver) + } else { + sde_session_to_aggregate_record(session, &mut resolver) + } + }) + .collect::>(); + if group == "agent_org" { + annotate_agent_org_root_rows(&mut rows)?; + } + rows } Some("human") => { - let human_filter = agent_core::session::SessionListFilter { - type_name: Some(session_type::HUMAN.to_string()), - limit: Some(limit), - offset: Some(offset), - ..Default::default() - }; - session_persistence::list_sessions(&human_filter) - .map_err(|err| format!("Failed to load Human session page: {err}"))? - .into_iter() - .map(human_session_to_aggregate_record) - .collect::>() + sidebar_queries::load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "human", + org_ids: org_ids.as_deref(), + repo_path: repo_path.as_deref(), + missing_repo_path: missing_repo_scope, + updated_after_ms: *updated_after_ms, + updated_before_ms: *updated_before_ms, + before, + limit, + offset, + })? + .into_iter() + .map(human_session_to_aggregate_record) + .collect::>() } None => return plain_directory_page(filter, limit, offset), _ => return Ok(None), @@ -586,9 +396,9 @@ fn plain_directory_page( ]; if include_external { sources.extend( - EXTERNAL_HISTORY_SOURCE_LOADERS + ImportedHistorySourceId::ALL .iter() - .map(|loader| loader.source) + .map(|source| source.as_str()) .filter(|source| !disabled_sources.contains(source)), ); } @@ -889,19 +699,68 @@ fn apply_filters( }); } + if let Some(updated_after_ms) = filter.updated_after_ms { + sessions.retain(|session| { + parse_epoch_millis(&session.updated_at) + .map(|updated_at_ms| updated_at_ms >= updated_after_ms) + .unwrap_or(false) + }); + } + + if let Some(updated_before_ms) = filter.updated_before_ms { + sessions.retain(|session| { + parse_epoch_millis(&session.updated_at) + .map(|updated_at_ms| updated_at_ms < updated_before_ms) + .unwrap_or(false) + }); + } + if let Some(ref repo_path) = filter.repo_path { + let normalized_repo_path = repo_path.trim_end_matches('/'); sessions.retain(|session| { session .repo_path .as_ref() - .map(|p| p.starts_with(repo_path)) + .map(|path| { + if filter.repo_path_exact == Some(true) { + path.trim_end_matches('/') == normalized_repo_path + } else { + path.starts_with(repo_path) + } + }) .unwrap_or(false) }); + } else if filter.missing_repo_path == Some(true) { + sessions.retain(|session| { + session + .repo_path + .as_deref() + .is_none_or(|path| path.trim().is_empty()) + }); } if let Some(ref org_id) = filter.org_id { sessions.retain(|session| session.org_id.as_deref() == Some(org_id.as_str())); } + if let Some(org_ids) = filter.org_ids.as_ref() { + let org_ids = org_ids + .iter() + .map(|value| value.trim()) + .filter(|value| !value.is_empty()) + .collect::>(); + if org_ids.is_empty() { + return Err("orgIds must contain at least one non-empty org id".to_string()); + } + sessions.retain(|session| { + let org_id = session + .org_id + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .unwrap_or(sidebar_queries::PERSONAL_ORG_ID); + org_ids.contains(org_id) + }); + } if let Some(ref project_slug) = filter.project_slug { sessions.retain(|session| session.project_slug.as_deref() == Some(project_slug.as_str())); @@ -922,6 +781,9 @@ fn apply_filters( if filter.active_only == Some(true) { sessions.retain(|session| session.is_active); } + if filter.pinned_only == Some(true) { + sessions.retain(|session| session.pinned); + } Ok(()) } @@ -930,7 +792,10 @@ fn apply_filters( // Sorting // ============================================================================ -fn apply_sorting(sessions: &mut [SessionAggregateRecord], filter: Option<&SessionFilter>) { +pub(super) fn apply_sorting( + sessions: &mut [SessionAggregateRecord], + filter: Option<&SessionFilter>, +) { let sort_by = filter .as_ref() .and_then(|f| f.sort_by.as_deref()) @@ -984,241 +849,9 @@ fn apply_pagination(sessions: &mut Vec, filter: &Session } } -fn agent_org_display_name(run: &AgentOrgRunRecord) -> String { - run.org_snapshot_json - .as_deref() - .and_then(|json| serde_json::from_str::(json).ok()) - .map(|org| org.name) - .unwrap_or_else(|| run.org_id.clone()) -} - -fn annotate_agent_org_root_rows(sessions: &mut [SessionAggregateRecord]) -> Result<(), String> { - let root_session_ids: std::collections::HashMap = - AgentOrgRunStore::list_runs(usize::MAX)? - .into_iter() - .filter_map(|run| { - let root_session_id = run.root_session_id.clone()?; - let org_name = agent_org_display_name(&run); - Some((root_session_id, (run.org_id, org_name))) - }) - .collect(); - if root_session_ids.is_empty() { - return Ok(()); - } - - for session in sessions { - if let Some((org_id, org_name)) = root_session_ids.get(&session.session_id) { - session.agent_icon_id = Some(AGENT_ORG_ICON_ID.to_string()); - session.agent_org_id = Some(org_id.clone()); - session.agent_org_name = Some(org_name.clone()); - } - } - - Ok(()) -} - // ============================================================================ // Tests // ============================================================================ #[cfg(test)] -mod tests { - use super::*; - use crate::agent_sessions::session_directory::display::generate_display_label; - use crate::agent_sessions::session_directory::status::is_active_status; - use crate::agent_sessions::session_directory::types::SessionCategory; - - fn make_session( - id: &str, - status: &str, - category: SessionCategory, - key_source: KeySource, - ) -> SessionAggregateRecord { - let name = format!("Session {}", id); - SessionAggregateRecord { - session_id: id.to_string(), - name: name.clone(), - status: status.to_string(), - created_at: "2024-01-01T00:00:00Z".to_string(), - updated_at: "2024-01-01T01:00:00Z".to_string(), - category, - external_history_source: None, - user_input: None, - repo_path: None, - repo_root_path: None, - repo_remote_urls: None, - storage_path: None, - repo_name: None, - branch: None, - model: Some("gpt-4".to_string()), - account_id: None, - cli_agent_type: None, - key_source, - tier: None, - pid: None, - total_tokens: 1000, - worktree_path: None, - worktree_branch: None, - base_branch: None, - merge_status: None, - background: false, - org_id: None, - project_id: None, - project_name: None, - project_slug: None, - work_item_id: None, - agent_role: None, - is_active: is_active_status(status), - display_label: generate_display_label(&name, None), - parent_session_id: None, - org_member_id: None, - agent_org_id: None, - agent_org_name: None, - agent_definition_id: None, - agent_icon_id: None, - agent_display_name: None, - agent_exec_mode: None, - draft_text: None, - reply_target_event_id: None, - pinned: false, - files_changed: None, - lines_added: None, - lines_removed: None, - touched_files: None, - } - } - - #[test] - fn apply_filters_accepts_known_key_source() { - let mut sessions = vec![ - make_session("1", "running", SessionCategory::Cli, KeySource::OwnKey), - make_session("2", "running", SessionCategory::Cli, KeySource::HostedKey), - ]; - - let filter = SessionFilter { - key_source: Some("hosted_key".to_string()), - ..Default::default() - }; - apply_filters(&mut sessions, &filter).expect("known key_source must be Ok"); - - assert_eq!(sessions.len(), 1); - assert_eq!(sessions[0].session_id, "2"); - } - - #[test] - fn apply_filters_matches_canonical_session_ids_exactly() { - let mut sessions = vec![ - make_session( - "session-1", - "completed", - SessionCategory::Cli, - KeySource::OwnKey, - ), - make_session( - "session-10", - "completed", - SessionCategory::Cli, - KeySource::OwnKey, - ), - ]; - let filter = SessionFilter { - session_ids: Some(vec!["session-1".to_string()]), - ..Default::default() - }; - - apply_filters(&mut sessions, &filter).expect("session ID filter"); - - assert_eq!(sessions.len(), 1); - assert_eq!(sessions[0].session_id, "session-1"); - } - - #[test] - fn apply_filters_rejects_unknown_key_source() { - let mut sessions = vec![make_session( - "1", - "running", - SessionCategory::Cli, - KeySource::OwnKey, - )]; - - let filter = SessionFilter { - // Typo: missing "_key" suffix. Previously silently mapped to - // OwnKey and mis-filtered the entire response. - key_source: Some("market".to_string()), - ..Default::default() - }; - let err = - apply_filters(&mut sessions, &filter).expect_err("unknown key_source must be rejected"); - assert!( - err.contains("Unknown key_source filter"), - "expected explicit rejection, got: {err}" - ); - } - - #[test] - fn pagination_does_not_append_org_member_children_for_visible_roots() { - let root = make_session( - "root-session", - "running", - SessionCategory::Agent, - KeySource::OwnKey, - ); - let mut paged_sessions = vec![root]; - let filter = SessionFilter { - limit: Some(1), - ..Default::default() - }; - apply_pagination(&mut paged_sessions, &filter); - - assert_eq!( - paged_sessions - .iter() - .map(|session| session.session_id.as_str()) - .collect::>(), - vec!["root-session"] - ); - } - - fn plain_page_filter() -> SessionFilter { - SessionFilter { - category: Some("cli".to_string()), - include_external_history: Some(false), - limit: Some(20), - offset: Some(0), - sort_by: Some("updated_at".to_string()), - sort_order: Some("desc".to_string()), - ..SessionFilter::default() - } - } - - #[test] - fn plain_native_page_rejects_non_plain_filters() { - // Missing filter entirely, or any shape the SQL page can't express, - // must fall through to the merge path (Ok(None)). - assert!(plain_native_page(None).unwrap().is_none()); - - let mut with_text = plain_page_filter(); - with_text.text_query = Some("bug".to_string()); - assert!(plain_native_page(Some(&with_text)).unwrap().is_none()); - - let mut with_status = plain_page_filter(); - with_status.status = Some("running".to_string()); - assert!(plain_native_page(Some(&with_status)).unwrap().is_none()); - - let mut with_external = plain_page_filter(); - with_external.include_external_history = Some(true); - assert!(plain_native_page(Some(&with_external)).unwrap().is_none()); - - let mut external_unset = plain_page_filter(); - external_unset.include_external_history = None; - assert!(plain_native_page(Some(&external_unset)).unwrap().is_none()); - - let mut multi_category = plain_page_filter(); - multi_category.category = Some("cli,agent".to_string()); - assert!(plain_native_page(Some(&multi_category)).unwrap().is_none()); - - let mut sorted_by_name = plain_page_filter(); - sorted_by_name.sort_by = Some("name".to_string()); - assert!(plain_native_page(Some(&sorted_by_name)).unwrap().is_none()); - } -} +mod tests; diff --git a/src-tauri/src/agent_sessions/session_directory/aggregation/tests.rs b/src-tauri/src/agent_sessions/session_directory/aggregation/tests.rs new file mode 100644 index 000000000..a1621285b --- /dev/null +++ b/src-tauri/src/agent_sessions/session_directory/aggregation/tests.rs @@ -0,0 +1,497 @@ +use super::*; +use crate::agent_sessions::session_directory::display::generate_display_label; +use crate::agent_sessions::session_directory::status::is_active_status; +use crate::agent_sessions::session_directory::types::SessionCategory; +use crate::test_utils::test_env; + +fn make_session( + id: &str, + status: &str, + category: SessionCategory, + key_source: KeySource, +) -> SessionAggregateRecord { + let name = format!("Session {}", id); + SessionAggregateRecord { + session_id: id.to_string(), + name: name.clone(), + status: status.to_string(), + created_at: "2024-01-01T00:00:00Z".to_string(), + updated_at: "2024-01-01T01:00:00Z".to_string(), + category, + external_history_source: None, + user_input: None, + repo_path: None, + repo_root_path: None, + repo_remote_urls: None, + storage_path: None, + repo_name: None, + branch: None, + model: Some("gpt-4".to_string()), + account_id: None, + cli_agent_type: None, + key_source, + tier: None, + pid: None, + total_tokens: 1000, + worktree_path: None, + worktree_branch: None, + base_branch: None, + merge_status: None, + background: false, + org_id: None, + project_id: None, + project_name: None, + project_slug: None, + work_item_id: None, + agent_role: None, + is_active: is_active_status(status), + display_label: generate_display_label(&name, None), + parent_session_id: None, + org_member_id: None, + agent_org_id: None, + agent_org_name: None, + agent_definition_id: None, + agent_icon_id: None, + agent_display_name: None, + agent_exec_mode: None, + draft_text: None, + reply_target_event_id: None, + pinned: false, + files_changed: None, + lines_added: None, + lines_removed: None, + touched_files: None, + } +} + +fn seed_native_root( + session_id: &str, + record_type: &str, + updated_at: &str, + parent_session_id: Option<&str>, +) { + let record = session_persistence::UnifiedSessionRecord { + session_id: session_id.to_string(), + name: session_id.to_string(), + status: agent_core::session::SessionStatus::Completed + .as_str() + .to_string(), + created_at: updated_at.to_string(), + updated_at: updated_at.to_string(), + session_type: record_type.to_string(), + parent_session_id: parent_session_id.map(str::to_string), + key_source: KeySource::OwnKey, + ..Default::default() + }; + session_persistence::upsert_session(&record).expect("seed native sidebar root"); +} + +fn set_native_workspace(session_id: &str, workspace_path: Option<&str>) { + let conn = get_connection().expect("open sandbox DB"); + conn.execute( + "UPDATE agent_sessions SET workspace_path = ?2 WHERE session_id = ?1", + rusqlite::params![session_id, workspace_path], + ) + .expect("set native workspace"); +} + +fn seed_cli_root(session_id: &str, updated_at: &str, repo_path: Option<&str>) { + let conn = get_connection().expect("open sandbox DB"); + conn.execute( + "INSERT INTO code_sessions + (session_id, name, status, flow, runner, cli_agent_type, + repo_path, created_at, updated_at) + VALUES (?1, ?1, 'completed', 'quick', 'local', 'opencode', + ?2, ?3, ?3)", + rusqlite::params![session_id, repo_path, updated_at], + ) + .expect("seed CLI sidebar root"); +} + +#[test] +fn rust_agent_group_pages_do_not_consume_each_others_offsets() { + let _sandbox = test_env::sandbox(); + seed_native_root( + "sdeagent-archived", + session_type::CODING, + "2026-07-26T13:00:00Z", + None, + ); + session_persistence::update_status( + "sdeagent-archived", + agent_core::session::SessionStatus::Archived, + ) + .expect("archive fixture"); + seed_native_root( + "sdeagent-child", + session_type::CODING, + "2026-07-26T12:00:00Z", + Some("sdeagent-standalone-1"), + ); + for (session_id, updated_at) in [ + ("sdeagent-standalone-1", "2026-07-26T11:00:00Z"), + ("sdeagent-org-1", "2026-07-26T10:00:00Z"), + ("sdeagent-standalone-2", "2026-07-26T09:00:00Z"), + ("sdeagent-org-2", "2026-07-26T08:00:00Z"), + ("sdeagent-standalone-3", "2026-07-26T07:00:00Z"), + ] { + seed_native_root(session_id, session_type::CODING, updated_at, None); + } + seed_native_root( + "wingman-1", + session_type::CODING, + "2026-07-26T06:00:00Z", + None, + ); + seed_native_root( + "dsagent-1", + session_type::CODING, + "2026-07-26T05:00:00Z", + None, + ); + seed_native_root( + "osagent-1", + session_type::DESKTOP, + "2026-07-26T04:00:00Z", + None, + ); + + let conn = get_connection().expect("open sandbox DB"); + for (run_id, root_session_id) in [("run-1", "sdeagent-org-1"), ("run-2", "sdeagent-org-2")] { + conn.execute( + "INSERT INTO agent_org_runs + (id, org_id, coordinator_agent_id, root_session_id, + entry_mode, status, created_at, updated_at) + VALUES (?1, 'org-alpha', 'coordinator', ?2, + 'standalone_session', 'completed', + '2026-07-26T00:00:00Z', '2026-07-26T00:00:00Z')", + rusqlite::params![run_id, root_session_id], + ) + .expect("seed Agent Org run"); + } + + let ids = |group: &str, limit: usize, offset: usize| { + load_rust_agent_group_page(group, limit, offset) + .expect("load group page") + .into_iter() + .map(|session| session.session_id) + .collect::>() + }; + assert_eq!( + ids("sde", 2, 0), + vec!["sdeagent-standalone-1", "sdeagent-standalone-2"] + ); + assert_eq!(ids("sde", 2, 2), vec!["sdeagent-standalone-3"]); + assert_eq!( + ids("agent_org", 10, 0), + vec!["sdeagent-org-1", "sdeagent-org-2"] + ); + assert_eq!(ids("wingman", 10, 0), vec!["wingman-1"]); + assert_eq!(ids("custom", 10, 0), vec!["dsagent-1"]); + assert_eq!(ids("os", 10, 0), vec!["osagent-1"]); +} + +#[test] +fn rust_agent_group_classification_matches_sidebar_sections() { + let _sandbox = test_env::sandbox(); + for (session_id, record_type, updated_at) in [ + ( + "sdeagent-modern", + session_type::CODING, + "2026-07-26T11:00:00Z", + ), + ( + "agentsession-legacy", + session_type::CODING, + "2026-07-26T10:00:00Z", + ), + ( + "wingman-specialist", + session_type::CODING, + "2026-07-26T09:00:00Z", + ), + ( + "random-custom-id", + session_type::CODING, + "2026-07-26T08:00:00Z", + ), + ( + "osagent-desktop", + session_type::DESKTOP, + "2026-07-26T07:00:00Z", + ), + ( + "random-agent-org-root", + session_type::CODING, + "2026-07-26T06:00:00Z", + ), + ( + "random-agent-org-os-root", + session_type::DESKTOP, + "2026-07-26T05:30:00Z", + ), + ( + "random-agent-org-human-root", + session_type::HUMAN, + "2026-07-26T05:00:00Z", + ), + ] { + seed_native_root(session_id, record_type, updated_at, None); + } + let conn = get_connection().expect("open sandbox DB"); + for (run_id, root_session_id) in [ + ("run-parity-coding", "random-agent-org-root"), + ("run-parity-os", "random-agent-org-os-root"), + ("run-parity-human", "random-agent-org-human-root"), + ] { + conn.execute( + "INSERT INTO agent_org_runs + (id, org_id, coordinator_agent_id, root_session_id, + entry_mode, status, created_at, updated_at) + VALUES (?1, 'org-parity', 'coordinator', ?2, + 'standalone_session', 'completed', + '2026-07-26T00:00:00Z', + '2026-07-26T00:00:00Z')", + rusqlite::params![run_id, root_session_id], + ) + .expect("seed parity Agent Org run"); + } + + let ids = |group: &str| { + load_rust_agent_group_page(group, 20, 0) + .expect("load classified group") + .into_iter() + .map(|session| session.session_id) + .collect::>() + }; + assert_eq!(ids("sde"), vec!["sdeagent-modern", "agentsession-legacy"]); + assert_eq!(ids("wingman"), vec!["wingman-specialist"]); + assert_eq!(ids("custom"), vec!["random-custom-id"]); + assert_eq!(ids("os"), vec!["osagent-desktop"]); + assert!(ids("human").is_empty()); + assert_eq!( + ids("agent_org"), + vec![ + "random-agent-org-root", + "random-agent-org-os-root", + "random-agent-org-human-root" + ] + ); +} + +#[test] +fn native_scope_filters_workspace_and_date_before_pagination() { + let _sandbox = test_env::sandbox(); + for (session_id, updated_at, workspace) in [ + ("sdeagent-a-new", "2026-07-26T11:00:00Z", Some("/repo-a/")), + ("sdeagent-b-new", "2026-07-26T10:00:00Z", Some("/repo-b")), + ("sdeagent-a-old", "2026-07-26T09:00:00Z", Some("/repo-a")), + ("sdeagent-missing", "2026-07-26T08:00:00Z", None), + ] { + seed_native_root(session_id, session_type::CODING, updated_at, None); + set_native_workspace(session_id, workspace); + } + let range_start = DateTime::parse_from_rfc3339("2026-07-26T08:30:00Z") + .expect("range start") + .timestamp_millis(); + let range_end = DateTime::parse_from_rfc3339("2026-07-26T12:00:00Z") + .expect("range end") + .timestamp_millis(); + let page = |offset| { + sidebar_queries::load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "sde", + org_ids: None, + repo_path: Some("/repo-a"), + missing_repo_path: false, + updated_after_ms: Some(range_start), + updated_before_ms: Some(range_end), + before: None, + limit: 1, + offset, + }) + .expect("scoped native page") + .into_iter() + .map(|session| session.session_id) + .collect::>() + }; + + assert_eq!(page(0), vec!["sdeagent-a-new"]); + assert_eq!(page(1), vec!["sdeagent-a-old"]); + assert_eq!( + sidebar_queries::load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "sde", + org_ids: None, + repo_path: None, + missing_repo_path: true, + updated_after_ms: None, + updated_before_ms: None, + before: None, + limit: 10, + offset: 0, + },) + .expect("missing-workspace native page") + .into_iter() + .map(|session| session.session_id) + .collect::>(), + vec!["sdeagent-missing"] + ); + + for (session_id, updated_at, workspace) in [ + ("cliagent-a-new", "2026-07-26T11:30:00Z", Some("/repo-a")), + ("cliagent-b-new", "2026-07-26T10:30:00Z", Some("/repo-b")), + ("cliagent-a-old", "2026-07-26T09:30:00Z", Some("/repo-a/")), + ] { + seed_cli_root(session_id, updated_at, workspace); + } + assert_eq!( + sidebar_queries::load_scoped_cli_page(CliPageRequest { + org_ids: None, + repo_path: Some("/repo-a"), + missing_repo_path: false, + updated_after_ms: Some(range_start), + updated_before_ms: Some(range_end), + before: None, + limit: 10, + offset: 0, + }) + .expect("scoped CLI page") + .into_iter() + .map(|session| session.session_id) + .collect::>(), + vec!["cliagent-a-new", "cliagent-a-old"] + ); +} + +#[test] +fn apply_filters_accepts_known_key_source() { + let mut sessions = vec![ + make_session("1", "running", SessionCategory::Cli, KeySource::OwnKey), + make_session("2", "running", SessionCategory::Cli, KeySource::HostedKey), + ]; + + let filter = SessionFilter { + key_source: Some("hosted_key".to_string()), + ..Default::default() + }; + apply_filters(&mut sessions, &filter).expect("known key_source must be Ok"); + + assert_eq!(sessions.len(), 1); + assert_eq!(sessions[0].session_id, "2"); +} + +#[test] +fn apply_filters_matches_canonical_session_ids_exactly() { + let mut sessions = vec![ + make_session( + "session-1", + "completed", + SessionCategory::Cli, + KeySource::OwnKey, + ), + make_session( + "session-10", + "completed", + SessionCategory::Cli, + KeySource::OwnKey, + ), + ]; + let filter = SessionFilter { + session_ids: Some(vec!["session-1".to_string()]), + ..Default::default() + }; + + apply_filters(&mut sessions, &filter).expect("session ID filter"); + + assert_eq!(sessions.len(), 1); + assert_eq!(sessions[0].session_id, "session-1"); +} + +#[test] +fn apply_filters_rejects_unknown_key_source() { + let mut sessions = vec![make_session( + "1", + "running", + SessionCategory::Cli, + KeySource::OwnKey, + )]; + + let filter = SessionFilter { + // Typo: missing "_key" suffix. Previously silently mapped to + // OwnKey and mis-filtered the entire response. + key_source: Some("market".to_string()), + ..Default::default() + }; + let err = + apply_filters(&mut sessions, &filter).expect_err("unknown key_source must be rejected"); + assert!( + err.contains("Unknown key_source filter"), + "expected explicit rejection, got: {err}" + ); +} + +#[test] +fn pagination_does_not_append_org_member_children_for_visible_roots() { + let root = make_session( + "root-session", + "running", + SessionCategory::Agent, + KeySource::OwnKey, + ); + let mut paged_sessions = vec![root]; + let filter = SessionFilter { + limit: Some(1), + ..Default::default() + }; + apply_pagination(&mut paged_sessions, &filter); + + assert_eq!( + paged_sessions + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + vec!["root-session"] + ); +} + +fn plain_page_filter() -> SessionFilter { + SessionFilter { + category: Some("cli".to_string()), + include_external_history: Some(false), + limit: Some(20), + offset: Some(0), + sort_by: Some("updated_at".to_string()), + sort_order: Some("desc".to_string()), + ..SessionFilter::default() + } +} + +#[test] +fn plain_native_page_rejects_non_plain_filters() { + // Missing filter entirely, or any shape the SQL page can't express, + // must fall through to the merge path (Ok(None)). + assert!(plain_native_page(None).unwrap().is_none()); + + let mut with_text = plain_page_filter(); + with_text.text_query = Some("bug".to_string()); + assert!(plain_native_page(Some(&with_text)).unwrap().is_none()); + + let mut with_status = plain_page_filter(); + with_status.status = Some("running".to_string()); + assert!(plain_native_page(Some(&with_status)).unwrap().is_none()); + + let mut with_external = plain_page_filter(); + with_external.include_external_history = Some(true); + assert!(plain_native_page(Some(&with_external)).unwrap().is_none()); + + let mut external_unset = plain_page_filter(); + external_unset.include_external_history = None; + assert!(plain_native_page(Some(&external_unset)).unwrap().is_none()); + + let mut multi_category = plain_page_filter(); + multi_category.category = Some("cli,agent".to_string()); + assert!(plain_native_page(Some(&multi_category)).unwrap().is_none()); + + let mut sorted_by_name = plain_page_filter(); + sorted_by_name.sort_by = Some("name".to_string()); + assert!(plain_native_page(Some(&sorted_by_name)).unwrap().is_none()); +} diff --git a/src-tauri/src/agent_sessions/session_directory/commands.rs b/src-tauri/src/agent_sessions/session_directory/commands.rs index a31e819ea..43b722a6e 100644 --- a/src-tauri/src/agent_sessions/session_directory/commands.rs +++ b/src-tauri/src/agent_sessions/session_directory/commands.rs @@ -9,15 +9,19 @@ use std::collections::HashSet; use database::db::get_connection; use orgtrack_core::sources::cursor_ide::history::CURSORIDE_SESSION_PREFIX; use orgtrack_core::sources::imported_history::{ - cache::query_imported_sidebar_page_from_conn, + cache::{query_imported_sidebar_scoped_page_from_conn, ImportedHistorySidebarPageQuery}, metadata::{is_imported_history_source, SOURCE_CURSOR_IDE}, }; use super::aggregation::list_all_sessions; +use super::sidebar_queries::{ + query_workspace_facets, WorkspaceFacetQuery, WorkspaceFacetSeekCursor, +}; use super::types::{ ExternalHistorySidebarBatchResponse, ExternalHistorySidebarBucketPage, - ExternalHistorySidebarResponse, ExternalHistorySidebarSourceRequest, SessionFilter, - SessionListResponse, + ExternalHistorySidebarCursor, ExternalHistorySidebarResponse, + ExternalHistorySidebarSourceRequest, SessionFilter, SessionListResponse, SessionWorkspaceFacet, + SessionWorkspaceFacetRequest, SessionWorkspaceFacetResponse, }; // ============================================================================ @@ -53,12 +57,20 @@ pub async fn session_external_history_sidebar_list( let mut seen_sources = HashSet::with_capacity(requests.len()); for source_request in requests { let source = source_request.source; + let repo_path = source_request.repo_path; + let missing_repo_path = source_request.missing_repo_path; if !seen_sources.insert(source.clone()) { return Err("External history sidebar sources must be unique".to_string()); } if !is_imported_history_source(&source) { return Err(format!("Unknown external history source: {source}")); } + if repo_path.is_some() && missing_repo_path { + return Err( + "External history sidebar scope cannot combine repoPath and missingRepoPath" + .to_string(), + ); + } let mut pages = Vec::with_capacity(source_request.buckets.len()); let mut seen_buckets = HashSet::with_capacity(source_request.buckets.len()); for request in source_request.buckets { @@ -80,14 +92,33 @@ pub async fn session_external_history_sidebar_list( ); } let limit = request.limit.min(EXTERNAL_HISTORY_SIDEBAR_BUCKET_MAX_LIMIT); - let mut page = query_imported_sidebar_page_from_conn( + let mut page = query_imported_sidebar_scoped_page_from_conn( &conn, - &source, - request.start_ms, - request.end_ms, - limit, - request.offset, + ImportedHistorySidebarPageQuery { + source: &source, + start_ms: request.start_ms, + end_ms: request.end_ms, + repo_path: repo_path.as_deref(), + missing_repo_path, + before_updated_at_ms: request + .before + .as_ref() + .map(|cursor| cursor.updated_at_ms), + before_session_id: request + .before + .as_ref() + .map(|cursor| cursor.session_id.as_str()), + limit, + offset: request.offset, + }, )?; + let next_cursor = + page.next_cursor + .as_ref() + .map(|cursor| ExternalHistorySidebarCursor { + updated_at_ms: cursor.updated_at_ms, + session_id: cursor.session_id.clone(), + }); if source == SOURCE_CURSOR_IDE { for session in &mut page.sessions { if !session.session_id.starts_with(CURSORIDE_SESSION_PREFIX) { @@ -114,6 +145,7 @@ pub async fn session_external_history_sidebar_list( bucket: request.bucket, sessions: page.sessions, has_more: page.has_more, + next_cursor, }); } sources.push(ExternalHistorySidebarResponse { @@ -126,3 +158,53 @@ pub async fn session_external_history_sidebar_list( .await .map_err(|err| format!("Task join error: {err}"))? } + +/// Discover bounded workspace groups without hydrating session transcripts. +#[tauri::command] +pub async fn session_sidebar_workspace_facets( + request: SessionWorkspaceFacetRequest, +) -> Result { + tokio::task::spawn_blocking(move || { + if request.org_ids.iter().all(|value| value.trim().is_empty()) { + return Err("Workspace facets require at least one org id".to_string()); + } + if request.limit == 0 { + return Err("Workspace facet limit must be positive".to_string()); + } + let page_limit = request.limit.min(50); + let conn = get_connection() + .map_err(|error| format!("Failed to open ORGII session cache: {error}"))?; + let mut rows = query_workspace_facets( + &conn, + WorkspaceFacetQuery { + org_ids: &request.org_ids, + include_external: request.include_external_history, + disabled_sources: &request.disabled_external_history_sources, + before: request + .before + .as_ref() + .map(|cursor| WorkspaceFacetSeekCursor { + last_updated_at_ms: cursor.last_updated_at_ms, + repo_path: cursor.repo_path.as_deref(), + }), + limit: page_limit.saturating_add(1), + offset: request.offset, + }, + )?; + let has_more = rows.len() > page_limit; + rows.truncate(page_limit); + Ok(SessionWorkspaceFacetResponse { + facets: rows + .into_iter() + .map(|row| SessionWorkspaceFacet { + repo_path: row.repo_path, + last_updated_at_ms: row.last_updated_at_ms, + session_count: row.session_count, + }) + .collect(), + has_more, + }) + }) + .await + .map_err(|error| format!("Task join error: {error}"))? +} diff --git a/src-tauri/src/agent_sessions/session_directory/conversion.rs b/src-tauri/src/agent_sessions/session_directory/conversion.rs index 910a5faec..e38c100f9 100644 --- a/src-tauri/src/agent_sessions/session_directory/conversion.rs +++ b/src-tauri/src/agent_sessions/session_directory/conversion.rs @@ -19,7 +19,10 @@ use orgtrack_core::sources::imported_history::ImportedHistorySessionRow; use super::display::generate_display_label; use super::status::is_active_status; use super::types::{SessionAggregateRecord, SessionCategory}; -use crate::orgtrack::impact_indexer::get_session_impact; +use crate::agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::is_snapshot_backed_native_fork; +use crate::orgtrack::impact_indexer::{ + get_persisted_session_impact, get_session_impact, SessionImpactStats, +}; pub struct AgentMetadataResolver { store: std::sync::Arc, @@ -45,7 +48,30 @@ fn warn_once_for_definition(def_id: &str, err: &str) { fn native_impact_fields( session_id: &str, ) -> (Option, Option, Option, Option>) { - match get_session_impact(session_id) { + let snapshot_probe = is_snapshot_backed_native_fork(session_id); + native_impact_fields_with( + session_id, + snapshot_probe, + || get_persisted_session_impact(session_id), + || get_session_impact(session_id), + ) +} + +fn native_impact_fields_with( + session_id: &str, + snapshot_probe: Result, + load_persisted_impact: impl FnOnce() -> Result, String>, + load_native_impact: impl FnOnce() -> Result, String>, +) -> (Option, Option, Option, Option>) { + let impact = match snapshot_probe { + Ok(true) => load_persisted_impact(), + Ok(false) => load_native_impact(), + Err(err) => { + tracing::debug!(session_id = %session_id, error = %err, "[session_directory] snapshot-backed fork probe unavailable"); + load_native_impact() + } + }; + match impact { Ok(Some(impact)) => ( Some(impact.files_changed), Some(impact.lines_added), @@ -514,3 +540,80 @@ pub fn human_session_to_aggregate_record( touched_files: None, } } + +#[cfg(test)] +mod tests { + use std::cell::Cell; + + use super::*; + + #[test] + fn snapshot_backed_native_fork_prefers_persisted_impact_without_rebuild() { + let load_called = Cell::new(false); + let fields = native_impact_fields_with( + "agentsession-cloud-fork", + Ok(true), + || { + Ok(Some(SessionImpactStats { + files_changed: 1, + lines_added: 5, + lines_removed: 2, + touched_files: vec!["src/snapshot.rs".to_string()], + })) + }, + || -> Result, String> { + load_called.set(true); + panic!("snapshot-backed fork must not load native turn impact") + }, + ); + + assert_eq!( + fields, + ( + Some(1), + Some(5), + Some(2), + Some(vec!["src/snapshot.rs".to_string()]), + ) + ); + assert!(!load_called.get()); + } + + #[test] + fn snapshot_backed_native_fork_without_persisted_impact_returns_none() { + let fields = native_impact_fields_with( + "agentsession-cloud-fork", + Ok(true), + || Ok(None), + || panic!("snapshot-backed fork must not rebuild native impact"), + ); + assert_eq!(fields, (None, None, None, None)); + } + + #[test] + fn ordinary_native_session_keeps_existing_impact_projection() { + let fields = native_impact_fields_with( + "agentsession-native", + Ok(false), + || panic!("ordinary native session must not use snapshot metadata"), + || { + Ok(Some(SessionImpactStats { + files_changed: 1, + lines_added: 3, + lines_removed: 2, + touched_files: vec!["src/lib.rs".to_string()], + })) + }, + ); + + assert_eq!( + fields, + ( + Some(1), + Some(3), + Some(2), + Some(vec!["src/lib.rs".to_string()]), + ) + ); + } +} diff --git a/src-tauri/src/agent_sessions/session_directory/mod.rs b/src-tauri/src/agent_sessions/session_directory/mod.rs index 8025331d0..5d3cec867 100644 --- a/src-tauri/src/agent_sessions/session_directory/mod.rs +++ b/src-tauri/src/agent_sessions/session_directory/mod.rs @@ -11,16 +11,21 @@ //! - `display` — Display label generation and text search //! - `conversion` — Backend record → directory record conversion //! - `aggregation` — Core merge + filter + sort + paginate logic +//! - `sidebar_queries` — Bounded search, pinned, grouping, and facet queries +//! - `sidebar_discovery`— Bounded search/pinned candidate hydration //! - `orgtrack_adapter`— Write-path mirror of session rows into orgtrack //! - `patch` — Per-session field mutations //! - `commands` — Tauri command handlers +mod agent_org_annotations; pub mod aggregation; pub mod commands; pub mod conversion; pub mod display; pub mod orgtrack_adapter; pub mod patch; +mod sidebar_discovery; +mod sidebar_queries; pub mod status; pub mod types; diff --git a/src-tauri/src/agent_sessions/session_directory/sidebar_discovery.rs b/src-tauri/src/agent_sessions/session_directory/sidebar_discovery.rs new file mode 100644 index 000000000..bf632f0b0 --- /dev/null +++ b/src-tauri/src/agent_sessions/session_directory/sidebar_discovery.rs @@ -0,0 +1,148 @@ +//! Bounded search and pinned-row hydration for the workstation sidebar. +//! +//! Candidate selection stays in compact SQL tables; only the bounded result +//! IDs are hydrated into the public aggregate record shape. + +use crate::agent_sessions::cli::persistence as cli_session_persistence; +use agent_core::session::persistence::{self as session_persistence, session_type}; +use database::db::get_connection; +use orgtrack_core::sources::imported_history::cache as imported_history_cache; + +use super::agent_org_annotations::annotate_agent_org_root_rows; +use super::aggregation::apply_sorting; +use super::conversion::{ + cli_session_to_aggregate_record, human_session_to_aggregate_record, + imported_history_to_aggregate_record, os_session_to_aggregate_record, + sde_session_to_aggregate_record, AgentMetadataResolver, +}; +use super::sidebar_queries::{ + self, SearchOrPinnedRequest, SidebarSeekCursor, SIDEBAR_COMPACT_PAGE_LIMIT, +}; +use super::status::decorate_imported_live_status; +use super::types::{SessionFilter, SessionListResponse}; + +pub(super) fn bounded_search_or_pinned_page( + filter: &SessionFilter, +) -> Result, String> { + let has_query = filter + .text_query + .as_deref() + .is_some_and(|query| !query.trim().is_empty()); + let pinned_only = filter.pinned_only == Some(true); + if !has_query && !pinned_only { + return Ok(None); + } + if filter.before_updated_at.is_some() != filter.before_session_id.is_some() { + return Err("beforeUpdatedAt and beforeSessionId must be provided together".to_string()); + } + if filter.category.is_some() + || filter.session_ids.is_some() + || filter.status.is_some() + || filter.key_source.is_some() + || filter.repo_path.is_some() + || filter.repo_path_exact.is_some() + || filter.missing_repo_path.is_some() + || filter.org_id.is_some() + || filter.project_slug.is_some() + || filter.work_item_id.is_some() + || filter.external_history_source.is_some() + || filter.created_after_ms.is_some() + || filter.created_before_ms.is_some() + || filter.updated_after_ms.is_some() + || filter.updated_before_ms.is_some() + || filter.active_only == Some(true) + || filter + .sort_by + .as_deref() + .is_some_and(|value| value != "updated_at") + || filter + .sort_order + .as_deref() + .is_some_and(|value| value != "desc") + { + return Ok(None); + } + + let conn = get_connection().map_err(|error| format!("Failed to open session DB: {error}"))?; + let candidates = sidebar_queries::query_search_or_pinned_candidates( + &conn, + SearchOrPinnedRequest { + query: filter.text_query.as_deref(), + pinned_only, + org_ids: filter.org_ids.as_deref(), + include_external: filter.include_external_history.unwrap_or(true), + disabled_sources: filter + .disabled_external_history_sources + .as_deref() + .unwrap_or(&[]), + before: filter + .before_updated_at + .as_deref() + .zip(filter.before_session_id.as_deref()) + .map(|(updated_at, session_id)| SidebarSeekCursor { + updated_at, + session_id, + }), + limit: filter.limit.unwrap_or(SIDEBAR_COMPACT_PAGE_LIMIT), + offset: filter.offset.unwrap_or(0), + }, + )?; + + let mut resolver = AgentMetadataResolver::new(); + let mut sessions = Vec::with_capacity(candidates.len()); + for candidate in candidates { + if candidate.source == orgtrack_core::canonical::SOURCE_ORGII_CLI_SESSIONS { + let Some(session) = cli_session_persistence::get_session(&candidate.session_id) + .map_err(|error| { + format!( + "Failed to hydrate bounded CLI sidebar row {}: {error}", + candidate.session_id + ) + })? + else { + continue; + }; + sessions.push(cli_session_to_aggregate_record(session)); + continue; + } + if candidate.source == orgtrack_core::canonical::SOURCE_ORGII_RUST_AGENTS { + let Some(session) = + session_persistence::get_session(&candidate.session_id).map_err(|error| { + format!( + "Failed to hydrate bounded agent sidebar row {}: {error}", + candidate.session_id + ) + })? + else { + continue; + }; + let record = match session.session_type.as_str() { + value if value == session_type::DESKTOP => { + os_session_to_aggregate_record(session, &mut resolver) + } + value if value == session_type::HUMAN => human_session_to_aggregate_record(session), + _ => sde_session_to_aggregate_record(session, &mut resolver), + }; + sessions.push(record); + continue; + } + + let Some((source, cached)) = + imported_history_cache::query_cached_session_by_session_id_from_conn( + &conn, + &candidate.session_id, + )? + else { + continue; + }; + if source != candidate.source { + continue; + } + let mut record = imported_history_to_aggregate_record(cached.to_row(), &source); + decorate_imported_live_status(std::slice::from_mut(&mut record)); + sessions.push(record); + } + annotate_agent_org_root_rows(&mut sessions)?; + apply_sorting(&mut sessions, Some(filter)); + Ok(Some(SessionListResponse { sessions })) +} diff --git a/src-tauri/src/agent_sessions/session_directory/sidebar_queries.rs b/src-tauri/src/agent_sessions/session_directory/sidebar_queries.rs new file mode 100644 index 000000000..a9b4a56fd --- /dev/null +++ b/src-tauri/src/agent_sessions/session_directory/sidebar_queries.rs @@ -0,0 +1,573 @@ +//! Bounded SQLite queries used by the workstation session sidebar. +//! +//! These queries deliberately read the three compact source tables directly: +//! `agent_sessions`, `code_sessions`, and +//! `imported_history_session_cache`. The canonical orgtrack directory is a +//! useful projection, but older native/managed rows can predate that mirror. +//! Sidebar discovery must therefore never depend on a backfill having run. + +use std::collections::BTreeSet; + +use agent_core::session::persistence::{self as session_persistence, session_type}; +use database::db::get_connection; +use orgtrack_core::canonical::{SOURCE_ORGII_CLI_SESSIONS, SOURCE_ORGII_RUST_AGENTS}; +use rusqlite::{params, Connection}; + +use crate::agent_sessions::cli::persistence as cli_session_persistence; + +pub(super) const PERSONAL_ORG_ID: &str = "personal-org"; +pub(super) const SIDEBAR_COMPACT_PAGE_LIMIT: usize = 51; +const LEGACY_SDE_SESSION_PREFIX: &str = "agentsession-"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct SidebarSessionCandidate { + pub session_id: String, + pub source: String, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct WorkspaceFacetRow { + pub repo_path: Option, + pub last_updated_at_ms: i64, + pub session_count: usize, +} + +pub(super) struct RustAgentGroupPageRequest<'a> { + pub group: &'a str, + pub org_ids: Option<&'a [String]>, + pub repo_path: Option<&'a str>, + pub missing_repo_path: bool, + pub updated_after_ms: Option, + pub updated_before_ms: Option, + pub before: Option>, + pub limit: usize, + pub offset: usize, +} + +pub(super) struct CliPageRequest<'a> { + pub org_ids: Option<&'a [String]>, + pub repo_path: Option<&'a str>, + pub missing_repo_path: bool, + pub updated_after_ms: Option, + pub updated_before_ms: Option, + pub before: Option>, + pub limit: usize, + pub offset: usize, +} + +/// Stable descending-page boundary for native and managed CLI rows. +/// +/// Values come from the exact `ORDER BY updated_at DESC, session_id DESC` +/// tuple. Imported history has a separate millisecond cursor because its +/// compact cache stores timestamps as integers. +#[derive(Clone, Copy)] +pub(super) struct SidebarSeekCursor<'a> { + pub updated_at: &'a str, + pub session_id: &'a str, +} + +pub(super) struct SearchOrPinnedRequest<'a> { + pub query: Option<&'a str>, + pub pinned_only: bool, + pub org_ids: Option<&'a [String]>, + pub include_external: bool, + pub disabled_sources: &'a [String], + pub before: Option>, + pub limit: usize, + pub offset: usize, +} + +pub(super) struct WorkspaceFacetQuery<'a> { + pub org_ids: &'a [String], + pub include_external: bool, + pub disabled_sources: &'a [String], + pub before: Option>, + pub limit: usize, + pub offset: usize, +} + +#[derive(Clone, Copy)] +pub(super) struct WorkspaceFacetSeekCursor<'a> { + pub last_updated_at_ms: i64, + pub repo_path: Option<&'a str>, +} + +fn normalize_org_ids(org_ids: Option<&[String]>) -> Result, String> { + let Some(org_ids) = org_ids else { + return Ok(None); + }; + let normalized = org_ids + .iter() + .map(|value| value.trim()) + .filter(|value| !value.is_empty()) + .collect::>(); + if normalized.is_empty() { + return Err("Sidebar org scope must contain at least one org id".to_string()); + } + serde_json::to_string(&normalized) + .map(Some) + .map_err(|error| format!("Failed to serialize sidebar org scope for SQLite: {error}")) +} + +fn disabled_sources_json(disabled_sources: &[String]) -> Result { + let normalized = disabled_sources + .iter() + .map(|value| value.trim()) + .filter(|value| !value.is_empty()) + .collect::>(); + serde_json::to_string(&normalized) + .map_err(|error| format!("Failed to serialize disabled sidebar sources: {error}")) +} + +fn escaped_like_pattern(query: Option<&str>) -> Option { + let query = query?.trim(); + if query.is_empty() { + return None; + } + let escaped = query + .to_lowercase() + .replace('\\', "\\\\") + .replace('%', "\\%") + .replace('_', "\\_"); + Some(format!("%{escaped}%")) +} + +pub(super) fn load_scoped_rust_agent_group_page( + request: RustAgentGroupPageRequest<'_>, +) -> Result, String> { + let RustAgentGroupPageRequest { + group, + org_ids, + repo_path, + missing_repo_path, + updated_after_ms, + updated_before_ms, + before, + limit, + offset, + } = request; + if repo_path.is_some() && missing_repo_path { + return Err( + "Native session page cannot combine repo_path and missing_repo_path".to_string(), + ); + } + let org_ids_json = normalize_org_ids(org_ids)?; + let conn = get_connection().map_err(|err| format!("Failed to open session DB: {err}"))?; + let sql = "SELECT s.session_id + FROM agent_sessions s + WHERE (s.parent_session_id IS NULL OR s.parent_session_id = '') + AND s.status != ?1 + AND ( + (?2 = 'sde' + AND s.session_type = ?3 + AND (s.session_id LIKE ?4 OR s.session_id LIKE ?5) + AND NOT EXISTS ( + SELECT 1 FROM agent_org_runs r + WHERE r.root_session_id = s.session_id + )) + OR (?2 = 'agent_org' + AND EXISTS ( + SELECT 1 FROM agent_org_runs r + WHERE r.root_session_id = s.session_id + )) + OR (?2 = 'os' + AND s.session_type = ?6 + AND NOT EXISTS ( + SELECT 1 FROM agent_org_runs r + WHERE r.root_session_id = s.session_id + )) + OR (?2 = 'wingman' + AND s.session_type = ?3 + AND s.session_id LIKE ?7 + AND NOT EXISTS ( + SELECT 1 FROM agent_org_runs r + WHERE r.root_session_id = s.session_id + )) + OR (?2 = 'custom' + AND s.session_type = ?3 + AND s.session_id NOT LIKE ?4 + AND s.session_id NOT LIKE ?5 + AND s.session_id NOT LIKE ?7 + AND NOT EXISTS ( + SELECT 1 FROM agent_org_runs r + WHERE r.root_session_id = s.session_id + )) + OR (?2 = 'human' + AND s.session_type = ?8 + AND NOT EXISTS ( + SELECT 1 FROM agent_org_runs r + WHERE r.root_session_id = s.session_id + )) + ) + AND (?9 IS NULL + OR COALESCE(NULLIF(TRIM(s.org_id), ''), 'personal-org') + IN (SELECT value FROM json_each(?9))) + AND (?10 IS NULL + OR RTRIM(COALESCE(s.workspace_path, ''), '/') = + RTRIM(?10, '/')) + AND (?11 = 0 OR TRIM(COALESCE(s.workspace_path, '')) = '') + AND (?12 IS NULL + OR CAST(strftime('%s', s.updated_at) AS INTEGER) * 1000 >= ?12) + AND (?13 IS NULL + OR CAST(strftime('%s', s.updated_at) AS INTEGER) * 1000 < ?13) + AND (?14 IS NULL + OR s.updated_at < ?14 + OR (s.updated_at = ?14 AND s.session_id < ?15)) + ORDER BY s.updated_at DESC, s.session_id DESC + LIMIT ?16 OFFSET ?17"; + let like_prefix = |prefix: &str| format!("{prefix}%"); + let mut stmt = conn + .prepare(sql) + .map_err(|err| format!("Failed to prepare Rust-agent group page: {err}"))?; + let session_ids = stmt + .query_map( + params![ + agent_core::session::SessionStatus::Archived.as_str(), + group, + session_type::CODING, + like_prefix(core_types::session::SDE_SESSION_PREFIX), + like_prefix(LEGACY_SDE_SESSION_PREFIX), + session_type::DESKTOP, + like_prefix(core_types::session::WINGMAN_SESSION_PREFIX), + session_type::HUMAN, + org_ids_json, + repo_path, + i64::from(missing_repo_path), + updated_after_ms, + updated_before_ms, + before.map(|cursor| cursor.updated_at), + before.map(|cursor| cursor.session_id), + limit.min(i64::MAX as usize) as i64, + if before.is_some() { + 0 + } else { + offset.min(i64::MAX as usize) as i64 + }, + ], + |row| row.get::<_, String>(0), + ) + .map_err(|err| format!("Failed to query Rust-agent group page: {err}"))? + .collect::, _>>() + .map_err(|err| format!("Failed to read Rust-agent group page: {err}"))?; + + session_ids + .into_iter() + .map(|session_id| { + session_persistence::get_session(&session_id) + .map_err(|err| format!("Failed to hydrate Rust-agent session {session_id}: {err}"))? + .ok_or_else(|| { + format!("Rust-agent session disappeared while listing: {session_id}") + }) + }) + .collect() +} + +pub(super) fn load_scoped_cli_page( + request: CliPageRequest<'_>, +) -> Result, String> { + let CliPageRequest { + org_ids, + repo_path, + missing_repo_path, + updated_after_ms, + updated_before_ms, + before, + limit, + offset, + } = request; + if repo_path.is_some() && missing_repo_path { + return Err("CLI page cannot combine repo_path and missing_repo_path".to_string()); + } + let org_ids_json = normalize_org_ids(org_ids)?; + let conn = get_connection().map_err(|err| format!("Failed to open session DB: {err}"))?; + let mut stmt = conn + .prepare( + "SELECT cs.session_id + FROM code_sessions cs + WHERE (cs.parent_session_id IS NULL OR cs.parent_session_id = '') + AND (?1 IS NULL + OR COALESCE(NULLIF(TRIM(cs.org_id), ''), 'personal-org') + IN (SELECT value FROM json_each(?1))) + AND (?2 IS NULL + OR RTRIM(COALESCE(cs.repo_path, ''), '/') = RTRIM(?2, '/')) + AND (?3 = 0 OR TRIM(COALESCE(cs.repo_path, '')) = '') + AND (?4 IS NULL + OR CAST(strftime('%s', cs.updated_at) AS INTEGER) * 1000 >= ?4) + AND (?5 IS NULL + OR CAST(strftime('%s', cs.updated_at) AS INTEGER) * 1000 < ?5) + AND (?6 IS NULL + OR cs.updated_at < ?6 + OR (cs.updated_at = ?6 AND cs.session_id < ?7)) + ORDER BY cs.updated_at DESC, cs.session_id DESC + LIMIT ?8 OFFSET ?9", + ) + .map_err(|err| format!("Failed to prepare scoped CLI page: {err}"))?; + let session_ids = stmt + .query_map( + params![ + org_ids_json, + repo_path, + i64::from(missing_repo_path), + updated_after_ms, + updated_before_ms, + before.map(|cursor| cursor.updated_at), + before.map(|cursor| cursor.session_id), + limit.min(i64::MAX as usize) as i64, + if before.is_some() { + 0 + } else { + offset.min(i64::MAX as usize) as i64 + }, + ], + |row| row.get::<_, String>(0), + ) + .map_err(|err| format!("Failed to query scoped CLI page: {err}"))? + .collect::, _>>() + .map_err(|err| format!("Failed to read scoped CLI page: {err}"))?; + + session_ids + .into_iter() + .map(|session_id| { + cli_session_persistence::get_session(&session_id) + .map_err(|err| format!("Failed to hydrate CLI session {session_id}: {err}"))? + .ok_or_else(|| format!("CLI session disappeared while listing: {session_id}")) + }) + .collect() +} + +pub(super) fn query_search_or_pinned_candidates( + conn: &Connection, + request: SearchOrPinnedRequest<'_>, +) -> Result, String> { + let SearchOrPinnedRequest { + query, + pinned_only, + org_ids, + include_external, + disabled_sources, + before, + limit, + offset, + } = request; + let search_pattern = escaped_like_pattern(query); + if search_pattern.is_none() && !pinned_only { + return Ok(Vec::new()); + } + let org_ids_json = normalize_org_ids(org_ids)?; + let disabled_sources_json = disabled_sources_json(disabled_sources)?; + let sql = format!( + "WITH candidates AS ( + SELECT s.session_id, + '{rust_source}' AS source, + COALESCE(CAST(strftime('%s', s.updated_at) AS INTEGER) * 1000, 0) + AS updated_ms + FROM agent_sessions s + WHERE (s.parent_session_id IS NULL OR s.parent_session_id = '') + AND s.status != 'archived' + AND s.session_type IN ('{coding}', '{org_member}', '{desktop}', '{human}') + AND (?1 IS NULL + OR COALESCE(NULLIF(TRIM(s.org_id), ''), 'personal-org') + IN (SELECT value FROM json_each(?1))) + AND (?2 IS NULL + OR LOWER(COALESCE(s.name, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(s.user_input, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(s.workspace_path, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(s.model, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(s.worktree_branch, '')) LIKE ?2 ESCAPE '\\') + AND (?3 = 0 OR COALESCE(s.pinned, 0) = 1) + UNION ALL + SELECT cs.session_id, + '{cli_source}' AS source, + COALESCE(CAST(strftime('%s', cs.updated_at) AS INTEGER) * 1000, 0) + AS updated_ms + FROM code_sessions cs + WHERE (cs.parent_session_id IS NULL OR cs.parent_session_id = '') + AND (?1 IS NULL + OR COALESCE(NULLIF(TRIM(cs.org_id), ''), 'personal-org') + IN (SELECT value FROM json_each(?1))) + AND (?2 IS NULL + OR LOWER(COALESCE(cs.name, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cs.user_input, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cs.repo_path, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cs.model, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cs.branch, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cs.cli_agent_type, '')) LIKE ?2 ESCAPE '\\') + AND (?3 = 0 OR COALESCE(cs.pinned, 0) = 1) + UNION ALL + SELECT cache.session_id, + cache.source, + cache.updated_at_ms + FROM imported_history_session_cache cache + WHERE ?3 = 0 + AND ?4 = 1 + AND (?1 IS NULL + OR EXISTS ( + SELECT 1 FROM json_each(?1) WHERE value = 'personal-org' + )) + AND cache.listable = 1 + AND cache.parent_session_id = '' + AND NOT EXISTS ( + SELECT 1 FROM json_each(?5) disabled + WHERE disabled.value = cache.source + ) + AND (?2 IS NULL + OR LOWER(COALESCE(cache.name, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cache.repo_path, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cache.model, '')) LIKE ?2 ESCAPE '\\' + OR LOWER(COALESCE(cache.branch, '')) LIKE ?2 ESCAPE '\\') + ) + SELECT session_id, source + FROM candidates + WHERE (?6 IS NULL + OR updated_ms < COALESCE(CAST(strftime('%s', ?6) AS INTEGER) * 1000, 0) + OR ( + updated_ms = COALESCE(CAST(strftime('%s', ?6) AS INTEGER) * 1000, 0) + AND session_id < ?7 + )) + ORDER BY updated_ms DESC, session_id DESC + LIMIT ?8 OFFSET ?9", + rust_source = SOURCE_ORGII_RUST_AGENTS, + cli_source = SOURCE_ORGII_CLI_SESSIONS, + coding = session_type::CODING, + org_member = session_type::ORG_MEMBER, + desktop = session_type::DESKTOP, + human = session_type::HUMAN, + ); + let mut stmt = conn + .prepare(&sql) + .map_err(|error| format!("Failed to prepare bounded sidebar search: {error}"))?; + let rows = stmt + .query_map( + params![ + org_ids_json, + search_pattern, + i64::from(pinned_only), + i64::from(include_external), + disabled_sources_json, + before.map(|cursor| cursor.updated_at), + before.map(|cursor| cursor.session_id), + limit.min(SIDEBAR_COMPACT_PAGE_LIMIT) as i64, + if before.is_some() { + 0 + } else { + offset.min(i64::MAX as usize) as i64 + }, + ], + |row| { + Ok(SidebarSessionCandidate { + session_id: row.get(0)?, + source: row.get(1)?, + }) + }, + ) + .map_err(|error| format!("Failed to query bounded sidebar search: {error}"))?; + rows.collect::, _>>() + .map_err(|error| format!("Failed to read bounded sidebar search: {error}")) +} + +pub(super) fn query_workspace_facets( + conn: &Connection, + query: WorkspaceFacetQuery<'_>, +) -> Result, String> { + let WorkspaceFacetQuery { + org_ids, + include_external, + disabled_sources, + before, + limit, + offset, + } = query; + let org_ids_json = normalize_org_ids(Some(org_ids))? + .ok_or_else(|| "Workspace facets require an org scope".to_string())?; + let disabled_sources_json = disabled_sources_json(disabled_sources)?; + let sql = format!( + "WITH source_rows AS ( + SELECT s.workspace_path AS workspace_path, + COALESCE(CAST(strftime('%s', s.updated_at) AS INTEGER) * 1000, 0) + AS updated_ms + FROM agent_sessions s + WHERE (s.parent_session_id IS NULL OR s.parent_session_id = '') + AND s.status != 'archived' + AND COALESCE(s.pinned, 0) = 0 + AND s.session_type IN ('{coding}', '{org_member}', '{desktop}', '{human}') + AND COALESCE(NULLIF(TRIM(s.org_id), ''), 'personal-org') + IN (SELECT value FROM json_each(?1)) + UNION ALL + SELECT cs.repo_path, + COALESCE(CAST(strftime('%s', cs.updated_at) AS INTEGER) * 1000, 0) + AS updated_ms + FROM code_sessions cs + WHERE (cs.parent_session_id IS NULL OR cs.parent_session_id = '') + AND COALESCE(cs.pinned, 0) = 0 + AND COALESCE(NULLIF(TRIM(cs.org_id), ''), 'personal-org') + IN (SELECT value FROM json_each(?1)) + UNION ALL + SELECT cache.repo_path, cache.updated_at_ms + FROM imported_history_session_cache cache + WHERE ?2 = 1 + AND EXISTS ( + SELECT 1 FROM json_each(?1) WHERE value = 'personal-org' + ) + AND cache.listable = 1 + AND cache.parent_session_id = '' + AND NOT EXISTS ( + SELECT 1 FROM json_each(?3) disabled + WHERE disabled.value = cache.source + ) + ), + normalized AS ( + SELECT CASE + WHEN TRIM(COALESCE(workspace_path, '')) = '' THEN '' + WHEN RTRIM(TRIM(workspace_path), '/') = '' THEN '/' + ELSE RTRIM(TRIM(workspace_path), '/') + END AS workspace_key, + updated_ms + FROM source_rows + ) + SELECT workspace_key, MAX(updated_ms), COUNT(*) + FROM normalized + GROUP BY workspace_key + HAVING (?4 IS NULL + OR MAX(updated_ms) < ?4 + OR (MAX(updated_ms) = ?4 AND workspace_key > ?5)) + ORDER BY MAX(updated_ms) DESC, workspace_key ASC + LIMIT ?6 OFFSET ?7", + coding = session_type::CODING, + org_member = session_type::ORG_MEMBER, + desktop = session_type::DESKTOP, + human = session_type::HUMAN, + ); + let mut stmt = conn + .prepare(&sql) + .map_err(|error| format!("Failed to prepare sidebar workspace facets: {error}"))?; + let rows = stmt + .query_map( + params![ + org_ids_json, + i64::from(include_external), + disabled_sources_json, + before.map(|cursor| cursor.last_updated_at_ms), + before.and_then(|cursor| cursor.repo_path).unwrap_or(""), + limit.min(SIDEBAR_COMPACT_PAGE_LIMIT) as i64, + if before.is_some() { + 0 + } else { + offset.min(i64::MAX as usize) as i64 + }, + ], + |row| { + let path: String = row.get(0)?; + Ok(WorkspaceFacetRow { + repo_path: if path.is_empty() { None } else { Some(path) }, + last_updated_at_ms: row.get(1)?, + session_count: row.get::<_, i64>(2)?.max(0) as usize, + }) + }, + ) + .map_err(|error| format!("Failed to query sidebar workspace facets: {error}"))?; + rows.collect::, _>>() + .map_err(|error| format!("Failed to read sidebar workspace facets: {error}")) +} + +#[cfg(test)] +mod tests; diff --git a/src-tauri/src/agent_sessions/session_directory/sidebar_queries/tests.rs b/src-tauri/src/agent_sessions/session_directory/sidebar_queries/tests.rs new file mode 100644 index 000000000..e89e5804d --- /dev/null +++ b/src-tauri/src/agent_sessions/session_directory/sidebar_queries/tests.rs @@ -0,0 +1,615 @@ +use super::*; +use crate::test_utils::test_env; +use core_types::key_source::KeySource; +use orgtrack_core::sources::imported_history::{ + cache::upsert_imported_session_cache_from_conn, + metadata::{ + ImportedHistoryCacheInput, ImportedHistoryImpactStats, SOURCE_CODEX_APP, SOURCE_OPENCODE, + }, +}; + +fn seed_native( + session_id: &str, + name: &str, + updated_at: &str, + org_id: Option<&str>, + workspace_path: Option<&str>, + pinned: bool, +) { + session_persistence::upsert_session(&session_persistence::UnifiedSessionRecord { + session_id: session_id.to_string(), + name: name.to_string(), + status: agent_core::session::SessionStatus::Completed + .as_str() + .to_string(), + created_at: updated_at.to_string(), + updated_at: updated_at.to_string(), + session_type: session_type::CODING.to_string(), + org_id: org_id.map(str::to_string), + workspace_path: workspace_path.map(str::to_string), + pinned, + key_source: KeySource::OwnKey, + ..Default::default() + }) + .expect("seed native sidebar discovery row"); +} + +fn seed_cli(session_id: &str, updated_at: &str) { + let conn = get_connection().expect("open sandbox DB"); + conn.execute( + "INSERT INTO code_sessions + (session_id, name, status, flow, runner, cli_agent_type, + created_at, updated_at) + VALUES (?1, ?1, 'completed', 'quick', 'local', 'opencode', ?2, ?2) + ON CONFLICT(session_id) DO UPDATE SET updated_at = excluded.updated_at", + params![session_id, updated_at], + ) + .expect("seed managed CLI sidebar row"); +} + +fn imported_input( + source: &'static str, + source_session_id: &str, + name: &str, + updated_at_ms: i64, + repo_path: Option<&str>, +) -> ImportedHistoryCacheInput { + ImportedHistoryCacheInput { + source, + source_session_id: source_session_id.to_string(), + session_id: format!("{source}-{source_session_id}"), + source_path: format!("/tmp/{source_session_id}.jsonl"), + source_record_key: source_session_id.to_string(), + source_mtime_ms: updated_at_ms, + source_size_bytes: 1, + source_fingerprint: format!("fingerprint-{source_session_id}"), + parser_version: 1, + name: name.to_string(), + created_at_ms: updated_at_ms, + updated_at_ms, + model: None, + input_tokens: 0, + output_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + repo_path: repo_path.map(str::to_string), + branch: None, + impact: ImportedHistoryImpactStats::default(), + listable: true, + source_metadata_json: None, + parent_session_id: None, + } +} + +fn candidate_ids(rows: Vec) -> Vec { + rows.into_iter().map(|row| row.session_id).collect() +} + +#[test] +fn org_scope_normalizes_personal_and_accepts_cloud_aliases() { + let _sandbox = test_env::sandbox(); + for (session_id, org_id) in [ + ("personal-null", None), + ("personal-empty", Some("")), + ("personal-explicit", Some(PERSONAL_ORG_ID)), + ("cloud-namespaced", Some("cloud:alpha")), + ("cloud-bare", Some("alpha")), + ("cloud-other", Some("beta")), + ] { + seed_native( + session_id, + "scope needle", + "2026-07-26T12:00:00Z", + org_id, + Some("/repo"), + false, + ); + } + let conn = get_connection().expect("open sandbox DB"); + let query = |org_ids: Vec| { + candidate_ids( + query_search_or_pinned_candidates( + &conn, + SearchOrPinnedRequest { + query: Some("scope needle"), + pinned_only: false, + org_ids: Some(&org_ids), + include_external: false, + disabled_sources: &[], + before: None, + limit: 50, + offset: 0, + }, + ) + .expect("query scoped search"), + ) + }; + + let personal = query(vec![PERSONAL_ORG_ID.to_string()]); + assert_eq!( + personal.into_iter().collect::>(), + [ + "personal-null".to_string(), + "personal-empty".to_string(), + "personal-explicit".to_string() + ] + .into_iter() + .collect() + ); + let cloud = query(vec!["cloud:alpha".to_string(), "alpha".to_string()]); + assert_eq!( + cloud.into_iter().collect::>(), + ["cloud-namespaced".to_string(), "cloud-bare".to_string()] + .into_iter() + .collect() + ); +} + +#[test] +fn search_and_pinned_predicates_run_before_the_page_limit() { + let _sandbox = test_env::sandbox(); + for index in 0..60 { + seed_native( + &format!("newer-{index:02}"), + "ordinary newer row", + &format!("2026-07-26T12:{index:02}:00Z"), + None, + Some("/new"), + false, + ); + } + seed_native( + "old-unique-search", + "uniquely searchable historical row", + "2026-07-01T00:00:00Z", + None, + Some("/old"), + false, + ); + seed_native( + "old-pinned", + "old pinned row", + "2026-06-03T00:00:00Z", + None, + Some("/old"), + true, + ); + seed_native( + "older-pinned", + "older pinned row", + "2026-06-02T00:00:00Z", + None, + Some("/old"), + true, + ); + seed_native( + "oldest-pinned", + "oldest pinned row", + "2026-06-01T00:00:00Z", + None, + Some("/old"), + true, + ); + let conn = get_connection().expect("open sandbox DB"); + let personal = vec![PERSONAL_ORG_ID.to_string()]; + + let search = query_search_or_pinned_candidates( + &conn, + SearchOrPinnedRequest { + query: Some("uniquely searchable"), + pinned_only: false, + org_ids: Some(&personal), + include_external: false, + disabled_sources: &[], + before: None, + limit: 50, + offset: 0, + }, + ) + .expect("search old row"); + assert_eq!(candidate_ids(search), vec!["old-unique-search"]); + + let pinned = query_search_or_pinned_candidates( + &conn, + SearchOrPinnedRequest { + query: None, + pinned_only: true, + org_ids: Some(&personal), + include_external: true, + disabled_sources: &[], + before: None, + limit: 2, + offset: 0, + }, + ) + .expect("query old pinned row"); + assert_eq!(candidate_ids(pinned), vec!["old-pinned", "older-pinned"]); + seed_native( + "new-pinned", + "new pinned row", + "2026-07-27T00:00:00Z", + None, + Some("/new"), + true, + ); + let pinned_tail = query_search_or_pinned_candidates( + &conn, + SearchOrPinnedRequest { + query: None, + pinned_only: true, + org_ids: Some(&personal), + include_external: false, + disabled_sources: &[], + before: Some(SidebarSeekCursor { + updated_at: "2026-06-02T00:00:00Z", + session_id: "older-pinned", + }), + limit: 50, + offset: 2, + }, + ) + .expect("query stable pinned tail"); + assert_eq!(candidate_ids(pinned_tail), vec!["oldest-pinned"]); +} + +#[test] +fn native_and_cli_seek_pages_ignore_newer_mutations_without_skipping_static_rows() { + let _sandbox = test_env::sandbox(); + for (session_id, updated_at) in [ + ("sdeagent-four", "2026-07-26T04:00:00Z"), + ("sdeagent-three", "2026-07-26T03:00:00Z"), + ("sdeagent-two", "2026-07-26T02:00:00Z"), + ("sdeagent-one", "2026-07-26T01:00:00Z"), + ] { + seed_native(session_id, session_id, updated_at, None, None, false); + } + let first = load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "sde", + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: None, + limit: 2, + offset: 0, + }) + .expect("first native seek page"); + assert_eq!( + first + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + ["sdeagent-four", "sdeagent-three"] + ); + let cursor_updated_at = first[1].updated_at.clone(); + let cursor_session_id = first[1].session_id.clone(); + + // A new top row and an already-consumed row moving to the top cannot + // shift a descending seek boundary or make page two repeat them. + seed_native( + "sdeagent-new", + "new", + "2026-07-26T06:00:00Z", + None, + None, + false, + ); + seed_native( + "sdeagent-four", + "four", + "2026-07-26T05:00:00Z", + None, + None, + false, + ); + let second = load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "sde", + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: Some(SidebarSeekCursor { + updated_at: &cursor_updated_at, + session_id: &cursor_session_id, + }), + limit: 2, + // A cursor must win over a stale compatibility offset. + offset: 99, + }) + .expect("second native seek page"); + assert_eq!( + second + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + ["sdeagent-two", "sdeagent-one"] + ); + + for (session_id, updated_at) in [ + ("cliagent-four", "2026-07-26T04:00:00Z"), + ("cliagent-three", "2026-07-26T03:00:00Z"), + ("cliagent-two", "2026-07-26T02:00:00Z"), + ("cliagent-one", "2026-07-26T01:00:00Z"), + ] { + seed_cli(session_id, updated_at); + } + let first_cli = load_scoped_cli_page(CliPageRequest { + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: None, + limit: 2, + offset: 0, + }) + .expect("first CLI seek page"); + let cli_cursor_updated_at = first_cli[1].updated_at.clone(); + let cli_cursor_session_id = first_cli[1].session_id.clone(); + seed_cli("cliagent-new", "2026-07-26T06:00:00Z"); + seed_cli("cliagent-four", "2026-07-26T05:00:00Z"); + let second_cli = load_scoped_cli_page(CliPageRequest { + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: Some(SidebarSeekCursor { + updated_at: &cli_cursor_updated_at, + session_id: &cli_cursor_session_id, + }), + limit: 2, + offset: 99, + }) + .expect("second CLI seek page"); + assert_eq!( + second_cli + .iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + ["cliagent-two", "cliagent-one"] + ); +} + +#[test] +fn refreshed_first_page_surfaces_an_unconsumed_row_that_moves_above_the_cursor() { + let _sandbox = test_env::sandbox(); + for (session_id, updated_at) in [ + ("sdeagent-three", "2026-07-26T03:00:00Z"), + ("sdeagent-two", "2026-07-26T02:00:00Z"), + ("sdeagent-one", "2026-07-26T01:00:00Z"), + ] { + seed_native(session_id, session_id, updated_at, None, None, false); + } + let first = load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "sde", + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: None, + limit: 1, + offset: 0, + }) + .expect("first page"); + let cursor_updated_at = first[0].updated_at.clone(); + let cursor_session_id = first[0].session_id.clone(); + + // Keyset pagination is a stable walk of the old tail. A row that was + // not consumed and then becomes newer than the cursor belongs to the + // live/refresh head, not to the old-tail continuation. + seed_native( + "sdeagent-two", + "two moved", + "2026-07-26T04:00:00Z", + None, + None, + false, + ); + let tail = load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "sde", + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: Some(SidebarSeekCursor { + updated_at: &cursor_updated_at, + session_id: &cursor_session_id, + }), + limit: 10, + offset: 0, + }) + .expect("stable old tail"); + assert_eq!( + tail.iter() + .map(|session| session.session_id.as_str()) + .collect::>(), + ["sdeagent-one"] + ); + let refreshed = load_scoped_rust_agent_group_page(RustAgentGroupPageRequest { + group: "sde", + org_ids: None, + repo_path: None, + missing_repo_path: false, + updated_after_ms: None, + updated_before_ms: None, + before: None, + limit: 1, + offset: 0, + }) + .expect("refreshed head"); + assert_eq!(refreshed[0].session_id, "sdeagent-two"); +} + +#[test] +fn workspace_facets_include_old_only_and_no_workspace_groups() { + let _sandbox = test_env::sandbox(); + seed_native( + "new-main", + "new main", + "2026-07-26T12:00:00Z", + None, + Some("/repo/main"), + false, + ); + seed_native( + "old-only", + "old only", + "2026-06-01T00:00:00Z", + None, + Some("/repo/old-only/"), + false, + ); + seed_native( + "no-workspace", + "no workspace", + "2026-05-01T00:00:00Z", + None, + None, + false, + ); + seed_native( + "pinned-only-workspace", + "pinned only workspace", + "2026-04-01T00:00:00Z", + None, + Some("/repo/pinned-only"), + true, + ); + let conn = get_connection().expect("open sandbox DB"); + let personal = vec![PERSONAL_ORG_ID.to_string()]; + let first = query_workspace_facets( + &conn, + WorkspaceFacetQuery { + org_ids: &personal, + include_external: false, + disabled_sources: &[], + before: None, + limit: 2, + offset: 0, + }, + ) + .expect("query workspace facets"); + assert_eq!( + first + .iter() + .map(|facet| facet.repo_path.as_deref()) + .collect::>(), + [Some("/repo/main"), Some("/repo/old-only")] + ); + seed_native( + "new-workspace", + "new workspace", + "2026-07-27T00:00:00Z", + None, + Some("/repo/new"), + false, + ); + let tail = query_workspace_facets( + &conn, + WorkspaceFacetQuery { + org_ids: &personal, + include_external: false, + disabled_sources: &[], + before: Some(WorkspaceFacetSeekCursor { + last_updated_at_ms: first[1].last_updated_at_ms, + repo_path: first[1].repo_path.as_deref(), + }), + limit: 50, + offset: 2, + }, + ) + .expect("query stable workspace tail"); + assert_eq!(tail.len(), 1); + assert_eq!(tail[0].repo_path, None); + assert!( + first + .iter() + .chain(tail.iter()) + .all(|facet| facet.repo_path.as_deref() != Some("/repo/pinned-only")), + "a workspace represented only by pinned rows belongs in Pinned, not an empty section" + ); +} + +#[test] +fn imported_discovery_is_personal_listable_and_source_gated() { + let _sandbox = test_env::sandbox(); + let mut conn = get_connection().expect("open sandbox DB"); + let mut hidden = imported_input( + SOURCE_CODEX_APP, + "hidden", + "imported needle hidden", + 300, + Some("/hidden"), + ); + hidden.listable = false; + let mut child = imported_input( + SOURCE_CODEX_APP, + "child", + "imported needle child", + 250, + Some("/child"), + ); + child.parent_session_id = Some("codex_app-root".to_string()); + upsert_imported_session_cache_from_conn( + &mut conn, + &[ + imported_input( + SOURCE_CODEX_APP, + "visible", + "imported needle visible", + 200, + Some("/visible"), + ), + imported_input( + SOURCE_OPENCODE, + "disabled", + "imported needle disabled", + 100, + Some("/disabled"), + ), + hidden, + child, + ], + ) + .expect("seed imported sidebar rows"); + + let personal = vec![PERSONAL_ORG_ID.to_string()]; + let disabled = vec![SOURCE_OPENCODE.to_string()]; + let rows = query_search_or_pinned_candidates( + &conn, + SearchOrPinnedRequest { + query: Some("imported needle"), + pinned_only: false, + org_ids: Some(&personal), + include_external: true, + disabled_sources: &disabled, + before: None, + limit: 50, + offset: 0, + }, + ) + .expect("query imported discovery"); + assert_eq!(candidate_ids(rows), vec!["codex_app-visible"]); + + let cloud = vec!["cloud:alpha".to_string(), "alpha".to_string()]; + let rows = query_search_or_pinned_candidates( + &conn, + SearchOrPinnedRequest { + query: Some("imported needle"), + pinned_only: false, + org_ids: Some(&cloud), + include_external: true, + disabled_sources: &[], + before: None, + limit: 50, + offset: 0, + }, + ) + .expect("query cloud imported discovery"); + assert!(rows.is_empty()); +} diff --git a/src-tauri/src/agent_sessions/session_directory/status.rs b/src-tauri/src/agent_sessions/session_directory/status.rs index aea2c9ead..eba0d895e 100644 --- a/src-tauri/src/agent_sessions/session_directory/status.rs +++ b/src-tauri/src/agent_sessions/session_directory/status.rs @@ -5,6 +5,37 @@ use agent_core::session::SessionStatus; +use super::types::SessionAggregateRecord; + +/// How long a hook-less imported CLI still counts as active after its +/// transcript changed. The focused scan cadence is 60 seconds, so this covers +/// roughly one to two scan ticks without inventing a durable running state. +const IMPORTED_MTIME_ACTIVE_WINDOW_MS: i64 = 60_000; + +pub(super) fn decorate_imported_live_status(records: &mut [SessionAggregateRecord]) { + let now_ms = chrono::Utc::now().timestamp_millis(); + for record in records.iter_mut() { + if let Some((status, _entry)) = + crate::orgtrack::agent_live_status::effective_live_status(&record.session_id) + { + record.status = status.to_string(); + record.is_active = is_active_status(status); + continue; + } + if record.status == orgtrack_core::sources::imported_history::IMPORTED_STATUS_COMPLETED { + let recently_updated = chrono::DateTime::parse_from_rfc3339(&record.updated_at) + .map(|updated| { + now_ms - updated.timestamp_millis() < IMPORTED_MTIME_ACTIVE_WINDOW_MS + }) + .unwrap_or(false); + if recently_updated { + record.status = "running".to_string(); + record.is_active = true; + } + } + } +} + // ============================================================================ // Status Classification // ============================================================================ diff --git a/src-tauri/src/agent_sessions/session_directory/types.rs b/src-tauri/src/agent_sessions/session_directory/types.rs index a56c2323d..cd4ff08ce 100644 --- a/src-tauri/src/agent_sessions/session_directory/types.rs +++ b/src-tauri/src/agent_sessions/session_directory/types.rs @@ -224,9 +224,23 @@ pub struct SessionFilter { /// Filter by repo path prefix #[serde(skip_serializing_if = "Option::is_none")] pub repo_path: Option, + /// Match `repo_path` exactly (ignoring trailing slashes) instead of using + /// the historical prefix semantics. Used by the By Workspace pager. + #[serde(skip_serializing_if = "Option::is_none")] + pub repo_path_exact: Option, + /// Return only rows without a repository/workspace path. This is distinct + /// from an omitted `repo_path`, which means no workspace filter. + #[serde(skip_serializing_if = "Option::is_none")] + pub missing_repo_path: Option, /// Filter by owning project/collaboration org ID #[serde(skip_serializing_if = "Option::is_none")] pub org_id: Option, + /// Exact canonical org IDs accepted by bounded sidebar cursors. + /// + /// Cloud scopes include both `cloud:` and the historical bare ID. + /// An omitted list preserves the legacy unscoped API behavior. + #[serde(skip_serializing_if = "Option::is_none")] + pub org_ids: Option>, /// Filter by linked project slug #[serde(skip_serializing_if = "Option::is_none")] pub project_slug: Option, @@ -239,6 +253,14 @@ pub struct SessionFilter { /// Skip first N sessions (for pagination) #[serde(skip_serializing_if = "Option::is_none")] pub offset: Option, + /// Descending sidebar seek boundary. Both cursor fields must be supplied + /// together; rows strictly older than this exact `(updated_at, + /// session_id)` tuple are returned. Legacy callers may continue using + /// `offset` when these fields are absent. + #[serde(skip_serializing_if = "Option::is_none")] + pub before_updated_at: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub before_session_id: Option, /// Text search query (searches name, user_input, repo_name — case-insensitive) #[serde(skip_serializing_if = "Option::is_none")] pub text_query: Option, @@ -264,9 +286,22 @@ pub struct SessionFilter { /// Only include sessions created at or before this epoch millisecond. #[serde(skip_serializing_if = "Option::is_none")] pub created_before_ms: Option, + /// Only include sessions updated at or after this epoch millisecond. + #[serde(skip_serializing_if = "Option::is_none")] + pub updated_after_ms: Option, + /// Only include sessions updated before this epoch millisecond. + /// + /// The upper bound is exclusive so adjacent sidebar date buckets cannot + /// overlap. + #[serde(skip_serializing_if = "Option::is_none")] + pub updated_before_ms: Option, /// Only return active (ongoing) sessions #[serde(skip_serializing_if = "Option::is_none")] pub active_only: Option, + /// Return only native/managed sessions pinned in their source table. + /// Imported application history is not pinnable. + #[serde(skip_serializing_if = "Option::is_none")] + pub pinned_only: Option, /// Exact-id lookups normally report continuation-superseded siblings as /// absent so hydration surfaces never re-add rows the listing demoted. /// Existence checks (the cloud vanished-session sweep) opt out: a @@ -295,6 +330,13 @@ pub enum ExternalHistorySidebarDateBucket { Older, } +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExternalHistorySidebarCursor { + pub updated_at_ms: i64, + pub session_id: String, +} + #[derive(Debug, Clone, Deserialize)] #[serde(rename_all = "camelCase")] pub struct ExternalHistorySidebarBucketRequest { @@ -303,12 +345,18 @@ pub struct ExternalHistorySidebarBucketRequest { pub end_ms: Option, pub limit: usize, pub offset: usize, + pub before: Option, } #[derive(Debug, Clone, Deserialize)] #[serde(rename_all = "camelCase")] pub struct ExternalHistorySidebarSourceRequest { pub source: String, + /// Optional exact workspace scope for every requested date bucket. + pub repo_path: Option, + /// Scope every requested bucket to sessions without a workspace. + #[serde(default)] + pub missing_repo_path: bool, pub buckets: Vec, } @@ -318,6 +366,8 @@ pub struct ExternalHistorySidebarBucketPage { pub bucket: ExternalHistorySidebarDateBucket, pub sessions: Vec, pub has_more: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub next_cursor: Option, } #[derive(Debug, Clone, Serialize)] @@ -332,3 +382,39 @@ pub struct ExternalHistorySidebarResponse { pub struct ExternalHistorySidebarBatchResponse { pub sources: Vec, } + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct SessionWorkspaceFacetRequest { + pub org_ids: Vec, + #[serde(default)] + pub include_external_history: bool, + #[serde(default)] + pub disabled_external_history_sources: Vec, + pub limit: usize, + #[serde(default)] + pub offset: usize, + pub before: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct SessionWorkspaceFacetCursor { + pub last_updated_at_ms: i64, + pub repo_path: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct SessionWorkspaceFacet { + pub repo_path: Option, + pub last_updated_at_ms: i64, + pub session_count: usize, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct SessionWorkspaceFacetResponse { + pub facets: Vec, + pub has_more: bool, +} diff --git a/src-tauri/src/api/agent/test/cli.rs b/src-tauri/src/api/agent/test/cli.rs index 1c9d441ea..8d5949034 100644 --- a/src-tauri/src/api/agent/test/cli.rs +++ b/src-tauri/src/api/agent/test/cli.rs @@ -3,18 +3,20 @@ #![cfg(debug_assertions)] use axum::Json; -use core_types::activity::ActivityChunk; +use core_types::session_event::{EventSource, SessionEvent}; use serde::Deserialize; use serde_json::json; use tokio::process::Command; use crate::agent_sessions::cli::commands::{ - cli_agent_chunks, cli_agent_create, cli_agent_message, cli_agent_resume, cli_agent_run, - cli_agent_status, + cli_agent_create, cli_agent_message, cli_agent_resume, cli_agent_run, cli_agent_status, }; use crate::agent_sessions::cli::persistence::{self, CreateCodeSessionParams}; use crate::agent_sessions::cli::session_runner; use crate::agent_sessions::cli::types::{KeySource, SessionStatus}; +use crate::agent_sessions::event_pipeline::commands::external_replay::{ + test_open_managed_replay_window, ExternalReplayWindow, +}; use key_vault::key_store::ModelType; const DEFAULT_TIMEOUT_SECS: u64 = 240; @@ -67,14 +69,13 @@ pub struct TestCodexCliAccountSwitchRequest { timeout_secs: Option, } -fn assistant_chunk_contains(chunks: &[ActivityChunk], needle: &str) -> bool { - chunks.iter().any(|chunk| { - matches!( - chunk.action_type.as_str(), - "assistant" | "assistant_delta" | "llm_response" - ) && serde_json::to_string(&chunk.result) - .map(|value| value.contains(needle)) - .unwrap_or(false) +fn assistant_event_contains(events: &[SessionEvent], needle: &str) -> bool { + events.iter().any(|event| { + event.source == EventSource::Assistant + && (event.display_text.contains(needle) + || serde_json::to_string(&event.result) + .map(|value| value.contains(needle)) + .unwrap_or(false)) }) } @@ -114,27 +115,27 @@ async fn wait_for_terminal_session_after_update( Err("CLI session timed out".to_string()) } -async fn wait_for_chunk_progress( +async fn wait_for_replay_progress( session_id: &str, - baseline_count: usize, + baseline_count: u64, expected_text: Option<&str>, timeout_secs: u64, -) -> Result, String> { +) -> Result { let deadline = std::time::Instant::now() + std::time::Duration::from_secs(timeout_secs); while std::time::Instant::now() < deadline { - let chunks = cli_agent_chunks(session_id.to_string()) + let replay = test_open_managed_replay_window(session_id.to_string()) .await - .map_err(|err| format!("cli_agent_chunks failed: {err}"))?; + .map_err(|err| format!("bounded managed replay failed: {err}"))?; if expected_text - .map(|text| assistant_chunk_contains(&chunks, text)) + .map(|text| assistant_event_contains(&replay.events, text)) .unwrap_or(false) - || chunks.len() > baseline_count + || replay.total_event_count > baseline_count { - return Ok(chunks); + return Ok(replay); } tokio::time::sleep(std::time::Duration::from_millis(POLL_INTERVAL_MS)).await; } - Err("CLI session chunk progress timed out".to_string()) + Err("CLI session bounded replay progress timed out".to_string()) } pub async fn test_cursor_cli_runtime( @@ -224,14 +225,14 @@ pub async fn test_cursor_cli_runtime( } }; - let chunks = match cli_agent_chunks(session_id.clone()).await { - Ok(chunks) => chunks, - Err(err) => return Json(json!({ "error": format!("cli_agent_chunks failed: {err}") })), + let replay = match test_open_managed_replay_window(session_id.clone()).await { + Ok(replay) => replay, + Err(err) => return Json(json!({ "error": format!("bounded replay failed: {err}") })), }; let expected_seen = request .expected_text .as_deref() - .map(|expected_text| assistant_chunk_contains(&chunks, expected_text)); + .map(|expected_text| assistant_event_contains(&replay.events, expected_text)); let expected_cursor_config_file = expected_cursor_config_dir.join("cli-config.json"); @@ -240,12 +241,12 @@ pub async fn test_cursor_cli_runtime( "status": session.status.as_ref(), "error_message": session.error_message, "cli_session_id": session.cli_session_id, - "chunk_count": chunks.len(), + "chunk_count": replay.total_event_count, "expected_seen": expected_seen, "cursor_config_dir": expected_cursor_config_dir.to_string_lossy(), "cursor_config_dir_exists": expected_cursor_config_dir.exists(), "cursor_config_file_exists": expected_cursor_config_file.exists(), - "chunks": chunks, + "chunks": replay.events, })) } @@ -435,11 +436,11 @@ pub async fn test_claude_code_cli_account_switch( Ok(session) => session, Err(err) => return Json(json!({ "error": err, "session_id": session_id })), }; - let initial_chunks = match cli_agent_chunks(session_id.clone()).await { - Ok(chunks) => chunks, - Err(err) => return Json(json!({ "error": format!("cli_agent_chunks failed: {err}") })), + let initial_replay = match test_open_managed_replay_window(session_id.clone()).await { + Ok(replay) => replay, + Err(err) => return Json(json!({ "error": format!("bounded replay failed: {err}") })), }; - let baseline_chunk_count = initial_chunks.len(); + let baseline_event_count = initial_replay.total_event_count; let baseline_updated_at = initial_session.updated_at.clone(); if let Err(err) = cli_agent_message( @@ -456,15 +457,15 @@ pub async fn test_claude_code_cli_account_switch( return Json(json!({ "error": format!("cli_agent_message failed: {err}") })); } - let _followup_chunks = match wait_for_chunk_progress( + let _followup_replay = match wait_for_replay_progress( &session_id, - baseline_chunk_count, + baseline_event_count, request.followup_expected_text.as_deref(), timeout_secs, ) .await { - Ok(chunks) => chunks, + Ok(replay) => replay, Err(err) => return Json(json!({ "error": err, "session_id": session_id })), }; @@ -483,9 +484,9 @@ pub async fn test_claude_code_cli_account_switch( Ok(None) => return Json(json!({ "error": "session disappeared after follow-up" })), Err(err) => return Json(json!({ "error": format!("DB error: {err}") })), }; - let chunks = match cli_agent_chunks(session_id.clone()).await { - Ok(chunks) => chunks, - Err(err) => return Json(json!({ "error": format!("cli_agent_chunks failed: {err}") })), + let replay = match test_open_managed_replay_window(session_id.clone()).await { + Ok(replay) => replay, + Err(err) => return Json(json!({ "error": format!("bounded replay failed: {err}") })), }; let initial_claude_config_dir = @@ -495,11 +496,11 @@ pub async fn test_claude_code_cli_account_switch( let initial_expected_seen = request .initial_expected_text .as_deref() - .map(|expected_text| assistant_chunk_contains(&chunks, expected_text)); + .map(|expected_text| assistant_event_contains(&replay.events, expected_text)); let followup_expected_seen = request .followup_expected_text .as_deref() - .map(|expected_text| assistant_chunk_contains(&chunks, expected_text)); + .map(|expected_text| assistant_event_contains(&replay.events, expected_text)); Json(json!({ "session_id": session_id, @@ -509,7 +510,7 @@ pub async fn test_claude_code_cli_account_switch( "followup_error_message": followup_session.error_message, "persisted_account_id": persisted_session.account_id, "persisted_model": persisted_session.model, - "chunk_count": chunks.len(), + "chunk_count": replay.total_event_count, "initial_expected_seen": initial_expected_seen, "followup_expected_seen": followup_expected_seen, "initial_claude_config_dir": initial_claude_config_dir.to_string_lossy(), @@ -594,11 +595,11 @@ pub async fn test_codex_cli_account_switch( Err(err) => return Json(json!({ "error": err, "session_id": session_id })), }; let initial_env_ready = initial_codex_home.exists(); - let initial_chunks = match cli_agent_chunks(session_id.clone()).await { - Ok(chunks) => chunks, - Err(err) => return Json(json!({ "error": format!("cli_agent_chunks failed: {err}") })), + let initial_replay = match test_open_managed_replay_window(session_id.clone()).await { + Ok(replay) => replay, + Err(err) => return Json(json!({ "error": format!("bounded replay failed: {err}") })), }; - let baseline_chunk_count = initial_chunks.len(); + let baseline_event_count = initial_replay.total_event_count; let baseline_updated_at = initial_session.updated_at.clone(); if let Err(err) = cli_agent_message( @@ -615,15 +616,15 @@ pub async fn test_codex_cli_account_switch( return Json(json!({ "error": format!("cli_agent_message failed: {err}") })); } - let _followup_chunks = match wait_for_chunk_progress( + let _followup_replay = match wait_for_replay_progress( &session_id, - baseline_chunk_count, + baseline_event_count, request.followup_expected_text.as_deref(), timeout_secs, ) .await { - Ok(chunks) => chunks, + Ok(replay) => replay, Err(err) => { return Json(json!({ "error": err, @@ -645,18 +646,18 @@ pub async fn test_codex_cli_account_switch( Ok(session) => session, Err(err) => return Json(json!({ "error": err, "session_id": session_id })), }; - let chunks = match cli_agent_chunks(session_id.clone()).await { - Ok(chunks) => chunks, - Err(err) => return Json(json!({ "error": format!("cli_agent_chunks failed: {err}") })), + let replay = match test_open_managed_replay_window(session_id.clone()).await { + Ok(replay) => replay, + Err(err) => return Json(json!({ "error": format!("bounded replay failed: {err}") })), }; let initial_expected_seen = request .initial_expected_text .as_deref() - .map(|expected_text| assistant_chunk_contains(&chunks, expected_text)); + .map(|expected_text| assistant_event_contains(&replay.events, expected_text)); let followup_expected_seen = request .followup_expected_text .as_deref() - .map(|expected_text| assistant_chunk_contains(&chunks, expected_text)); + .map(|expected_text| assistant_event_contains(&replay.events, expected_text)); let persisted_session = match persistence::get_session(&session_id) { Ok(Some(session)) => session, @@ -673,7 +674,7 @@ pub async fn test_codex_cli_account_switch( "followup_error_message": followup_session.error_message, "persisted_account_id": persisted_session.account_id, "persisted_model": persisted_session.model, - "chunk_count": chunks.len(), + "chunk_count": replay.total_event_count, "initial_expected_seen": initial_expected_seen, "followup_expected_seen": followup_expected_seen, "initial_codex_home": initial_codex_home.to_string_lossy(), diff --git a/src-tauri/src/commands/handler_list.inc b/src-tauri/src/commands/handler_list.inc index 1ec9088fb..5c24ad4de 100644 --- a/src-tauri/src/commands/handler_list.inc +++ b/src-tauri/src/commands/handler_list.inc @@ -273,8 +273,6 @@ infrastructure::archive::create_folder_archive, infrastructure::archive::get_folder_info, // Session Persistence commands (SQLite for fast local storage) session_persistence::cache_save_events, -session_persistence::cache_load_events, -session_persistence::cache_load_turn_index, orgtrack::history_commands::orgtrack_session_turn_metadata_index, // Usage dashboard (chat pane → Runtime → Usage): read-only rollups orgtrack::usage_dashboard_commands::usage_dashboard_summary, @@ -533,7 +531,6 @@ agent_sessions::cli::commands::cli_agent_history_mutation, agent_sessions::cli::commands::cli_agent_cancel, agent_sessions::cli::commands::cli_agent_tui_release, agent_sessions::cli::commands::cli_agent_list, -agent_sessions::cli::commands::cli_agent_chunks, agent_sessions::cli::commands::cli_agent_transcript_path, agent_sessions::cli::commands::cli_agent_truncate_after_chunk, agent_sessions::cli::commands::cli_agent_delete, @@ -1007,50 +1004,51 @@ agent_core::mcp::registries::glama::glama_detail, // Tokenizer // Orgtrack history sources orgtrack::history_commands::orgtrack_get_cursor_sessions, -orgtrack::history_commands::cursor_ide_chunks, -orgtrack::history_commands::cursor_ide_composer_last_updated_at, -orgtrack::history_commands::cursor_ide_initial_window, -orgtrack::history_commands::cursor_ide_full_refresh, -orgtrack::history_commands::cursor_ide_turn_window, -orgtrack::history_commands::codex_app_chunks, -orgtrack::history_commands::codex_app_initial_window, -orgtrack::history_commands::codex_app_turn_window, orgtrack::history_commands::codex_app_recent_paths, -orgtrack::history_commands::claude_code_history_chunks, -orgtrack::history_commands::claude_code_history_stat, -orgtrack::history_commands::imported_history_stat, orgtrack::history_commands::claude_code_recent_paths, -orgtrack::history_commands::opencode_history_chunks, orgtrack::history_commands::opencode_recent_paths, -orgtrack::history_commands::cursor_cli_history_chunks, -orgtrack::history_commands::cursor_cli_history_stat, orgtrack::history_commands::cursor_cli_recent_paths, -orgtrack::history_commands::warp_history_chunks, orgtrack::history_commands::warp_recent_paths, -orgtrack::history_commands::zcode_history_chunks, orgtrack::history_commands::zcode_recent_paths, -orgtrack::history_commands::qoder_history_chunks, orgtrack::history_commands::qoder_recent_paths, -orgtrack::history_commands::mimo_code_history_chunks, orgtrack::history_commands::mimo_code_recent_paths, -orgtrack::history_commands::omp_history_chunks, orgtrack::history_commands::omp_recent_paths, -orgtrack::history_commands::qoder_cli_history_chunks, orgtrack::history_commands::qoder_cli_recent_paths, orgtrack::history_commands::external_cli_sources_detect, orgtrack::history_commands::external_cli_source_probe, orgtrack::history_commands::external_history_rescan_source, orgtrack::history_commands::external_history_rescan_sources, +orgtrack::history_commands::external_history_recent_paths, orgtrack::history_commands::external_history_auto_import_recent_paths, -orgtrack::history_commands::windsurf_history_chunks, orgtrack::history_commands::windsurf_recent_paths, -orgtrack::history_commands::trae_history_chunks, orgtrack::history_commands::trae_recent_paths, -orgtrack::history_commands::cline_history_chunks, orgtrack::history_commands::cline_recent_paths, -orgtrack::history_commands::workbuddy_history_chunks, orgtrack::history_commands::workbuddy_recent_paths, orgtrack::history_commands::external_history_source_stats, +// Bounded external/managed CLI replay (#443). Foreground open/poll/read own +// EventStore delivery; query_window is explicitly side-effect-free for hover +// and raw/export previews, while prewarm_window owns one guarded Rust apply. +agent_sessions::event_pipeline::commands::external_replay::external_replay_open_window, +agent_sessions::event_pipeline::commands::external_replay::external_replay_poll_delta, +agent_sessions::event_pipeline::commands::external_replay::external_replay_read_window, +agent_sessions::event_pipeline::commands::external_replay::external_replay_query_window, +agent_sessions::event_pipeline::commands::external_replay::external_replay_handoff, +agent_sessions::event_pipeline::commands::external_replay::external_replay_prewarm_window, +agent_sessions::event_pipeline::commands::external_replay::external_replay_release, +agent_sessions::event_pipeline::commands::external_replay::external_replay_read_payload_range, +agent_sessions::event_pipeline::commands::external_replay::external_replay_stream_export, +agent_sessions::event_pipeline::commands::external_replay::external_replay_cloud_prepare, +agent_sessions::event_pipeline::commands::external_replay::external_replay_cloud_read_batch, +agent_sessions::event_pipeline::commands::external_replay::external_replay_cloud_prefix_hash, +agent_sessions::event_pipeline::commands::external_replay::external_replay_cloud_release, +// Byte-bounded Cloud collaboration snapshot ingest. Opaque compressed pages +// are staged in Rust and published to sessions.db atomically. +agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_ingest_begin, +agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_ingest_apply_wire_page, +agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_ingest_commit, +agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_ingest_abort, +agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_ingest_get_cursor, +agent_sessions::event_pipeline::commands::collaboration_snapshot_ingest::collaboration_snapshot_secondary_probe, // Remote SQL client commands (PostgreSQL + MySQL via sqlx). // Crate: src-tauri/crates/db-clients. Owns the `sqlx` dep tree privately // so it never enters `cargo check -p org2`. See the crate's Cargo.toml @@ -1101,6 +1099,7 @@ agent_sessions::event_pipeline::commands::cache_save_session_events, agent_sessions::event_pipeline::commands::cache_append_session_event_import, agent_sessions::event_pipeline::commands::cache_finalize_session_event_import, agent_sessions::event_pipeline::commands::cache_load_session_events, +agent_sessions::event_pipeline::commands::cache_load_native_cached_events, agent_sessions::event_pipeline::commands::cache_load_session_turn_body, agent_sessions::event_pipeline::commands::cache_load_session_initial_turn_window, agent_sessions::event_pipeline::commands::cache_search_session_events, @@ -1118,8 +1117,9 @@ agent_sessions::event_pipeline::commands::es_paginate_events, agent_sessions::event_pipeline::commands::es_paginate_cached_events, agent_sessions::event_pipeline::commands::es_count_matching_events, agent_sessions::event_pipeline::commands::es_get_distinct_functions, -// Durable shell replay (range reads never block Session Replay playback). -agent_core::tools::impls::coding::exec::shell_replay::shell_replay_read_range, +// Durable shell replay. External CLI manifests use canonical replay payload +// artifacts; native SDE sessions fall through to the original #425 `.slog`. +agent_sessions::event_pipeline::commands::external_replay::shell_replay_read_range, // Batch update commands agent_sessions::event_pipeline::commands::es_complete_last_running, agent_sessions::event_pipeline::commands::es_patch_by_ids, @@ -1148,6 +1148,7 @@ agent_sessions::event_pipeline::commands::debug_seed_child_session, // Unified Session API commands agent_sessions::session_directory::commands::session_aggregate_list, agent_sessions::session_directory::commands::session_external_history_sidebar_list, +agent_sessions::session_directory::commands::session_sidebar_workspace_facets, agent_sessions::session_directory::patch::session_patch, // Flow Awareness commands (user activity tracking for intent inference) agent_core::flow_awareness::commands::flow_record_activity, diff --git a/src-tauri/src/orgtrack/builder_profile_commands.rs b/src-tauri/src/orgtrack/builder_profile_commands.rs index 54827ac7d..048cd7704 100644 --- a/src-tauri/src/orgtrack/builder_profile_commands.rs +++ b/src-tauri/src/orgtrack/builder_profile_commands.rs @@ -213,8 +213,8 @@ pub async fn builder_profile_extract(limit: Option) -> Result = provenance .iter() .map(|row| row.session_id.clone()) @@ -379,15 +386,11 @@ pub fn export_orgtrack( } if tier.includes_trajectory() { - let trajectory = OrgtrackSessionTrajectory { - schema_version: ORGTRACK_SCHEMA_VERSION, - tier, - session_id: session_id.clone(), - raw_events: load_raw_events(&conn, session_id)?, - }; - paths::write_json_pretty( + write_trajectory_atomic( + &mut conn, &paths::session_trajectory_path(repo_path, session_id), - &trajectory, + tier, + session_id, )?; } @@ -555,3 +558,202 @@ pub fn export_orgtrack( manifest_version, }) } + +fn write_trajectory_atomic( + conn: &mut rusqlite::Connection, + target: &Path, + tier: OrgtrackTier, + session_id: &str, +) -> Result<(), String> { + atomic_write_buffered(target, |writer| { + write_session_trajectory(conn, writer, ORGTRACK_SCHEMA_VERSION, tier, session_id) + }) +} + +/// Write beside the destination, durably publish with one rename, and remove +/// the private UUID temp file on every pre-publication failure. The existing +/// trajectory is never opened for writing and therefore remains valid unless +/// the replacement is fully flushed and synced. +fn atomic_write_buffered( + target: &Path, + write_value: impl FnOnce(&mut BufWriter) -> Result<(), String>, +) -> Result<(), String> { + let parent = target + .parent() + .ok_or_else(|| format!("Trajectory path has no parent: {}", target.display()))?; + fs::create_dir_all(parent) + .map_err(|err| format!("Failed to create {}: {err}", parent.display()))?; + let temp = trajectory_temp_path(target)?; + let result = (|| { + let file = OpenOptions::new() + .create_new(true) + .write(true) + .open(&temp) + .map_err(|err| format!("Failed to create {}: {err}", temp.display()))?; + let mut writer = BufWriter::with_capacity(TRAJECTORY_WRITER_BUFFER_BYTES, file); + write_value(&mut writer)?; + writer + .flush() + .map_err(|err| format!("Failed to flush {}: {err}", temp.display()))?; + writer + .get_ref() + .sync_all() + .map_err(|err| format!("Failed to sync {}: {err}", temp.display()))?; + drop(writer); + atomic_replace(&temp, target).map_err(|err| { + format!( + "Failed to atomically replace {} from {}: {err}", + target.display(), + temp.display() + ) + })?; + // The file contents are already durable. Directory sync is best + // effort because some supported platforms do not allow opening a + // directory as a file descriptor. + if let Ok(directory) = File::open(parent) { + let _ = directory.sync_all(); + } + Ok(()) + })(); + if result.is_err() { + match fs::remove_file(&temp) { + Ok(()) => {} + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => tracing::warn!( + path = %temp.display(), + error = %err, + "Failed to clean interrupted trajectory export temp file" + ), + } + } + result +} + +#[cfg(not(windows))] +fn atomic_replace(temp: &Path, target: &Path) -> std::io::Result<()> { + fs::rename(temp, target) +} + +/// `std::fs::rename` does not replace an existing destination on Windows. +/// `MoveFileExW(REPLACE_EXISTING | WRITE_THROUGH)` provides the same-volume +/// atomic replacement contract needed by repeat trajectory exports, without +/// first deleting the last valid target. +#[cfg(windows)] +fn atomic_replace(temp: &Path, target: &Path) -> std::io::Result<()> { + use std::os::windows::ffi::OsStrExt; + + const MOVEFILE_REPLACE_EXISTING: u32 = 0x0000_0001; + const MOVEFILE_WRITE_THROUGH: u32 = 0x0000_0008; + #[link(name = "Kernel32")] + unsafe extern "system" { + fn MoveFileExW(existing: *const u16, replacement: *const u16, flags: u32) -> i32; + } + + let existing = temp + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + let replacement = target + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + // SAFETY: both pointers reference NUL-terminated UTF-16 buffers that live + // for the duration of the call; flags request an in-place same-volume + // replacement and do not retain either pointer. + let moved = unsafe { + MoveFileExW( + existing.as_ptr(), + replacement.as_ptr(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH, + ) + }; + if moved == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } +} + +fn trajectory_temp_path(target: &Path) -> Result { + let parent = target + .parent() + .ok_or_else(|| format!("Trajectory path has no parent: {}", target.display()))?; + let file_name = target + .file_name() + .and_then(|name| name.to_str()) + .ok_or_else(|| { + format!( + "Trajectory path has no UTF-8 file name: {}", + target.display() + ) + })?; + Ok(parent.join(format!( + ".{file_name}.{}.tmp", + uuid::Uuid::new_v4().simple() + ))) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn atomic_trajectory_failure_preserves_old_target_and_cleans_temp() { + let directory = tempfile::tempdir().expect("trajectory temp directory"); + let target = directory.path().join("session.trajectory.json"); + fs::write(&target, b"old-valid-trajectory").expect("old trajectory"); + + let error = atomic_write_buffered(&target, |writer| { + writer.write_all(b"partial-new-trajectory").unwrap(); + Err("injected trajectory failure".to_string()) + }) + .expect_err("injected writer failure"); + + assert!(error.contains("injected trajectory failure")); + assert_eq!( + fs::read(&target).expect("preserved old trajectory"), + b"old-valid-trajectory" + ); + let names = fs::read_dir(directory.path()) + .expect("trajectory directory") + .map(|entry| entry.unwrap().file_name()) + .collect::>(); + assert_eq!(names, vec![target.file_name().unwrap()]); + } + + #[test] + fn atomic_trajectory_repeat_export_replaces_existing_target() { + let directory = tempfile::tempdir().expect("trajectory temp directory"); + let target = directory.path().join("session.trajectory.json"); + fs::write(&target, b"old-valid-trajectory").expect("old trajectory"); + + atomic_write_buffered(&target, |writer| { + writer + .write_all(b"new-complete-trajectory") + .map_err(|err| format!("test trajectory write failed: {err}")) + }) + .expect("replace existing trajectory"); + + assert_eq!( + fs::read(&target).expect("replacement trajectory"), + b"new-complete-trajectory" + ); + assert_eq!( + fs::read_dir(directory.path()) + .expect("trajectory directory") + .count(), + 1, + "UUID temp must be consumed by atomic replace" + ); + } + + #[test] + fn trajectory_writer_buffer_is_hard_capped_below_two_mib() { + const { + assert!(TRAJECTORY_WRITER_BUFFER_BYTES <= 2 * 1024 * 1024); + } + assert_eq!(TRAJECTORY_WRITER_BUFFER_BYTES, 64 * 1024); + } +} diff --git a/src-tauri/src/orgtrack/exporter/loaders.rs b/src-tauri/src/orgtrack/exporter/loaders.rs index 711a408f1..23ad6fcd2 100644 --- a/src-tauri/src/orgtrack/exporter/loaders.rs +++ b/src-tauri/src/orgtrack/exporter/loaders.rs @@ -2,19 +2,32 @@ //! sessions, commit links, and raw trajectory events, plus schema introspection. use std::collections::{BTreeMap, BTreeSet}; +use std::io::Write; use std::path::Path; use chrono::{DateTime, Utc}; +use orgtrack_core::projectors::turn_metadata::metadata_projection_requirements; +use orgtrack_core::sources::imported_history::replay::{ + self, ImportedHistorySourceId, ReplayCursor, ReplayIndexedChunk, ReplayLimits, + ReplayPayloadDescriptor, +}; use rusqlite::params; +use serde::Serialize; +use serde_json::Value; use super::file_paths::{ extract_file_paths_from_json, is_file_edit_function, path_belongs_to_repo, }; use super::{LocalEditRow, ProvenanceRow, SessionRow}; -use crate::orgtrack::types::{OrgtrackRawEvent, OrgtrackRawEventSource}; +use crate::orgtrack::types::OrgtrackTier; + +/// SQLite BLOB and replay payload reads are kept independently bounded even +/// when one trajectory field is hundreds of MiB. The outer BufWriter is +/// capped separately in `export.rs`. +pub(super) const TRAJECTORY_PAYLOAD_RANGE_BYTES: usize = 256 * 1024; pub(super) fn load_local_edit_rows( - conn: &rusqlite::Connection, + conn: &mut rusqlite::Connection, repo_path: &Path, ) -> Result, String> { let mut rows = Vec::new(); @@ -26,24 +39,44 @@ pub(super) fn load_local_edit_rows( ORDER BY created_at ASC", ) .map_err(|err| format!("Prepare failed: {}", err))?; - let mapped = stmt - .query_map([], |row| { - Ok(( - row.get::<_, String>(0)?, - row.get::<_, String>(1)?, - row.get::<_, Option>(2)?, - row.get::<_, String>(3)?, - row.get::<_, String>(4)?, - row.get::<_, String>(5)?, - )) - }) + let mut mapped = stmt + .query([]) .map_err(|err| format!("Query failed: {}", err))?; - for row in mapped { - let (event_id, session_id, function_name, args_json, result_json, created_at) = - row.map_err(|err| format!("Row decode failed: {}", err))?; + while let Some(row) = mapped + .next() + .map_err(|err| format!("Row decode failed: {}", err))? + { + let event_id = row + .get::<_, String>(0) + .map_err(|err| format!("Row decode failed: {}", err))?; + let session_id = row + .get::<_, String>(1) + .map_err(|err| format!("Row decode failed: {}", err))?; + let function_name = row + .get::<_, Option>(2) + .map_err(|err| format!("Row decode failed: {}", err))?; let Some(function_name_value) = function_name.as_deref() else { continue; }; + // Read the lightweight tool discriminator before materializing + // either JSON column. Known non-edit tools (assistant/thinking, + // Grep, Node REPL, reads, shell, and so on) cannot contribute a + // local-edit row, while unknown tools retain the conservative + // old path through the shared projection classifier. + if !metadata_projection_requirements(Some(function_name_value)) + .projects_modified_files() + { + continue; + } + let args_json = row + .get::<_, String>(3) + .map_err(|err| format!("Row decode failed: {}", err))?; + let result_json = row + .get::<_, String>(4) + .map_err(|err| format!("Row decode failed: {}", err))?; + let created_at = row + .get::<_, String>(5) + .map_err(|err| format!("Row decode failed: {}", err))?; if !is_file_edit_function(function_name_value) { continue; } @@ -71,21 +104,34 @@ pub(super) fn load_local_edit_rows( ORDER BY sequence ASC", ) .map_err(|err| format!("Prepare failed: {}", err))?; - let mapped = stmt - .query_map([], |row| { - Ok(( - row.get::<_, String>(0)?, - row.get::<_, String>(1)?, - row.get::<_, String>(2)?, - row.get::<_, String>(3)?, - row.get::<_, String>(4)?, - row.get::<_, String>(5)?, - )) - }) + let mut mapped = stmt + .query([]) .map_err(|err| format!("Query failed: {}", err))?; - for row in mapped { - let (event_id, session_id, function_name, args_json, result_json, created_at) = - row.map_err(|err| format!("Row decode failed: {}", err))?; + while let Some(row) = mapped + .next() + .map_err(|err| format!("Row decode failed: {}", err))? + { + let event_id = row + .get::<_, String>(0) + .map_err(|err| format!("Row decode failed: {}", err))?; + let session_id = row + .get::<_, String>(1) + .map_err(|err| format!("Row decode failed: {}", err))?; + let function_name = row + .get::<_, String>(2) + .map_err(|err| format!("Row decode failed: {}", err))?; + if !metadata_projection_requirements(Some(&function_name)).projects_modified_files() { + continue; + } + let args_json = row + .get::<_, String>(3) + .map_err(|err| format!("Row decode failed: {}", err))?; + let result_json = row + .get::<_, String>(4) + .map_err(|err| format!("Row decode failed: {}", err))?; + let created_at = row + .get::<_, String>(5) + .map_err(|err| format!("Row decode failed: {}", err))?; if !is_file_edit_function(&function_name) { continue; } @@ -124,33 +170,39 @@ pub(super) fn load_local_edit_rows( else { continue; }; - let Ok(Some(chunks)) = - orgtrack_core::sources::imported_history::load_activity_chunks_for_session( - conn, - &imported_id, - ) - else { - continue; - }; - for chunk in chunks { - if !is_file_edit_function(&chunk.function) { - continue; - } - let args_json = chunk.args.to_string(); - let result_json = chunk.result.to_string(); - for file_path in - extract_file_paths_from_json(&chunk.function, &args_json, &result_json) - { - if path_belongs_to_repo(conn, repo_path, &session_id, &file_path)? { - rows.push(LocalEditRow { - event_id: chunk.chunk_id.clone(), - session_id: session_id.clone(), - file_path, - function_name: Some(chunk.function.clone()), - created_at: parse_timestamp(&chunk.created_at), - }); + let replay_result = for_each_imported_replay_chunk( + conn, + &imported_id, + |replay_conn, _generation, indexed| { + let chunk = indexed.chunk; + if !is_file_edit_function(&chunk.function) { + return Ok(()); } - } + let args_json = chunk.args.to_string(); + let result_json = chunk.result.to_string(); + for file_path in + extract_file_paths_from_json(&chunk.function, &args_json, &result_json) + { + if path_belongs_to_repo(replay_conn, repo_path, &session_id, &file_path)? { + rows.push(LocalEditRow { + event_id: chunk.chunk_id.clone(), + session_id: session_id.clone(), + file_path, + function_name: Some(chunk.function.clone()), + created_at: parse_timestamp(&chunk.created_at), + }); + } + } + Ok(()) + }, + ); + if let Err(err) = replay_result { + tracing::warn!( + session_id, + imported_id, + error = %err, + "Skipping managed native transcript during edit export" + ); } } } @@ -163,6 +215,221 @@ pub(super) fn load_local_edit_rows( Ok(rows) } +#[cfg(test)] +mod local_edit_projection_tests { + use super::*; + use rusqlite::{params, Connection}; + + fn test_connection() -> Connection { + let conn = Connection::open_in_memory().expect("open test database"); + conn.execute_batch( + "CREATE TABLE events ( + id TEXT NOT NULL, + session_id TEXT NOT NULL, + function_name TEXT, + args_json, + result_json, + created_at TEXT NOT NULL + ); + CREATE TABLE code_session_chunks ( + chunk_id TEXT NOT NULL, + session_id TEXT NOT NULL, + function TEXT NOT NULL, + args_json, + result_json, + created_at TEXT NOT NULL, + sequence INTEGER NOT NULL + );", + ) + .expect("create test schema"); + conn + } + + fn insert_valid_rows(conn: &Connection) { + let large_text = format!(r#"{{"text":"{}"}}"#, "x".repeat(128 * 1024)); + for (id, function_name) in [ + ("assistant", "assistant"), + ("thinking", "thinking"), + ("grep", "Grep"), + ("node", "node_repl"), + ("unknown", "future_provider_tool"), + ] { + conn.execute( + "INSERT INTO events + (id, session_id, function_name, args_json, result_json, created_at) + VALUES (?1, 'session', ?2, ?3, ?3, '2026-01-01T00:00:00Z')", + params![id, function_name, large_text], + ) + .expect("insert event fixture"); + } + conn.execute( + "INSERT INTO events + (id, session_id, function_name, args_json, result_json, created_at) + VALUES ('edit', 'session', 'edit_file', + '{\"file_path\":\"src/event.rs\"}', '{}', + '2026-01-01T00:00:01Z')", + [], + ) + .expect("insert event edit"); + conn.execute( + "INSERT INTO code_session_chunks + (chunk_id, session_id, function, args_json, result_json, created_at, sequence) + VALUES ('patch', 'session', 'apply_patch', + '{\"patch_text\":\"*** Update File: src/chunk.rs\\n\"}', '{}', + '2026-01-01T00:00:02Z', 1)", + [], + ) + .expect("insert chunk edit"); + conn.execute( + "INSERT INTO code_session_chunks + (chunk_id, session_id, function, args_json, result_json, created_at, sequence) + VALUES ('chunk-unknown', 'session', 'future_chunk_tool', + '{\"file_path\":\"src/ignored.rs\"}', '{}', + '2026-01-01T00:00:03Z', 2)", + [], + ) + .expect("insert unknown chunk"); + } + + /// Test-only copy of the pre-short-circuit projection. It intentionally + /// materializes both JSON columns before inspecting the tool name. + fn load_local_edit_rows_legacy( + conn: &Connection, + repo_path: &Path, + ) -> Result, String> { + let mut stmt = conn + .prepare( + "SELECT id, session_id, function_name, args_json, result_json, created_at + FROM events + UNION ALL + SELECT chunk_id, session_id, function, args_json, result_json, created_at + FROM code_session_chunks", + ) + .map_err(|err| err.to_string())?; + let mapped = stmt + .query_map([], |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, Option>(2)?, + row.get::<_, String>(3)?, + row.get::<_, String>(4)?, + row.get::<_, String>(5)?, + )) + }) + .map_err(|err| err.to_string())?; + let mut rows = Vec::new(); + for row in mapped { + let (event_id, session_id, function_name, args_json, result_json, created_at) = + row.map_err(|err| err.to_string())?; + let Some(function_name_value) = function_name.as_deref() else { + continue; + }; + if !is_file_edit_function(function_name_value) { + continue; + } + for file_path in + extract_file_paths_from_json(function_name_value, &args_json, &result_json) + { + if path_belongs_to_repo(conn, repo_path, &session_id, &file_path)? { + rows.push(LocalEditRow { + event_id: event_id.clone(), + session_id: session_id.clone(), + file_path, + function_name: function_name.clone(), + created_at: parse_timestamp(&created_at), + }); + } + } + } + rows.sort_by(|left, right| { + left.created_at + .cmp(&right.created_at) + .then_with(|| left.event_id.cmp(&right.event_id)) + }); + Ok(rows) + } + + fn row_signatures( + rows: Vec, + ) -> Vec<(String, String, String, Option, i64)> { + rows.into_iter() + .map(|row| { + ( + row.event_id, + row.session_id, + row.file_path, + row.function_name, + row.created_at, + ) + }) + .collect() + } + + #[test] + fn local_edit_projection_matches_legacy_for_edits_and_unknown_tools() { + let mut conn = test_connection(); + insert_valid_rows(&conn); + + let legacy = load_local_edit_rows_legacy(&conn, Path::new("/repo")) + .expect("legacy projection succeeds"); + let projected = + load_local_edit_rows(&mut conn, Path::new("/repo")).expect("projection succeeds"); + + assert_eq!(row_signatures(projected), row_signatures(legacy)); + } + + #[test] + fn large_known_non_edit_payloads_are_not_read_but_unknown_tools_remain_conservative() { + let mut conn = test_connection(); + let large_blob = vec![b'x'; 1024 * 1024]; + for (index, function_name) in ["assistant", "thinking", "Grep", "node_repl"] + .into_iter() + .enumerate() + { + conn.execute( + "INSERT INTO events + (id, session_id, function_name, args_json, result_json, created_at) + VALUES (?1, 'session', ?2, ?3, ?3, '2026-01-01T00:00:00Z')", + params![format!("event-{index}"), function_name, &large_blob], + ) + .expect("insert large event payload"); + conn.execute( + "INSERT INTO code_session_chunks + (chunk_id, session_id, function, args_json, result_json, created_at, sequence) + VALUES (?1, 'session', ?2, ?3, ?3, '2026-01-01T00:00:00Z', ?4)", + params![format!("chunk-{index}"), function_name, &large_blob, index], + ) + .expect("insert large chunk payload"); + } + conn.execute( + "INSERT INTO events + (id, session_id, function_name, args_json, result_json, created_at) + VALUES ('edit', 'session', 'edit_file', + '{\"file_path\":\"src/kept.rs\"}', '{}', + '2026-01-01T00:00:01Z')", + [], + ) + .expect("insert edit event"); + + let rows = load_local_edit_rows(&mut conn, Path::new("/repo")) + .expect("known non-edit BLOB columns must not be decoded as strings"); + assert_eq!(row_signatures(rows).len(), 1); + + conn.execute( + "INSERT INTO events + (id, session_id, function_name, args_json, result_json, created_at) + VALUES ('unknown', 'session', 'future_provider_tool', ?1, ?1, + '2026-01-01T00:00:02Z')", + params![&large_blob], + ) + .expect("insert unknown payload"); + let err = load_local_edit_rows(&mut conn, Path::new("/repo")) + .expect_err("unknown tools must retain conservative payload materialization"); + assert!(err.contains("Row decode failed"), "unexpected error: {err}"); + } +} + pub(super) fn load_provenance_rows( conn: &rusqlite::Connection, repo_path: &Path, @@ -344,88 +611,784 @@ pub(super) fn load_commit_links( Ok(links) } -pub(super) fn load_raw_events( +/// Stream the legacy trajectory schema without ever constructing a +/// session-sized `Vec` or JSON string. Field order, +/// indentation, enum spelling and camelCase names intentionally match +/// `serde_json::to_string_pretty(OrgtrackSessionTrajectory)` byte-for-byte. +pub(super) fn write_session_trajectory( + conn: &mut rusqlite::Connection, + writer: &mut W, + schema_version: u32, + tier: OrgtrackTier, + session_id: &str, +) -> Result<(), String> { + let imported_id = + crate::agent_sessions::cli::native_transcript::imported_transcript_id_for_managed_session( + session_id, + ); + write_session_trajectory_with_imported_id( + conn, + writer, + schema_version, + tier, + session_id, + imported_id.as_deref(), + ) +} + +fn write_session_trajectory_with_imported_id( + conn: &mut rusqlite::Connection, + writer: &mut W, + schema_version: u32, + tier: OrgtrackTier, + session_id: &str, + imported_id: Option<&str>, +) -> Result<(), String> { + writer + .write_all(b"{\n \"schemaVersion\": ") + .map_err(write_error)?; + write_json(writer, &schema_version)?; + writer.write_all(b",\n \"tier\": ").map_err(write_error)?; + write_json(writer, &tier)?; + writer + .write_all(b",\n \"sessionId\": ") + .map_err(write_error)?; + write_json(writer, &session_id)?; + writer + .write_all(b",\n \"rawEvents\": [") + .map_err(write_error)?; + + let mut wrote_event = false; + write_native_events(conn, writer, session_id, &mut wrote_event)?; + write_native_code_session_chunks(conn, writer, session_id, &mut wrote_event)?; + + // Native-mode managed sessions persist no chunk rows. Their canonical + // transcript remains in the CLI store, so stream its compact replay index + // with a generation+revision pin and restore every deferred payload range. + if let Some(imported_id) = imported_id { + for_each_imported_replay_chunk(conn, imported_id, |replay_conn, generation, indexed| { + write_imported_event( + replay_conn, + writer, + imported_id, + generation, + indexed, + &mut wrote_event, + ) + })?; + } + + if wrote_event { + writer.write_all(b"\n ]\n}").map_err(write_error)?; + } else { + writer.write_all(b"]\n}").map_err(write_error)?; + } + Ok(()) +} + +fn write_native_events( conn: &rusqlite::Connection, + writer: &mut W, session_id: &str, -) -> Result, String> { - let mut raw_events = Vec::new(); - if table_exists(conn, "events")? { - let mut stmt = conn - .prepare( - "SELECT function_name, args_json, result_json, history_sequence, created_at - FROM events - WHERE session_id = ?1 - ORDER BY COALESCE(history_sequence, 0) ASC, created_at ASC", + wrote_event: &mut bool, +) -> Result<(), String> { + if !table_exists(conn, "events")? { + return Ok(()); + } + let mut args_range = conn + .prepare( + "SELECT substr(CAST(args_json AS BLOB), ?2, ?3) + FROM events WHERE rowid = ?1", + ) + .map_err(|err| format!("Prepare event args range failed: {err}"))?; + let mut result_range = conn + .prepare( + "SELECT substr(CAST(result_json AS BLOB), ?2, ?3) + FROM events WHERE rowid = ?1", + ) + .map_err(|err| format!("Prepare event result range failed: {err}"))?; + let mut stmt = conn + .prepare( + "SELECT rowid, function_name, + length(CAST(args_json AS BLOB)), + length(CAST(result_json AS BLOB)), + history_sequence, created_at + FROM events + WHERE session_id = ?1 + ORDER BY COALESCE(history_sequence, 0) ASC, created_at ASC", + ) + .map_err(|err| format!("Prepare events failed: {err}"))?; + let mut rows = stmt + .query([session_id]) + .map_err(|err| format!("Query events failed: {err}"))?; + while let Some(row) = rows + .next() + .map_err(|err| format!("Read event row failed: {err}"))? + { + let rowid = row.get::<_, i64>(0).map_err(row_error)?; + let name = row.get::<_, Option>(1).map_err(row_error)?; + let args_bytes = optional_length(row, 2)?; + let result_bytes = optional_length(row, 3)?; + let sequence = row.get::<_, Option>(4).map_err(row_error)?; + let created_at = row.get::<_, Option>(5).map_err(row_error)?; + begin_raw_event(writer, wrote_event, "event", name.as_deref())?; + write_sqlite_blob_json_string(writer, &mut args_range, rowid, args_bytes, "event args")?; + writer + .write_all(b",\n \"resultJson\": ") + .map_err(write_error)?; + write_sqlite_blob_json_string( + writer, + &mut result_range, + rowid, + result_bytes, + "event result", + )?; + finish_raw_event(writer, sequence, created_at.as_deref())?; + } + Ok(()) +} + +fn write_native_code_session_chunks( + conn: &rusqlite::Connection, + writer: &mut W, + session_id: &str, + wrote_event: &mut bool, +) -> Result<(), String> { + if !table_exists(conn, "code_session_chunks")? { + return Ok(()); + } + let mut args_range = conn + .prepare( + "SELECT substr(CAST(args_json AS BLOB), ?2, ?3) + FROM code_session_chunks WHERE rowid = ?1", + ) + .map_err(|err| format!("Prepare code-session args range failed: {err}"))?; + let mut result_range = conn + .prepare( + "SELECT substr(CAST(result_json AS BLOB), ?2, ?3) + FROM code_session_chunks WHERE rowid = ?1", + ) + .map_err(|err| format!("Prepare code-session result range failed: {err}"))?; + let mut stmt = conn + .prepare( + "SELECT rowid, function, + length(CAST(args_json AS BLOB)), + length(CAST(result_json AS BLOB)), + sequence, created_at + FROM code_session_chunks + WHERE session_id = ?1 + ORDER BY sequence ASC", + ) + .map_err(|err| format!("Prepare code-session chunks failed: {err}"))?; + let mut rows = stmt + .query([session_id]) + .map_err(|err| format!("Query code-session chunks failed: {err}"))?; + while let Some(row) = rows + .next() + .map_err(|err| format!("Read code-session row failed: {err}"))? + { + let rowid = row.get::<_, i64>(0).map_err(row_error)?; + let name = row.get::<_, Option>(1).map_err(row_error)?; + let args_bytes = optional_length(row, 2)?; + let result_bytes = optional_length(row, 3)?; + let sequence = row.get::<_, Option>(4).map_err(row_error)?; + let created_at = row.get::<_, Option>(5).map_err(row_error)?; + begin_raw_event(writer, wrote_event, "code_session_chunk", name.as_deref())?; + write_sqlite_blob_json_string( + writer, + &mut args_range, + rowid, + args_bytes, + "code-session args", + )?; + writer + .write_all(b",\n \"resultJson\": ") + .map_err(write_error)?; + write_sqlite_blob_json_string( + writer, + &mut result_range, + rowid, + result_bytes, + "code-session result", + )?; + finish_raw_event(writer, sequence, created_at.as_deref())?; + } + Ok(()) +} + +fn optional_length(row: &rusqlite::Row<'_>, index: usize) -> Result, String> { + row.get::<_, Option>(index) + .map_err(row_error) + .and_then(|length| match length { + Some(length) if length < 0 => Err("SQLite reported a negative JSON length".to_string()), + Some(length) => Ok(Some(length as u64)), + None => Ok(None), + }) +} + +fn write_sqlite_blob_json_string( + writer: &mut W, + range_stmt: &mut rusqlite::Statement<'_>, + rowid: i64, + total_bytes: Option, + field_label: &str, +) -> Result<(), String> { + let Some(total_bytes) = total_bytes else { + return writer.write_all(b"null").map_err(write_error); + }; + writer.write_all(b"\"").map_err(write_error)?; + let mut offset = 0_u64; + let mut pending_utf8 = Vec::with_capacity(4); + while offset < total_bytes { + let requested = (total_bytes - offset).min(TRAJECTORY_PAYLOAD_RANGE_BYTES as u64) as usize; + let bytes = range_stmt + .query_row( + params![rowid, offset.saturating_add(1) as i64, requested as i64], + |row| row.get::<_, Option>>(0), ) - .map_err(|err| format!("Prepare failed: {}", err))?; - let rows = stmt - .query_map([session_id], |row| { - Ok(OrgtrackRawEvent { - source: OrgtrackRawEventSource::Event, - name: row.get(0)?, - args_json: row.get(1)?, - result_json: row.get(2)?, - sequence: row.get(3)?, - created_at: row.get(4)?, - }) - }) - .map_err(|err| format!("Query failed: {}", err))?; - for row in rows { - raw_events.push(row.map_err(|err| format!("Row decode failed: {}", err))?); + .map_err(|err| format!("Read {field_label} range failed: {err}"))? + .unwrap_or_default(); + if bytes.len() != requested { + return Err(format!( + "{field_label} changed during trajectory export: expected {requested} bytes at offset {offset}, read {}", + bytes.len() + )); + } + offset = offset.saturating_add(bytes.len() as u64); + if pending_utf8.is_empty() { + write_utf8_json_content(writer, &bytes, &mut pending_utf8, field_label)?; + } else { + let mut joined = Vec::with_capacity(pending_utf8.len() + bytes.len()); + joined.extend_from_slice(&pending_utf8); + joined.extend_from_slice(&bytes); + pending_utf8.clear(); + write_utf8_json_content(writer, &joined, &mut pending_utf8, field_label)?; } } - if table_exists(conn, "code_session_chunks")? { - let mut stmt = conn - .prepare( - "SELECT function, args_json, result_json, sequence, created_at - FROM code_session_chunks - WHERE session_id = ?1 - ORDER BY sequence ASC", - ) - .map_err(|err| format!("Prepare failed: {}", err))?; - let rows = stmt - .query_map([session_id], |row| { - Ok(OrgtrackRawEvent { - source: OrgtrackRawEventSource::CodeSessionChunk, - name: row.get(0)?, - args_json: row.get(1)?, - result_json: row.get(2)?, - sequence: row.get(3)?, - created_at: row.get(4)?, - }) - }) - .map_err(|err| format!("Query failed: {}", err))?; - for row in rows { - raw_events.push(row.map_err(|err| format!("Row decode failed: {}", err))?); + if !pending_utf8.is_empty() { + return Err(format!("{field_label} ended with incomplete UTF-8")); + } + writer.write_all(b"\"").map_err(write_error) +} + +fn write_utf8_json_content( + writer: &mut W, + bytes: &[u8], + pending: &mut Vec, + field_label: &str, +) -> Result<(), String> { + match std::str::from_utf8(bytes) { + Ok(text) => write_escaped_json_content(writer, text), + Err(err) if err.error_len().is_none() => { + let valid = err.valid_up_to(); + write_escaped_json_content( + writer, + std::str::from_utf8(&bytes[..valid]).map_err(|decode| decode.to_string())?, + )?; + pending.extend_from_slice(&bytes[valid..]); + if pending.len() > 3 { + return Err(format!("{field_label} contains invalid UTF-8")); + } + Ok(()) } + Err(err) => Err(format!("{field_label} contains invalid UTF-8: {err}")), } - // Native-mode managed sessions persist no chunk rows — replay the - // imported transcript into the same raw-event shape so exported - // trajectories stay complete. - if let Some(imported_id) = - crate::agent_sessions::cli::native_transcript::imported_transcript_id_for_managed_session( +} + +fn write_imported_event( + conn: &mut rusqlite::Connection, + writer: &mut W, + imported_session_id: &str, + generation: &str, + indexed: ReplayIndexedChunk, + wrote_event: &mut bool, +) -> Result<(), String> { + let source = ImportedHistorySourceId::from_session_id(imported_session_id) + .ok_or_else(|| format!("Unknown imported transcript id: {imported_session_id}"))?; + let ReplayIndexedChunk { + sequence, + chunk, + payloads, + .. + } = indexed; + begin_raw_event( + writer, + wrote_event, + "code_session_chunk", + Some(&chunk.function), + )?; + write_replay_value_json_string( + conn, + writer, + source, + imported_session_id, + generation, + &chunk.chunk_id, + "args", + &chunk.args, + &payloads, + )?; + writer + .write_all(b",\n \"resultJson\": ") + .map_err(write_error)?; + write_replay_value_json_string( + conn, + writer, + source, + imported_session_id, + generation, + &chunk.chunk_id, + "result", + &chunk.result, + &payloads, + )?; + finish_raw_event(writer, Some(sequence), Some(&chunk.created_at)) +} + +#[allow(clippy::too_many_arguments)] +fn write_replay_value_json_string( + conn: &mut rusqlite::Connection, + writer: &mut W, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + root: &str, + value: &Value, + payloads: &[ReplayPayloadDescriptor], +) -> Result<(), String> { + writer.write_all(b"\"").map_err(write_error)?; + write_value_inside_outer_json_string( + conn, writer, source, session_id, generation, event_id, root, value, payloads, + )?; + writer.write_all(b"\"").map_err(write_error) +} + +#[allow(clippy::too_many_arguments)] +fn write_value_inside_outer_json_string( + conn: &mut rusqlite::Connection, + writer: &mut W, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + path: &str, + value: &Value, + payloads: &[ReplayPayloadDescriptor], +) -> Result<(), String> { + if let Some(payload) = payloads.iter().find(|payload| payload.field_path == path) { + return match payload.resolved_encoding() { + replay::ReplayPayloadEncoding::JsonValue => stream_replay_payload( + conn, writer, source, session_id, generation, event_id, path, false, + ), + replay::ReplayPayloadEncoding::Utf8Text => { + write_escaped_json_content(writer, "\"")?; + stream_replay_payload( + conn, writer, source, session_id, generation, event_id, path, true, + )?; + write_escaped_json_content(writer, "\"") + } + replay::ReplayPayloadEncoding::LegacyPathInferred => { + unreachable!("resolved replay payload encoding cannot remain legacy") + } + }; + } + match value { + Value::Null => write_escaped_json_content(writer, "null"), + Value::Bool(value) => { + write_escaped_json_content(writer, if *value { "true" } else { "false" }) + } + Value::Number(value) => write_escaped_json_content(writer, &value.to_string()), + Value::String(value) => write_inner_json_string(writer, value), + Value::Array(values) => { + write_escaped_json_content(writer, "[")?; + for (index, value) in values.iter().enumerate() { + if index > 0 { + write_escaped_json_content(writer, ",")?; + } + write_value_inside_outer_json_string( + conn, + writer, + source, + session_id, + generation, + event_id, + &format!("{path}.{index}"), + value, + payloads, + )?; + } + write_escaped_json_content(writer, "]") + } + Value::Object(values) => { + write_escaped_json_content(writer, "{")?; + let mut wrote_field = false; + for (key, value) in values { + if replay::is_compact_only_replay_field(key) { + continue; + } + if wrote_field { + write_escaped_json_content(writer, ",")?; + } + wrote_field = true; + write_inner_json_string(writer, key)?; + write_escaped_json_content(writer, ":")?; + let child_path = format!("{path}.{key}"); + write_value_inside_outer_json_string( + conn, + writer, + source, + session_id, + generation, + event_id, + &child_path, + value, + payloads, + )?; + } + write_escaped_json_content(writer, "}") + } + } +} + +#[allow(clippy::too_many_arguments)] +fn stream_replay_payload( + conn: &mut rusqlite::Connection, + writer: &mut W, + source: ImportedHistorySourceId, + session_id: &str, + generation: &str, + event_id: &str, + field_path: &str, + nested_string: bool, +) -> Result<(), String> { + let mut offset = 0_u64; + loop { + let range = replay::read_payload_range( + conn, + source, session_id, - ) - { - if let Some(chunks) = - orgtrack_core::sources::imported_history::load_activity_chunks_for_session( - conn, - &imported_id, - )? - { - for (index, chunk) in chunks.into_iter().enumerate() { - raw_events.push(OrgtrackRawEvent { - source: OrgtrackRawEventSource::CodeSessionChunk, - name: Some(chunk.function), - args_json: Some(chunk.args.to_string()), - result_json: Some(chunk.result.to_string()), - sequence: Some(index as i64), - created_at: Some(chunk.created_at), - }); + generation, + event_id, + field_path, + offset, + Some(TRAJECTORY_PAYLOAD_RANGE_BYTES), + )?; + if range.offset != offset { + return Err(format!( + "Replay payload {event_id}:{field_path} skipped from {offset} to {}", + range.offset + )); + } + if nested_string { + write_double_escaped_json_content(writer, &range.text)?; + } else { + write_escaped_json_content(writer, &range.text)?; + } + if range.next_offset <= offset && !range.eof { + return Err(format!( + "Replay payload {event_id}:{field_path} made no progress at {offset}" + )); + } + offset = range.next_offset; + if range.eof { + if offset != range.total_bytes { + return Err(format!( + "Replay payload {event_id}:{field_path} ended at {offset}, expected {}", + range.total_bytes + )); } + break; } } - Ok(raw_events) + Ok(()) +} + +fn begin_raw_event( + writer: &mut W, + wrote_event: &mut bool, + source: &str, + name: Option<&str>, +) -> Result<(), String> { + if *wrote_event { + writer.write_all(b",\n").map_err(write_error)?; + } else { + writer.write_all(b"\n").map_err(write_error)?; + *wrote_event = true; + } + writer + .write_all(b" {\n \"source\": ") + .map_err(write_error)?; + write_json(writer, &source)?; + writer + .write_all(b",\n \"name\": ") + .map_err(write_error)?; + write_json(writer, &name)?; + writer + .write_all(b",\n \"argsJson\": ") + .map_err(write_error) +} + +fn finish_raw_event( + writer: &mut W, + sequence: Option, + created_at: Option<&str>, +) -> Result<(), String> { + writer + .write_all(b",\n \"sequence\": ") + .map_err(write_error)?; + write_json(writer, &sequence)?; + writer + .write_all(b",\n \"createdAt\": ") + .map_err(write_error)?; + write_json(writer, &created_at)?; + writer.write_all(b"\n }").map_err(write_error) +} + +fn write_inner_json_string(writer: &mut W, text: &str) -> Result<(), String> { + write_escaped_json_content(writer, "\"")?; + write_double_escaped_json_content(writer, text)?; + write_escaped_json_content(writer, "\"") +} + +fn write_double_escaped_json_content(writer: &mut W, text: &str) -> Result<(), String> { + let mut plain_start = 0_usize; + for (index, character) in text.char_indices() { + if !needs_json_escape(character) { + continue; + } + if plain_start < index { + writer + .write_all(&text.as_bytes()[plain_start..index]) + .map_err(write_error)?; + } + let mut encoded = [0_u8; 4]; + let mut control = [0_u8; 6]; + let fragment = json_escape_fragment(character, &mut encoded, &mut control); + write_escaped_json_content(writer, fragment)?; + plain_start = index + character.len_utf8(); + } + if plain_start < text.len() { + writer + .write_all(&text.as_bytes()[plain_start..]) + .map_err(write_error)?; + } + Ok(()) +} + +fn write_escaped_json_content(writer: &mut W, text: &str) -> Result<(), String> { + let mut plain_start = 0_usize; + for (index, character) in text.char_indices() { + if !needs_json_escape(character) { + continue; + } + if plain_start < index { + writer + .write_all(&text.as_bytes()[plain_start..index]) + .map_err(write_error)?; + } + let mut encoded = [0_u8; 4]; + let mut control = [0_u8; 6]; + writer + .write_all(json_escape_fragment(character, &mut encoded, &mut control).as_bytes()) + .map_err(write_error)?; + plain_start = index + character.len_utf8(); + } + if plain_start < text.len() { + writer + .write_all(&text.as_bytes()[plain_start..]) + .map_err(write_error)?; + } + Ok(()) +} + +fn needs_json_escape(character: char) -> bool { + matches!( + character, + '"' | '\\' | '\u{08}' | '\u{0c}' | '\n' | '\r' | '\t' + ) || character <= '\u{1f}' +} + +fn json_escape_fragment<'a>( + character: char, + encoded: &'a mut [u8; 4], + control: &'a mut [u8; 6], +) -> &'a str { + match character { + '"' => "\\\"", + '\\' => "\\\\", + '\u{08}' => "\\b", + '\u{0c}' => "\\f", + '\n' => "\\n", + '\r' => "\\r", + '\t' => "\\t", + character if character <= '\u{1f}' => { + const HEX: &[u8; 16] = b"0123456789abcdef"; + let value = character as usize; + *control = [b'\\', b'u', b'0', b'0', HEX[value >> 4], HEX[value & 0x0f]]; + // The array is constructed from ASCII bytes only. + std::str::from_utf8(control).expect("ASCII JSON escape") + } + character => character.encode_utf8(encoded), + } +} + +fn write_json(writer: &mut W, value: &impl Serialize) -> Result<(), String> { + serde_json::to_writer(writer, value) + .map_err(|err| format!("Serialize trajectory failed: {err}")) +} + +fn write_error(err: std::io::Error) -> String { + format!("Write trajectory failed: {err}") +} + +fn row_error(err: rusqlite::Error) -> String { + format!("Decode trajectory row failed: {err}") +} + +fn for_each_imported_replay_chunk( + conn: &mut rusqlite::Connection, + imported_session_id: &str, + mut visit: impl FnMut(&mut rusqlite::Connection, &str, ReplayIndexedChunk) -> Result<(), String>, +) -> Result<(), String> { + let source = ImportedHistorySourceId::from_session_id(imported_session_id) + .ok_or_else(|| format!("Unknown imported transcript id: {imported_session_id}"))?; + let limits = ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: replay::HARD_MAX_EVENTS, + max_ipc_bytes: replay::HARD_MAX_IPC_BYTES, + }; + let prepared = { + // Synchronization and lazy-turn materialization can write the compact + // replay index, so keep only that preparation under the process-wide + // writer lock. The potentially long export below is read-only. + let _writer = database::db::sessions_writer_guard(); + replay::prepare_pinned_scan(conn, source, imported_session_id, limits)? + }; + let generation = prepared.generation; + let revision = prepared.revision; + if let Some(mut read_conn) = open_export_read_connection(conn)? { + visit_pinned_replay_chunks( + &mut read_conn, + source, + imported_session_id, + &generation, + revision, + limits, + &mut visit, + )?; + } else { + // In-memory SQLite is used by unit tests and has no second connection + // target. It still runs outside the writer guard. + visit_pinned_replay_chunks( + conn, + source, + imported_session_id, + &generation, + revision, + limits, + &mut visit, + )?; + } + + // Re-synchronize briefly after every payload range has been emitted. A + // generation or same-generation revision change rejects the UUID temp + // before the caller publishes it, without blocking unrelated writers + // during the streamed file I/O. + let final_scan = { + let _writer = database::db::sessions_writer_guard(); + replay::scan_window_after(conn, source, imported_session_id, i64::MAX, limits)? + }; + validate_pinned_replay_cursor( + imported_session_id, + &generation, + revision, + &final_scan.cursor, + )?; + if final_scan.has_more || !final_scan.chunks.is_empty() { + return Err(format!( + "Imported transcript grew after the pinned trajectory scan: {imported_session_id}" + )); + } + Ok(()) +} + +fn open_export_read_connection( + conn: &rusqlite::Connection, +) -> Result, String> { + let Some(path) = conn.path().filter(|path| !path.is_empty()) else { + return Ok(None); + }; + let read_conn = rusqlite::Connection::open_with_flags( + path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY + | rusqlite::OpenFlags::SQLITE_OPEN_URI + | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX, + ) + .map_err(|err| format!("Open replay export read connection failed: {err}"))?; + read_conn + .busy_timeout(std::time::Duration::from_secs(15)) + .map_err(|err| format!("Configure replay export read connection failed: {err}"))?; + Ok(Some(read_conn)) +} + +#[allow( + clippy::too_many_arguments, + reason = "Pinned replay scan keeps the immutable source identity and bounded visitor explicit" +)] +fn visit_pinned_replay_chunks( + conn: &mut rusqlite::Connection, + source: ImportedHistorySourceId, + imported_session_id: &str, + generation: &str, + revision: u64, + limits: ReplayLimits, + visit: &mut impl FnMut(&mut rusqlite::Connection, &str, ReplayIndexedChunk) -> Result<(), String>, +) -> Result<(), String> { + let mut after_sequence = -1_i64; + loop { + let batch = replay::scan_window_after_generation( + conn, + source, + imported_session_id, + generation, + revision, + after_sequence, + limits, + )?; + if batch.chunks.is_empty() + && batch.has_more + && batch.cursor.through_sequence <= after_sequence + { + return Err(format!( + "Bounded replay scan made no progress for {imported_session_id} after sequence {after_sequence}" + )); + } + let next_sequence = batch.cursor.through_sequence; + for indexed in batch.chunks { + visit(conn, generation, indexed)?; + } + after_sequence = next_sequence; + if !batch.has_more { + break; + } + } + Ok(()) +} + +fn validate_pinned_replay_cursor( + imported_session_id: &str, + expected_generation: &str, + expected_revision: u64, + cursor: &ReplayCursor, +) -> Result<(), String> { + if cursor.generation != expected_generation || cursor.revision != expected_revision { + return Err(format!( + "Imported transcript changed during trajectory export: expected {expected_generation}@{expected_revision}, found {}@{} for {imported_session_id}", + cursor.generation, cursor.revision + )); + } + Ok(()) } pub(super) fn table_exists(conn: &rusqlite::Connection, table: &str) -> Result { @@ -471,3 +1434,7 @@ fn get_optional_column( }; row.get(*index) } + +#[cfg(test)] +#[path = "loaders/tests.rs"] +mod tests; diff --git a/src-tauri/src/orgtrack/exporter/loaders/tests.rs b/src-tauri/src/orgtrack/exporter/loaders/tests.rs new file mode 100644 index 000000000..7c66d87fc --- /dev/null +++ b/src-tauri/src/orgtrack/exporter/loaders/tests.rs @@ -0,0 +1,576 @@ +use sha2::{Digest, Sha256}; + +use super::*; +use crate::orgtrack::types::{OrgtrackRawEvent, OrgtrackRawEventSource, OrgtrackSessionTrajectory}; + +fn create_native_trajectory_tables(conn: &rusqlite::Connection) { + conn.execute_batch( + "CREATE TABLE events ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + function_name TEXT, + args_json TEXT, + result_json TEXT, + history_sequence INTEGER, + created_at TEXT + ); + CREATE TABLE code_session_chunks ( + chunk_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + function TEXT, + args_json TEXT, + result_json TEXT, + sequence INTEGER, + created_at TEXT + );", + ) + .expect("native trajectory tables"); +} + +#[test] +fn streaming_trajectory_matches_legacy_pretty_json_golden() { + let mut conn = rusqlite::Connection::open_in_memory().expect("trajectory DB"); + create_native_trajectory_tables(&conn); + let args = "{\"command\":\"printf \\\"你🙂\\n\\u0000\"\"}"; + conn.execute( + "INSERT INTO events( + id,session_id,function_name,args_json,result_json,history_sequence,created_at + ) VALUES('event-1','plain-session','Shell',?1,NULL,2,'2026-07-22T01:02:03Z')", + [args], + ) + .expect("event fixture"); + let chunk_result = "{\"output\":\"done\\t\\\\quoted\"}"; + conn.execute( + "INSERT INTO code_session_chunks( + chunk_id,session_id,function,args_json,result_json,sequence,created_at + ) VALUES('chunk-1','plain-session','Read',NULL,?1,7,NULL)", + [chunk_result], + ) + .expect("chunk fixture"); + + let expected = OrgtrackSessionTrajectory { + schema_version: 1, + tier: OrgtrackTier::Trajectory, + session_id: "plain-session".to_string(), + raw_events: vec![ + OrgtrackRawEvent { + source: OrgtrackRawEventSource::Event, + name: Some("Shell".to_string()), + args_json: Some(args.to_string()), + result_json: None, + sequence: Some(2), + created_at: Some("2026-07-22T01:02:03Z".to_string()), + }, + OrgtrackRawEvent { + source: OrgtrackRawEventSource::CodeSessionChunk, + name: Some("Read".to_string()), + args_json: None, + result_json: Some(chunk_result.to_string()), + sequence: Some(7), + created_at: None, + }, + ], + }; + let legacy = serde_json::to_string_pretty(&expected).expect("legacy trajectory JSON"); + let mut streamed = Vec::new(); + write_session_trajectory( + &mut conn, + &mut streamed, + 1, + OrgtrackTier::Trajectory, + "plain-session", + ) + .expect("stream trajectory"); + + assert_eq!(String::from_utf8(streamed).unwrap(), legacy); +} + +#[test] +fn native_ten_mib_payload_stream_matches_legacy_hash_across_utf8_boundary() { + let mut conn = rusqlite::Connection::open_in_memory().expect("trajectory DB"); + create_native_trajectory_tables(&conn); + let mut large = "A".repeat(TRAJECTORY_PAYLOAD_RANGE_BYTES - 1); + large.push('🙂'); + large.push_str(&"B".repeat(10 * 1024 * 1024 - large.len())); + let result_json = format!("{{\"output\":\"{large}\"}}"); + conn.execute( + "INSERT INTO events( + id,session_id,function_name,args_json,result_json,history_sequence,created_at + ) VALUES('large','plain-large','Shell','{}',?1,1,'2026-07-22T00:00:00Z')", + [&result_json], + ) + .expect("large event fixture"); + + let expected = OrgtrackSessionTrajectory { + schema_version: 1, + tier: OrgtrackTier::Trajectory, + session_id: "plain-large".to_string(), + raw_events: vec![OrgtrackRawEvent { + source: OrgtrackRawEventSource::Event, + name: Some("Shell".to_string()), + args_json: Some("{}".to_string()), + result_json: Some(result_json), + sequence: Some(1), + created_at: Some("2026-07-22T00:00:00Z".to_string()), + }], + }; + let mut legacy_hash = Sha256::new(); + serde_json::to_writer_pretty(&mut legacy_hash, &expected).expect("legacy hash"); + + let mut streamed_hash = Sha256::new(); + write_session_trajectory( + &mut conn, + &mut streamed_hash, + 1, + OrgtrackTier::Trajectory, + "plain-large", + ) + .expect("stream large trajectory"); + + assert_eq!(streamed_hash.finalize(), legacy_hash.finalize()); +} + +#[test] +fn imported_deferred_payload_is_exported_in_full_without_preview_marker() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let mut conn = rusqlite::Connection::open_in_memory().expect("replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay index tables"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache tables"); + let directory = tempfile::tempdir().expect("Codex trajectory fixture"); + let transcript_path = directory.path().join("rollout.jsonl"); + let imported_id = "codexapp-trajectory-deferred"; + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES('codex_app','trajectory-deferred',?1,?2)", + params![imported_id, transcript_path.to_string_lossy()], + ) + .expect("Codex cache fixture"); + let output = format!("BEGIN:{}:END", "payload🙂".repeat(1024 * 1024)); + let line = |payload: Value| { + serde_json::json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"event_msg", + "payload":payload + }) + .to_string() + }; + let transcript = format!( + "{}\n{}\n{}\n", + line(serde_json::json!({"type":"user_message","message":"run"})), + line(serde_json::json!({ + "type":"function_call", + "name":"shell_command", + "arguments":"{\"command\":\"printf payload\"}", + "call_id":"call-trajectory" + })), + line(serde_json::json!({ + "type":"function_call_output", + "call_id":"call-trajectory", + "output":output + })) + ); + std::fs::write(&transcript_path, transcript).expect("Codex transcript"); + + let mut streamed = Vec::new(); + write_session_trajectory_with_imported_id( + &mut conn, + &mut streamed, + 1, + OrgtrackTier::Trajectory, + "managed-session", + Some(imported_id), + ) + .expect("stream imported trajectory"); + let trajectory: OrgtrackSessionTrajectory = + serde_json::from_slice(&streamed).expect("valid trajectory JSON"); + let shell = trajectory + .raw_events + .iter() + .find(|event| event.name.as_deref() == Some("run_command_line")) + .expect("Shell trajectory event"); + let result: Value = + serde_json::from_str(shell.result_json.as_deref().unwrap()).expect("Shell result JSON"); + let restored = result + .get("output") + .and_then(Value::as_str) + .expect("restored Shell output"); + assert_eq!(restored.len(), output.len()); + assert_eq!( + Sha256::digest(restored.as_bytes()), + Sha256::digest(output.as_bytes()) + ); + assert!(!restored.contains("[payload truncated]")); + assert_eq!( + TRAJECTORY_PAYLOAD_RANGE_BYTES, + replay::HARD_MAX_PAYLOAD_RANGE_BYTES + ); +} + +#[test] +fn imported_export_releases_the_sessions_writer_lock_while_visiting_pages() { + use std::sync::mpsc; + use std::time::Duration; + + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let directory = tempfile::tempdir().expect("export lock fixture"); + let cache_path = directory.path().join("sessions.db"); + let transcript_path = directory.path().join("rollout.jsonl"); + let mut conn = rusqlite::Connection::open(&cache_path).expect("file-backed replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay index tables"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache tables"); + let imported_id = "codexapp-export-writer-lock"; + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES('codex_app','export-writer-lock',?1,?2)", + params![imported_id, transcript_path.to_string_lossy()], + ) + .expect("Codex cache fixture"); + let line = |payload: Value| { + serde_json::json!({ + "timestamp":"2026-07-22T00:00:00Z", + "type":"event_msg", + "payload":payload + }) + .to_string() + }; + std::fs::write( + &transcript_path, + format!( + "{}\n{}\n", + line(serde_json::json!({"type":"user_message","message":"run"})), + line(serde_json::json!({"type":"agent_message","message":"done"})) + ), + ) + .expect("Codex transcript"); + + let mut probed = false; + for_each_imported_replay_chunk(&mut conn, imported_id, |_read_conn, _generation, _chunk| { + if !probed { + probed = true; + let (acquired_tx, acquired_rx) = mpsc::sync_channel(1); + let writer = std::thread::spawn(move || { + let _guard = database::db::sessions_writer_guard(); + acquired_tx.send(()).expect("report writer acquisition"); + }); + acquired_rx + .recv_timeout(Duration::from_secs(5)) + .expect("streaming visitor must not hold the sessions writer lock"); + writer.join().expect("writer probe thread"); + } + Ok(()) + }) + .expect("pinned replay scan"); + assert!(probed, "fixture must produce at least one replay event"); +} + +#[test] +fn imported_nested_array_payload_uses_indexed_path_and_explicit_encoding() { + use orgtrack_core::sources::imported_history::replay::{ + ReplayPayloadEncoding, ReplayPayloadKind, + }; + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let mut conn = rusqlite::Connection::open_in_memory().expect("array replay DB"); + SqliteRecordStore::init_tables(&conn).expect("replay index tables"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("source cache tables"); + let source_session_id = "array-source"; + let session_id = "codexapp-array-source"; + let generation = "array-generation"; + let event_id = "array-event"; + let field_path = "result.content.0.text"; + let canonical = "FULL_ARRAY_\"quoted\"\\path\nnext"; + let descriptor = ReplayPayloadDescriptor { + field_path: field_path.to_string(), + kind: ReplayPayloadKind::AssistantContent, + encoding: ReplayPayloadEncoding::Utf8Text, + body_projection: None, + spans: Vec::new(), + total_bytes: canonical.len() as u64, + source_ordinal: None, + source_key: None, + }; + conn.execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES('codex_app',?1,?2,'/unused/provider.jsonl')", + params![source_session_id, session_id], + ) + .expect("array source binding"); + conn.execute( + "INSERT INTO imported_replay_events( + source,source_session_id,generation,sequence,event_id,turn_index, + action_type,function_name,created_at,payloads_json,content_hash + ) VALUES('codex_app',?1,?2,1,?3,0,'assistant','assistant', + '2026-07-23T00:00:00Z',?4,'event-hash')", + params![ + source_session_id, + generation, + event_id, + serde_json::to_string(&vec![descriptor.clone()]).expect("payload descriptor JSON") + ], + ) + .expect("array replay event"); + conn.execute( + "INSERT INTO imported_replay_payload_artifacts( + source,source_session_id,generation,content_hash,payload + ) VALUES('codex_app',?1,?2,'payload-hash',?3)", + params![source_session_id, generation, canonical.as_bytes()], + ) + .expect("array payload artifact"); + conn.execute( + "INSERT INTO imported_replay_payload_artifact_refs( + source,source_session_id,generation,event_id,field_path,content_hash + ) VALUES('codex_app',?1,?2,?3,?4,'payload-hash')", + params![source_session_id, generation, event_id, field_path], + ) + .expect("array payload ref"); + + let compact = serde_json::json!({ + "content":[{"text":"ARRAY_PREVIEW","kind":"text"}] + }); + let mut encoded = Vec::new(); + write_replay_value_json_string( + &mut conn, + &mut encoded, + ImportedHistorySourceId::CodexApp, + session_id, + generation, + event_id, + "result", + &compact, + &[descriptor], + ) + .expect("serialize nested array payload"); + let inner: String = serde_json::from_slice(&encoded).expect("outer resultJson string"); + let restored: Value = serde_json::from_str(&inner).expect("inner result JSON"); + assert_eq!(restored["content"][0]["text"], canonical); + assert_eq!(restored["content"][0]["kind"], "text"); + assert!(!inner.contains("ARRAY_PREVIEW")); +} + +#[test] +fn sqlite_root_descriptor_restores_path_and_ten_mib_omitted_sibling_exactly() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + let directory = tempfile::tempdir().expect("OpenCode trajectory fixture"); + let source_path = directory.path().join("opencode.db"); + let source_conn = rusqlite::Connection::open(&source_path).expect("OpenCode source DB"); + source_conn + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE session( + id TEXT PRIMARY KEY,time_created INTEGER,time_updated INTEGER + ); + CREATE TABLE message( + id TEXT PRIMARY KEY,session_id TEXT,time_created INTEGER,data TEXT + ); + CREATE TABLE part( + id TEXT PRIMARY KEY,message_id TEXT,session_id TEXT, + time_created INTEGER,data TEXT + ); + INSERT INTO session VALUES('s1',1,1); + INSERT INTO message VALUES( + 'message-1','s1',1,'{\"role\":\"assistant\"}' + );", + ) + .expect("OpenCode source schema"); + let mut content = "C".repeat(10 * 1024 * 1024 - 4); + content.push('🙂'); + let source_args = serde_json::json!({ + "path":"src/large.txt", + "content":content, + }); + let part = serde_json::json!({ + "type":"tool", + "tool":"edit", + "callID":"call-large-edit", + "state":{ + "status":"completed", + "input":source_args, + "output":"updated" + } + }); + source_conn + .execute( + "INSERT INTO part(id,message_id,session_id,time_created,data) + VALUES('part-1','message-1','s1',1,?1)", + [part.to_string()], + ) + .expect("OpenCode large edit row"); + + let mut cache = rusqlite::Connection::open_in_memory().expect("replay cache DB"); + SqliteRecordStore::init_tables(&cache).expect("replay index tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source cache tables"); + let imported_id = "opencodeapp-s1"; + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES('opencode','s1',?1,?2)", + params![imported_id, source_path.to_string_lossy()], + ) + .expect("OpenCode cache fixture"); + + let mut streamed = Vec::new(); + write_session_trajectory_with_imported_id( + &mut cache, + &mut streamed, + 1, + OrgtrackTier::Trajectory, + "managed-opencode", + Some(imported_id), + ) + .expect("stream OpenCode trajectory"); + let trajectory: OrgtrackSessionTrajectory = + serde_json::from_slice(&streamed).expect("valid OpenCode trajectory JSON"); + let edit = trajectory + .raw_events + .iter() + .find(|event| event.name.as_deref() == Some("edit_file_by_replace")) + .expect("edit trajectory event"); + let restored: Value = + serde_json::from_str(edit.args_json.as_deref().unwrap()).expect("restored edit args JSON"); + let expected = serde_json::json!({ + "action":"edit", + "file_path":"src/large.txt", + "payload":source_args, + }); + assert_eq!( + restored, expected, + "root range must restore omitted siblings" + ); + let restored_content = restored + .pointer("/payload/content") + .and_then(Value::as_str) + .expect("restored edit content"); + assert_eq!(restored_content.len(), content.len()); + assert_eq!( + Sha256::digest(restored_content.as_bytes()), + Sha256::digest(content.as_bytes()) + ); + assert!(restored.get("_replayTruncated").is_none()); + assert!(restored.get("_preview").is_none()); +} + +#[test] +fn imported_trajectory_prepares_three_lazy_kv_turns_before_strict_scan() { + use orgtrack_core::store::sqlite::SqliteRecordStore; + + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let directory = tempfile::tempdir().expect("lazy trajectory fixture"); + let source_path = directory.path().join(format!("{}.db", source.as_str())); + let source_conn = rusqlite::Connection::open(&source_path).expect("KV source DB"); + source_conn + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE cursorDiskKV(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("KV source schema"); + let headers = (0..6) + .map(|index| { + serde_json::json!({ + "bubbleId":format!("b{index}"), + "type":if index % 2 == 0 { 1 } else { 2 }, + }) + }) + .collect::>(); + source_conn + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES('composerData:c1',?1)", + [serde_json::json!({ + "composerId":"c1", + "createdAt":1, + "lastUpdatedAt":6, + "fullConversationHeadersOnly":headers, + }) + .to_string()], + ) + .expect("composer row"); + for index in 0..6 { + source_conn + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2)", + params![ + format!("bubbleId:c1:b{index}"), + serde_json::json!({ + "bubbleId":format!("b{index}"), + "type":if index % 2 == 0 { 1 } else { 2 }, + "createdAt":format!("2026-07-22T00:00:{index:02}Z"), + "text":format!("message {index}"), + }) + .to_string(), + ], + ) + .expect("bubble row"); + } + + let mut cache = rusqlite::Connection::open_in_memory().expect("replay cache"); + SqliteRecordStore::init_tables(&cache).expect("replay tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source cache tables"); + let imported_id = format!("{}c1", source.descriptor().session_prefix); + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,'c1',?2,?3)", + params![source.as_str(), imported_id, source_path.to_string_lossy()], + ) + .expect("cache source path"); + + let mut sequences = Vec::new(); + for_each_imported_replay_chunk(&mut cache, &imported_id, |_conn, _generation, chunk| { + sequences.push(chunk.sequence); + Ok(()) + }) + .expect("strict lazy trajectory scan"); + assert_eq!(sequences, vec![0, 1, 2, 3, 4, 5], "{}", source.as_str()); + } +} + +#[test] +fn same_generation_revision_change_rejects_pinned_trajectory_cursor() { + let cursor = ReplayCursor { + source_id: "codex_app".to_string(), + session_id: "codexapp-revision".to_string(), + generation: "same-generation".to_string(), + revision: 12, + through_sequence: 99, + }; + let error = validate_pinned_replay_cursor("codexapp-revision", "same-generation", 11, &cursor) + .expect_err("same-generation revision mutation must abort export"); + assert!(error.contains("same-generation@11")); + assert!(error.contains("same-generation@12")); +} + +#[test] +fn imported_trajectory_omits_compact_only_replay_markers() { + let mut conn = rusqlite::Connection::open_in_memory().expect("marker DB"); + let value = serde_json::json!({ + "output":"legacy output", + "_replayTruncated":true, + "_replayGitArtifacts":[{"kind":"commit","sha":"abc1234"}] + }); + let mut encoded = Vec::new(); + write_replay_value_json_string( + &mut conn, + &mut encoded, + ImportedHistorySourceId::CodexApp, + "codexapp-marker", + "generation", + "event", + "result", + &value, + &[], + ) + .expect("serialize compatible replay result"); + let inner: String = serde_json::from_slice(&encoded).expect("outer argsJson string"); + let restored: Value = serde_json::from_str(&inner).expect("inner result JSON"); + assert_eq!(restored, serde_json::json!({"output":"legacy output"})); +} diff --git a/src-tauri/src/orgtrack/history_commands.rs b/src-tauri/src/orgtrack/history_commands.rs index 9ec0107d1..64af3614f 100644 --- a/src-tauri/src/orgtrack/history_commands.rs +++ b/src-tauri/src/orgtrack/history_commands.rs @@ -1,8 +1,4 @@ -use std::{ - collections::{HashSet, VecDeque}, - path::Path, - sync::{Mutex, OnceLock}, -}; +use std::{collections::HashSet, path::Path, sync::OnceLock}; use database::db::get_connection; use orgtrack_core::pricing; @@ -10,10 +6,9 @@ use orgtrack_core::sources::claude_code::history as claude_code_history; use orgtrack_core::sources::cline::history as cline_history; use orgtrack_core::sources::codex::app as codex_app; use orgtrack_core::sources::cursor_cli::history as cursor_cli_history; -use orgtrack_core::sources::cursor_ide::{ - db as cursor_db, disk_reads as cursor_disk_reads, history as cursor_db_history, -}; +use orgtrack_core::sources::cursor_ide::db as cursor_db; use orgtrack_core::sources::imported_history; +use orgtrack_core::sources::imported_history::replay::{self, ImportedHistorySourceId}; use orgtrack_core::sources::mimo_code::history as mimo_code_history; use orgtrack_core::sources::omp::history as omp_history; use orgtrack_core::sources::opencode::history as opencode_history; @@ -43,146 +38,6 @@ async fn acquire_external_history_scan_permit( .map_err(|_| "External history scan queue closed".to_string()) } -const CODEX_TURN_PROJECTION_CACHE_CAPACITY: usize = 8; -const CODEX_TURN_PROJECTION_LIMIT_PER_SESSION: usize = 4_096; -const CODEX_INITIAL_RECENT_TURN_COUNT: usize = 1; - -#[derive(Debug)] -struct CodexTurnProjectionCacheEntry { - session_id: String, - signature: (i64, u64), - projected: Vec, -} - -#[derive(Debug, Default)] -struct CodexTurnProjectionCache { - entries: VecDeque, -} - -impl CodexTurnProjectionCache { - fn get( - &mut self, - session_id: &str, - signature: (i64, u64), - ) -> Option> { - let index = self - .entries - .iter() - .position(|entry| entry.session_id == session_id)?; - let entry = self.entries.remove(index)?; - if entry.signature != signature { - return None; - } - let projected = entry.projected.clone(); - self.entries.push_back(entry); - Some(projected) - } - - fn insert( - &mut self, - session_id: String, - signature: (i64, u64), - projected: Vec, - ) { - if let Some(index) = self - .entries - .iter() - .position(|entry| entry.session_id == session_id) - { - self.entries.remove(index); - } - let projected = if projected.len() > CODEX_TURN_PROJECTION_LIMIT_PER_SESSION { - projected - .into_iter() - .rev() - .take(CODEX_TURN_PROJECTION_LIMIT_PER_SESSION) - .collect::>() - .into_iter() - .rev() - .collect() - } else { - projected - }; - self.entries.push_back(CodexTurnProjectionCacheEntry { - session_id, - signature, - projected, - }); - while self.entries.len() > CODEX_TURN_PROJECTION_CACHE_CAPACITY { - self.entries.pop_front(); - } - } -} - -fn codex_turn_projection_cache() -> &'static Mutex { - static CACHE: OnceLock> = OnceLock::new(); - CACHE.get_or_init(|| Mutex::new(CodexTurnProjectionCache::default())) -} - -fn codex_transcript_signature( - conn: &rusqlite::Connection, - session_id: &str, -) -> Result, String> { - if !session_id.starts_with(orgtrack_core::sources::codex::SESSION_PREFIX) { - return Ok(None); - } - imported_history::cache::stat_imported_transcript_by_session_id_from_conn( - conn, - imported_history::metadata::SOURCE_CODEX_APP, - session_id, - ) -} - -fn remember_codex_turn_projection( - session_id: &str, - signature_before: Option<(i64, u64)>, - signature_after: Option<(i64, u64)>, - projected: Vec, -) { - let (Some(before), Some(after)) = (signature_before, signature_after) else { - return; - }; - // Do not cache a parse that raced a transcript append. The next read will - // parse the now-stable file instead of treating an incomplete projection - // as current. - if before != after { - return; - } - codex_turn_projection_cache() - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()) - .insert(session_id.to_string(), after, projected); -} - -fn load_projected_turn_metadata( - conn: &rusqlite::Connection, - session_id: &str, -) -> Result>, String> { - let signature_before = codex_transcript_signature(conn, session_id)?; - if let Some(signature) = signature_before { - if let Some(projected) = codex_turn_projection_cache() - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()) - .get(session_id, signature) - { - return Ok(Some(projected)); - } - } - - let Some(chunks) = imported_history::load_activity_chunks_for_session(conn, session_id)? else { - return Ok(None); - }; - let projected = orgtrack_core::projectors::turn_metadata::project_activity_chunks(&chunks); - let signature_after = codex_transcript_signature(conn, session_id)?; - remember_codex_turn_projection( - session_id, - signature_before, - signature_after, - projected.clone(), - ); - Ok(Some(projected)) -} - fn projected_rounds_to_cached_turns( session_id: &str, projected: Vec, @@ -220,10 +75,16 @@ fn projected_rounds_to_cached_turns( .collect() } -/// Unified per-round metadata read surface. Native/managed sessions use the -/// versioned local turn cache; read-only imported sessions are projected -/// directly from their existing provider loader and never copied into -/// `sessions.db.events`. +fn metadata_cache_miss_requires_sync(source: ImportedHistorySourceId) -> bool { + !matches!( + source, + ImportedHistorySourceId::CursorIde | ImportedHistorySourceId::Windsurf + ) +} + +/// Unified per-round metadata read surface. Native SDE sessions keep using the +/// versioned local turn cache. Imported and managed CLI sessions project only +/// visible turns from the compact replay index and never hydrate a transcript. #[tauri::command] pub async fn orgtrack_session_turn_metadata_index( session_id: String, @@ -236,7 +97,6 @@ pub async fn orgtrack_session_turn_metadata_index( { return Err("At most 500 turn summaries can be loaded at once".to_string()); } - let conn = open_cache_conn()?; // Managed native-transcript sessions project from the CLI's own // store: remap the managed id to its imported transcript id first. let transcript_session_id = @@ -244,13 +104,36 @@ pub async fn orgtrack_session_turn_metadata_index( &session_id, ) .unwrap_or_else(|| session_id.clone()); - if let Some(projected) = load_projected_turn_metadata(&conn, &transcript_session_id)? { - let mut turns = projected_rounds_to_cached_turns(&session_id, projected); - if let Some(turn_ids) = turn_ids.as_ref() { - let requested = turn_ids.iter().collect::>(); - turns.retain(|turn| requested.contains(&turn.turn_id)); + if let Some(source) = ImportedHistorySourceId::from_session_id(&transcript_session_id) { + let cached = { + let conn = open_cache_conn()?; + replay::project_cached_turn_metadata( + &conn, + source, + &transcript_session_id, + turn_ids.as_deref(), + )? + }; + if let Some(projected) = cached { + return Ok(projected_rounds_to_cached_turns(&session_id, projected)); + } + // Cursor/Windsurf materialize old KV turn bodies lazily. A + // metadata-only caller must wait for the foreground compact index + // instead of taking the application writer lock and hydrating the + // requested body as a cache-miss fallback. + if !metadata_cache_miss_requires_sync(source) { + return Ok(Vec::new()); } - return Ok(turns); + let projected = database::db::with_sessions_writer(|| { + let mut conn = open_cache_conn()?; + replay::project_turn_metadata( + &mut conn, + source, + &transcript_session_id, + turn_ids.as_deref(), + ) + })?; + return Ok(projected_rounds_to_cached_turns(&session_id, projected)); } if let Some(turn_ids) = turn_ids.as_ref() { return session_persistence::load_turn_summaries(&session_id, turn_ids) @@ -272,32 +155,41 @@ fn estimate_cost_blended(total_tokens: i64, model: &str) -> f64 { } fn imported_recent_paths() -> Result, String> { - let mut conn = open_cache_conn()?; - let mut paths = codex_app::list_codex_app_recent_paths(&mut conn, 0)?; - paths.extend(claude_code_history::list_claude_code_recent_paths( - &mut conn, 0, - )?); - paths.extend(cursor_cli_history::list_cursor_cli_recent_paths( - &mut conn, 0, - )?); - paths.extend(opencode_history::list_opencode_recent_paths(&mut conn, 0)?); - paths.extend(windsurf_history::list_windsurf_recent_paths(&mut conn, 0)?); - paths.extend(workbuddy_history::list_workbuddy_recent_paths( - &mut conn, 0, - )?); - paths.extend(trae_history::list_trae_recent_paths(&mut conn, 0)?); - paths.extend(cline_history::list_cline_recent_paths(&mut conn, 0)?); - paths.extend(warp_history::list_warp_recent_paths(&mut conn, 0)?); - paths.extend(zcode_history::list_zcode_recent_paths(&mut conn, 0)?); - paths.extend(qoder_history::list_qoder_recent_paths(&mut conn, 0)?); - paths.extend(mimo_code_history::list_mimo_code_recent_paths( - &mut conn, 0, - )?); - paths.extend(omp_history::list_omp_recent_paths(&mut conn, 0)?); - paths.extend(qoder_cli_history::list_qoder_cli_recent_paths( - &mut conn, 0, - )?); - Ok(imported_history::recent_paths_from_paths(&paths)) + database::db::with_sessions_writer(|| { + let mut conn = open_cache_conn()?; + let mut paths = Vec::new(); + for source in ImportedHistorySourceId::ALL { + imported_history::catalog::refresh_source(&mut conn, source)?; + paths.extend( + imported_history::cache::query_imported_recent_paths_from_conn( + &conn, + source.as_str(), + 0, + )?, + ); + } + Ok(imported_history::recent_paths_from_paths(&paths)) + }) +} + +/// Refresh one source's compact catalog and return only grouped path rows. +/// Source-specific legacy `list_*_recent_paths` functions historically +/// rebuilt changed transcripts before answering this lightweight request; +/// routing every command through the replay registry prevents a settings or +/// spotlight view from re-reading a growing JSONL/SQLite history in full. +fn compact_recent_paths( + source: ImportedHistorySourceId, + limit: usize, +) -> Result, String> { + database::db::with_sessions_writer(|| { + let mut conn = open_cache_conn()?; + imported_history::catalog::refresh_source(&mut conn, source)?; + imported_history::cache::query_imported_recent_paths_from_conn( + &conn, + source.as_str(), + limit, + ) + }) } /// Rescan one external history source. @@ -329,26 +221,27 @@ pub async fn external_history_rescan_source( } let _permit = acquire_external_history_scan_permit().await?; tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - let changes_before = conn.total_changes(); - // `clear`: wipe the source's cached rows so every session is re-parsed - // from scratch (drops stale rows / forces a full re-parse even when - // file signatures are unchanged). Otherwise this is an incremental - // "update" — only sessions whose signature changed are re-parsed. - if clear { - imported_history::cache::prune_missing_records_from_conn(&conn, &source, &[])?; - } - // Always re-read the on-disk store and repopulate the cache. The old - // behavior only pruned, leaving the count at 0 until a later lazy load. - let changed = + database::db::with_sessions_writer(|| { + let mut conn = open_cache_conn()?; + let changes_before = conn.total_changes(); + // `clear`: wipe the source's cached rows so every session is re-parsed + // from scratch (drops stale rows / forces a full re-parse even when + // file signatures are unchanged). Otherwise this is an incremental + // "update" — only sessions whose signature changed are re-parsed. + if clear { + imported_history::cache::prune_missing_records_from_conn(&conn, &source, &[])?; + } + // Always re-read the on-disk store and repopulate the compact cache. crate::agent_sessions::session_directory::aggregation::resync_external_history_source( &mut conn, &source, - )? || conn.total_changes() > changes_before; - let signature = - imported_history::cache::query_source_cache_signature_from_conn(&conn, &source)?; - Ok(ExternalHistoryScanResultWire { - changed_sources: changed.then_some(source.clone()).into_iter().collect(), - source_signatures: std::iter::once((source, signature)).collect(), + )?; + let changed = conn.total_changes() > changes_before; + let signature = + imported_history::cache::query_source_cache_signature_from_conn(&conn, &source)?; + Ok(ExternalHistoryScanResultWire { + changed_sources: changed.then_some(source.clone()).into_iter().collect(), + source_signatures: std::iter::once((source, signature)).collect(), + }) }) }) .await @@ -377,34 +270,61 @@ pub async fn external_history_rescan_sources( let _permit = acquire_external_history_scan_permit().await?; tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - let mut changed_sources = Vec::new(); - let mut source_signatures = std::collections::HashMap::new(); - for source in sources { - let changes_before = conn.total_changes(); - if clear { - imported_history::cache::prune_missing_records_from_conn(&conn, &source, &[])?; - } - let changed = crate::agent_sessions::session_directory::aggregation::resync_external_history_source( - &mut conn, &source, - )? || conn.total_changes() > changes_before; - source_signatures.insert( - source.clone(), - imported_history::cache::query_source_cache_signature_from_conn(&conn, &source)?, - ); - if changed { - changed_sources.push(source); + database::db::with_sessions_writer(|| { + let mut conn = open_cache_conn()?; + let mut changed_sources = Vec::new(); + let mut source_signatures = std::collections::HashMap::new(); + for source in sources { + let changes_before = conn.total_changes(); + if clear { + imported_history::cache::prune_missing_records_from_conn( + &conn, + &source, + &[], + )?; + } + crate::agent_sessions::session_directory::aggregation::resync_external_history_source( + &mut conn, + &source, + )?; + let changed = conn.total_changes() > changes_before; + source_signatures.insert( + source.clone(), + imported_history::cache::query_source_cache_signature_from_conn( + &conn, + &source, + )?, + ); + if changed { + changed_sources.push(source); + } } - } - Ok(ExternalHistoryScanResultWire { - changed_sources, - source_signatures, + Ok(ExternalHistoryScanResultWire { + changed_sources, + source_signatures, + }) }) }) .await .map_err(|err| format!("Task join error: {err}"))? } +/// Source-neutral compact recent-path query used by settings and workspace +/// discovery. The source registry owns refresh semantics for all 15 adapters; +/// adding a source without a catalog adapter fails in the exhaustive Rust +/// match instead of silently falling back to a transcript loader. +#[tauri::command] +pub async fn external_history_recent_paths( + source: String, + limit: Option, +) -> Result, String> { + let source = ImportedHistorySourceId::parse(&source)?; + let limit = limit.unwrap_or(20); + tokio::task::spawn_blocking(move || compact_recent_paths(source, limit)) + .await + .map_err(|err| format!("Task join error: {err}"))? +} + #[tauri::command] pub async fn orgtrack_get_cursor_sessions( start_date: String, @@ -422,128 +342,13 @@ pub async fn orgtrack_get_cursor_sessions( .map_err(|err| format!("Task join error: {err}"))? } -#[tauri::command] -pub async fn cursor_ide_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || cursor_db_history::load_history_for_session(&session_id)) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -/// Freshness signal for an open read-only Cursor session — the frontend compares -/// snapshots to decide whether to reload chunks. Reads Cursor's `state.vscdb`. -#[tauri::command] -pub async fn cursor_ide_composer_last_updated_at( - composer_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - cursor_disk_reads::cursor_composer_last_updated_at(&composer_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn cursor_ide_initial_window( - session_id: String, - recent_limit: Option, -) -> Result { - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - cursor_db_history::load_initial_window_for_session(&mut conn, &session_id, recent_limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn cursor_ide_full_refresh( - session_id: String, -) -> Result { - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - cursor_db_history::load_full_refresh_for_session(&mut conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn cursor_ide_turn_window( - session_id: String, - user_bubble_id: String, -) -> Result { - tokio::task::spawn_blocking(move || { - cursor_db_history::load_turn_window_for_session(&session_id, &user_bubble_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn codex_app_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - let signature_before = codex_transcript_signature(&conn, &session_id)?; - let chunks = codex_app::load_codex_app_for_session(&conn, &session_id)?; - let projected = orgtrack_core::projectors::turn_metadata::project_activity_chunks(&chunks); - let signature_after = codex_transcript_signature(&conn, &session_id)?; - remember_codex_turn_projection(&session_id, signature_before, signature_after, projected); - Ok(chunks) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn codex_app_initial_window( - session_id: String, -) -> Result { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - let signature_before = codex_transcript_signature(&conn, &session_id)?; - let window = codex_app::load_codex_app_initial_window_for_session( - &conn, - &session_id, - CODEX_INITIAL_RECENT_TURN_COUNT, - )?; - let signature_after = codex_transcript_signature(&conn, &session_id)?; - remember_codex_turn_projection( - &session_id, - signature_before, - signature_after, - window.turns.clone(), - ); - Ok(window) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn codex_app_turn_window( - session_id: String, - turn_id: String, -) -> Result { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - codex_app::load_codex_app_turn_for_session(&conn, &session_id, &turn_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - #[tauri::command] pub async fn codex_app_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - codex_app::list_codex_app_recent_paths(&mut conn, limit) + compact_recent_paths(ImportedHistorySourceId::CodexApp, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? @@ -585,119 +390,13 @@ pub async fn external_history_auto_import_recent_paths( Ok(imported) } -#[tauri::command] -pub async fn claude_code_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - claude_code_history::load_claude_code_history_for_session(&conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -/// Freshness snapshot of one imported transcript's source file. -#[derive(serde::Serialize)] -#[serde(rename_all = "camelCase")] -pub struct ImportedTranscriptStat { - pub mtime_ms: i64, - pub size_bytes: u64, -} - -/// Cheap freshness probe for the replay auto-refresh: returns the transcript -/// file's `(mtime, size)` so the frontend can skip the full -/// read → parse → merge pipeline when nothing changed. `None` when the -/// source file is missing. -#[tauri::command] -pub async fn claude_code_history_stat( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - Ok( - claude_code_history::stat_claude_code_history_for_session(&conn, &session_id)?.map( - |(mtime_ms, size_bytes)| ImportedTranscriptStat { - mtime_ms, - size_bytes, - }, - ), - ) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -/// Source-agnostic freshness probe for the replay auto-refresh: resolves the -/// session's transcript path from the imported-history cache and stats it -/// (folding in the SQLite `-wal` sibling for WAL-mode stores, whose main db -/// mtime doesn't move between checkpoints). `None` when the session is -/// uncached or the file is missing — the frontend then falls back to the -/// full refresh, which re-syncs the cache. -#[tauri::command] -pub async fn imported_history_stat( - source_id: String, - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - Ok( - imported_history::cache::stat_imported_transcript_by_session_id_from_conn( - &conn, - &source_id, - &session_id, - )? - .map(|(mtime_ms, size_bytes)| ImportedTranscriptStat { - mtime_ms, - size_bytes, - }), - ) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - #[tauri::command] pub async fn claude_code_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - claude_code_history::list_claude_code_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn cursor_cli_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - cursor_cli_history::load_cursor_cli_history_for_session(&conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -/// Cheap freshness probe for the replay auto-refresh, folding the store's -/// `-wal` sidecar in (a WAL commit doesn't touch the main file's mtime). -#[tauri::command] -pub async fn cursor_cli_history_stat( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - Ok( - cursor_cli_history::stat_cursor_cli_history_for_session(&conn, &session_id)?.map( - |(mtime_ms, size_bytes)| ImportedTranscriptStat { - mtime_ms, - size_bytes, - }, - ), - ) + compact_recent_paths(ImportedHistorySourceId::ClaudeCode, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? @@ -709,19 +408,7 @@ pub async fn cursor_cli_recent_paths( ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - cursor_cli_history::list_cursor_cli_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn opencode_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - opencode_history::load_opencode_history_for_session(&session_id) + compact_recent_paths(ImportedHistorySourceId::CursorCli, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? @@ -733,40 +420,18 @@ pub async fn opencode_recent_paths( ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - opencode_history::list_opencode_recent_paths(&mut conn, limit) + compact_recent_paths(ImportedHistorySourceId::OpenCode, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? } -#[tauri::command] -pub async fn warp_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || warp_history::load_warp_history_for_session(&session_id)) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - #[tauri::command] pub async fn warp_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - warp_history::list_warp_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn zcode_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || zcode_history::load_zcode_history_for_session(&session_id)) + tokio::task::spawn_blocking(move || compact_recent_paths(ImportedHistorySourceId::Warp, limit)) .await .map_err(|err| format!("Task join error: {err}"))? } @@ -776,24 +441,9 @@ pub async fn zcode_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - zcode_history::list_zcode_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn qoder_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - qoder_history::load_qoder_history_for_session(&conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? + tokio::task::spawn_blocking(move || compact_recent_paths(ImportedHistorySourceId::ZCode, limit)) + .await + .map_err(|err| format!("Task join error: {err}"))? } #[tauri::command] @@ -801,24 +451,9 @@ pub async fn qoder_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - qoder_history::list_qoder_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn mimo_code_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - mimo_code_history::load_mimo_code_history_for_session(&conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? + tokio::task::spawn_blocking(move || compact_recent_paths(ImportedHistorySourceId::Qoder, limit)) + .await + .map_err(|err| format!("Task join error: {err}"))? } #[tauri::command] @@ -827,20 +462,7 @@ pub async fn mimo_code_recent_paths( ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - mimo_code_history::list_mimo_code_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn omp_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - omp_history::load_omp_history_for_session(&conn, &session_id) + compact_recent_paths(ImportedHistorySourceId::MimoCode, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? @@ -851,24 +473,9 @@ pub async fn omp_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - omp_history::list_omp_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn qoder_cli_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - qoder_cli_history::load_qoder_cli_history_for_session(&conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? + tokio::task::spawn_blocking(move || compact_recent_paths(ImportedHistorySourceId::Omp, limit)) + .await + .map_err(|err| format!("Task join error: {err}"))? } #[tauri::command] @@ -877,19 +484,7 @@ pub async fn qoder_cli_recent_paths( ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - qoder_cli_history::list_qoder_cli_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn windsurf_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - windsurf_history::load_windsurf_history_for_session(&session_id) + compact_recent_paths(ImportedHistorySourceId::QoderCli, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? @@ -901,20 +496,7 @@ pub async fn windsurf_recent_paths( ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - windsurf_history::list_windsurf_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn trae_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - trae_history::load_trae_history_for_session(&conn, &session_id) + compact_recent_paths(ImportedHistorySourceId::Windsurf, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? @@ -925,24 +507,9 @@ pub async fn trae_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - trae_history::list_trae_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn cline_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - cline_history::load_cline_history_for_session(&conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? + tokio::task::spawn_blocking(move || compact_recent_paths(ImportedHistorySourceId::Trae, limit)) + .await + .map_err(|err| format!("Task join error: {err}"))? } #[tauri::command] @@ -950,24 +517,9 @@ pub async fn cline_recent_paths( limit: Option, ) -> Result, String> { let limit = limit.unwrap_or(20); - tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - cline_history::list_cline_recent_paths(&mut conn, limit) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? -} - -#[tauri::command] -pub async fn workbuddy_history_chunks( - session_id: String, -) -> Result, String> { - tokio::task::spawn_blocking(move || { - let conn = open_cache_conn()?; - workbuddy_history::load_workbuddy_history_for_session(&conn, &session_id) - }) - .await - .map_err(|err| format!("Task join error: {err}"))? + tokio::task::spawn_blocking(move || compact_recent_paths(ImportedHistorySourceId::Cline, limit)) + .await + .map_err(|err| format!("Task join error: {err}"))? } #[tauri::command] @@ -976,8 +528,7 @@ pub async fn workbuddy_recent_paths( ) -> Result, String> { let limit = limit.unwrap_or(20); tokio::task::spawn_blocking(move || { - let mut conn = open_cache_conn()?; - workbuddy_history::list_workbuddy_recent_paths(&mut conn, limit) + compact_recent_paths(ImportedHistorySourceId::WorkBuddy, limit) }) .await .map_err(|err| format!("Task join error: {err}"))? @@ -1084,40 +635,18 @@ mod tests { } #[test] - fn codex_projection_cache_is_bounded_and_rejects_stale_signatures() { - let mut cache = CodexTurnProjectionCache::default(); - for index in 0..=CODEX_TURN_PROJECTION_CACHE_CAPACITY { - cache.insert( - format!("codexapp-{index}"), - (index as i64, index as u64), - vec![projected(&format!("user-{index}"), index as i64)], - ); - } - - assert_eq!(cache.entries.len(), CODEX_TURN_PROJECTION_CACHE_CAPACITY); - assert!(cache.get("codexapp-0", (0, 0)).is_none()); - assert!(cache.get("codexapp-1", (999, 999)).is_none()); - assert!(cache.get("codexapp-2", (2, 2)).is_some()); - } - - #[test] - fn codex_projection_cache_bounds_turns_per_session() { - let mut cache = CodexTurnProjectionCache::default(); - cache.insert( - "codexapp-large".to_string(), - (1, 2), - (0..=CODEX_TURN_PROJECTION_LIMIT_PER_SESSION) - .map(|index| projected(&format!("user-{index}"), index as i64)) - .collect(), - ); - - let projected = cache - .get("codexapp-large", (1, 2)) - .expect("cached projection"); - assert_eq!(projected.len(), CODEX_TURN_PROJECTION_LIMIT_PER_SESSION); - assert_eq!( - projected.first().map(|turn| turn.turn_id.as_str()), - Some("user-1") - ); + fn lazy_kv_metadata_cache_misses_never_take_the_sync_fallback() { + assert!(!metadata_cache_miss_requires_sync( + ImportedHistorySourceId::CursorIde + )); + assert!(!metadata_cache_miss_requires_sync( + ImportedHistorySourceId::Windsurf + )); + assert!(metadata_cache_miss_requires_sync( + ImportedHistorySourceId::OpenCode + )); + assert!(metadata_cache_miss_requires_sync( + ImportedHistorySourceId::CodexApp + )); } } diff --git a/src-tauri/src/orgtrack/impact_indexer.rs b/src-tauri/src/orgtrack/impact_indexer.rs index b4d3e4c96..2bcc9fb8c 100644 --- a/src-tauri/src/orgtrack/impact_indexer.rs +++ b/src-tauri/src/orgtrack/impact_indexer.rs @@ -23,22 +23,94 @@ pub fn get_session_impact(session_id: &str) -> Result Ok(summarize_turns(&turns)) } +/// Read only already-materialized impact metadata. +/// +/// Unlike `get_session_impact`, this never checks freshness and can therefore +/// never rebuild a turn index from transcript events. Snapshot-backed native +/// forks use it for sidebar metadata: snapshot publication deletes stale turn +/// rows, so no rows means "metadata unavailable" rather than "reparse all +/// inherited history". +pub fn get_persisted_session_impact( + session_id: &str, +) -> Result, String> { + let conn = database::db::get_connection() + .map_err(|error| format!("open persisted session impact database: {error}"))?; + persisted_session_impact_from_connection(&conn, session_id) +} + +fn persisted_session_impact_from_connection( + conn: &rusqlite::Connection, + session_id: &str, +) -> Result, String> { + let mut statement = conn + .prepare_cached( + "SELECT modified_files_json FROM session_turns + WHERE session_id=?1 ORDER BY start_sequence ASC", + ) + .map_err(|error| format!("prepare persisted session impact: {error}"))?; + let mut rows = statement + .query([session_id]) + .map_err(|error| format!("query persisted session impact: {error}"))?; + let mut touched_files = BTreeSet::new(); + let mut lines_added = 0_i64; + let mut lines_removed = 0_i64; + while let Some(row) = rows + .next() + .map_err(|error| format!("read persisted session impact: {error}"))? + { + let raw: String = row + .get(0) + .map_err(|error| format!("decode persisted session impact column: {error}"))?; + let files = serde_json::from_str::>(&raw) + .map_err(|error| format!("decode persisted session impact JSON: {error}"))?; + fold_modified_files( + &mut touched_files, + &mut lines_added, + &mut lines_removed, + &files, + ); + } + Ok(finish_impact(touched_files, lines_added, lines_removed)) +} + fn summarize_turns(turns: &[CachedTurnSummary]) -> Option { let mut touched_files = BTreeSet::new(); let mut lines_added = 0_i64; let mut lines_removed = 0_i64; for turn in turns { - for file in &turn.modified_files { - if file.path.trim().is_empty() { - continue; - } - touched_files.insert(file.path.clone()); - lines_added = lines_added.saturating_add(i64::from(file.additions)); - lines_removed = lines_removed.saturating_add(i64::from(file.deletions)); + fold_modified_files( + &mut touched_files, + &mut lines_added, + &mut lines_removed, + &turn.modified_files, + ); + } + + finish_impact(touched_files, lines_added, lines_removed) +} + +fn fold_modified_files( + touched_files: &mut BTreeSet, + lines_added: &mut i64, + lines_removed: &mut i64, + files: &[session_persistence::TurnModifiedFile], +) { + for file in files { + if file.path.trim().is_empty() { + continue; } + touched_files.insert(file.path.clone()); + *lines_added = lines_added.saturating_add(i64::from(file.additions)); + *lines_removed = lines_removed.saturating_add(i64::from(file.deletions)); } +} +fn finish_impact( + touched_files: BTreeSet, + lines_added: i64, + lines_removed: i64, +) -> Option { if touched_files.is_empty() && lines_added == 0 && lines_removed == 0 { return None; } @@ -112,4 +184,45 @@ mod tests { assert_eq!(summary.lines_removed, 1); assert_eq!(summary.touched_files, vec!["src/a.ts", "src/b.ts"]); } + + #[test] + fn persisted_impact_streams_materialized_metadata_without_turn_index_refresh() { + let conn = rusqlite::Connection::open_in_memory().expect("impact database"); + session_persistence::init_session_tables(&conn).expect("session schema"); + let files = serde_json::to_string(&vec![TurnModifiedFile { + path: "src/persisted.ts".to_string(), + file_name: "persisted.ts".to_string(), + status: "modified".to_string(), + additions: 7, + deletions: 4, + }]) + .expect("serialize materialized files"); + conn.execute( + "INSERT INTO session_turns( + session_id,turn_id,start_sequence,started_at,status,updated_at, + modified_files_json + ) VALUES('agentsession-cloud-fork','turn-1',0, + '2026-07-23T00:00:00Z','completed', + '2026-07-23T00:00:01Z',?1)", + [files], + ) + .expect("insert materialized turn metadata"); + + let impact = persisted_session_impact_from_connection(&conn, "agentsession-cloud-fork") + .expect("load persisted impact") + .expect("persisted impact exists"); + assert_eq!(impact.files_changed, 1); + assert_eq!(impact.lines_added, 7); + assert_eq!(impact.lines_removed, 4); + assert_eq!(impact.touched_files, vec!["src/persisted.ts"]); + let index_state_rows: i64 = conn + .query_row( + "SELECT COUNT(*) FROM session_turn_index_state + WHERE session_id='agentsession-cloud-fork'", + [], + |row| row.get(0), + ) + .expect("count turn index states"); + assert_eq!(index_state_rows, 0); + } } diff --git a/src-tauri/src/orgtrack/session_provenance.rs b/src-tauri/src/orgtrack/session_provenance.rs index d8238247c..cfc604ceb 100644 --- a/src-tauri/src/orgtrack/session_provenance.rs +++ b/src-tauri/src/orgtrack/session_provenance.rs @@ -25,11 +25,15 @@ pub(crate) use hook_capture::notify_hook_inbox_ready; #[cfg(test)] use hook_capture::quarantine_invalid_envelope; pub(crate) use hook_capture::spawn_hook_inbox_drain_loop; +#[cfg(test)] +use interaction_store::persist_activity_chunks; use interaction_store::persist_file_interaction; pub(crate) use interaction_store::persist_native_event_interactions; #[cfg(test)] use interaction_store::resource_interaction_id; -pub(super) use interaction_store::{cached_event_to_activity_chunk, persist_activity_chunks}; +pub(super) use interaction_store::{ + persist_activity_chunks_with_turn_state, persist_cached_event_interactions_streaming, +}; pub(super) use path_resolution::{canonicalize_existing_prefix, resolve_file_resource}; use std::path::Path; @@ -42,14 +46,21 @@ use orgtrack_core::canonical::{ }; use orgtrack_core::privacy::ORGTRACK_SCHEMA_VERSION; use orgtrack_core::repo_sync::paths::record_id; -use orgtrack_core::sources::codex::app::{ - load_codex_app_from_path, resolve_codex_transcript_for_thread_id_near_path, -}; +use orgtrack_core::sources::codex::app::resolve_codex_transcript_for_thread_id_near_path; use orgtrack_core::sources::imported_history::metadata::SOURCE_CODEX_APP; +use orgtrack_core::sources::imported_history::replay::{ + self, ImportedHistorySourceId, ReplayLimits, +}; use orgtrack_core::store::{sqlite::SqliteRecordStore, RecentHookSignal, RecordStore}; +use rusqlite::Connection; pub(crate) const RESOURCE_INTERACTIONS_CHANGED_EVENT: &str = "orgtrack:resource-interactions-changed"; +const ACTOR_REPLAY_LIMITS: ReplayLimits = ReplayLimits { + max_turns: 10, + max_events: 200, + max_ipc_bytes: 4 * 1024 * 1024, +}; #[tauri::command] pub async fn session_provenance_recent_signals( @@ -59,9 +70,10 @@ pub async fn session_provenance_recent_signals( } fn persist_actor_lifecycle( - store: &SqliteRecordStore<'_>, + conn: &mut Connection, envelope: &SessionActorLifecycleEnvelopeV1, ) -> Result<(), String> { + let store = SqliteRecordStore::new(conn); let existing = store.get_session_actor_by_source_identity( &envelope.source, &envelope.source_session_id, @@ -236,23 +248,15 @@ fn persist_actor_lifecycle( ) { let path = Path::new(transcript_path); if path.is_file() { - match load_codex_app_from_path(transcript_session_id, path) { - Ok(chunks) => { - store.delete_reconciled_resource_interactions( - &envelope.source, - transcript_session_id, - )?; - persist_activity_chunks( - store, - &envelope.source, - Some(source_session_id), - transcript_session_id, - Some(&envelope.actor_id), - &envelope.cwd, - AttributionPrecision::Exact, - &chunks, - )?; - } + match reconcile_codex_actor_transcript( + conn, + source_session_id, + transcript_session_id, + path, + &envelope.actor_id, + &envelope.cwd, + ) { + Ok(()) => {} Err(err) => tracing::warn!( actor_id = %envelope.actor_id, transcript_path, @@ -266,6 +270,133 @@ fn persist_actor_lifecycle( Ok(()) } +/// Rebuild one stopped Codex actor's reconciled interactions through the +/// bounded replay index. The old path decoded the whole JSONL into one +/// `Vec`; this keeps only a <=200-event page in memory and +/// atomically publishes one generation/revision snapshot. +fn reconcile_codex_actor_transcript( + conn: &mut Connection, + source_session_id: &str, + transcript_session_id: &str, + transcript_path: &Path, + actor_id: &str, + workspace_path: &str, +) -> Result<(), String> { + let _writer = database::db::sessions_writer_guard(); + let source = ImportedHistorySourceId::CodexApp; + replay::bind_source_path( + conn, + source, + source_session_id, + transcript_session_id, + transcript_path, + )?; + + // Materialize lazy compact turns through bounded pages, retry a changing + // source at most twice, then publish one strict immutable snapshot. + let prepared = + replay::prepare_pinned_scan(conn, source, transcript_session_id, ACTOR_REPLAY_LIMITS)?; + + publish_codex_actor_reconciliation( + conn, + source_session_id, + transcript_session_id, + actor_id, + workspace_path, + &prepared.generation, + prepared.revision, + ) +} + +#[allow(clippy::too_many_arguments)] +fn publish_codex_actor_reconciliation( + conn: &mut Connection, + source_session_id: &str, + transcript_session_id: &str, + actor_id: &str, + workspace_path: &str, + expected_generation: &str, + expected_revision: u64, +) -> Result<(), String> { + let source = ImportedHistorySourceId::CodexApp; + conn.execute_batch("BEGIN IMMEDIATE") + .map_err(|err| format!("begin Codex actor reconciliation: {err}"))?; + let publish = (|| { + let store = SqliteRecordStore::new(conn); + store.delete_reconciled_resource_interactions(SOURCE_CODEX_APP, transcript_session_id)?; + let mut after_sequence = -1_i64; + let mut current_turn_id = None; + loop { + let batch = replay::scan_window_after_generation( + conn, + source, + transcript_session_id, + expected_generation, + expected_revision, + after_sequence, + ACTOR_REPLAY_LIMITS, + )?; + if batch.chunks.is_empty() + && batch.has_more + && batch.cursor.through_sequence <= after_sequence + { + return Err(format!( + "Pinned Codex actor replay made no progress for {transcript_session_id} after sequence {after_sequence}" + )); + } + after_sequence = batch.cursor.through_sequence; + if !batch.chunks.is_empty() { + let chunks = batch + .chunks + .into_iter() + .map(|indexed| indexed.chunk) + .collect::>(); + let store = SqliteRecordStore::new(conn); + persist_activity_chunks_with_turn_state( + &store, + SOURCE_CODEX_APP, + Some(source_session_id), + transcript_session_id, + Some(actor_id), + workspace_path, + AttributionPrecision::Exact, + &chunks, + &mut current_turn_id, + )?; + } + if !batch.has_more { + break; + } + } + Ok::<(), String>(()) + })(); + match publish { + Ok(()) => match conn.execute_batch("COMMIT") { + Ok(()) => Ok(()), + Err(commit_error) => { + let rollback = conn.execute_batch("ROLLBACK"); + match rollback { + Ok(()) => Err(format!( + "commit Codex actor reconciliation: {commit_error}" + )), + Err(rollback_error) => Err(format!( + "commit Codex actor reconciliation: {commit_error}; failed to roll back: {rollback_error}" + )), + } + } + }, + Err(error) => { + let rollback = conn.execute_batch("ROLLBACK"); + match rollback { + Ok(()) => Err(error), + Err(rollback_error) => Err(format!( + "{error}; failed to roll back Codex actor reconciliation: {rollback_error}" + )), + } + } + } +} + fn merge_earliest_timestamp(current: Option, incoming: Option<&str>) -> Option { match (current, incoming) { (Some(current), Some(incoming)) if incoming < current.as_str() => { @@ -415,7 +546,9 @@ fn persist_envelope( mod tests { use super::*; use orgtrack_core::canonical::ResourceAction; - use orgtrack_core::sources::codex::app::load_codex_app_for_session; + use orgtrack_core::sources::codex::app::{ + load_codex_app_for_session, load_codex_app_from_path, + }; use orgtrack_core::sources::imported_history::metadata::SOURCE_CLAUDE_CODE; use rusqlite::Connection; use std::fs; @@ -491,9 +624,10 @@ mod tests { #[test] fn codex_lifecycle_maps_actor_to_independently_loadable_transcript() { - let conn = Connection::open_in_memory().expect("in-memory SQLite"); + let mut conn = Connection::open_in_memory().expect("in-memory SQLite"); SqliteRecordStore::init_tables(&conn).expect("initialize orgtrack schema"); - let store = SqliteRecordStore::new(&conn); + SqliteRecordStore::init_source_cache_tables(&conn) + .expect("initialize imported replay schema"); let temp = tempfile::tempdir().expect("Codex session root"); let sessions_dir = temp .path() @@ -525,7 +659,7 @@ mod tests { let child_session_id = orgtrack_core::sources::codex::canonical_session_id(&child_stem); persist_actor_lifecycle( - &store, + &mut conn, &SessionActorLifecycleEnvelopeV1 { schema_version: SESSION_ACTOR_SCHEMA_VERSION, source: SOURCE_CODEX_APP.to_string(), @@ -544,7 +678,7 @@ mod tests { ) .expect("persist stop first"); persist_actor_lifecycle( - &store, + &mut conn, &SessionActorLifecycleEnvelopeV1 { schema_version: SESSION_ACTOR_SCHEMA_VERSION, source: SOURCE_CODEX_APP.to_string(), @@ -563,6 +697,7 @@ mod tests { ) .expect("persist late start"); + let store = SqliteRecordStore::new(&conn); let actor = store .get_session_actor(SOURCE_CODEX_APP, &parent_session_id, "agent-1") .expect("query actor") @@ -614,6 +749,160 @@ mod tests { })); } + #[test] + fn bounded_actor_reconciliation_matches_legacy_projection_and_rolls_back_cursor_drift() { + let temp = tempfile::tempdir().expect("Codex actor fixture"); + let source_session_id = "rollout-2026-07-22T10-00-00-differential"; + let session_id = orgtrack_core::sources::codex::canonical_session_id(source_session_id); + let path = temp.path().join(format!("{source_session_id}.jsonl")); + let mut fixture = vec![serde_json::json!({ + "timestamp":"2026-07-22T10:00:00Z", + "type":"event_msg", + "payload":{"type":"user_message","message":"update the file"} + }) + .to_string()]; + // 205 edits force at least two compact replay pages while one logical + // turn remains open, exercising the carried turn attribution state. + for index in 0..205 { + let call_id = format!("call_patch_{index}"); + let patch = format!( + "*** Begin Patch\n*** Update File: src/file_{index}.rs\n@@\n-old\n+new\n*** End Patch" + ); + let arguments = serde_json::json!({ "patch": patch }).to_string(); + fixture.push( + serde_json::json!({ + "timestamp":format!("2026-07-22T10:{:02}:01Z", index % 60), + "type":"response_item", + "payload":{ + "type":"function_call", + "name":"apply_patch", + "arguments":arguments, + "call_id":call_id + } + }) + .to_string(), + ); + fixture.push( + serde_json::json!({ + "timestamp":format!("2026-07-22T10:{:02}:02Z", index % 60), + "type":"response_item", + "payload":{ + "type":"function_call_output", + "call_id":call_id, + "output":"Done" + } + }) + .to_string(), + ); + } + fs::write(&path, format!("{}\n", fixture.join("\n"))).expect("write differential fixture"); + + let legacy_conn = Connection::open_in_memory().expect("legacy DB"); + SqliteRecordStore::init_tables(&legacy_conn).expect("legacy schema"); + SqliteRecordStore::init_source_cache_tables(&legacy_conn).expect("legacy replay schema"); + let legacy_chunks = load_codex_app_from_path(&session_id, &path).expect("legacy decode"); + let legacy_store = SqliteRecordStore::new(&legacy_conn); + persist_activity_chunks( + &legacy_store, + SOURCE_CODEX_APP, + Some(source_session_id), + &session_id, + Some("actor-differential"), + "/repo", + AttributionPrecision::Exact, + &legacy_chunks, + ) + .expect("legacy interaction projection"); + + let mut bounded_conn = Connection::open_in_memory().expect("bounded DB"); + SqliteRecordStore::init_tables(&bounded_conn).expect("bounded schema"); + SqliteRecordStore::init_source_cache_tables(&bounded_conn).expect("bounded replay schema"); + reconcile_codex_actor_transcript( + &mut bounded_conn, + source_session_id, + &session_id, + &path, + "actor-differential", + "/repo", + ) + .expect("bounded interaction projection"); + + let read_interactions = |conn: &Connection| { + let mut statement = conn + .prepare( + "SELECT interaction.payload_json,resource.repository_id, + resource.workspace_path,resource.repo_relative_path,resource.path_hash + FROM orgtrack_core_resource_interactions AS interaction + JOIN orgtrack_core_file_resources AS resource + ON resource.resource_id=interaction.resource_id + WHERE interaction.source=?1 AND interaction.session_id=?2 + ORDER BY interaction.interaction_id", + ) + .expect("prepare interactions"); + statement + .query_map(rusqlite::params![SOURCE_CODEX_APP, session_id], |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, String>(2)?, + row.get::<_, String>(3)?, + row.get::<_, String>(4)?, + )) + }) + .expect("query interactions") + .map(|row| row.expect("decode interaction")) + .collect::>() + }; + let legacy = read_interactions(&legacy_conn); + let bounded = read_interactions(&bounded_conn); + assert_eq!(bounded.len(), 205, "fixture must cross one replay page"); + assert_eq!( + bounded, legacy, + "bounded projection must preserve semantics" + ); + + let (generation, revision): (String, i64) = bounded_conn + .query_row( + "SELECT generation,revision FROM imported_replay_state + WHERE source=?1 AND source_session_id=?2", + rusqlite::params![SOURCE_CODEX_APP, source_session_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("read replay cursor"); + let before_failed_publish = read_interactions(&bounded_conn); + let error = publish_codex_actor_reconciliation( + &mut bounded_conn, + source_session_id, + &session_id, + "actor-differential", + "/repo", + &generation, + revision.max(0) as u64 + 1, + ) + .expect_err("revision drift must reject the replacement"); + assert!(error.contains("Replay cursor changed")); + assert_eq!( + read_interactions(&bounded_conn), + before_failed_publish, + "failed replacement must roll back the previous valid snapshot" + ); + + fs::write(&path, b"").expect("truncate actor transcript"); + reconcile_codex_actor_transcript( + &mut bounded_conn, + source_session_id, + &session_id, + &path, + "actor-differential", + "/repo", + ) + .expect("publish empty replacement generation"); + assert!( + read_interactions(&bounded_conn).is_empty(), + "an empty replacement must atomically clear stale reconciled rows" + ); + } + #[cfg(unix)] #[test] fn resolves_symlink_aliases_for_not_yet_created_files() { diff --git a/src-tauri/src/orgtrack/session_provenance/collaboration_replay.rs b/src-tauri/src/orgtrack/session_provenance/collaboration_replay.rs index 8a22c50ed..edb05d8e3 100644 --- a/src-tauri/src/orgtrack/session_provenance/collaboration_replay.rs +++ b/src-tauri/src/orgtrack/session_provenance/collaboration_replay.rs @@ -9,19 +9,14 @@ use std::path::{Component, Path}; use chrono::Utc; use database::db::{begin_immediate, get_connection, with_sessions_writer}; use orgtrack_core::canonical::{ - AgentMetadata, AttributionPrecision, CollaborationSessionOrigin, - ResourceInteractionCaptureMethod, SessionRecord, SOURCE_ORGII_CLOUD_REPLAY, + AgentMetadata, AttributionPrecision, CollaborationSessionOrigin, SessionRecord, + SOURCE_ORGII_CLOUD_REPLAY, }; use orgtrack_core::privacy::ORGTRACK_SCHEMA_VERSION; -use orgtrack_core::resource_interaction::{ - activity_chunk_source_event_id, file_interactions_from_activity_chunk, - interaction_outcome_from_activity_chunk, -}; -use orgtrack_core::sources::imported_history::FUNCTION_USER_MESSAGE; use orgtrack_core::store::{sqlite::SqliteRecordStore, RecordStore}; use sha2::{Digest, Sha256}; -use super::interaction_store::{cached_event_to_activity_chunk, persist_file_interaction}; +use super::interaction_store::persist_cached_event_interactions_streaming; const COLLABORATION_REPLAY_PARSER_VERSION: i64 = 2; @@ -110,24 +105,6 @@ pub(crate) fn index_collaboration_replay( }, }; - let preflight_current = { - let conn = get_connection().map_err(|err| err.to_string())?; - SqliteRecordStore::new(&conn).interaction_import_is_current( - SOURCE_ORGII_CLOUD_REPLAY, - local_session_id, - &fingerprint, - COLLABORATION_REPLAY_PARSER_VERSION, - )? - }; - let mut events = if preflight_current { - None - } else { - Some( - session_persistence::load_events(local_session_id) - .map_err(|err| format!("Failed to load collaboration replay events: {err}"))?, - ) - }; - with_sessions_writer(|| { let conn = get_connection().map_err(|err| err.to_string())?; let tx = begin_immediate(&conn).map_err(|err| err.to_string())?; @@ -143,52 +120,22 @@ pub(crate) fn index_collaboration_replay( return Ok(0); } - let events = match events.take() { - Some(events) => events, - None => session_persistence::load_events(local_session_id) - .map_err(|err| format!("Failed to load collaboration replay events: {err}"))?, - }; store .delete_reconciled_resource_interactions(SOURCE_ORGII_CLOUD_REPLAY, local_session_id)?; - let mut persisted = 0; - let mut current_turn_id: Option = None; - for event in &events { - let chunk = cached_event_to_activity_chunk(event); - if chunk.function == FUNCTION_USER_MESSAGE { - current_turn_id = Some(chunk.chunk_id); - continue; - } - let outcome = interaction_outcome_from_activity_chunk(&chunk); - for mut interaction in file_interactions_from_activity_chunk(&chunk) { - let Some(mapped_path) = remap_collaboration_file_path( - &interaction.file_path, - source_workspace_path, - workspace_path, - ) else { - continue; - }; - interaction.file_path = mapped_path; - let source_event_id = activity_chunk_source_event_id(&chunk, &interaction); - persist_file_interaction( - &store, - SOURCE_ORGII_CLOUD_REPLAY, - Some(source_session_id), - local_session_id, - Some(&source_event_id), - current_turn_id.as_deref(), - Some(owner_member_id), - workspace_path, - &interaction.file_path, - interaction.action, - outcome, - &chunk.created_at, - ResourceInteractionCaptureMethod::Reconciled, - AttributionPrecision::Exact, - )?; - persisted += 1; - } - } + let persisted = persist_cached_event_interactions_streaming( + &tx, + &store, + SOURCE_ORGII_CLOUD_REPLAY, + Some(source_session_id), + local_session_id, + Some(owner_member_id), + workspace_path, + AttributionPrecision::Exact, + |file_path| { + remap_collaboration_file_path(file_path, source_workspace_path, workspace_path) + }, + )?; store.mark_interaction_imported( SOURCE_ORGII_CLOUD_REPLAY, local_session_id, diff --git a/src-tauri/src/orgtrack/session_provenance/historical_backfill.rs b/src-tauri/src/orgtrack/session_provenance/historical_backfill.rs index c38e233e0..390f3a76b 100644 --- a/src-tauri/src/orgtrack/session_provenance/historical_backfill.rs +++ b/src-tauri/src/orgtrack/session_provenance/historical_backfill.rs @@ -14,28 +14,24 @@ use database::db::get_connection; use orgtrack_core::canonical::{ AttributionPrecision, SessionRecord, SOURCE_ORGII_CLI_SESSIONS, SOURCE_ORGII_RUST_AGENTS, }; -use orgtrack_core::sources::claude_code::history::{ - list_claude_code_history_sessions_paginated, load_claude_code_history_for_session, -}; use orgtrack_core::sources::codex::app::{ codex_thread_id_from_file_stem, list_codex_app_reconciliation_sessions, - list_codex_app_sessions_paginated, load_codex_app_for_session, -}; -use orgtrack_core::sources::cursor_ide::history::{ - list_cursor_ide_sessions_paginated, load_history_for_session as load_cursor_history_for_session, }; use orgtrack_core::sources::imported_history::cache::{ query_cached_sessions_for_repo_from_conn, query_recent_cached_sessions_for_source_from_conn, ImportedHistoryCachedSession, }; -use orgtrack_core::sources::imported_history::metadata::{ - SOURCE_CLAUDE_CODE, SOURCE_CODEX_APP, SOURCE_CURSOR_IDE, +use orgtrack_core::sources::imported_history::catalog::refresh_source as refresh_imported_catalog; +use orgtrack_core::sources::imported_history::metadata::SOURCE_CODEX_APP; +use orgtrack_core::sources::imported_history::replay::{ + self, ImportedHistorySourceId, ReplayLimits, }; use orgtrack_core::store::{sqlite::SqliteRecordStore, RecordStore}; use rusqlite::Connection; use super::{ - cached_event_to_activity_chunk, canonicalize_existing_prefix, persist_activity_chunks, + canonicalize_existing_prefix, persist_activity_chunks_with_turn_state, + persist_cached_event_interactions_streaming, }; // v3: repository ids now come from filesystem git discovery instead of @@ -441,6 +437,13 @@ fn failed_backfill_snapshot(error: String) -> crate::orgtrack::types::FileSessio } } +/// Keep every historical-provenance entry point tied to the authoritative +/// replay registry. Adding a new imported source must not require finding and +/// updating another hand-maintained allow-list in this module. +fn historical_imported_sources() -> impl Iterator { + ImportedHistorySourceId::ALL.into_iter() +} + fn reconcile_historical_interactions( conn: &mut Connection, repo_path: &str, @@ -451,7 +454,8 @@ fn reconcile_historical_interactions( let mut discovery_failures = sync_imported_history_caches(conn); let mut imported_sessions = Vec::new(); - for source in [SOURCE_CLAUDE_CODE, SOURCE_CODEX_APP, SOURCE_CURSOR_IDE] { + for replay_source in historical_imported_sources() { + let source = replay_source.as_str(); match imported_sessions_for_repo(conn, source, repo_path, &canonical_repo) { Ok(sessions) => { imported_sessions.extend(sessions.into_iter().map(|session| (source, session))) @@ -530,17 +534,15 @@ fn reconcile_historical_interactions( fn sync_imported_history_caches(conn: &mut Connection) -> usize { let mut failures = 0; - if let Err(err) = list_claude_code_history_sessions_paginated(conn, 1, 0) { - failures += 1; - tracing::warn!(error = %err, "[SessionProvenance] Claude history discovery failed"); - } - if let Err(err) = list_codex_app_sessions_paginated(conn, 1, 0) { - failures += 1; - tracing::warn!(error = %err, "[SessionProvenance] Codex history discovery failed"); - } - if let Err(err) = list_cursor_ide_sessions_paginated(conn, 1, 0) { - failures += 1; - tracing::warn!(error = %err, "[SessionProvenance] Cursor history discovery failed"); + for source in historical_imported_sources() { + if let Err(err) = refresh_imported_catalog(conn, source) { + failures += 1; + tracing::warn!( + source = source.as_str(), + error = %err, + "[SessionProvenance] Imported history catalog discovery failed" + ); + } } failures } @@ -630,6 +632,17 @@ fn session_is_quiescent(session: &ImportedHistoryCachedSession, now_ms: i64) -> source_age_ms(session.source_mtime_ms, now_ms) >= SESSION_QUIESCENCE_MS } +fn periodic_codex_reconciliation_needed( + session: &ImportedHistoryCachedSession, + session_start_active: bool, + now_ms: i64, +) -> bool { + // This loop is live-hook recovery, not historical migration. Completed + // transcripts stay compact-catalog-only until a provenance/file-history + // request explicitly asks the bounded backfill worker to index them. + !session_start_active && !session_is_quiescent(session, now_ms) +} + fn active_codex_recovery_is_quiet_enough(source_mtime_value: i64, now_ms: i64) -> bool { source_age_ms(source_mtime_value, now_ms) >= ACTIVE_CODEX_MIN_QUIET_MS } @@ -650,7 +663,8 @@ fn priority_file_needs_backfill( ) -> bool { let store = SqliteRecordStore::new(conn); let now_ms = Utc::now().timestamp_millis(); - for source in [SOURCE_CLAUDE_CODE, SOURCE_CODEX_APP, SOURCE_CURSOR_IDE] { + for replay_source in historical_imported_sources() { + let source = replay_source.as_str(); let Ok(sessions) = imported_sessions_for_repo(conn, source, repo_path, canonical_repo) else { return true; @@ -712,21 +726,24 @@ fn backlog_sessions_needing_work( } fn reconcile_imported_session( - conn: &Connection, + conn: &mut Connection, source: &str, canonical_repo: &Path, session: &ImportedHistoryCachedSession, active_session_policy: ActiveSessionPolicy, ) -> Result { - let store = SqliteRecordStore::new(conn); + let _writer = database::db::sessions_writer_guard(); let fingerprint = imported_session_fingerprint(session); - if store.interaction_import_is_current( - source, - &session.session_id, - &fingerprint, - HISTORICAL_INTERACTION_PARSER_VERSION, - )? { - return Ok(false); + { + let store = SqliteRecordStore::new(conn); + if store.interaction_import_is_current( + source, + &session.session_id, + &fingerprint, + HISTORICAL_INTERACTION_PARSER_VERSION, + )? { + return Ok(false); + } } if active_session_policy == ActiveSessionPolicy::QuiescentOnly && !session_is_quiescent(session, Utc::now().timestamp_millis()) @@ -735,13 +752,7 @@ fn reconcile_imported_session( // quiet will index it. return Ok(false); } - let chunks = match source { - SOURCE_CLAUDE_CODE => load_claude_code_history_for_session(conn, &session.session_id), - SOURCE_CODEX_APP => load_codex_app_for_session(conn, &session.session_id), - SOURCE_CURSOR_IDE => load_cursor_history_for_session(&session.session_id), - _ => return Err(format!("Unsupported imported history source: {source}")), - }?; - store.delete_reconciled_resource_interactions(source, &session.session_id)?; + let replay_source = ImportedHistorySourceId::parse(source)?; let actor_id = session .parent_session_id .as_ref() @@ -751,27 +762,92 @@ fn reconcile_imported_session( } else { AttributionPrecision::SessionOnly }; - persist_activity_chunks( - &store, - source, - Some(&session.source_session_id), - &session.session_id, - actor_id, - session - .repo_path - .as_deref() - .or_else(|| canonical_repo.to_str()) - .unwrap_or("."), - precision, - &chunks, - )?; - store.mark_interaction_imported( - source, - &session.session_id, - &fingerprint, - HISTORICAL_INTERACTION_PARSER_VERSION, - &Utc::now().to_rfc3339(), - )?; + let workspace_path = session + .repo_path + .as_deref() + .or_else(|| canonical_repo.to_str()) + .unwrap_or(".") + .to_string(); + let limits = ReplayLimits { + max_turns: replay::HARD_MAX_TURNS, + max_events: replay::HARD_MAX_EVENTS, + max_ipc_bytes: replay::HARD_MAX_IPC_BYTES, + }; + // Finish a bounded materialization pass before deleting the previous read + // model. The helper retries a changing source at most twice and returns a + // generation/revision that the transaction can replay strictly. + let prepared = replay::prepare_pinned_scan(conn, replay_source, &session.session_id, limits)?; + let expected_generation = prepared.generation; + let expected_revision = prepared.revision; + + conn.execute_batch("BEGIN IMMEDIATE") + .map_err(|err| format!("begin provenance snapshot publish: {err}"))?; + let publish = (|| { + let store = SqliteRecordStore::new(conn); + store.delete_reconciled_resource_interactions(source, &session.session_id)?; + let mut after_sequence = -1_i64; + let mut current_turn_id = None; + loop { + let batch = replay::scan_window_after_generation( + conn, + replay_source, + &session.session_id, + &expected_generation, + expected_revision, + after_sequence, + limits, + )?; + if batch.chunks.is_empty() + && batch.has_more + && batch.cursor.through_sequence <= after_sequence + { + return Err(format!( + "Pinned replay scan made no progress for {} after sequence {}", + session.session_id, after_sequence + )); + } + after_sequence = batch.cursor.through_sequence; + if !batch.chunks.is_empty() { + let chunks = batch + .chunks + .into_iter() + .map(|indexed| indexed.chunk) + .collect::>(); + let store = SqliteRecordStore::new(conn); + persist_activity_chunks_with_turn_state( + &store, + source, + Some(&session.source_session_id), + &session.session_id, + actor_id, + &workspace_path, + precision, + &chunks, + &mut current_turn_id, + )?; + } + if !batch.has_more { + break; + } + } + let store = SqliteRecordStore::new(conn); + store.mark_interaction_imported( + source, + &session.session_id, + &fingerprint, + HISTORICAL_INTERACTION_PARSER_VERSION, + &Utc::now().to_rfc3339(), + ) + })(); + match publish { + Ok(()) => conn + .execute_batch("COMMIT") + .map_err(|err| format!("commit provenance snapshot publish: {err}"))?, + Err(error) => { + let _ = conn.execute_batch("ROLLBACK"); + return Err(error); + } + } Ok(true) } @@ -921,7 +997,7 @@ fn reconcile_recent_codex_sessions( let session_start_active = agent_cli::session_provenance::codex_session_start_is_active(hook_session_id) .unwrap_or(false); - if session_start_active { + if !periodic_codex_reconciliation_needed(&session, session_start_active, now_ms) { throttle.clear(&session.session_id); continue; } @@ -932,12 +1008,11 @@ fn reconcile_recent_codex_sessions( continue; } let is_quiescent = session_is_quiescent(&session, now_ms); - if !is_quiescent { - if !active_codex_recovery_is_quiet_enough(session.source_mtime_ms, now_ms) - || active_reconciliations >= ACTIVE_CODEX_RECONCILIATION_BATCH_PER_PASS - { - continue; - } + if !is_quiescent + && (!active_codex_recovery_is_quiet_enough(session.source_mtime_ms, now_ms) + || active_reconciliations >= ACTIVE_CODEX_RECONCILIATION_BATCH_PER_PASS) + { + continue; } if !throttle.should_attempt(&session.session_id, is_quiescent, now_ms) { continue; @@ -948,7 +1023,7 @@ fn reconcile_recent_codex_sessions( let repo = session.repo_path.as_deref().unwrap_or("."); let canonical_repo = canonicalize_existing_prefix(Path::new(repo)); match reconcile_imported_session( - &conn, + &mut conn, SOURCE_CODEX_APP, &canonical_repo, &session, @@ -1028,12 +1103,6 @@ fn reconcile_native_session( )? { return Ok(()); } - let events = - session_persistence::load_events(&session.session_id).map_err(|err| err.to_string())?; - let chunks = events - .iter() - .map(cached_event_to_activity_chunk) - .collect::>(); let actor_id = session.org_member_id.as_deref().or_else(|| { session .parent_session_id @@ -1046,19 +1115,21 @@ fn reconcile_native_session( AttributionPrecision::SessionOnly }; store.delete_reconciled_resource_interactions(&session.source, &session.session_id)?; - persist_activity_chunks( + let workspace_path = session + .workspace_path + .as_deref() + .or_else(|| canonical_repo.to_str()) + .unwrap_or("."); + persist_cached_event_interactions_streaming( + conn, &store, &session.source, Some(&session.source_session_id), &session.session_id, actor_id, - session - .workspace_path - .as_deref() - .or_else(|| canonical_repo.to_str()) - .unwrap_or("."), + workspace_path, precision, - &chunks, + |file_path| Some(file_path.to_string()), )?; store.mark_interaction_imported( &session.source, @@ -1070,144 +1141,4 @@ fn reconcile_native_session( } #[cfg(test)] -mod tests { - use super::*; - use orgtrack_core::store::sqlite::SqliteRecordStore; - - #[test] - fn durable_backfill_claim_joins_current_process_and_reclaims_previous_process() { - let mut conn = Connection::open_in_memory().expect("in-memory SQLite"); - SqliteRecordStore::init_tables(&conn).expect("initialize Orgtrack schema"); - - let (first, first_token) = - claim_backfill_job(&mut conn, "/repo").expect("claim first durable backfill"); - let first_token = first_token.expect("first request owns the job"); - assert_eq!(first.status, HistoricalBackfillStatus::Queued); - - let (joined, joined_token) = - claim_backfill_job(&mut conn, "/repo").expect("join active durable backfill"); - assert!(joined.is_active()); - assert_eq!(joined.run_token, first_token); - assert!(joined_token.is_none()); - - conn.execute( - "UPDATE orgtrack_core_interaction_backfill_jobs - SET status = 'indexing', run_token = 'previous-process:run', updated_at_ms = ?1 - WHERE repo_key = '/repo'", - [Utc::now().timestamp_millis()], - ) - .expect("simulate previous process lease"); - let (reclaimed, reclaimed_token) = - claim_backfill_job(&mut conn, "/repo").expect("reclaim previous process backfill"); - assert_eq!(reclaimed.status, HistoricalBackfillStatus::Queued); - assert_ne!(reclaimed.run_token, "previous-process:run"); - assert_eq!( - reclaimed_token.as_deref(), - Some(reclaimed.run_token.as_str()) - ); - } - - #[test] - fn active_codex_reconciliation_is_sparse_but_quiescence_flushes_immediately() { - let mut throttle = ActiveCodexReconciliationThrottle::default(); - let now_ms = 1_000_000; - - assert!(throttle.should_attempt("active", false, now_ms)); - assert!(!throttle.should_attempt( - "active", - false, - now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS - 1 - )); - assert!(throttle.should_attempt( - "active", - false, - now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS - )); - - // The final quiet transcript is never delayed by the active throttle. - assert!(throttle.should_attempt("active", true, now_ms + 1)); - assert!(!throttle.attempted_at_ms.contains_key("active")); - } - - #[test] - fn active_codex_reconciliation_throttle_is_bounded_to_recent_sessions() { - let mut throttle = ActiveCodexReconciliationThrottle::default(); - assert!(throttle.should_attempt("old", false, 1)); - assert!(throttle.should_attempt( - "recent", - false, - 1 + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS - )); - - throttle.retain_recent(["recent"]); - - assert_eq!(throttle.attempted_at_ms.len(), 1); - assert!(throttle.attempted_at_ms.contains_key("recent")); - } - - #[test] - fn active_codex_reconciliation_is_globally_serialized() { - let mut throttle = ActiveCodexReconciliationThrottle::default(); - let now_ms = 1_000_000; - - assert!(throttle.should_attempt("first", false, now_ms)); - assert!(!throttle.should_attempt( - "second", - false, - now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS - 1 - )); - assert!(throttle.should_attempt( - "second", - false, - now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS - )); - } - - #[test] - fn background_codex_reconciliation_rejects_giant_rollouts() { - assert!(background_codex_reconciliation_source_is_bounded( - MAX_BACKGROUND_CODEX_RECONCILIATION_SOURCE_BYTES - )); - assert!(!background_codex_reconciliation_source_is_bounded( - MAX_BACKGROUND_CODEX_RECONCILIATION_SOURCE_BYTES + 1 - )); - } - - #[test] - fn codex_discovery_refresh_is_low_frequency() { - let mut throttle = ActiveCodexReconciliationThrottle::default(); - let now_ms = 1_000_000; - - assert!(throttle.should_refresh_discovery(now_ms)); - assert!( - !throttle.should_refresh_discovery(now_ms + CODEX_DISCOVERY_REFRESH_INTERVAL_MS - 1) - ); - assert!(throttle.should_refresh_discovery(now_ms + CODEX_DISCOVERY_REFRESH_INTERVAL_MS)); - } - - #[test] - fn active_codex_recovery_waits_for_a_short_writer_quiet_window() { - let now_ms = 1_000_000; - - assert!(!active_codex_recovery_is_quiet_enough( - now_ms - ACTIVE_CODEX_MIN_QUIET_MS + 1, - now_ms - )); - assert!(active_codex_recovery_is_quiet_enough( - now_ms - ACTIVE_CODEX_MIN_QUIET_MS, - now_ms - )); - } - - #[test] - fn imported_history_source_age_accepts_legacy_milliseconds_and_current_nanoseconds() { - let now_ms = 1_750_000_000_000; - let source_ms = now_ms - ACTIVE_CODEX_MIN_QUIET_MS; - let source_ns = source_ms * 1_000_000; - - assert_eq!(source_age_ms(source_ms, now_ms), ACTIVE_CODEX_MIN_QUIET_MS); - assert_eq!(source_age_ms(source_ns, now_ms), ACTIVE_CODEX_MIN_QUIET_MS); - assert!(active_codex_recovery_is_quiet_enough(source_ms, now_ms)); - assert!(active_codex_recovery_is_quiet_enough(source_ns, now_ms)); - } -} +mod tests; diff --git a/src-tauri/src/orgtrack/session_provenance/historical_backfill/tests.rs b/src-tauri/src/orgtrack/session_provenance/historical_backfill/tests.rs new file mode 100644 index 000000000..ac5d34ae0 --- /dev/null +++ b/src-tauri/src/orgtrack/session_provenance/historical_backfill/tests.rs @@ -0,0 +1,346 @@ +use super::*; +use orgtrack_core::sources::imported_history::metadata::ImportedHistoryImpactStats; +use orgtrack_core::store::sqlite::SqliteRecordStore; + +#[test] +fn historical_backfill_uses_every_registered_imported_source() { + assert_eq!( + historical_imported_sources().collect::>(), + ImportedHistorySourceId::ALL + ); +} + +fn imported_session_with_touched_files(touched_files: Vec) -> ImportedHistoryCachedSession { + ImportedHistoryCachedSession { + source_session_id: "source-session".to_string(), + session_id: "codexapp-source-session".to_string(), + source_path: "/tmp/rollout.jsonl".to_string(), + source_record_key: "source-session".to_string(), + source_mtime_ms: 1, + source_size_bytes: 1, + source_fingerprint: "fingerprint".to_string(), + parser_version: 1, + name: "Imported session".to_string(), + created_at_ms: 1, + updated_at_ms: 1, + model: None, + input_tokens: 0, + output_tokens: 0, + repo_path: Some("/repo".to_string()), + repo_root_path: None, + repo_remote_urls: Vec::new(), + branch: None, + impact: ImportedHistoryImpactStats { + files_changed: touched_files.len() as i64, + touched_files, + ..ImportedHistoryImpactStats::default() + }, + listable: true, + source_metadata_json: None, + parent_session_id: None, + } +} + +#[test] +fn quiescence_normalizes_current_nanoseconds_and_legacy_milliseconds() { + let now_ms = 1_800_000_000_000_i64; + let quiet_ms = now_ms - SESSION_QUIESCENCE_MS - 1; + let active_ms = now_ms - SESSION_QUIESCENCE_MS + 1; + let mut session = imported_session_with_touched_files(Vec::new()); + + session.source_mtime_ms = quiet_ms; + assert!(session_is_quiescent(&session, now_ms)); + session.source_mtime_ms = quiet_ms * 1_000_000; + assert!(session_is_quiescent(&session, now_ms)); + + session.source_mtime_ms = active_ms; + assert!(!session_is_quiescent(&session, now_ms)); + session.source_mtime_ms = active_ms * 1_000_000; + assert!(!session_is_quiescent(&session, now_ms)); +} + +#[test] +fn periodic_codex_recovery_skips_hooked_and_completed_sessions() { + let now_ms = 1_800_000_000_000_i64; + let mut session = imported_session_with_touched_files(Vec::new()); + session.source_mtime_ms = (now_ms - 60_000) * 1_000_000; + + assert!(periodic_codex_reconciliation_needed( + &session, false, now_ms + )); + assert!(!periodic_codex_reconciliation_needed( + &session, true, now_ms + )); + + session.source_mtime_ms = (now_ms - SESSION_QUIESCENCE_MS - 1) * 1_000_000; + assert!(!periodic_codex_reconciliation_needed( + &session, false, now_ms + )); +} + +#[test] +fn priority_file_matching_requires_the_catalog_to_preserve_relative_paths() { + let repo = Path::new("/repo"); + let relative = imported_session_with_touched_files(vec!["src/new.rs".to_string()]); + let absolute = imported_session_with_touched_files(vec!["/repo/src/new.rs".to_string()]); + let basename_only = imported_session_with_touched_files(vec!["new.rs".to_string()]); + + assert!(session_touches_priority_file(&relative, repo, "src/new.rs")); + assert!(session_touches_priority_file(&absolute, repo, "src/new.rs")); + assert!(!session_touches_priority_file( + &basename_only, + repo, + "src/new.rs" + )); +} + +#[test] +fn historical_backfill_prepares_three_lazy_kv_turns_before_publish() { + for source in [ + ImportedHistorySourceId::CursorIde, + ImportedHistorySourceId::Windsurf, + ] { + let directory = tempfile::tempdir().expect("lazy backfill fixture"); + let source_path = directory.path().join(format!("{}.db", source.as_str())); + let source_conn = Connection::open(&source_path).expect("KV source DB"); + source_conn + .execute_batch( + "PRAGMA journal_mode=WAL; + CREATE TABLE cursorDiskKV(key TEXT PRIMARY KEY,value TEXT);", + ) + .expect("KV source schema"); + let headers = (0..6) + .map(|index| { + serde_json::json!({ + "bubbleId":format!("b{index}"), + "type":if index % 2 == 0 { 1 } else { 2 }, + }) + }) + .collect::>(); + source_conn + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES('composerData:c1',?1)", + [serde_json::json!({ + "composerId":"c1", + "createdAt":1, + "lastUpdatedAt":6, + "fullConversationHeadersOnly":headers, + }) + .to_string()], + ) + .expect("composer row"); + for index in 0..6 { + source_conn + .execute( + "INSERT INTO cursorDiskKV(key,value) VALUES(?1,?2)", + rusqlite::params![ + format!("bubbleId:c1:b{index}"), + serde_json::json!({ + "bubbleId":format!("b{index}"), + "type":if index % 2 == 0 { 1 } else { 2 }, + "createdAt":format!("2026-07-22T00:00:{index:02}Z"), + "text":format!("message {index}"), + }) + .to_string(), + ], + ) + .expect("bubble row"); + } + + let mut cache = Connection::open_in_memory().expect("replay cache"); + SqliteRecordStore::init_tables(&cache).expect("Orgtrack tables"); + SqliteRecordStore::init_source_cache_tables(&cache).expect("source cache tables"); + let session_id = format!("{}c1", source.descriptor().session_prefix); + cache + .execute( + "INSERT INTO imported_history_session_cache( + source,source_session_id,session_id,source_path + ) VALUES(?1,'c1',?2,?3)", + rusqlite::params![source.as_str(), session_id, source_path.to_string_lossy()], + ) + .expect("cache source path"); + let session = ImportedHistoryCachedSession { + source_session_id: "c1".to_string(), + session_id: session_id.clone(), + source_path: source_path.to_string_lossy().into_owned(), + source_record_key: "c1".to_string(), + source_mtime_ms: 1, + source_size_bytes: std::fs::metadata(&source_path) + .expect("source metadata") + .len() as i64, + source_fingerprint: "lazy-three-turns".to_string(), + parser_version: source.descriptor().parser_version as i64, + name: "Lazy KV session".to_string(), + created_at_ms: 1, + updated_at_ms: 6, + model: None, + input_tokens: 0, + output_tokens: 0, + repo_path: Some(directory.path().to_string_lossy().into_owned()), + repo_root_path: None, + repo_remote_urls: Vec::new(), + branch: None, + impact: ImportedHistoryImpactStats::default(), + listable: true, + source_metadata_json: None, + parent_session_id: None, + }; + + assert!(reconcile_imported_session( + &mut cache, + source.as_str(), + directory.path(), + &session, + ActiveSessionPolicy::AllowActive, + ) + .expect("publish strict lazy provenance snapshot")); + let replay_events = cache + .query_row( + "SELECT COUNT(*) FROM imported_replay_events + WHERE source=?1 AND source_session_id='c1'", + [source.as_str()], + |row| row.get::<_, i64>(0), + ) + .expect("count prepared replay events"); + assert_eq!(replay_events, 6, "{}", source.as_str()); + assert!(SqliteRecordStore::new(&cache) + .interaction_import_is_current( + source.as_str(), + &session_id, + &imported_session_fingerprint(&session), + HISTORICAL_INTERACTION_PARSER_VERSION, + ) + .expect("backfill checkpoint")); + } +} + +#[test] +fn durable_backfill_claim_joins_current_process_and_reclaims_previous_process() { + let mut conn = Connection::open_in_memory().expect("in-memory SQLite"); + SqliteRecordStore::init_tables(&conn).expect("initialize Orgtrack schema"); + + let (first, first_token) = + claim_backfill_job(&mut conn, "/repo").expect("claim first durable backfill"); + let first_token = first_token.expect("first request owns the job"); + assert_eq!(first.status, HistoricalBackfillStatus::Queued); + + let (joined, joined_token) = + claim_backfill_job(&mut conn, "/repo").expect("join active durable backfill"); + assert!(joined.is_active()); + assert_eq!(joined.run_token, first_token); + assert!(joined_token.is_none()); + + conn.execute( + "UPDATE orgtrack_core_interaction_backfill_jobs + SET status = 'indexing', run_token = 'previous-process:run', updated_at_ms = ?1 + WHERE repo_key = '/repo'", + [Utc::now().timestamp_millis()], + ) + .expect("simulate previous process lease"); + let (reclaimed, reclaimed_token) = + claim_backfill_job(&mut conn, "/repo").expect("reclaim previous process backfill"); + assert_eq!(reclaimed.status, HistoricalBackfillStatus::Queued); + assert_ne!(reclaimed.run_token, "previous-process:run"); + assert_eq!( + reclaimed_token.as_deref(), + Some(reclaimed.run_token.as_str()) + ); +} + +#[test] +fn active_codex_reconciliation_is_sparse_but_quiescence_flushes_immediately() { + let mut throttle = ActiveCodexReconciliationThrottle::default(); + let now_ms = 1_000_000; + + assert!(throttle.should_attempt("active", false, now_ms)); + assert!(!throttle.should_attempt( + "active", + false, + now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS - 1 + )); + assert!(throttle.should_attempt( + "active", + false, + now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS + )); + + // The final quiet transcript is never delayed by the active throttle. + assert!(throttle.should_attempt("active", true, now_ms + 1)); + assert!(!throttle.attempted_at_ms.contains_key("active")); +} + +#[test] +fn active_codex_reconciliation_throttle_is_bounded_to_recent_sessions() { + let mut throttle = ActiveCodexReconciliationThrottle::default(); + assert!(throttle.should_attempt("old", false, 1)); + assert!(throttle.should_attempt("recent", false, 1 + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS)); + + throttle.retain_recent(["recent"]); + + assert_eq!(throttle.attempted_at_ms.len(), 1); + assert!(throttle.attempted_at_ms.contains_key("recent")); +} + +#[test] +fn active_codex_reconciliation_is_globally_serialized() { + let mut throttle = ActiveCodexReconciliationThrottle::default(); + let now_ms = 1_000_000; + + assert!(throttle.should_attempt("first", false, now_ms)); + assert!(!throttle.should_attempt( + "second", + false, + now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS - 1 + )); + assert!(throttle.should_attempt( + "second", + false, + now_ms + ACTIVE_CODEX_RECONCILIATION_INTERVAL_MS + )); +} + +#[test] +fn background_codex_reconciliation_rejects_giant_rollouts() { + assert!(background_codex_reconciliation_source_is_bounded( + MAX_BACKGROUND_CODEX_RECONCILIATION_SOURCE_BYTES + )); + assert!(!background_codex_reconciliation_source_is_bounded( + MAX_BACKGROUND_CODEX_RECONCILIATION_SOURCE_BYTES + 1 + )); +} + +#[test] +fn codex_discovery_refresh_is_low_frequency() { + let mut throttle = ActiveCodexReconciliationThrottle::default(); + let now_ms = 1_000_000; + + assert!(throttle.should_refresh_discovery(now_ms)); + assert!(!throttle.should_refresh_discovery(now_ms + CODEX_DISCOVERY_REFRESH_INTERVAL_MS - 1)); + assert!(throttle.should_refresh_discovery(now_ms + CODEX_DISCOVERY_REFRESH_INTERVAL_MS)); +} + +#[test] +fn active_codex_recovery_waits_for_a_short_writer_quiet_window() { + let now_ms = 1_000_000; + + assert!(!active_codex_recovery_is_quiet_enough( + now_ms - ACTIVE_CODEX_MIN_QUIET_MS + 1, + now_ms + )); + assert!(active_codex_recovery_is_quiet_enough( + now_ms - ACTIVE_CODEX_MIN_QUIET_MS, + now_ms + )); +} + +#[test] +fn imported_history_source_age_accepts_legacy_milliseconds_and_current_nanoseconds() { + let now_ms = 1_750_000_000_000; + let source_ms = now_ms - ACTIVE_CODEX_MIN_QUIET_MS; + let source_ns = source_ms * 1_000_000; + + assert_eq!(source_age_ms(source_ms, now_ms), ACTIVE_CODEX_MIN_QUIET_MS); + assert_eq!(source_age_ms(source_ns, now_ms), ACTIVE_CODEX_MIN_QUIET_MS); + assert!(active_codex_recovery_is_quiet_enough(source_ms, now_ms)); + assert!(active_codex_recovery_is_quiet_enough(source_ns, now_ms)); +} diff --git a/src-tauri/src/orgtrack/session_provenance/hook_capture.rs b/src-tauri/src/orgtrack/session_provenance/hook_capture.rs index 75255cf58..f12659417 100644 --- a/src-tauri/src/orgtrack/session_provenance/hook_capture.rs +++ b/src-tauri/src/orgtrack/session_provenance/hook_capture.rs @@ -110,10 +110,10 @@ pub fn capture_hook_stdin(source: &str) -> Result { Ok(envelopes.len() + usize::from(lifecycle.is_some())) } -fn codex_session_start_source_session_id<'a>( +fn codex_session_start_source_session_id( source: HookSource, - payload: &'a serde_json::Value, -) -> Option<&'a str> { + payload: &serde_json::Value, +) -> Option<&str> { if source != HookSource::Codex { return None; } @@ -232,10 +232,7 @@ fn spool_has_capacity(inbox: &Path, incoming_bytes: u64) -> Result continue; }; let path = entry.path(); - if !path - .extension() - .is_some_and(|extension| extension == "json") - { + if path.extension().is_none_or(|extension| extension != "json") { continue; } file_count = file_count.saturating_add(1); @@ -306,8 +303,7 @@ pub(crate) fn drain_hook_inbox() -> Result { } let files = collect_drain_batch(&inbox)?; - let conn = get_connection().map_err(|err| err.to_string())?; - let store = SqliteRecordStore::new(&conn); + let mut conn = get_connection().map_err(|err| err.to_string())?; let mut drained = 0; for path in files { let bytes = fs::read(&path).map_err(|err| err.to_string())?; @@ -316,6 +312,7 @@ pub(crate) fn drain_hook_inbox() -> Result { if envelope.validate().is_err() { false } else { + let store = SqliteRecordStore::new(&conn); persist_envelope(&store, &envelope)?; true } @@ -325,7 +322,7 @@ pub(crate) fn drain_hook_inbox() -> Result { if envelope.validate().is_err() { false } else { - persist_actor_lifecycle(&store, &envelope)?; + persist_actor_lifecycle(&mut conn, &envelope)?; true } } else { diff --git a/src-tauri/src/orgtrack/session_provenance/interaction_store.rs b/src-tauri/src/orgtrack/session_provenance/interaction_store.rs index 59c36c27e..6533c3076 100644 --- a/src-tauri/src/orgtrack/session_provenance/interaction_store.rs +++ b/src-tauri/src/orgtrack/session_provenance/interaction_store.rs @@ -9,6 +9,7 @@ use orgtrack_core::canonical::{ ResourceInteractionOutcome, ResourceInteractionRecord, SessionRecord, RESOURCE_INTERACTION_SCHEMA_VERSION, }; +use orgtrack_core::projectors::turn_metadata::metadata_projection_requirements; use orgtrack_core::repo_sync::paths::{path_hash, record_id}; use orgtrack_core::resource_interaction::{ activity_chunk_source_event_id, file_interactions_from_activity_chunk, @@ -16,6 +17,7 @@ use orgtrack_core::resource_interaction::{ }; use orgtrack_core::sources::imported_history::FUNCTION_USER_MESSAGE; use orgtrack_core::store::RecordStore; +use rusqlite::Connection; use session_persistence::CachedEvent; use super::path_resolution::resolve_file_resource; @@ -122,6 +124,7 @@ pub(crate) fn persist_native_event_interactions( } #[allow(clippy::too_many_arguments)] +#[cfg(test)] pub(crate) fn persist_activity_chunks( store: &dyn RecordStore, source: &str, @@ -131,12 +134,40 @@ pub(crate) fn persist_activity_chunks( workspace_path: &str, precision: AttributionPrecision, chunks: &[ActivityChunk], +) -> Result { + let mut current_turn_id = None; + persist_activity_chunks_with_turn_state( + store, + source, + source_session_id, + session_id, + actor_id, + workspace_path, + precision, + chunks, + &mut current_turn_id, + ) +} + +/// Persist one bounded replay batch while carrying the user-turn boundary +/// across batches. This prevents backend metadata consumers from rebuilding a +/// session-sized `Vec` merely to preserve turn attribution. +#[allow(clippy::too_many_arguments)] +pub(crate) fn persist_activity_chunks_with_turn_state( + store: &dyn RecordStore, + source: &str, + source_session_id: Option<&str>, + session_id: &str, + actor_id: Option<&str>, + workspace_path: &str, + precision: AttributionPrecision, + chunks: &[ActivityChunk], + current_turn_id: &mut Option, ) -> Result { let mut persisted = 0; - let mut current_turn_id: Option<&str> = None; for chunk in chunks { if chunk.function == FUNCTION_USER_MESSAGE { - current_turn_id = Some(&chunk.chunk_id); + *current_turn_id = Some(chunk.chunk_id.clone()); continue; } let outcome = interaction_outcome_from_activity_chunk(chunk); @@ -148,7 +179,122 @@ pub(crate) fn persist_activity_chunks( source_session_id, session_id, Some(&source_event_id), - current_turn_id, + current_turn_id.as_deref(), + actor_id, + workspace_path, + &interaction.file_path, + interaction.action, + outcome, + &chunk.created_at, + ResourceInteractionCaptureMethod::Reconciled, + precision, + )?; + persisted += 1; + } + } + Ok(persisted) +} + +/// Stream resource metadata out of the session event cache without ever +/// building a session-sized `Vec` or `Vec`. +/// +/// The projection classifier runs before either JSON column is copied from +/// SQLite. Known assistant/thinking/REPL rows therefore cost only their small +/// discriminator fields; Grep reads its compact args but never its potentially +/// huge result. Unknown tools deliberately keep the conservative old path. +#[allow(clippy::too_many_arguments)] +pub(crate) fn persist_cached_event_interactions_streaming( + conn: &Connection, + store: &dyn RecordStore, + source: &str, + source_session_id: Option<&str>, + session_id: &str, + actor_id: Option<&str>, + workspace_path: &str, + precision: AttributionPrecision, + mut map_path: impl FnMut(&str) -> Option, +) -> Result { + let mut statement = conn + .prepare_cached( + "SELECT id, session_id, event_type, function_name, thread_id, + created_at, history_sequence, args_json, result_json + FROM events + WHERE session_id = ?1 + ORDER BY COALESCE(history_sequence, rowid) ASC, created_at ASC, id ASC", + ) + .map_err(|err| format!("Prepare cached event projection stream failed: {err}"))?; + let mut rows = statement + .query([session_id]) + .map_err(|err| format!("Query cached event projection stream failed: {err}"))?; + let mut persisted = 0; + let mut current_turn_id: Option = None; + while let Some(row) = rows + .next() + .map_err(|err| format!("Read cached event projection row failed: {err}"))? + { + let id = row + .get::<_, String>(0) + .map_err(|err| format!("Read cached event id failed: {err}"))?; + let function_name = row + .get::<_, Option>(3) + .map_err(|err| format!("Read cached event function failed: {err}"))?; + if function_name.as_deref() == Some(FUNCTION_USER_MESSAGE) { + current_turn_id = Some(id); + continue; + } + let requirements = metadata_projection_requirements(function_name.as_deref()); + if !requirements.projects_resource_interactions() { + continue; + } + + let event = CachedEvent { + id, + session_id: row + .get(1) + .map_err(|err| format!("Read cached event session id failed: {err}"))?, + event_type: row + .get(2) + .map_err(|err| format!("Read cached event type failed: {err}"))?, + function_name, + thread_id: row + .get(4) + .map_err(|err| format!("Read cached event thread id failed: {err}"))?, + args_json: if requirements.needs_args_json() { + row.get(7) + .map_err(|err| format!("Read cached event args failed: {err}"))? + } else { + "null".to_string() + }, + result_json: if requirements.needs_result_json() { + row.get(8) + .map_err(|err| format!("Read cached event result failed: {err}"))? + } else { + "null".to_string() + }, + content: String::new(), + created_at: row + .get(5) + .map_err(|err| format!("Read cached event timestamp failed: {err}"))?, + meta_json: None, + history_sequence: row + .get(6) + .map_err(|err| format!("Read cached event sequence failed: {err}"))?, + }; + let chunk = cached_event_to_activity_chunk(&event); + let outcome = interaction_outcome_from_activity_chunk(&chunk); + for mut interaction in file_interactions_from_activity_chunk(&chunk) { + let Some(mapped_path) = map_path(&interaction.file_path) else { + continue; + }; + interaction.file_path = mapped_path; + let source_event_id = activity_chunk_source_event_id(&chunk, &interaction); + persist_file_interaction( + store, + source, + source_session_id, + session_id, + Some(&source_event_id), + current_turn_id.as_deref(), actor_id, workspace_path, &interaction.file_path, @@ -270,3 +416,87 @@ pub(super) fn resource_interaction_id( capture_method.as_str(), ]) } + +#[cfg(test)] +mod tests { + use super::*; + use orgtrack_core::store::sqlite::SqliteRecordStore; + use rusqlite::{params, Connection}; + + const INSERT_EVENT: &str = "INSERT INTO events ( + id, session_id, event_type, function_name, thread_id, + args_json, result_json, content, created_at, meta_json, history_sequence + ) VALUES (?1, 'snapshot', 'tool_call', ?2, NULL, ?3, ?4, '', ?5, NULL, ?6)"; + + #[test] + fn cached_projection_stream_skips_unneeded_blob_payloads() { + let conn = Connection::open_in_memory().expect("open DB"); + session_persistence::init_session_tables(&conn).expect("session tables"); + SqliteRecordStore::init_tables(&conn).expect("orgtrack tables"); + + conn.execute( + INSERT_EVENT, + params![ + "user-1", + FUNCTION_USER_MESSAGE, + "null", + "null", + "2026-07-22T00:00:00Z", + 0_i64 + ], + ) + .expect("user row"); + conn.execute( + INSERT_EVENT, + params![ + "assistant-1", + "assistant", + vec![0xff_u8; 1024 * 1024], + vec![0xfe_u8; 1024 * 1024], + "2026-07-22T00:00:01Z", + 1_i64 + ], + ) + .expect("assistant blob row"); + conn.execute( + INSERT_EVENT, + params![ + "grep-1", + "Grep", + r#"{"path":"src"}"#, + vec![0xfd_u8; 1024 * 1024], + "2026-07-22T00:00:02Z", + 2_i64 + ], + ) + .expect("grep blob result row"); + conn.execute( + INSERT_EVENT, + params![ + "edit-1", + "write_file", + r#"{"file_path":"src/main.rs","content":"ok"}"#, + r#"{"success":true}"#, + "2026-07-22T00:00:03Z", + 3_i64 + ], + ) + .expect("edit row"); + + let store = SqliteRecordStore::new(&conn); + let persisted = persist_cached_event_interactions_streaming( + &conn, + &store, + "test-source", + Some("source-session"), + "snapshot", + Some("actor"), + "/repo", + AttributionPrecision::Exact, + |path| Some(path.to_string()), + ) + .expect("stream projection"); + + assert_eq!(persisted, 1); + } +} diff --git a/src-tauri/tests/issue_443_replay_export_perf.rs b/src-tauri/tests/issue_443_replay_export_perf.rs new file mode 100644 index 000000000..367cedb94 --- /dev/null +++ b/src-tauri/tests/issue_443_replay_export_perf.rs @@ -0,0 +1,225 @@ +//! Ignored real-path acceptance harness for issue #443 streamed exports. +//! +//! The regular replay unit suite proves individual range limits. This test +//! exercises the public Tauri command against a production-shaped sessions DB +//! and a deterministic source whose exported JSON is at least 300 MiB. Keep it +//! ignored in normal CI: it intentionally performs several hundred MiB of +//! disk I/O and samples process peak RSS. + +#![cfg(unix)] + +use std::fs::{File, OpenOptions}; +use std::io::{BufReader, BufWriter, Read, Write}; +use std::path::Path; + +use app_lib::agent_sessions::event_pipeline::commands::external_replay::{ + stream_replay_export_to_path, ReplayExportFormat, +}; +use orgtrack_core::sources::imported_history::replay::{ + open_window, scan_window_after, scan_window_after_generation, ImportedHistorySourceId, + ReplayLimits, +}; +use orgtrack_core::store::sqlite::SqliteRecordStore; +use sha2::{Digest, Sha256}; + +const MIB: usize = 1024 * 1024; +const SOURCE_BYTES: u64 = 160 * MIB as u64; +const MIN_EXPORT_BYTES: u64 = 300 * MIB as u64; +const HASH_BUFFER_BYTES: usize = MIB; +const MAX_EXPORT_RSS_GROWTH_BYTES: usize = 64 * MIB; + +fn assistant_line(turn: u64, padding: &str) -> Vec { + let body = format!("export-turn-{turn:08}:{padding}"); + let mut line = serde_json::json!({ + "timestamp": "2026-07-23T00:00:01Z", + "type": "event_msg", + "payload": { "type": "agent_message", "message": body }, + }) + .to_string() + .into_bytes(); + line.push(b'\n'); + line +} + +fn write_source(path: &Path) { + let file = OpenOptions::new() + .create_new(true) + .write(true) + .open(path) + .expect("create deterministic export source"); + let mut writer = BufWriter::with_capacity(MIB, file); + let padding = "e".repeat(32 * 1024); + let mut written = 0_u64; + let mut turn = 0_u64; + while written < SOURCE_BYTES { + // Keep the fixture production-shaped: a user row starts each turn, + // followed by one complete assistant message. Thousands of adjacent + // assistant rows in one synthetic turn are streaming snapshots and + // are intentionally consolidated by normal ingestion. + let user = serde_json::json!({ + "timestamp": "2026-07-23T00:00:00Z", + "type": "event_msg", + "payload": { + "type": "user_message", + "message": format!("next export turn {turn:08}"), + }, + }) + .to_string(); + let line = assistant_line(turn, &padding); + writer.write_all(user.as_bytes()).expect("write user row"); + writer.write_all(b"\n").expect("finish user row"); + writer.write_all(&line).expect("write assistant row"); + written = written.saturating_add((user.len() + 1 + line.len()) as u64); + turn = turn.saturating_add(1); + } + writer.flush().expect("flush deterministic export source"); +} + +fn peak_rss_bytes() -> usize { + let mut usage = std::mem::MaybeUninit::::zeroed(); + let result = unsafe { libc::getrusage(libc::RUSAGE_SELF, usage.as_mut_ptr()) }; + assert_eq!(result, 0, "getrusage failed"); + let peak = unsafe { usage.assume_init() }.ru_maxrss as usize; + #[cfg(target_os = "macos")] + { + peak + } + #[cfg(not(target_os = "macos"))] + { + peak.saturating_mul(1024) + } +} + +fn file_sha256(path: &Path) -> (u64, String) { + let file = File::open(path).expect("open streamed export"); + let mut reader = BufReader::with_capacity(HASH_BUFFER_BYTES, file); + let mut buffer = vec![0_u8; HASH_BUFFER_BYTES]; + let mut bytes = 0_u64; + let mut digest = Sha256::new(); + loop { + let read = reader.read(&mut buffer).expect("hash streamed export"); + if read == 0 { + break; + } + bytes = bytes.saturating_add(read as u64); + digest.update(&buffer[..read]); + } + (bytes, format!("{:x}", digest.finalize())) +} + +#[tokio::test(flavor = "current_thread")] +#[ignore = "#443 serial 300 MiB streamed-export/RSS acceptance"] +async fn production_export_streams_three_hundred_mib_with_exact_hash_and_bounded_rss() { + let sandbox = test_helpers::test_env::sandbox(); + let source_path = sandbox.path().join("codex-export.jsonl"); + let destination = sandbox.path().join("codex-export.json"); + write_source(&source_path); + + let mut conn = database::db::get_connection().expect("open sandbox sessions DB"); + SqliteRecordStore::init_tables(&conn).expect("initialize orgtrack tables"); + SqliteRecordStore::init_source_cache_tables(&conn).expect("initialize imported replay cache"); + conn.execute( + "INSERT INTO imported_history_session_cache ( + source, source_session_id, session_id, source_path + ) VALUES ('codex_app', 'issue-443-export', 'codexapp-issue-443-export', ?1)", + [source_path.to_string_lossy().as_ref()], + ) + .expect("register deterministic Codex source"); + let opened = open_window( + &mut conn, + ImportedHistorySourceId::CodexApp, + "codexapp-issue-443-export", + ReplayLimits::default(), + ) + .expect("pre-index deterministic export source"); + let expected_event_count = opened.total_event_count; + let source_bytes = std::fs::metadata(&source_path) + .expect("stat deterministic export source") + .len(); + let (indexed_events, indexed_max_sequence): (u64, i64) = conn + .query_row( + "SELECT COUNT(*), COALESCE(MAX(sequence), -1) + FROM imported_replay_events + WHERE source='codex_app' AND source_session_id='issue-443-export'", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("inspect deterministic replay index"); + eprintln!( + "#443 export setup: source={:.1} MiB, parsed={:.1} MiB/{} rows, indexed={} events through sequence {}", + source_bytes as f64 / MIB as f64, + opened.stats.parsed_bytes as f64 / MIB as f64, + opened.stats.parsed_rows, + indexed_events, + indexed_max_sequence, + ); + assert_eq!(indexed_events, expected_event_count); + let scan_limits = ReplayLimits { + max_turns: 10, + max_events: 200, + max_ipc_bytes: 256 * 1024, + }; + let first_scan = scan_window_after( + &mut conn, + ImportedHistorySourceId::CodexApp, + "codexapp-issue-443-export", + -1, + scan_limits, + ) + .expect("probe first production export scan"); + eprintln!( + "#443 export scan 0: chunks={}, through={}, has_more={}", + first_scan.chunks.len(), + first_scan.cursor.through_sequence, + first_scan.has_more, + ); + let second_scan = scan_window_after_generation( + &mut conn, + ImportedHistorySourceId::CodexApp, + "codexapp-issue-443-export", + &first_scan.cursor.generation, + first_scan.cursor.revision, + first_scan.cursor.through_sequence, + scan_limits, + ) + .expect("probe second production export scan"); + eprintln!( + "#443 export scan 1: chunks={}, through={}, has_more={}", + second_scan.chunks.len(), + second_scan.cursor.through_sequence, + second_scan.has_more, + ); + drop(conn); + + let rss_before_export = peak_rss_bytes(); + let result = stream_replay_export_to_path( + "codex_app", + "codexapp-issue-443-export", + &destination, + ReplayExportFormat::Json, + None, + ) + .expect("run production streamed export command"); + let rss_after_export = peak_rss_bytes(); + let export_rss_growth = rss_after_export.saturating_sub(rss_before_export); + + let (bytes, hash) = file_sha256(&destination); + eprintln!( + "#443 export: bytes={:.1} MiB, events={}, sha256={}, export-only peak RSS growth={:.1} MiB", + bytes as f64 / MIB as f64, + result.event_count, + hash, + export_rss_growth as f64 / MIB as f64, + ); + assert!( + bytes >= MIN_EXPORT_BYTES, + "acceptance export must be at least 300 MiB, got {bytes} bytes" + ); + assert_eq!(result.bytes_written, bytes); + assert_eq!(result.sha256, hash); + assert_eq!(result.event_count, expected_event_count); + assert!( + export_rss_growth <= MAX_EXPORT_RSS_GROWTH_BYTES, + "300 MiB streamed export grew peak RSS by {export_rss_growth} bytes" + ); +} diff --git a/src/api/tauri/collaborationSnapshotIngest.ts b/src/api/tauri/collaborationSnapshotIngest.ts new file mode 100644 index 000000000..e3b035716 --- /dev/null +++ b/src/api/tauri/collaborationSnapshotIngest.ts @@ -0,0 +1,61 @@ +import { rpc, schemas } from "./rpc"; + +export type CollaborationSnapshotCursor = + schemas.collaborationSnapshotIngest.CollaborationSnapshotCursor; +export type CollaborationSnapshotIngestBeginRequest = + schemas.collaborationSnapshotIngest.CollaborationSnapshotIngestBeginRequest; +export type CollaborationSnapshotIngestGetCursorRequest = + schemas.collaborationSnapshotIngest.CollaborationSnapshotIngestGetCursorRequest; +export type CollaborationSnapshotIngestPageRequest = + schemas.collaborationSnapshotIngest.CollaborationSnapshotIngestPageRequest; +export type CollaborationSnapshotIngestProgress = + schemas.collaborationSnapshotIngest.CollaborationSnapshotIngestProgress; +export type CollaborationSnapshotIngestCommitResult = + schemas.collaborationSnapshotIngest.CollaborationSnapshotIngestCommitResult; + +/** Start one single-use, crash-safe staged snapshot publication. */ +export function collaborationSnapshotIngestBegin( + request: CollaborationSnapshotIngestBeginRequest +): Promise<{ token: string }> { + return rpc.collaborationSnapshotIngest.begin({ request }); +} + +/** Return the trusted local cursor only when the imported snapshot is intact. */ +export function collaborationSnapshotIngestGetCursor( + localSessionId: CollaborationSnapshotIngestGetCursorRequest["localSessionId"] +): Promise { + return rpc.collaborationSnapshotIngest.getCursor({ + request: { localSessionId }, + }); +} + +/** True only for a native fork with an intact collaboration snapshot index. */ +export function collaborationSnapshotSecondaryProbe( + sessionId: string +): Promise { + return rpc.collaborationSnapshotIngest.probeSecondary({ + request: { sessionId }, + }); +} + +/** + * Persist one bounded page of opaque Cloud wires in Rust. The compressed + * payloads cross IPC once; decoded SessionEvent arrays never return to JS. + */ +export function collaborationSnapshotIngestApplyWirePage( + request: CollaborationSnapshotIngestPageRequest +): Promise { + return rpc.collaborationSnapshotIngest.applyWirePage({ request }); +} + +/** Atomically publish the verified staged snapshot into sessions.db. */ +export function collaborationSnapshotIngestCommit( + token: string +): Promise { + return rpc.collaborationSnapshotIngest.commit({ request: { token } }); +} + +/** Discard a staged snapshot without changing the currently visible copy. */ +export function collaborationSnapshotIngestAbort(token: string): Promise { + return rpc.collaborationSnapshotIngest.abort({ request: { token } }); +} diff --git a/src/api/tauri/externalHistory/__tests__/replay.secondary.test.ts b/src/api/tauri/externalHistory/__tests__/replay.secondary.test.ts new file mode 100644 index 000000000..27731771a --- /dev/null +++ b/src/api/tauri/externalHistory/__tests__/replay.secondary.test.ts @@ -0,0 +1,59 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import { + resolveExternalReplayTarget, + resolveSecondaryReplayTarget, +} from "../replay"; + +const mocks = vi.hoisted(() => ({ + probe: vi.fn(), +})); + +vi.mock("@src/api/tauri/collaborationSnapshotIngest", () => ({ + collaborationSnapshotSecondaryProbe: mocks.probe, +})); + +describe("secondary replay target resolution", () => { + beforeEach(() => { + vi.resetAllMocks(); + mocks.probe.mockResolvedValue(false); + }); + + it("keeps native Agent sessions out of the primary replay registry", () => { + expect(resolveExternalReplayTarget("agentsession-cloud-fork")).toBeNull(); + expect(resolveExternalReplayTarget("sdeagent-native")).toBeNull(); + }); + + it("admits an agentsession fork only after the Rust snapshot probe succeeds", async () => { + mocks.probe.mockResolvedValue(true); + + await expect( + resolveSecondaryReplayTarget("agentsession-cloud-fork") + ).resolves.toEqual({ + sourceId: "collaboration_snapshot", + sessionId: "agentsession-cloud-fork", + }); + expect(mocks.probe).toHaveBeenCalledOnce(); + expect(mocks.probe).toHaveBeenCalledWith("agentsession-cloud-fork"); + }); + + it("falls back to the ordinary native path when the snapshot proof is absent", async () => { + await expect( + resolveSecondaryReplayTarget("agentsession-native") + ).resolves.toBeNull(); + expect(mocks.probe).toHaveBeenCalledWith("agentsession-native"); + }); + + it("never probes SDE or already registered external sessions", async () => { + await expect( + resolveSecondaryReplayTarget("sdeagent-native") + ).resolves.toBeNull(); + await expect( + resolveSecondaryReplayTarget("codexapp-session-1") + ).resolves.toEqual({ + sourceId: "codex_app", + sessionId: "codexapp-session-1", + }); + expect(mocks.probe).not.toHaveBeenCalled(); + }); +}); diff --git a/src/api/tauri/externalHistory/__tests__/replayRequestGuard.test.ts b/src/api/tauri/externalHistory/__tests__/replayRequestGuard.test.ts new file mode 100644 index 000000000..da1c73da5 --- /dev/null +++ b/src/api/tauri/externalHistory/__tests__/replayRequestGuard.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from "vitest"; + +import { ReplayRequestGuard } from "../replayRequestGuard"; + +describe("ReplayRequestGuard", () => { + it("rejects an older request after a newer request starts", () => { + const guard = new ReplayRequestGuard(); + const older = guard.begin(); + const newer = guard.begin(); + + expect(guard.isCurrent(older)).toBe(false); + expect(guard.isCurrent(newer)).toBe(true); + }); + + it("rejects an in-flight request when the component episode changes", () => { + const guard = new ReplayRequestGuard(); + const inFlight = guard.begin(); + + guard.invalidate(); + + expect(guard.isCurrent(inFlight)).toBe(false); + }); +}); diff --git a/src/api/tauri/externalHistory/cursorIde/index.ts b/src/api/tauri/externalHistory/cursorIde/index.ts deleted file mode 100644 index e13d077b0..000000000 --- a/src/api/tauri/externalHistory/cursorIde/index.ts +++ /dev/null @@ -1,96 +0,0 @@ -/** - * Cursor IDE history — Tauri API wrappers. - * - * These commands surface Cursor IDE chat history (read from Cursor's - * `~/.../state.vscdb`) as read-only sessions in our session list. - * Frontend never sees the bare composer UUID — every session id is - * prefixed with `cursoride-` (see `CURSOR_IDE_SESSION_PREFIX`). - */ -import { invoke } from "@tauri-apps/api/core"; - -import type { ActivityChunk } from "@src/types/session/session"; - -export interface CursorIdeTurnSummary { - turnId: string; - nextTurnId: string | null; - turnIndex: number; - startedAt: string; - endedAt: string | null; - durationMs: number | null; - userPreview: string; - eventCount: number; - bodyEventCount: number; -} - -export interface CursorIdeInitialWindow { - chunks: ActivityChunk[]; - turns: CursorIdeTurnSummary[]; - totalBubbleCount: number; - userBubbleCount: number; - recentBubbleCount: number; - recentStartCursor: string | null; - recentEndCursor: string | null; - hasUnloadedMiddle: boolean; -} - -export interface CursorIdeFullRefresh { - chunks: ActivityChunk[]; - turns: CursorIdeTurnSummary[]; -} - -export interface CursorIdeTurnWindow { - chunks: ActivityChunk[]; - userBubbleId: string; - nextUserBubbleId: string | null; - loadedBubbleCount: number; -} - -/** - * Read all bubbles for one Cursor IDE composer, returned as `ActivityChunk[]` - * ready to feed through the standard event pipeline (`processChunksRust` → - * `eventStoreProxy` → `ChatHistory`). - */ -export async function cursorIdeChunks( - sessionId: string -): Promise { - return invoke("cursor_ide_chunks", { sessionId }); -} - -/** - * A composer's last-updated timestamp from Cursor's `state.vscdb` — a cheap - * freshness signal for reloading an open read-only Cursor session when it - * changes. `null` when Cursor isn't installed or the composer is unknown. - */ -export async function cursorIdeComposerLastUpdatedAt( - composerId: string -): Promise { - return invoke("cursor_ide_composer_last_updated_at", { - composerId, - }); -} - -export async function cursorIdeInitialWindow(args: { - sessionId: string; - recentLimit?: number; -}): Promise { - return invoke("cursor_ide_initial_window", { - sessionId: args.sessionId, - recentLimit: args.recentLimit, - }); -} - -export async function cursorIdeFullRefresh( - sessionId: string -): Promise { - return invoke("cursor_ide_full_refresh", { sessionId }); -} - -export async function cursorIdeTurnWindow(args: { - sessionId: string; - userBubbleId: string; -}): Promise { - return invoke("cursor_ide_turn_window", { - sessionId: args.sessionId, - userBubbleId: args.userBubbleId, - }); -} diff --git a/src/api/tauri/externalHistory/externalReplayTurnSummary.ts b/src/api/tauri/externalHistory/externalReplayTurnSummary.ts new file mode 100644 index 000000000..a98ee0712 --- /dev/null +++ b/src/api/tauri/externalHistory/externalReplayTurnSummary.ts @@ -0,0 +1,19 @@ +/** Compact turn shape retained by the bounded replay presentation. */ +export interface ExternalReplayTurnSummary { + /** Backend locator used by bounded replay read requests. */ + turnId: string; + /** + * User SessionEvent id that owns the rendered chat group for this turn. + * Provider turn locators and normalized event ids are separate identities + * for sources such as Codex, so presentation code must not compare them. + */ + renderedUserEventId: string | null; + nextTurnId: string | null; + turnIndex: number; + startedAt: string; + endedAt: string | null; + durationMs: number | null; + userPreview: string; + eventCount: number; + bodyEventCount: number; +} diff --git a/src/api/tauri/externalHistory/imported/__tests__/sources.test.ts b/src/api/tauri/externalHistory/imported/__tests__/sources.test.ts index b80e43c1c..6510a8062 100644 --- a/src/api/tauri/externalHistory/imported/__tests__/sources.test.ts +++ b/src/api/tauri/externalHistory/imported/__tests__/sources.test.ts @@ -1,63 +1,93 @@ -import { describe, expect, it, vi } from "vitest"; +import { existsSync, readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { describe, expect, it } from "vitest"; import { IMPORTED_HISTORY_SOURCES, getImportedHistorySourceByListCategory, getImportedHistorySourceBySessionId, isImportedHistoryListCategory, + resolveExternalReplayTarget, } from "@src/api/tauri/externalHistory"; -const cursorLoaders = vi.hoisted(() => ({ - preview: vi.fn(), - full: vi.fn(), -})); -const codexLoaders = vi.hoisted(() => ({ - preview: vi.fn(), - full: vi.fn(), -})); +function loadRustReplayRegistry(): Map { + const rustRoot = resolve( + process.cwd(), + "src-tauri/crates/orgtrack-core/src/sources" + ); + const metadata = readFileSync( + resolve(rustRoot, "imported_history/metadata.rs"), + "utf8" + ); + const sourceConstants = new Map(); + for (const match of metadata.matchAll( + /pub const (SOURCE_[A-Z0-9_]+):\s*&str\s*=\s*"([^"]+)"\s*;/g + )) { + sourceConstants.set(match[1], match[2]); + } -vi.mock("../../cursorIde", () => ({ - cursorIdeInitialWindow: cursorLoaders.preview, - cursorIdeChunks: cursorLoaders.full, -})); + const registry = readFileSync( + resolve(rustRoot, "imported_history/replay/registry.rs"), + "utf8" + ); + const descriptors = new Map(); + for (const match of registry.matchAll( + /source_id:\s*(SOURCE_[A-Z0-9_]+),\s*session_prefix:\s*(sources::[A-Za-z0-9_:]+),/g + )) { + const sourceId = sourceConstants.get(match[1]); + if (!sourceId) throw new Error(`Unknown Rust source constant ${match[1]}`); -vi.mock("../../sources/codexApp", () => ({ - codexAppInitialWindow: codexLoaders.preview, - codexAppChunks: codexLoaders.full, -})); + const pathParts = match[2].split("::").slice(1); + const constantName = pathParts.pop(); + if (!constantName) throw new Error(`Invalid Rust prefix path ${match[2]}`); + const base = resolve(rustRoot, ...pathParts); + const candidateFiles = [`${base}.rs`, resolve(base, "mod.rs")]; + const sourceFile = candidateFiles.find(existsSync); + if (!sourceFile) { + throw new Error(`Cannot resolve Rust prefix module ${match[2]}`); + } + const moduleSource = readFileSync(sourceFile, "utf8"); + const constantMatch = moduleSource.match( + new RegExp( + `(?:pub\\s+)?const\\s+${constantName}:\\s*&str\\s*=\\s*"([^"]+)"\\s*;` + ) + ); + if (!constantMatch) { + throw new Error(`Cannot resolve Rust prefix constant ${match[2]}`); + } + descriptors.set(sourceId, constantMatch[1]); + } + return descriptors; +} describe("imported history source registry", () => { - it("keeps Cursor's local preview window separate from cloud's full transcript", async () => { - cursorLoaders.preview.mockResolvedValue({ chunks: [{ id: "preview" }] }); - cursorLoaders.full.mockResolvedValue([{ id: "full" }]); - const cursor = getImportedHistorySourceBySessionId("cursoride-session-1"); - - await expect( - cursor?.loadPreviewChunks("cursoride-session-1") - ).resolves.toEqual([{ id: "preview" }]); - await expect( - cursor?.loadFullTranscriptChunks("cursoride-session-1") - ).resolves.toEqual([{ id: "full" }]); - expect(cursorLoaders.preview).toHaveBeenCalledWith({ - sessionId: "cursoride-session-1", - recentLimit: 100, + it("routes ORGII collaboration snapshots without adding a sixteenth vendor source", () => { + expect( + resolveExternalReplayTarget("imported-session-0123456789abcdef") + ).toEqual({ + sourceId: "collaboration_snapshot", + sessionId: "imported-session-0123456789abcdef", }); - expect(cursorLoaders.full).toHaveBeenCalledWith("cursoride-session-1"); + expect(resolveExternalReplayTarget("sdeagent-native-1")).toBeNull(); + expect(resolveExternalReplayTarget("agentsession-cloud-fork-1")).toBeNull(); }); - it("keeps Codex's bounded preview separate from cloud's full transcript", async () => { - codexLoaders.preview.mockResolvedValue({ chunks: [{ id: "preview" }] }); - codexLoaders.full.mockResolvedValue([{ id: "full" }]); - const codex = getImportedHistorySourceBySessionId("codexapp-session-1"); + it("keeps the renderer metadata mirror exhaustive with the Rust authority", () => { + const rust = loadRustReplayRegistry(); + const renderer = new Map( + IMPORTED_HISTORY_SOURCES.map((source) => [source.sourceId, source.prefix]) + ); - await expect( - codex?.loadPreviewChunks("codexapp-session-1") - ).resolves.toEqual([{ id: "preview" }]); - await expect( - codex?.loadFullTranscriptChunks("codexapp-session-1") - ).resolves.toEqual([{ id: "full" }]); - expect(codexLoaders.preview).toHaveBeenCalledWith("codexapp-session-1"); - expect(codexLoaders.full).toHaveBeenCalledWith("codexapp-session-1"); + expect(rust.size).toBe(15); + expect([...renderer.entries()].sort()).toEqual([...rust.entries()].sort()); + }); + + it("does not expose a renderer-side full transcript fallback", () => { + const cursor = getImportedHistorySourceBySessionId("cursoride-session-1"); + expect(cursor).toBeDefined(); + expect(cursor).not.toHaveProperty("loadPreviewChunks"); + expect(cursor).not.toHaveProperty("loadFullTranscriptChunks"); + expect(cursor).not.toHaveProperty("statTranscript"); }); it("registers source-specific external history providers", () => { @@ -98,8 +128,8 @@ describe("imported history source registry", () => { "external_history:qoder_cli", ]); for (const source of IMPORTED_HISTORY_SOURCES) { - expect(source.loadPreviewChunks).toBeTypeOf("function"); - expect(source.loadFullTranscriptChunks).toBeTypeOf("function"); + expect(source).not.toHaveProperty("loadPreviewChunks"); + expect(source).not.toHaveProperty("loadFullTranscriptChunks"); } }); diff --git a/src/api/tauri/externalHistory/imported/descriptors.ts b/src/api/tauri/externalHistory/imported/descriptors.ts index 632dfd893..ff182f023 100644 --- a/src/api/tauri/externalHistory/imported/descriptors.ts +++ b/src/api/tauri/externalHistory/imported/descriptors.ts @@ -14,7 +14,6 @@ export interface ImportedHistorySourceDescriptor { groupLabel: string; listable: true; replayable: true; - supportsWindowedReplay: boolean; } export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySourceDescriptor[] = @@ -28,7 +27,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Cursor App", listable: true, replayable: true, - supportsWindowedReplay: true, }, { sourceId: "cursor_cli", @@ -39,7 +37,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Cursor CLI", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "codex_app", @@ -50,7 +47,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Codex App", listable: true, replayable: true, - supportsWindowedReplay: true, }, { sourceId: "claude_code", @@ -61,7 +57,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Claude App", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "opencode", @@ -72,7 +67,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "OpenCode", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "windsurf", @@ -83,7 +77,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Windsurf", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "workbuddy", @@ -94,7 +87,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "WorkBuddy", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "trae", @@ -105,7 +97,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Trae", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "cline", @@ -116,7 +107,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Cline", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "warp", @@ -127,7 +117,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Warp", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "zcode", @@ -138,7 +127,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "ZCode", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "qoder", @@ -149,7 +137,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Qoder", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "mimo_code", @@ -160,7 +147,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Mimo Code", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "omp", @@ -171,7 +157,6 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "OMP", listable: true, replayable: true, - supportsWindowedReplay: false, }, { sourceId: "qoder_cli", @@ -182,6 +167,5 @@ export const IMPORTED_HISTORY_SOURCE_DESCRIPTORS: readonly ImportedHistorySource groupLabel: "Qoder CLI", listable: true, replayable: true, - supportsWindowedReplay: false, }, ]; diff --git a/src/api/tauri/externalHistory/imported/index.ts b/src/api/tauri/externalHistory/imported/index.ts index f2244092c..393b454c4 100644 --- a/src/api/tauri/externalHistory/imported/index.ts +++ b/src/api/tauri/externalHistory/imported/index.ts @@ -1,33 +1,13 @@ -import type { ActivityChunk } from "@src/types/session/session"; +import { invoke } from "@tauri-apps/api/core"; import type { DispatchCategory } from "../../session"; -import { cursorIdeChunks, cursorIdeInitialWindow } from "../cursorIde"; import type { ExternalCliSourceProbe } from "../detection"; -import { - type ImportedTranscriptStat, - claudeCodeHistoryChunks, - claudeCodeHistoryStat, -} from "../sources/claudeCode"; -import { clineHistoryChunks } from "../sources/cline"; -import { codexAppChunks, codexAppInitialWindow } from "../sources/codexApp"; -import { cursorCliHistoryChunks } from "../sources/cursorCli"; -import { mimoCodeHistoryChunks } from "../sources/mimoCode"; -import { ompHistoryChunks } from "../sources/omp"; -import { opencodeHistoryChunks } from "../sources/opencode"; -import { qoderHistoryChunks } from "../sources/qoder"; -import { qoderCliHistoryChunks } from "../sources/qoderCli"; -import { traeHistoryChunks } from "../sources/trae"; -import { warpHistoryChunks } from "../sources/warp"; -import { windsurfHistoryChunks } from "../sources/windsurf"; -import { workBuddyHistoryChunks } from "../sources/workbuddy"; -import { zcodeHistoryChunks } from "../sources/zcode"; import { IMPORTED_HISTORY_SOURCE_DESCRIPTORS, type ImportedHistoryListCategory, type ImportedHistorySourceDescriptor, type ImportedHistorySourceId, } from "./descriptors"; -import { importedHistoryStat } from "./stat"; export type { ImportedHistoryListCategory, @@ -36,153 +16,38 @@ export type { }; export { IMPORTED_HISTORY_SOURCE_DESCRIPTORS }; -export type { ImportedTranscriptStat }; - export interface ImportedHistorySource extends ImportedHistorySourceDescriptor { dispatchCategory: Extract; - /** Fast/windowed transcript used when the user opens the local history. */ - loadPreviewChunks(sessionId: string): Promise; - /** Complete source transcript used for cloud replay/fork publication. */ - loadFullTranscriptChunks(sessionId: string): Promise; - /** - * Optional freshness probe (one backend `stat`). When present, the replay - * auto-refresh compares it against the previous tick and skips the full - * read/parse/merge pipeline while the transcript is unchanged. Sources - * without it simply refresh unconditionally. - */ - statTranscript?(sessionId: string): Promise; } -const CURSOR_IDE_INITIAL_RECENT_BUBBLE_LIMIT = 100; +export interface ImportedHistoryRecentPath { + path: string; + name?: string; + lastUsedAt: string; + sessionCount: number; +} -function descriptorFor( - sourceId: ImportedHistorySourceId -): ImportedHistorySourceDescriptor { - const descriptor = IMPORTED_HISTORY_SOURCE_DESCRIPTORS.find( - (entry) => entry.sourceId === sourceId - ); - if (!descriptor) { - throw new Error(`Missing imported history source descriptor: ${sourceId}`); - } - return descriptor; +/** Read grouped paths from the source's compact catalog, never its transcript. */ +export async function importedHistoryRecentPaths( + source: ImportedHistorySourceId, + options?: { limit?: number } +): Promise { + return invoke("external_history_recent_paths", { + source, + limit: options?.limit, + }); } -export const IMPORTED_HISTORY_SOURCES: readonly ImportedHistorySource[] = [ - { - ...descriptorFor("cursor_ide"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("cursor_ide", sessionId), - async loadPreviewChunks(sessionId) { - return ( - await cursorIdeInitialWindow({ - sessionId, - recentLimit: CURSOR_IDE_INITIAL_RECENT_BUBBLE_LIMIT, - }) - ).chunks; - }, - loadFullTranscriptChunks: cursorIdeChunks, - }, - { - ...descriptorFor("cursor_cli"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("cursor_cli", sessionId), - loadPreviewChunks: cursorCliHistoryChunks, - loadFullTranscriptChunks: cursorCliHistoryChunks, - }, - { - ...descriptorFor("codex_app"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("codex_app", sessionId), - async loadPreviewChunks(sessionId) { - return (await codexAppInitialWindow(sessionId)).chunks; - }, - loadFullTranscriptChunks: codexAppChunks, - }, - { - ...descriptorFor("claude_code"), - dispatchCategory: "external_history", - loadPreviewChunks: claudeCodeHistoryChunks, - loadFullTranscriptChunks: claudeCodeHistoryChunks, - statTranscript: claudeCodeHistoryStat, - }, - { - ...descriptorFor("opencode"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("opencode", sessionId), - loadPreviewChunks: opencodeHistoryChunks, - loadFullTranscriptChunks: opencodeHistoryChunks, - }, - { - ...descriptorFor("windsurf"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("windsurf", sessionId), - loadPreviewChunks: windsurfHistoryChunks, - loadFullTranscriptChunks: windsurfHistoryChunks, - }, - { - ...descriptorFor("workbuddy"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("workbuddy", sessionId), - loadPreviewChunks: workBuddyHistoryChunks, - loadFullTranscriptChunks: workBuddyHistoryChunks, - }, - { - ...descriptorFor("trae"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("trae", sessionId), - loadPreviewChunks: traeHistoryChunks, - loadFullTranscriptChunks: traeHistoryChunks, - }, - { - ...descriptorFor("cline"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("cline", sessionId), - loadPreviewChunks: clineHistoryChunks, - loadFullTranscriptChunks: clineHistoryChunks, - }, - { - ...descriptorFor("warp"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("warp", sessionId), - loadPreviewChunks: warpHistoryChunks, - loadFullTranscriptChunks: warpHistoryChunks, - }, - { - ...descriptorFor("zcode"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("zcode", sessionId), - loadPreviewChunks: zcodeHistoryChunks, - loadFullTranscriptChunks: zcodeHistoryChunks, - }, - { - ...descriptorFor("qoder"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("qoder", sessionId), - loadPreviewChunks: qoderHistoryChunks, - loadFullTranscriptChunks: qoderHistoryChunks, - }, - { - ...descriptorFor("mimo_code"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("mimo_code", sessionId), - loadPreviewChunks: mimoCodeHistoryChunks, - loadFullTranscriptChunks: mimoCodeHistoryChunks, - }, - { - ...descriptorFor("omp"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("omp", sessionId), - loadPreviewChunks: ompHistoryChunks, - loadFullTranscriptChunks: ompHistoryChunks, - }, - { - ...descriptorFor("qoder_cli"), - dispatchCategory: "external_history", - statTranscript: (sessionId) => importedHistoryStat("qoder_cli", sessionId), - loadPreviewChunks: qoderCliHistoryChunks, - loadFullTranscriptChunks: qoderCliHistoryChunks, - }, -]; +/** + * Metadata-only source registry. Transcript access is deliberately absent: + * every renderer consumer must use the source-neutral bounded replay API. + * This makes an accidental JS full-history fallback a type error. + */ +export const IMPORTED_HISTORY_SOURCES: readonly ImportedHistorySource[] = + IMPORTED_HISTORY_SOURCE_DESCRIPTORS.map((descriptor) => ({ + ...descriptor, + dispatchCategory: "external_history" as const, + })); export function getImportedHistorySourceBySessionId( sessionId: string | null | undefined diff --git a/src/api/tauri/externalHistory/imported/stat.ts b/src/api/tauri/externalHistory/imported/stat.ts deleted file mode 100644 index 651776f5c..000000000 --- a/src/api/tauri/externalHistory/imported/stat.ts +++ /dev/null @@ -1,20 +0,0 @@ -import { invoke } from "@tauri-apps/api/core"; - -import type { ImportedTranscriptStat } from "../sources/claudeCode"; - -/** - * Source-agnostic transcript freshness probe. Resolves the session's source - * file from the backend imported-history cache and stats it (including the - * SQLite `-wal` sibling for WAL-mode stores). `null` when the session is - * uncached or the file is missing — the auto-refresh then falls back to a - * full reload, which re-syncs the cache. - */ -export async function importedHistoryStat( - sourceId: string, - sessionId: string -): Promise { - return invoke("imported_history_stat", { - sourceId, - sessionId, - }); -} diff --git a/src/api/tauri/externalHistory/index.ts b/src/api/tauri/externalHistory/index.ts index b78bb6deb..1c1702af1 100644 --- a/src/api/tauri/externalHistory/index.ts +++ b/src/api/tauri/externalHistory/index.ts @@ -17,8 +17,9 @@ export { fetchExternalSourceStatsBatch, type ExternalSourceStats, } from "./sourceStats"; -export * from "./cursorIde"; +export * from "./externalReplayTurnSummary"; export * from "./imported"; +export * from "./replay"; export * from "./sources/claudeCode"; export * from "./sources/codexApp"; export * from "./sources/cursorCli"; diff --git a/src/api/tauri/externalHistory/replay.ts b/src/api/tauri/externalHistory/replay.ts new file mode 100644 index 000000000..f09fce535 --- /dev/null +++ b/src/api/tauri/externalHistory/replay.ts @@ -0,0 +1,311 @@ +import { collaborationSnapshotSecondaryProbe } from "@src/api/tauri/collaborationSnapshotIngest"; +import { rpc, schemas } from "@src/api/tauri/rpc"; +import { + getExternalHistorySourceId, + isCliSession, + isCollaborationSnapshotSession, +} from "@src/util/session/sessionDispatch"; + +export const EXTERNAL_REPLAY_INVALIDATED_EVENT = + "external-replay://invalidated" as const; + +export type ExternalReplaySourceId = + schemas.externalReplay.ExternalReplaySourceId; +export type ExternalReplayCursor = schemas.externalReplay.ExternalReplayCursor; +export type ExternalReplayWindow = schemas.externalReplay.ExternalReplayWindow; +export type ExternalReplayDelta = schemas.externalReplay.ExternalReplayDelta; +export type ExternalReplayHandoff = + schemas.externalReplay.ExternalReplayHandoff; +export type ExternalReplayPayloadRange = + schemas.externalReplay.ExternalReplayPayloadRange; +export type ExternalReplayExportFormat = + schemas.externalReplay.ExternalReplayExportFormat; +export type ExternalReplayExportResult = + schemas.externalReplay.ExternalReplayExportResult; +export type ExternalReplayOrgiiEnvelope = + schemas.externalReplay.ExternalReplayOrgiiEnvelope; +export type ExternalReplayInvalidation = + schemas.externalReplay.ExternalReplayInvalidation; +export type ExternalReplayLimits = Exclude< + schemas.externalReplay.ExternalReplayLimits, + undefined +>; +export type ExternalReplayCloudManifest = + schemas.externalReplay.ExternalReplayCloudManifest; +export type ExternalReplayCloudBatch = + schemas.externalReplay.ExternalReplayCloudBatch; + +export interface ExternalReplayTarget { + sourceId: ExternalReplaySourceId; + sessionId: string; +} + +const SNAPSHOT_BACKED_FORK_PREFIX = "agentsession-"; + +/** + * Resolve the public session id to the source-aware replay transport. + * + * Managed CLI sessions deliberately stay keyed by their ORGII id. Rust owns + * the native binding lookup (and the legacy chunks-mode SQL driver), so the + * frontend never guesses a vendor-native id from stale status data. + */ +export function resolveExternalReplayTarget( + sessionId: string +): ExternalReplayTarget | null { + if (isCollaborationSnapshotSession(sessionId)) { + return { sourceId: "collaboration_snapshot", sessionId }; + } + if (isCliSession(sessionId)) { + return { sourceId: "managed_cli", sessionId }; + } + const sourceId = getExternalHistorySourceId(sessionId); + return sourceId ? { sourceId, sessionId } : null; +} + +/** + * Resolve a replay target for read-only secondary consumers only. + * + * A normal native Agent remains absent from the primary replay registry. An + * `agentsession-*` is admitted here only after Rust proves that an intact + * collaboration snapshot index exists for its inherited prefix. + */ +export async function resolveSecondaryReplayTarget( + sessionId: string +): Promise { + const primary = resolveExternalReplayTarget(sessionId); + if (primary) return primary; + if ( + !sessionId.startsWith(SNAPSHOT_BACKED_FORK_PREFIX) || + sessionId.length <= SNAPSHOT_BACKED_FORK_PREFIX.length + ) { + return null; + } + return (await collaborationSnapshotSecondaryProbe(sessionId)) + ? { sourceId: "collaboration_snapshot", sessionId } + : null; +} + +function requireExternalReplayTarget(sessionId: string): ExternalReplayTarget { + const target = resolveExternalReplayTarget(sessionId); + if (!target) { + throw new Error(`Session does not support bounded replay: ${sessionId}`); + } + return target; +} + +export function externalReplayOpenWindow( + sessionId: string, + episodeId: number, + limits?: ExternalReplayLimits +): Promise { + return rpc.externalReplay.openWindow({ + ...requireExternalReplayTarget(sessionId), + episodeId, + limits, + }); +} + +export function externalReplayPollDelta( + sessionId: string, + episodeId: number, + cursor: ExternalReplayCursor, + limits?: ExternalReplayLimits +): Promise { + return rpc.externalReplay.pollDelta({ + ...requireExternalReplayTarget(sessionId), + episodeId, + cursor, + limits, + }); +} + +export function externalReplayReadWindow(options: { + sessionId: string; + episodeId: number; + beforeSequence?: number; + turnId?: string; + turnIndex?: number; + limits?: ExternalReplayLimits; +}): Promise { + const { sessionId, episodeId, beforeSequence, turnId, turnIndex, limits } = + options; + return rpc.externalReplay.readWindow({ + ...requireExternalReplayTarget(sessionId), + episodeId, + beforeSequence, + turnId, + turnIndex, + limits, + }); +} + +/** + * Pure bounded source query. Unlike foreground open/poll/read, this never + * creates a watcher, mutates EventStore, emits `es:changed`, or invalidates a + * visible replay episode. + */ +export function externalReplayQueryWindow(options: { + sessionId: string; + beforeSequence?: number; + turnId?: string; + turnIndex?: number; + limits?: ExternalReplayLimits; +}): Promise { + const { sessionId, beforeSequence, turnId, turnIndex, limits } = options; + return externalReplayQueryWindowForTarget({ + target: requireExternalReplayTarget(sessionId), + beforeSequence, + turnId, + turnIndex, + limits, + }); +} + +export function externalReplayQueryWindowForTarget(options: { + target: ExternalReplayTarget; + beforeSequence?: number; + turnId?: string; + turnIndex?: number; + limits?: ExternalReplayLimits; +}): Promise { + const { target, beforeSequence, turnId, turnIndex, limits } = options; + return rpc.externalReplay.queryWindow({ + ...target, + beforeSequence, + turnId, + turnIndex, + limits, + }); +} + +/** + * Load and apply one latest bounded window entirely in Rust. The episode id + * belongs to the independent prewarm lifecycle, not the foreground watcher. + */ +export function externalReplayPrewarmWindow( + sessionId: string, + episodeId: number, + limits?: ExternalReplayLimits +): Promise { + return rpc.externalReplay.prewarmWindow({ + ...requireExternalReplayTarget(sessionId), + episodeId, + limits, + }); +} + +/** + * Pure backend handoff fold for Fork. Rust pages the compact replay index and + * returns at most 80 prompt-ready strings; no SessionEvent[] crosses IPC. + */ +export function externalReplayHandoff(options: { + sessionId: string; + sourceName: string; +}): Promise { + const { sessionId, sourceName } = options; + return rpc.externalReplay.handoff({ + ...requireExternalReplayTarget(sessionId), + sourceName, + }); +} + +export function externalReplayRelease( + sessionId: string, + episodeId: number +): Promise { + return rpc.externalReplay.release({ + ...requireExternalReplayTarget(sessionId), + episodeId, + }); +} + +export function externalReplayReadPayloadRange(options: { + sessionId: string; + generation: string; + eventId: string; + fieldPath: string; + offset: number; + maxBytes?: number; +}): Promise { + const { sessionId, ...range } = options; + return externalReplayReadPayloadRangeForTarget({ + target: requireExternalReplayTarget(sessionId), + ...range, + }); +} + +export function externalReplayReadPayloadRangeForTarget(options: { + target: ExternalReplayTarget; + generation: string; + eventId: string; + fieldPath: string; + offset: number; + maxBytes?: number; +}): Promise { + const { target, ...range } = options; + return rpc.externalReplay.readPayloadRange({ + ...target, + ...range, + }); +} + +export function externalReplayStreamExport(options: { + sessionId: string; + suggestedFileName?: string; + format: ExternalReplayExportFormat; + orgiiEnvelope?: ExternalReplayOrgiiEnvelope; +}): Promise { + const { sessionId, ...exportOptions } = options; + return externalReplayStreamExportForTarget({ + target: requireExternalReplayTarget(sessionId), + ...exportOptions, + }); +} + +export function externalReplayStreamExportForTarget(options: { + target: ExternalReplayTarget; + suggestedFileName?: string; + format: ExternalReplayExportFormat; + orgiiEnvelope?: ExternalReplayOrgiiEnvelope; +}): Promise { + const { target, ...exportOptions } = options; + return rpc.externalReplay.streamExport({ + ...target, + ...exportOptions, + }); +} + +export function externalReplayCloudPrepare( + sessionId: string +): Promise { + return externalReplayCloudPrepareForTarget( + requireExternalReplayTarget(sessionId) + ); +} + +export function externalReplayCloudPrepareForTarget( + target: ExternalReplayTarget +): Promise { + return rpc.externalReplay.cloudPrepare(target); +} + +export function externalReplayCloudReadBatch(options: { + token: string; + startEventIndex: number; + endEventIndex: number; + startSegmentIndex?: number; + maxBytes?: number; +}): Promise { + return rpc.externalReplay.cloudReadBatch(options); +} + +export function externalReplayCloudPrefixHash(options: { + token: string; + eventCount: number; +}): Promise { + return rpc.externalReplay.cloudPrefixHash(options); +} + +export function externalReplayCloudRelease(token: string): Promise { + return rpc.externalReplay.cloudRelease({ token }); +} diff --git a/src/api/tauri/externalHistory/replayRequestGuard.ts b/src/api/tauri/externalHistory/replayRequestGuard.ts new file mode 100644 index 000000000..7bb6a8b76 --- /dev/null +++ b/src/api/tauri/externalHistory/replayRequestGuard.ts @@ -0,0 +1,22 @@ +/** + * Monotonic guard for replay payload requests whose underlying + * session/generation/event identity can change while an async IPC call is in + * flight. Only the newest request in the current component episode may + * publish state. + */ +export class ReplayRequestGuard { + private epoch = 0; + + begin(): number { + this.epoch += 1; + return this.epoch; + } + + invalidate(): void { + this.epoch += 1; + } + + isCurrent(requestEpoch: number): boolean { + return this.epoch === requestEpoch; + } +} diff --git a/src/api/tauri/externalHistory/sources/claudeCode/index.ts b/src/api/tauri/externalHistory/sources/claudeCode/index.ts index 12eb6ed4d..737b6b142 100644 --- a/src/api/tauri/externalHistory/sources/claudeCode/index.ts +++ b/src/api/tauri/externalHistory/sources/claudeCode/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface ClaudeCodeRecentPath { path: string; name?: string; @@ -16,27 +14,3 @@ export async function claudeCodeRecentPaths(args?: { limit: args?.limit, }); } - -export async function claudeCodeHistoryChunks( - sessionId: string -): Promise { - return invoke("claude_code_history_chunks", { sessionId }); -} - -export interface ImportedTranscriptStat { - mtimeMs: number; - sizeBytes: number; -} - -/** - * Cheap freshness probe (a single `stat` backend-side). Lets the replay - * auto-refresh skip the full read/parse/merge pipeline when the transcript - * file hasn't changed. `null` when the source file is missing. - */ -export async function claudeCodeHistoryStat( - sessionId: string -): Promise { - return invoke("claude_code_history_stat", { - sessionId, - }); -} diff --git a/src/api/tauri/externalHistory/sources/cline/index.ts b/src/api/tauri/externalHistory/sources/cline/index.ts index 67783a93a..8a1b8a10c 100644 --- a/src/api/tauri/externalHistory/sources/cline/index.ts +++ b/src/api/tauri/externalHistory/sources/cline/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface ClineRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function clineRecentPaths(args?: { limit: args?.limit, }); } - -export async function clineHistoryChunks( - sessionId: string -): Promise { - return invoke("cline_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/codexApp/index.ts b/src/api/tauri/externalHistory/sources/codexApp/index.ts index 0a84a3459..3f67fe96b 100644 --- a/src/api/tauri/externalHistory/sources/codexApp/index.ts +++ b/src/api/tauri/externalHistory/sources/codexApp/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface CodexAppRecentPath { path: string; name?: string; @@ -9,27 +7,6 @@ export interface CodexAppRecentPath { sessionCount: number; } -export interface CodexAppInitialWindow { - chunks: ActivityChunk[]; - /** Backend-only projection cache; intentionally omitted from the IPC wire. */ - turns?: Array<{ - turnId: string; - startSequence: number; - startedAt: string; - endedAt: string | null; - status: string; - userPreview: string; - eventCount: number; - bodyEventCount: number; - }>; -} - -export interface CodexAppTurnWindow { - chunks: ActivityChunk[]; - turnId: string; - loadedEventCount: number; -} - export async function codexAppRecentPaths(args?: { limit?: number; }): Promise { @@ -37,24 +14,3 @@ export async function codexAppRecentPaths(args?: { limit: args?.limit, }); } - -export async function codexAppChunks( - sessionId: string -): Promise { - return invoke("codex_app_chunks", { sessionId }); -} - -export async function codexAppInitialWindow( - sessionId: string -): Promise { - return invoke("codex_app_initial_window", { - sessionId, - }); -} - -export async function codexAppTurnWindow(args: { - sessionId: string; - turnId: string; -}): Promise { - return invoke("codex_app_turn_window", args); -} diff --git a/src/api/tauri/externalHistory/sources/cursorCli/index.ts b/src/api/tauri/externalHistory/sources/cursorCli/index.ts index 857f0daa2..8cad0e54a 100644 --- a/src/api/tauri/externalHistory/sources/cursorCli/index.ts +++ b/src/api/tauri/externalHistory/sources/cursorCli/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface CursorCliRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function cursorCliRecentPaths(args?: { limit: args?.limit, }); } - -export async function cursorCliHistoryChunks( - sessionId: string -): Promise { - return invoke("cursor_cli_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/mimoCode/index.ts b/src/api/tauri/externalHistory/sources/mimoCode/index.ts index c18113a4c..9afab2638 100644 --- a/src/api/tauri/externalHistory/sources/mimoCode/index.ts +++ b/src/api/tauri/externalHistory/sources/mimoCode/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface MimoCodeRecentPath { path: string; name?: string | null; @@ -16,9 +14,3 @@ export async function mimoCodeRecentPaths(args?: { limit: args?.limit, }); } - -export async function mimoCodeHistoryChunks( - sessionId: string -): Promise { - return invoke("mimo_code_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/omp/index.ts b/src/api/tauri/externalHistory/sources/omp/index.ts index d6862a5b6..36d24539e 100644 --- a/src/api/tauri/externalHistory/sources/omp/index.ts +++ b/src/api/tauri/externalHistory/sources/omp/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface OmpRecentPath { path: string; name?: string | null; @@ -16,9 +14,3 @@ export async function ompRecentPaths(args?: { limit: args?.limit, }); } - -export async function ompHistoryChunks( - sessionId: string -): Promise { - return invoke("omp_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/opencode/index.ts b/src/api/tauri/externalHistory/sources/opencode/index.ts index 33d2188f0..653fcef2d 100644 --- a/src/api/tauri/externalHistory/sources/opencode/index.ts +++ b/src/api/tauri/externalHistory/sources/opencode/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface OpenCodeRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function opencodeRecentPaths(args?: { limit: args?.limit, }); } - -export async function opencodeHistoryChunks( - sessionId: string -): Promise { - return invoke("opencode_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/qoder/index.ts b/src/api/tauri/externalHistory/sources/qoder/index.ts index fcdd9fd77..ca6f67983 100644 --- a/src/api/tauri/externalHistory/sources/qoder/index.ts +++ b/src/api/tauri/externalHistory/sources/qoder/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface QoderRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function qoderRecentPaths(args?: { limit: args?.limit, }); } - -export async function qoderHistoryChunks( - sessionId: string -): Promise { - return invoke("qoder_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/qoderCli/index.ts b/src/api/tauri/externalHistory/sources/qoderCli/index.ts index c148956d3..fa93d1386 100644 --- a/src/api/tauri/externalHistory/sources/qoderCli/index.ts +++ b/src/api/tauri/externalHistory/sources/qoderCli/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface QoderCliRecentPath { path: string; name?: string | null; @@ -16,9 +14,3 @@ export async function qoderCliRecentPaths(args?: { limit: args?.limit, }); } - -export async function qoderCliHistoryChunks( - sessionId: string -): Promise { - return invoke("qoder_cli_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/trae/index.ts b/src/api/tauri/externalHistory/sources/trae/index.ts index ef30bb761..d351cbd5a 100644 --- a/src/api/tauri/externalHistory/sources/trae/index.ts +++ b/src/api/tauri/externalHistory/sources/trae/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface TraeRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function traeRecentPaths(args?: { limit: args?.limit, }); } - -export async function traeHistoryChunks( - sessionId: string -): Promise { - return invoke("trae_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/warp/index.ts b/src/api/tauri/externalHistory/sources/warp/index.ts index c75e8f5fa..83c123914 100644 --- a/src/api/tauri/externalHistory/sources/warp/index.ts +++ b/src/api/tauri/externalHistory/sources/warp/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface WarpRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function warpRecentPaths(args?: { limit: args?.limit, }); } - -export async function warpHistoryChunks( - sessionId: string -): Promise { - return invoke("warp_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/windsurf/index.ts b/src/api/tauri/externalHistory/sources/windsurf/index.ts index cb20f5fb7..5f91d1fd2 100644 --- a/src/api/tauri/externalHistory/sources/windsurf/index.ts +++ b/src/api/tauri/externalHistory/sources/windsurf/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface WindsurfRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function windsurfRecentPaths(args?: { limit: args?.limit, }); } - -export async function windsurfHistoryChunks( - sessionId: string -): Promise { - return invoke("windsurf_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/workbuddy/index.ts b/src/api/tauri/externalHistory/sources/workbuddy/index.ts index 86d261b77..5c6b67e9c 100644 --- a/src/api/tauri/externalHistory/sources/workbuddy/index.ts +++ b/src/api/tauri/externalHistory/sources/workbuddy/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface WorkBuddyRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function workBuddyRecentPaths(args?: { limit: args?.limit, }); } - -export async function workBuddyHistoryChunks( - sessionId: string -): Promise { - return invoke("workbuddy_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/externalHistory/sources/zcode/index.ts b/src/api/tauri/externalHistory/sources/zcode/index.ts index 3bf47ce4b..e6f23905e 100644 --- a/src/api/tauri/externalHistory/sources/zcode/index.ts +++ b/src/api/tauri/externalHistory/sources/zcode/index.ts @@ -1,7 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { ActivityChunk } from "@src/types/session/session"; - export interface ZCodeRecentPath { path: string; name?: string; @@ -16,9 +14,3 @@ export async function zcodeRecentPaths(args?: { limit: args?.limit, }); } - -export async function zcodeHistoryChunks( - sessionId: string -): Promise { - return invoke("zcode_history_chunks", { sessionId }); -} diff --git a/src/api/tauri/rpc/__tests__/collaborationSnapshotIngestSchemas.test.ts b/src/api/tauri/rpc/__tests__/collaborationSnapshotIngestSchemas.test.ts new file mode 100644 index 000000000..3627a7913 --- /dev/null +++ b/src/api/tauri/rpc/__tests__/collaborationSnapshotIngestSchemas.test.ts @@ -0,0 +1,113 @@ +import { describe, expect, it } from "vitest"; + +import * as schemas from "../schemas/collaborationSnapshotIngest"; + +const HASH = "a".repeat(64); + +describe("collaboration snapshot ingest RPC contracts", () => { + it("exposes read-only cursors only for imported snapshots", () => { + expect( + schemas.CollaborationSnapshotIngestGetCursorInputSchema.safeParse({ + request: { localSessionId: "imported-session-cloud-copy" }, + }).success + ).toBe(true); + expect( + schemas.CollaborationSnapshotIngestGetCursorInputSchema.safeParse({ + request: { localSessionId: "agentsession-native-fork" }, + }).success + ).toBe(false); + expect( + schemas.CollaborationSnapshotCursorSchema.nullable().parse(null) + ).toBeNull(); + }); + + it("allows a full native fork target but rejects native incremental ingest", () => { + const fullFork = { + localSessionId: "agentsession-cloud-fork", + epoch: 1, + expectedCount: 1, + expectedFrozenSeq: 1, + tailHash: null, + replace: true, + }; + expect( + schemas.CollaborationSnapshotIngestBeginRequestSchema.safeParse(fullFork) + .success + ).toBe(true); + expect( + schemas.CollaborationSnapshotIngestBeginRequestSchema.safeParse({ + ...fullFork, + replace: false, + previous: { + epoch: 1, + frozenSeq: 0, + count: 0, + frozenCount: 0, + tailHash: null, + }, + }).success + ).toBe(false); + }); + + it("exposes the secondary capability probe only for native Agent sessions", () => { + expect( + schemas.CollaborationSnapshotSecondaryProbeInputSchema.safeParse({ + request: { sessionId: "agentsession-cloud-fork" }, + }).success + ).toBe(true); + expect( + schemas.CollaborationSnapshotSecondaryProbeInputSchema.safeParse({ + request: { sessionId: "sdeagent-native" }, + }).success + ).toBe(false); + expect( + schemas.CollaborationSnapshotSecondaryProbeInputSchema.safeParse({ + request: { sessionId: "imported-session-cloud-copy" }, + }).success + ).toBe(false); + }); + + it("accepts a bounded physical-wire page without SessionEvent arrays", () => { + const parsed = schemas.CollaborationSnapshotIngestPageRequestSchema.parse({ + token: "00000000-0000-4000-8000-000000000001", + epoch: 3, + frozenSeq: 1, + count: 1, + tailHash: null, + cursor: { direction: "forward", afterSeq: 0, throughSeq: 1 }, + nextCursor: null, + tailIncluded: false, + hasMore: false, + returnedWireBytes: 128, + segments: [ + { + seq: 1, + payloadGz: "opaque-compressed-base64", + eventCount: 1, + segmentHash: HASH, + }, + ], + }); + + expect(parsed.segments[0]).not.toHaveProperty("events"); + }); + + it("rejects unbounded pages and broken continuation symmetry", () => { + const oversized = + schemas.CollaborationSnapshotIngestPageRequestSchema.safeParse({ + token: "00000000-0000-4000-8000-000000000001", + epoch: 3, + frozenSeq: 0, + count: 0, + tailHash: null, + cursor: { direction: "backward" }, + nextCursor: null, + tailIncluded: false, + hasMore: true, + returnedWireBytes: 4 * 1024 * 1024 + 1, + segments: [], + }); + + expect(oversized.success).toBe(false); + }); +}); diff --git a/src/api/tauri/rpc/__tests__/externalReplaySchemas.test.ts b/src/api/tauri/rpc/__tests__/externalReplaySchemas.test.ts new file mode 100644 index 000000000..58bb7fcc5 --- /dev/null +++ b/src/api/tauri/rpc/__tests__/externalReplaySchemas.test.ts @@ -0,0 +1,202 @@ +import { describe, expect, it } from "vitest"; + +import { + ExternalReplayCloudBatchSchema, + ExternalReplayCloudReadBatchInput, + ExternalReplayCursorSchema, + ExternalReplayHandoffInput, + ExternalReplayHandoffSchema, + ExternalReplayOpenWindowInput, + ExternalReplayPrewarmWindowInput, + ExternalReplayQueryWindowInput, + ExternalReplayReadPayloadRangeInput, + ExternalReplayReadWindowInput, + ExternalReplayStreamExportInput, + ExternalReplayWindowSchema, +} from "../schemas/externalReplay"; + +describe("external replay wire schemas", () => { + it("accepts the bounded managed-CLI not-ready sentinel", () => { + expect( + ExternalReplayCursorSchema.parse({ + sourceId: "managed_cli", + sessionId: "cliagent-1", + generation: "pending", + revision: 0, + throughSequence: -1, + }).throughSequence + ).toBe(-1); + }); + + it("accepts the ORGII-owned collaboration snapshot source", () => { + expect( + ExternalReplayCursorSchema.parse({ + sourceId: "collaboration_snapshot", + sessionId: "imported-session-abc", + generation: "collaboration-v1-0", + revision: 3, + throughSequence: 2, + }).sourceId + ).toBe("collaboration_snapshot"); + }); + + it("enforces window and payload hard limits before IPC", () => { + expect(() => + ExternalReplayOpenWindowInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + episodeId: 1, + limits: { maxTurns: 11 }, + }) + ).toThrow(); + expect(() => + ExternalReplayReadPayloadRangeInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + generation: "g1", + eventId: "event-1", + fieldPath: "result.output", + offset: 0, + maxBytes: 256 * 1024 + 1, + }) + ).toThrow(); + expect(() => + ExternalReplayReadWindowInput.parse({ + sourceId: "cursor_ide", + sessionId: "cursoride-1", + episodeId: 1, + beforeSequence: 10, + turnIndex: 2, + }) + ).toThrow("Choose only one replay window locator"); + expect( + ExternalReplayQueryWindowInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + limits: { maxEvents: 1 }, + }) + ).not.toHaveProperty("episodeId"); + expect( + ExternalReplayPrewarmWindowInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + episodeId: 2, + limits: { maxTurns: 1, maxEvents: 200 }, + }).episodeId + ).toBe(2); + expect(() => + ExternalReplayPrewarmWindowInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + episodeId: 2, + limits: { maxIpcBytes: 4 * 1024 * 1024 + 1 }, + }) + ).toThrow(); + }); + + it("validates bounded windows containing normalized SessionEvents", () => { + const parsed = ExternalReplayWindowSchema.parse({ + cursor: { + sourceId: "codex_app", + sessionId: "codexapp-1", + generation: "g1", + revision: 1, + throughSequence: 0, + }, + events: [], + windowStartSequence: null, + turnHeaders: [], + totalEventCount: 0, + totalTurnCount: 0, + hasOlder: false, + watcherAvailable: false, + stats: { + parsedBytes: 0, + parsedRows: 0, + normalizedEvents: 0, + upsertedEvents: 0, + removedEvents: 0, + ipcBytes: 0, + notReady: false, + }, + }); + expect(parsed.cursor.generation).toBe("g1"); + }); + + it("keeps Fork handoff prompt-ready and bounded without SessionEvents", () => { + expect( + ExternalReplayHandoffInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + sourceName: "Codex App", + }) + ).not.toHaveProperty("episodeId"); + const handoff = ExternalReplayHandoffSchema.parse({ + items: ["User: continue", "Assistant: done"], + generation: "g1", + scannedBytes: 4096, + scannedEvents: 2, + }); + expect(handoff.items).toHaveLength(2); + expect(handoff).not.toHaveProperty("events"); + expect(() => + ExternalReplayHandoffSchema.parse({ + items: ["x".repeat(1201)], + generation: "g1", + scannedBytes: 0, + scannedEvents: 1, + }) + ).toThrow(); + }); + + it("caps cloud spool IPC and accepts the small compatible export envelope", () => { + expect(() => + ExternalReplayCloudReadBatchInput.parse({ + token: "spool-1", + startEventIndex: 0, + endEventIndex: 10, + maxBytes: 256 * 1024 + 1, + }) + ).toThrow(); + expect( + ExternalReplayCloudBatchSchema.parse({ + segments: [ + { + payloadGz: "H4sI", + eventCount: 1, + segmentHash: "hash", + wireBytes: 128, + }, + ], + startEventIndex: 0, + nextEventIndex: 1, + startSegmentIndex: 0, + nextSegmentIndex: 1, + eof: true, + serializedBytes: 128, + }).segments + ).toHaveLength(1); + expect( + ExternalReplayStreamExportInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + suggestedFileName: "../../session.orgii-session.json", + format: "orgii_session_json", + orgiiEnvelope: { + exportedAt: "2026-07-22T00:00:00Z", + session: { session_id: "codexapp-1" }, + originalCategory: "cli_agent", + specs: [], + }, + }).format + ).toBe("orgii_session_json"); + expect( + ExternalReplayStreamExportInput.parse({ + sourceId: "codex_app", + sessionId: "codexapp-1", + destinationPath: "/tmp/renderer-chosen.json", + format: "json", + }) + ).not.toHaveProperty("destinationPath"); + }); +}); diff --git a/src/api/tauri/rpc/__tests__/sessionAggregateSchemas.test.ts b/src/api/tauri/rpc/__tests__/sessionAggregateSchemas.test.ts index 78f11d2f5..b2ee7fc24 100644 --- a/src/api/tauri/rpc/__tests__/sessionAggregateSchemas.test.ts +++ b/src/api/tauri/rpc/__tests__/sessionAggregateSchemas.test.ts @@ -4,6 +4,7 @@ import { ExternalHistorySidebarBatchResponseSchema, ExternalHistorySidebarListInput, SessionAggregateRecordSchema, + SessionWorkspaceFacetResponseSchema, } from "../schemas/sessionAggregate"; describe("session aggregate category schemas", () => { @@ -33,7 +34,16 @@ describe("external history sidebar schemas", () => { { source: "codex_app", buckets: [ - { bucket: "today", startMs: 200, limit: 10, offset: 0 }, + { + bucket: "today", + startMs: 200, + limit: 10, + offset: 0, + before: { + updatedAtMs: 300, + sessionId: "raw-codex-session", + }, + }, { bucket: "yesterday", startMs: 100, @@ -76,6 +86,28 @@ describe("external history sidebar schemas", () => { ).toThrow(); }); + it("accepts one workspace scope and rejects conflicting workspace scopes", () => { + const request = { + source: "codex_app", + buckets: [{ bucket: "today" as const, limit: 10, offset: 0 }], + }; + expect( + ExternalHistorySidebarListInput.parse({ + requests: [{ ...request, repoPath: "/repo-a" }], + }).requests[0].repoPath + ).toBe("/repo-a"); + expect( + ExternalHistorySidebarListInput.parse({ + requests: [{ ...request, missingRepoPath: true }], + }).requests[0].missingRepoPath + ).toBe(true); + expect(() => + ExternalHistorySidebarListInput.parse({ + requests: [{ ...request, repoPath: "/repo-a", missingRepoPath: true }], + }) + ).toThrow(); + }); + it("validates the lightweight response shape", () => { const parsed = ExternalHistorySidebarBatchResponseSchema.parse({ sources: [ @@ -93,6 +125,10 @@ describe("external history sidebar schemas", () => { }, ], hasMore: false, + nextCursor: { + updatedAtMs: 100, + sessionId: "raw-codex-session", + }, }, ], }, @@ -105,5 +141,26 @@ describe("external history sidebar schemas", () => { createdAt: "2026-07-11T01:00:00Z", updatedAt: "2026-07-11T02:00:00Z", }); + expect(parsed.sources[0].buckets[0].nextCursor).toEqual({ + updatedAtMs: 100, + sessionId: "raw-codex-session", + }); + }); +}); + +describe("workspace facet schemas", () => { + it("accepts the null repoPath emitted for the No Workspace facet", () => { + const parsed = SessionWorkspaceFacetResponseSchema.parse({ + facets: [ + { + repoPath: null, + lastUpdatedAtMs: 1, + sessionCount: 2, + }, + ], + hasMore: false, + }); + + expect(parsed.facets[0].repoPath).toBeNull(); }); }); diff --git a/src/api/tauri/rpc/__tests__/sessionCoreSchemas.test.ts b/src/api/tauri/rpc/__tests__/sessionCoreSchemas.test.ts index 33561f1f6..e39b5e027 100644 --- a/src/api/tauri/rpc/__tests__/sessionCoreSchemas.test.ts +++ b/src/api/tauri/rpc/__tests__/sessionCoreSchemas.test.ts @@ -68,6 +68,42 @@ describe("sessionCore RPC schemas", () => { ).toBe(128); }); + it("preserves explicit external replay payload encoding and accepts legacy refs", () => { + const explicit = makeEvent( + "event-explicit", + { output: "preview" }, + "2026-07-23T00:00:00.000Z" + ); + explicit.payloadRefs = [ + { + eventId: "event-explicit", + fieldPath: "args", + preview: "preview", + fullSizeBytes: 1024, + truncated: true, + replayEncoding: "json_value", + }, + ]; + const legacy = makeEvent( + "event-legacy", + { output: "preview" }, + "2026-07-23T00:00:01.000Z" + ); + legacy.payloadRefs = [ + { + eventId: "event-legacy", + fieldPath: "result.output", + preview: "preview", + fullSizeBytes: 1024, + truncated: true, + }, + ]; + + const parsed = SessionEventArraySchema.parse([explicit, legacy]); + expect(parsed[0].payloadRefs?.[0].replayEncoding).toBe("json_value"); + expect(parsed[1].payloadRefs?.[0].replayEncoding).toBeUndefined(); + }); + it("caps shell replay range reads at 256 KiB", () => { expect(() => ShellReplayRangeInput.parse({ diff --git a/src/api/tauri/rpc/__tests__/tauriUnit.test.ts b/src/api/tauri/rpc/__tests__/tauriUnit.test.ts new file mode 100644 index 000000000..6bd6cf02d --- /dev/null +++ b/src/api/tauri/rpc/__tests__/tauriUnit.test.ts @@ -0,0 +1,14 @@ +import { describe, expect, it } from "vitest"; + +import { TauriUnitSchema } from "../schemas/tauriUnit"; + +describe("Tauri unit response schema", () => { + it("normalizes Rust unit null to TypeScript undefined", () => { + expect(TauriUnitSchema.parse(null)).toBeUndefined(); + }); + + it("rejects a missing or non-unit response", () => { + expect(TauriUnitSchema.safeParse(undefined).success).toBe(false); + expect(TauriUnitSchema.safeParse({}).success).toBe(false); + }); +}); diff --git a/src/api/tauri/rpc/procedures/collaborationSnapshotIngest.ts b/src/api/tauri/rpc/procedures/collaborationSnapshotIngest.ts new file mode 100644 index 000000000..19f90a9ab --- /dev/null +++ b/src/api/tauri/rpc/procedures/collaborationSnapshotIngest.ts @@ -0,0 +1,34 @@ +import { z } from "zod/v4"; + +import { defineProcedure } from "../invoke"; +import * as schemas from "../schemas/collaborationSnapshotIngest"; +import { TauriUnitSchema } from "../schemas/tauriUnit"; + +export const collaborationSnapshotIngest = { + begin: defineProcedure("collaboration_snapshot_ingest_begin") + .input(schemas.CollaborationSnapshotIngestBeginInputSchema) + .output(schemas.CollaborationSnapshotIngestBeginResultSchema) + .build(), + getCursor: defineProcedure("collaboration_snapshot_ingest_get_cursor") + .input(schemas.CollaborationSnapshotIngestGetCursorInputSchema) + .output(schemas.CollaborationSnapshotCursorSchema.nullable()) + .build(), + probeSecondary: defineProcedure("collaboration_snapshot_secondary_probe") + .input(schemas.CollaborationSnapshotSecondaryProbeInputSchema) + .output(z.boolean()) + .build(), + applyWirePage: defineProcedure( + "collaboration_snapshot_ingest_apply_wire_page" + ) + .input(schemas.CollaborationSnapshotIngestPageInputSchema) + .output(schemas.CollaborationSnapshotIngestProgressSchema) + .build(), + commit: defineProcedure("collaboration_snapshot_ingest_commit") + .input(schemas.CollaborationSnapshotIngestTokenInputSchema) + .output(schemas.CollaborationSnapshotIngestCommitResultSchema) + .build(), + abort: defineProcedure("collaboration_snapshot_ingest_abort") + .input(schemas.CollaborationSnapshotIngestTokenInputSchema) + .output(TauriUnitSchema) + .build(), +} as const; diff --git a/src/api/tauri/rpc/procedures/externalReplay.ts b/src/api/tauri/rpc/procedures/externalReplay.ts new file mode 100644 index 000000000..854770062 --- /dev/null +++ b/src/api/tauri/rpc/procedures/externalReplay.ts @@ -0,0 +1,58 @@ +import { defineProcedure } from "../invoke"; +import * as schemas from "../schemas/externalReplay"; +import { TauriUnitSchema } from "../schemas/tauriUnit"; + +export const externalReplay = { + openWindow: defineProcedure("external_replay_open_window") + .input(schemas.ExternalReplayOpenWindowInput) + .output(schemas.ExternalReplayWindowSchema) + .build(), + pollDelta: defineProcedure("external_replay_poll_delta") + .input(schemas.ExternalReplayPollDeltaInput) + .output(schemas.ExternalReplayDeltaSchema) + .build(), + readWindow: defineProcedure("external_replay_read_window") + .input(schemas.ExternalReplayReadWindowInput) + .output(schemas.ExternalReplayWindowSchema) + .build(), + queryWindow: defineProcedure("external_replay_query_window") + .input(schemas.ExternalReplayQueryWindowInput) + .output(schemas.ExternalReplayWindowSchema) + .build(), + handoff: defineProcedure("external_replay_handoff") + .input(schemas.ExternalReplayHandoffInput) + .output(schemas.ExternalReplayHandoffSchema) + .build(), + prewarmWindow: defineProcedure("external_replay_prewarm_window") + .input(schemas.ExternalReplayPrewarmWindowInput) + .output(schemas.ExternalReplayWindowSchema) + .build(), + release: defineProcedure("external_replay_release") + .input(schemas.ExternalReplayReleaseInput) + .output(TauriUnitSchema) + .build(), + readPayloadRange: defineProcedure("external_replay_read_payload_range") + .input(schemas.ExternalReplayReadPayloadRangeInput) + .output(schemas.ExternalReplayPayloadRangeSchema) + .build(), + streamExport: defineProcedure("external_replay_stream_export") + .input(schemas.ExternalReplayStreamExportInput) + .output(schemas.ExternalReplayExportResultSchema.nullable()) + .build(), + cloudPrepare: defineProcedure("external_replay_cloud_prepare") + .input(schemas.ExternalReplayCloudPrepareInput) + .output(schemas.ExternalReplayCloudManifestSchema) + .build(), + cloudReadBatch: defineProcedure("external_replay_cloud_read_batch") + .input(schemas.ExternalReplayCloudReadBatchInput) + .output(schemas.ExternalReplayCloudBatchSchema) + .build(), + cloudPrefixHash: defineProcedure("external_replay_cloud_prefix_hash") + .input(schemas.ExternalReplayCloudPrefixHashInput) + .output(schemas.ExternalReplayCloudPrefixHashSchema) + .build(), + cloudRelease: defineProcedure("external_replay_cloud_release") + .input(schemas.ExternalReplayCloudReleaseInput) + .output(TauriUnitSchema) + .build(), +} as const; diff --git a/src/api/tauri/rpc/procedures/index.ts b/src/api/tauri/rpc/procedures/index.ts index c52fbf185..0552eba31 100644 --- a/src/api/tauri/rpc/procedures/index.ts +++ b/src/api/tauri/rpc/procedures/index.ts @@ -1,6 +1,7 @@ export { agentDef } from "./agentDef"; export { agentOrgs } from "./agentOrgs"; export { agentSession } from "./agentSession"; +export { collaborationSnapshotIngest } from "./collaborationSnapshotIngest"; export { diff } from "./diff"; export { flow } from "./flow"; export { humanSession } from "./humanSession"; @@ -13,6 +14,7 @@ export { searchRegex } from "./searchRegex"; export { searchSymbol } from "./searchSymbol"; export { sessionAggregate } from "./sessionAggregate"; export { sessionCore } from "./sessionCore"; +export { externalReplay } from "./externalReplay"; export { settings } from "./settings"; export { terminal } from "./terminal"; export { tools } from "./tools"; diff --git a/src/api/tauri/rpc/procedures/sessionAggregate.ts b/src/api/tauri/rpc/procedures/sessionAggregate.ts index 176bbc5ee..509e50444 100644 --- a/src/api/tauri/rpc/procedures/sessionAggregate.ts +++ b/src/api/tauri/rpc/procedures/sessionAggregate.ts @@ -14,6 +14,11 @@ export const sessionAggregate = { .output(schemas.sessionAggregate.ExternalHistorySidebarBatchResponseSchema) .build(), + workspaceFacets: defineProcedure("session_sidebar_workspace_facets") + .input(schemas.sessionAggregate.SessionWorkspaceFacetListInput) + .output(schemas.sessionAggregate.SessionWorkspaceFacetResponseSchema) + .build(), + /** * Patch in-session mutable fields for a single session row. * diff --git a/src/api/tauri/rpc/procedures/sessionCore.ts b/src/api/tauri/rpc/procedures/sessionCore.ts index 35ea1d78c..edf5144bf 100644 --- a/src/api/tauri/rpc/procedures/sessionCore.ts +++ b/src/api/tauri/rpc/procedures/sessionCore.ts @@ -29,7 +29,7 @@ const cache = { .input(schemas.sessionCore.SaveCachedEventsInput) .build(), - loadCachedEvents: defineProcedure("cache_load_events") + loadCachedEvents: defineProcedure("cache_load_native_cached_events") .input(schemas.sessionCore.SessionIdInput) .output(schemas.sessionCore.CachedEventRowsSchema) .build(), diff --git a/src/api/tauri/rpc/router.ts b/src/api/tauri/rpc/router.ts index f524ef47b..c2a392255 100644 --- a/src/api/tauri/rpc/router.ts +++ b/src/api/tauri/rpc/router.ts @@ -35,9 +35,11 @@ export const procedures = { agentDef: p.agentDef, agentOrgs: p.agentOrgs, agentSession: p.agentSession, + collaborationSnapshotIngest: p.collaborationSnapshotIngest, integrations: p.integrations, sessionAggregate: p.sessionAggregate, sessionCore: p.sessionCore, + externalReplay: p.externalReplay, learning: p.learning, workspaceMemory: p.workspaceMemory, lineage: p.lineage, diff --git a/src/api/tauri/rpc/schemas/collaborationSnapshotIngest.ts b/src/api/tauri/rpc/schemas/collaborationSnapshotIngest.ts new file mode 100644 index 000000000..c732bcb11 --- /dev/null +++ b/src/api/tauri/rpc/schemas/collaborationSnapshotIngest.ts @@ -0,0 +1,202 @@ +import { z } from "zod/v4"; + +import replayBudgets from "@src/shared/externalReplayBudgets.json"; + +const SafeU64Schema = z + .number() + .int() + .nonnegative() + .max(Number.MAX_SAFE_INTEGER); + +const Sha256Schema = z.string().regex(/^[0-9a-f]{64}$/i); + +const ImportedSnapshotSessionIdSchema = z + .string() + .refine( + (value) => + value.startsWith("imported-session-") && + value.length > "imported-session-".length, + "Snapshot cursor target must be an imported-session" + ); + +export const CollaborationSnapshotCursorSchema = z.object({ + epoch: SafeU64Schema, + frozenSeq: SafeU64Schema, + count: SafeU64Schema, + frozenCount: SafeU64Schema, + tailHash: Sha256Schema.nullable(), +}); + +export const CollaborationSnapshotWireCursorSchema = z.discriminatedUnion( + "direction", + [ + z.object({ + direction: z.literal("forward"), + afterSeq: SafeU64Schema, + throughSeq: SafeU64Schema.optional(), + }), + z.object({ + direction: z.literal("backward"), + beforeSeq: SafeU64Schema.positive().optional(), + }), + ] +); + +export const CollaborationSnapshotWireSchema = z.object({ + seq: SafeU64Schema, + payloadGz: z.string(), + eventCount: SafeU64Schema, + segmentHash: Sha256Schema, +}); + +export const CollaborationSnapshotIngestBeginRequestSchema = z + .object({ + localSessionId: z + .string() + .refine( + (value) => + (value.startsWith("imported-session-") && + value.length > "imported-session-".length) || + (value.startsWith("agentsession-") && + value.length > "agentsession-".length), + "Snapshot target must be an imported-session or agentsession" + ), + epoch: SafeU64Schema, + expectedCount: SafeU64Schema, + expectedFrozenSeq: SafeU64Schema, + tailHash: Sha256Schema.nullable(), + replace: z.boolean(), + previous: CollaborationSnapshotCursorSchema.optional(), + }) + .superRefine((value, context) => { + if (!value.replace && value.previous === undefined) { + context.addIssue({ + code: "custom", + message: "Incremental snapshot ingest requires a previous cursor", + path: ["previous"], + }); + } + if ( + value.localSessionId.startsWith("agentsession-") && + (!value.replace || value.previous !== undefined) + ) { + context.addIssue({ + code: "custom", + message: + "Native fork snapshot ingest must be an unconditional full replacement", + path: ["replace"], + }); + } + }); + +export const CollaborationSnapshotIngestBeginInputSchema = z.object({ + request: CollaborationSnapshotIngestBeginRequestSchema, +}); + +export const CollaborationSnapshotIngestBeginResultSchema = z.object({ + token: z.uuid(), +}); + +export const CollaborationSnapshotIngestGetCursorInputSchema = z.object({ + request: z.object({ + localSessionId: ImportedSnapshotSessionIdSchema, + }), +}); + +export const CollaborationSnapshotSecondaryProbeInputSchema = z.object({ + request: z.object({ + sessionId: z + .string() + .refine( + (value) => + value.startsWith("agentsession-") && + value.length > "agentsession-".length, + "Secondary snapshot target must be an agentsession" + ), + }), +}); + +export const CollaborationSnapshotIngestPageRequestSchema = z + .object({ + token: z.uuid(), + epoch: SafeU64Schema, + frozenSeq: SafeU64Schema, + count: SafeU64Schema, + tailHash: Sha256Schema.nullable(), + cursor: CollaborationSnapshotWireCursorSchema, + nextCursor: CollaborationSnapshotWireCursorSchema.nullable(), + tailIncluded: z.boolean(), + hasMore: z.boolean(), + returnedWireBytes: SafeU64Schema.max( + replayBudgets.cloudPageMaxBytes + + replayBudgets.cloudLegacyV1SegmentMaxBytes + ), + segments: z.array(CollaborationSnapshotWireSchema).max(200), + }) + .superRefine((value, context) => { + if (value.hasMore !== (value.nextCursor !== null)) { + context.addIssue({ + code: "custom", + message: "hasMore and nextCursor must agree", + path: ["nextCursor"], + }); + } + }); + +export const CollaborationSnapshotIngestPageInputSchema = z.object({ + request: CollaborationSnapshotIngestPageRequestSchema, +}); + +export const CollaborationSnapshotIngestProgressSchema = z.object({ + acceptedPhysicalRows: SafeU64Schema, + acceptedLogicalEvents: SafeU64Schema, + complete: z.boolean(), +}); + +export const CollaborationSnapshotIngestTokenInputSchema = z.object({ + request: z.object({ token: z.uuid() }), +}); + +export const CollaborationSnapshotIngestCommitResultSchema = z.object({ + localSessionId: z + .string() + .refine( + (value) => + value.startsWith("imported-session-") || + value.startsWith("agentsession-"), + "Snapshot target must be an imported-session or agentsession" + ), + epoch: SafeU64Schema, + frozenSeq: SafeU64Schema, + eventCount: SafeU64Schema, + frozenEventCount: SafeU64Schema, + tailHash: Sha256Schema.nullable(), + handoffItems: z.array(z.string().max(1_200)).max(80), + handoffScannedBytes: SafeU64Schema.max(4 * 1024 * 1024), + handoffScannedEvents: SafeU64Schema, +}); + +export type CollaborationSnapshotCursor = z.infer< + typeof CollaborationSnapshotCursorSchema +>; +export type CollaborationSnapshotWireCursor = z.infer< + typeof CollaborationSnapshotWireCursorSchema +>; +export type CollaborationSnapshotWire = z.infer< + typeof CollaborationSnapshotWireSchema +>; +export type CollaborationSnapshotIngestBeginRequest = z.infer< + typeof CollaborationSnapshotIngestBeginRequestSchema +>; +export type CollaborationSnapshotIngestGetCursorRequest = z.infer< + typeof CollaborationSnapshotIngestGetCursorInputSchema +>["request"]; +export type CollaborationSnapshotIngestPageRequest = z.infer< + typeof CollaborationSnapshotIngestPageRequestSchema +>; +export type CollaborationSnapshotIngestProgress = z.infer< + typeof CollaborationSnapshotIngestProgressSchema +>; +export type CollaborationSnapshotIngestCommitResult = z.infer< + typeof CollaborationSnapshotIngestCommitResultSchema +>; diff --git a/src/api/tauri/rpc/schemas/externalReplay.ts b/src/api/tauri/rpc/schemas/externalReplay.ts new file mode 100644 index 000000000..7aeb03838 --- /dev/null +++ b/src/api/tauri/rpc/schemas/externalReplay.ts @@ -0,0 +1,307 @@ +import { z } from "zod/v4"; + +import replayBudgets from "@src/shared/externalReplayBudgets.json"; +import { IMPORTED_HISTORY_SOURCE_IDS } from "@src/types/session/externalHistory"; + +import { SessionEventArraySchema } from "./sessionCore"; + +const SafeIntegerSchema = z + .number() + .int() + .min(Number.MIN_SAFE_INTEGER) + .max(Number.MAX_SAFE_INTEGER); +const SafeU64Schema = SafeIntegerSchema.nonnegative(); + +export const ExternalReplaySourceIdSchema = z.enum([ + ...IMPORTED_HISTORY_SOURCE_IDS, + "managed_cli", + // ORGII-owned, already-persisted collaboration snapshots. This is + // deliberately not part of the 15-vendor ImportedHistorySourceId mirror: + // its source of truth is our own sessions.db `events` table. + "collaboration_snapshot", +]); + +export const ExternalReplayLimitsSchema = z + .object({ + maxTurns: z + .number() + .int() + .positive() + .max(replayBudgets.replayMaxTurns) + .optional(), + maxEvents: z + .number() + .int() + .positive() + .max(replayBudgets.replayMaxEvents) + .optional(), + maxIpcBytes: z + .number() + .int() + .positive() + .max(replayBudgets.replayMaxIpcBytes) + .optional(), + }) + .optional(); + +export const ExternalReplayCursorSchema = z.object({ + sourceId: ExternalReplaySourceIdSchema, + sessionId: z.string().min(1), + generation: z.string(), + revision: SafeU64Schema, + // `-1` is reserved for a managed native CLI whose transcript binding is + // not available yet. No source bytes have been consumed in that state. + throughSequence: SafeIntegerSchema, +}); + +export const ExternalReplayTurnHeaderSchema = z.object({ + turnId: z.string(), + turnIndex: SafeU64Schema, + startSequence: SafeU64Schema, + endSequence: SafeU64Schema.nullable(), + startedAt: z.string(), + endedAt: z.string().nullable(), + eventCount: SafeU64Schema, +}); + +export const ExternalReplayStatsSchema = z.object({ + parsedBytes: SafeU64Schema, + parsedRows: SafeU64Schema, + normalizedEvents: SafeU64Schema, + upsertedEvents: SafeU64Schema, + removedEvents: SafeU64Schema, + ipcBytes: SafeU64Schema, + notReady: z.boolean(), +}); + +export const ExternalReplayWindowSchema = z.object({ + cursor: ExternalReplayCursorSchema, + events: SessionEventArraySchema, + windowStartSequence: SafeU64Schema.nullable(), + turnHeaders: z.array(ExternalReplayTurnHeaderSchema), + totalEventCount: SafeU64Schema, + totalTurnCount: SafeU64Schema, + hasOlder: z.boolean(), + watcherAvailable: z.boolean(), + stats: ExternalReplayStatsSchema, +}); + +export const ExternalReplayDeltaSchema = z.object({ + cursor: ExternalReplayCursorSchema, + events: SessionEventArraySchema, + removedEventIds: z.array(z.string()), + resetRequired: z.boolean(), + watcherAvailable: z.boolean(), + stats: ExternalReplayStatsSchema, +}); + +const ExternalReplayTargetInput = z.object({ + sourceId: ExternalReplaySourceIdSchema, + sessionId: z.string().min(1), +}); + +export const ExternalReplayOpenWindowInput = ExternalReplayTargetInput.extend({ + episodeId: SafeU64Schema, + limits: ExternalReplayLimitsSchema, +}); + +export const ExternalReplayPollDeltaInput = ExternalReplayTargetInput.extend({ + episodeId: SafeU64Schema, + cursor: ExternalReplayCursorSchema, + limits: ExternalReplayLimitsSchema, +}); + +export const ExternalReplayReadWindowInput = ExternalReplayTargetInput.extend({ + episodeId: SafeU64Schema, + beforeSequence: SafeU64Schema.optional(), + turnId: z.string().min(1).optional(), + turnIndex: SafeU64Schema.optional(), + limits: ExternalReplayLimitsSchema, +}).refine( + ({ beforeSequence, turnId, turnIndex }) => + [beforeSequence, turnId, turnIndex].filter((value) => value !== undefined) + .length <= 1, + { message: "Choose only one replay window locator" } +); + +export const ExternalReplayQueryWindowInput = ExternalReplayTargetInput.extend({ + beforeSequence: SafeU64Schema.optional(), + turnId: z.string().min(1).optional(), + turnIndex: SafeU64Schema.optional(), + limits: ExternalReplayLimitsSchema, +}).refine( + ({ beforeSequence, turnId, turnIndex }) => + [beforeSequence, turnId, turnIndex].filter((value) => value !== undefined) + .length <= 1, + { message: "Choose only one replay window locator" } +); + +export const ExternalReplayPrewarmWindowInput = + ExternalReplayTargetInput.extend({ + episodeId: SafeU64Schema, + limits: ExternalReplayLimitsSchema, + }); + +export const ExternalReplayHandoffInput = ExternalReplayTargetInput.extend({ + sourceName: z.string().trim().min(1).max(200), +}); + +export const ExternalReplayHandoffSchema = z.object({ + items: z.array(z.string().max(1200)).max(80), + generation: z.string(), + scannedBytes: SafeU64Schema.max(4 * 1024 * 1024), + scannedEvents: SafeU64Schema, +}); + +export const ExternalReplayReleaseInput = ExternalReplayTargetInput.extend({ + episodeId: SafeU64Schema, +}); + +export const ExternalReplayReadPayloadRangeInput = + ExternalReplayTargetInput.extend({ + generation: z.string(), + eventId: z.string().min(1), + fieldPath: z.string().min(1), + offset: SafeU64Schema, + maxBytes: z + .number() + .int() + .positive() + .max(replayBudgets.payloadRangeMaxBytes) + .optional(), + }); + +export const ExternalReplayPayloadRangeSchema = z.object({ + eventId: z.string(), + fieldPath: z.string(), + offset: SafeU64Schema, + nextOffset: SafeU64Schema, + eof: z.boolean(), + totalBytes: SafeU64Schema, + text: z.string(), +}); + +export const ExternalReplayExportFormatSchema = z.enum([ + "json", + "markdown", + "orgii_session_json", +]); + +export const ExternalReplayOrgiiEnvelopeSchema = z.object({ + exportedAt: z.string(), + session: z.record(z.string(), z.unknown()), + originalCategory: z.string(), + specs: z.array(z.unknown()).optional(), +}); + +export const ExternalReplayStreamExportInput = ExternalReplayTargetInput.extend( + { + suggestedFileName: z.string().min(1).max(255).optional(), + format: ExternalReplayExportFormatSchema, + orgiiEnvelope: ExternalReplayOrgiiEnvelopeSchema.optional(), + } +); + +export const ExternalReplayExportResultSchema = z.object({ + destinationPath: z.string(), + bytesWritten: SafeU64Schema, + eventCount: SafeU64Schema, + sha256: z.string(), +}); + +export const ExternalReplayCloudPrepareInput = ExternalReplayTargetInput; + +export const ExternalReplayCloudManifestSchema = z.object({ + token: z.string().min(1), + generation: z.string(), + totalCount: SafeU64Schema, + frozenEventCount: SafeU64Schema, + tailEventCount: SafeU64Schema, + frozenChainHash: z.string(), + tailHash: z.string().nullable(), +}); + +export const ExternalReplayCloudReadBatchInput = z.object({ + token: z.string().min(1), + startEventIndex: SafeU64Schema, + endEventIndex: SafeU64Schema, + startSegmentIndex: SafeU64Schema.optional(), + maxBytes: z + .number() + .int() + .positive() + .max(replayBudgets.cloudSegmentMaxBytes) + .optional(), +}); + +export const ExternalReplayCloudSegmentSchema = z.object({ + payloadGz: z.string(), + eventCount: SafeU64Schema, + segmentHash: z.string(), + wireBytes: SafeU64Schema.max(replayBudgets.cloudSegmentMaxBytes), +}); + +export const ExternalReplayCloudBatchSchema = z.object({ + segments: z + .array(ExternalReplayCloudSegmentSchema) + .max(replayBudgets.cloudPageMaxSegments), + startEventIndex: SafeU64Schema, + nextEventIndex: SafeU64Schema, + startSegmentIndex: SafeU64Schema, + nextSegmentIndex: SafeU64Schema, + eof: z.boolean(), + serializedBytes: SafeU64Schema, +}); + +export const ExternalReplayCloudPrefixHashInput = z.object({ + token: z.string().min(1), + eventCount: SafeU64Schema, +}); + +export const ExternalReplayCloudPrefixHashSchema = z.object({ + eventCount: SafeU64Schema, + frozenChainHash: z.string(), +}); + +export const ExternalReplayCloudReleaseInput = z.object({ + token: z.string().min(1), +}); + +export const ExternalReplayInvalidationSchema = z.object({ + sessionId: z.string(), + sourceId: ExternalReplaySourceIdSchema, + generation: z.string().optional(), +}); + +export type ExternalReplaySourceId = z.infer< + typeof ExternalReplaySourceIdSchema +>; +export type ExternalReplayLimits = z.infer; +export type ExternalReplayCursor = z.infer; +export type ExternalReplayWindow = z.infer; +export type ExternalReplayDelta = z.infer; +export type ExternalReplayHandoff = z.infer; +export type ExternalReplayPayloadRange = z.infer< + typeof ExternalReplayPayloadRangeSchema +>; +export type ExternalReplayExportFormat = z.infer< + typeof ExternalReplayExportFormatSchema +>; +export type ExternalReplayExportResult = z.infer< + typeof ExternalReplayExportResultSchema +>; +export type ExternalReplayOrgiiEnvelope = z.infer< + typeof ExternalReplayOrgiiEnvelopeSchema +>; +export type ExternalReplayInvalidation = z.infer< + typeof ExternalReplayInvalidationSchema +>; +export type ExternalReplayCloudManifest = z.infer< + typeof ExternalReplayCloudManifestSchema +>; +export type ExternalReplayCloudBatch = z.infer< + typeof ExternalReplayCloudBatchSchema +>; +export type ExternalReplayCloudPrefixHash = z.infer< + typeof ExternalReplayCloudPrefixHashSchema +>; diff --git a/src/api/tauri/rpc/schemas/index.ts b/src/api/tauri/rpc/schemas/index.ts index bc53d4a11..615149653 100644 --- a/src/api/tauri/rpc/schemas/index.ts +++ b/src/api/tauri/rpc/schemas/index.ts @@ -13,6 +13,7 @@ export * as gateway from "./gateway"; export * as agentDef from "./agentDef"; export * as agentOrgs from "./agentOrgs"; export * as agentSession from "./agentSession"; +export * as collaborationSnapshotIngest from "./collaborationSnapshotIngest"; export * as integrations from "./integrations"; export * as sessionAggregate from "./sessionAggregate"; export * as learning from "./learning"; @@ -25,3 +26,4 @@ export * as mcp from "./mcp"; export * as flow from "./flow"; export * as humanSession from "./humanSession"; export * as sessionCore from "./sessionCore"; +export * as externalReplay from "./externalReplay"; diff --git a/src/api/tauri/rpc/schemas/sessionAggregate.ts b/src/api/tauri/rpc/schemas/sessionAggregate.ts index e476a1a44..72a8eb94b 100644 --- a/src/api/tauri/rpc/schemas/sessionAggregate.ts +++ b/src/api/tauri/rpc/schemas/sessionAggregate.ts @@ -41,18 +41,26 @@ export const SessionFilterInput = z.object({ status: z.string().optional(), keySource: z.string().optional(), repoPath: z.string().optional(), + repoPathExact: z.boolean().optional(), + missingRepoPath: z.boolean().optional(), orgId: z.string().optional(), + orgIds: z.array(z.string().min(1)).min(1).optional(), projectSlug: z.string().optional(), workItemId: z.string().optional(), limit: z.number().int().optional(), offset: z.number().int().optional(), + beforeUpdatedAt: z.string().min(1).optional(), + beforeSessionId: z.string().min(1).optional(), textQuery: z.string().optional(), sortBy: z.string().optional(), sortOrder: z.enum(["asc", "desc"]).optional(), includeExternalHistory: z.boolean().optional(), externalHistorySource: z.string().optional(), disabledExternalHistorySources: z.array(z.string()).optional(), + updatedAfterMs: z.number().int().optional(), + updatedBeforeMs: z.number().int().optional(), activeOnly: z.boolean().optional(), + pinnedOnly: z.boolean().optional(), includeContinuationSuperseded: z.boolean().optional(), }); @@ -67,31 +75,45 @@ export const ExternalHistorySidebarDateBucketSchema = z.enum([ "older", ]); -export const ExternalHistorySidebarSourceRequestSchema = z.object({ - source: z.string().min(1), - buckets: z - .array( - z - .object({ - bucket: ExternalHistorySidebarDateBucketSchema, - startMs: z.number().int().optional(), - endMs: z.number().int().optional(), - limit: z.number().int().min(1).max(50), - offset: z.number().int().min(0), - }) - .refine( - ({ startMs, endMs }) => - startMs === undefined || endMs === undefined || startMs < endMs, - { message: "startMs must precede endMs" } - ) - ) - .refine( - (buckets) => - new Set(buckets.map(({ bucket }) => bucket)).size === buckets.length, - { message: "date buckets must be unique" } - ), +export const ExternalHistorySidebarCursorSchema = z.object({ + updatedAtMs: z.number().int(), + sessionId: z.string().min(1), }); +export const ExternalHistorySidebarSourceRequestSchema = z + .object({ + source: z.string().min(1), + repoPath: z.string().min(1).optional(), + missingRepoPath: z.boolean().optional(), + buckets: z + .array( + z + .object({ + bucket: ExternalHistorySidebarDateBucketSchema, + startMs: z.number().int().optional(), + endMs: z.number().int().optional(), + limit: z.number().int().min(1).max(50), + offset: z.number().int().min(0), + before: ExternalHistorySidebarCursorSchema.optional(), + }) + .refine( + ({ startMs, endMs }) => + startMs === undefined || endMs === undefined || startMs < endMs, + { message: "startMs must precede endMs" } + ) + ) + .refine( + (buckets) => + new Set(buckets.map(({ bucket }) => bucket)).size === buckets.length, + { message: "date buckets must be unique" } + ), + }) + .refine( + ({ repoPath, missingRepoPath }) => + repoPath === undefined || missingRepoPath !== true, + { message: "repoPath and missingRepoPath are mutually exclusive" } + ); + export const ExternalHistorySidebarListInput = z.object({ requests: z .array(ExternalHistorySidebarSourceRequestSchema) @@ -264,6 +286,7 @@ export const ExternalHistorySidebarResponseSchema = z.object({ bucket: ExternalHistorySidebarDateBucketSchema, sessions: z.array(ExternalHistorySidebarRowSchema), hasMore: z.boolean(), + nextCursor: ExternalHistorySidebarCursorSchema.optional(), }) ), }); @@ -272,6 +295,33 @@ export const ExternalHistorySidebarBatchResponseSchema = z.object({ sources: z.array(ExternalHistorySidebarResponseSchema), }); +export const SessionWorkspaceFacetListInput = z.object({ + request: z.object({ + orgIds: z.array(z.string().min(1)).min(1), + includeExternalHistory: z.boolean().optional(), + disabledExternalHistorySources: z.array(z.string()).optional(), + limit: z.number().int().min(1).max(50), + offset: z.number().int().min(0).optional(), + before: z + .object({ + lastUpdatedAtMs: z.number().int(), + repoPath: z.string().nullable(), + }) + .optional(), + }), +}); + +export const SessionWorkspaceFacetResponseSchema = z.object({ + facets: z.array( + z.object({ + repoPath: z.string().nullable().optional(), + lastUpdatedAtMs: z.number().int(), + sessionCount: z.number().int().min(0), + }) + ), + hasMore: z.boolean(), +}); + export type SessionFilter = z.input; export type SessionAggregateRecord = z.output< typeof SessionAggregateRecordSchema @@ -280,6 +330,9 @@ export type SessionListResponse = z.output; export type ExternalHistorySidebarDateBucket = z.output< typeof ExternalHistorySidebarDateBucketSchema >; +export type ExternalHistorySidebarCursor = z.output< + typeof ExternalHistorySidebarCursorSchema +>; export type ExternalHistorySidebarListRequest = z.input< typeof ExternalHistorySidebarListInput >; @@ -292,4 +345,10 @@ export type ExternalHistorySidebarResponse = z.output< export type ExternalHistorySidebarBatchResponse = z.output< typeof ExternalHistorySidebarBatchResponseSchema >; +export type SessionWorkspaceFacetListRequest = z.input< + typeof SessionWorkspaceFacetListInput +>["request"]; +export type SessionWorkspaceFacetResponse = z.output< + typeof SessionWorkspaceFacetResponseSchema +>; export type SessionPatchPayload = z.input; diff --git a/src/api/tauri/rpc/schemas/sessionCore.ts b/src/api/tauri/rpc/schemas/sessionCore.ts index d0090adee..1ec72e570 100644 --- a/src/api/tauri/rpc/schemas/sessionCore.ts +++ b/src/api/tauri/rpc/schemas/sessionCore.ts @@ -1,6 +1,7 @@ import { z } from "zod/v4"; import type { SessionEvent } from "@src/engines/SessionCore/core/types"; +import replayBudgets from "@src/shared/externalReplayBudgets.json"; const UnknownRecordSchema = z.record(z.string(), z.unknown()); const SafeU64NumberSchema = z @@ -56,6 +57,10 @@ export const PayloadRefSchema = z.object({ preview: z.string(), fullSizeBytes: z.number(), truncated: z.boolean(), + replayEncoding: z.enum(["json_value", "utf8_text"]).optional(), + replaySourceId: z.string().optional(), + replayGeneration: z.string().optional(), + replaySourceEventId: z.string().optional(), }); export const EventPayloadBodySchema = z.object({ @@ -101,7 +106,7 @@ export const ShellReplayRangeInput = z.object({ .number() .int() .positive() - .max(256 * 1024), + .max(replayBudgets.shellReplayRangeMaxBytes), }); export const ShellReplayFrameSchema = z.object({ diff --git a/src/api/tauri/rpc/schemas/tauriUnit.ts b/src/api/tauri/rpc/schemas/tauriUnit.ts new file mode 100644 index 000000000..213de0e2e --- /dev/null +++ b/src/api/tauri/rpc/schemas/tauriUnit.ts @@ -0,0 +1,9 @@ +import { z } from "zod/v4"; + +/** + * Rust `()` crosses the Tauri boundary as JSON `null`. + * + * Keep that wire detail in one schema so fire-and-forget commands do not + * report a false validation error after the backend action already succeeded. + */ +export const TauriUnitSchema = z.null().transform(() => undefined); diff --git a/src/api/tauri/session/index.ts b/src/api/tauri/session/index.ts index f0a129941..4af46c3a7 100644 --- a/src/api/tauri/session/index.ts +++ b/src/api/tauri/session/index.ts @@ -7,6 +7,7 @@ import { IMPORTED_HISTORY_SOURCE_DESCRIPTORS } from "@src/api/tauri/externalHist import { rpc } from "@src/api/tauri/rpc"; import type { ExternalHistorySidebarBatchResponse, + ExternalHistorySidebarCursor, ExternalHistorySidebarDateBucket, ExternalHistorySidebarListRequest, ExternalHistorySidebarResponse, @@ -14,6 +15,8 @@ import type { SessionAggregateRecord, SessionFilter, SessionListResponse, + SessionWorkspaceFacetListRequest, + SessionWorkspaceFacetResponse, } from "@src/api/tauri/rpc/schemas/sessionAggregate"; import { normalizeAgentExecMode } from "@src/config/sessionCreatorConfig"; import type { Session } from "@src/store/session/sessionAtom/types"; @@ -43,6 +46,7 @@ export type { // Re-export session aggregate types from RPC schemas (single source of truth). export type { ExternalHistorySidebarBatchResponse, + ExternalHistorySidebarCursor, ExternalHistorySidebarDateBucket, ExternalHistorySidebarListRequest, ExternalHistorySidebarResponse, @@ -50,6 +54,8 @@ export type { SessionAggregateRecord, SessionFilter, SessionListResponse, + SessionWorkspaceFacetListRequest, + SessionWorkspaceFacetResponse, }; // ============================================================================ @@ -74,6 +80,12 @@ export async function externalHistorySidebarList( return rpc.sessionAggregate.externalHistorySidebarList(request); } +export async function sessionWorkspaceFacets( + request: SessionWorkspaceFacetListRequest +): Promise { + return rpc.sessionAggregate.workspaceFacets({ request }); +} + // ============================================================================ // Helper Functions // ============================================================================ diff --git a/src/app/root/e2e/helpers/sessionHelpers/__tests__/openBoundedReplaySession.test.ts b/src/app/root/e2e/helpers/sessionHelpers/__tests__/openBoundedReplaySession.test.ts new file mode 100644 index 000000000..0d157420d --- /dev/null +++ b/src/app/root/e2e/helpers/sessionHelpers/__tests__/openBoundedReplaySession.test.ts @@ -0,0 +1,129 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import { IMPORTED_HISTORY_SOURCE_DESCRIPTORS } from "@src/api/tauri/externalHistory/imported"; + +import { openBoundedReplaySessionForE2E } from "../openBoundedReplaySession"; + +const mocks = vi.hoisted(() => ({ + activate: vi.fn(), + deactivate: vi.fn(), + getEvents: vi.fn(), + mergeTurnWindow: vi.fn(), + open: vi.fn(), + startTurnEpisode: vi.fn(), +})); + +vi.mock("@src/engines/SessionCore/core/store/EventStoreProxy", () => ({ + eventStoreProxy: { getEvents: mocks.getEvents }, +})); + +vi.mock("@src/engines/SessionCore/sync/externalReplayTransport", () => ({ + activateExternalReplaySession: mocks.activate, + deactivateExternalReplaySession: mocks.deactivate, + openExternalReplaySession: mocks.open, +})); + +vi.mock("@src/engines/SessionCore/sync/externalReplayTurnState", () => ({ + mergeExternalReplayTurnWindow: mocks.mergeTurnWindow, + startExternalReplayTurnEpisode: mocks.startTurnEpisode, +})); + +describe("openBoundedReplaySessionForE2E", () => { + beforeEach(() => { + vi.clearAllMocks(); + mocks.activate.mockImplementation((sessionId: string) => ({ + sessionId, + epoch: 1, + signal: new AbortController().signal, + })); + mocks.open.mockImplementation(async (lease: { sessionId: string }) => ({ + cursor: { + sourceId: "codex_app", + sessionId: lease.sessionId, + generation: "generation-1", + revision: 1, + throughSequence: 1, + }, + events: [], + turnHeaders: [], + totalTurnCount: 0, + hasOlder: false, + watcherAvailable: true, + stats: { notReady: false }, + })); + }); + + it.each([ + ...IMPORTED_HISTORY_SOURCE_DESCRIPTORS.map(({ prefix, sourceId }) => ({ + sessionId: `${prefix}e2e-production-open`, + sourceId, + })), + { + sessionId: "imported-session-e2e-collaboration", + sourceId: "collaboration_snapshot", + }, + ])( + "opens $sourceId through the production foreground transport", + async ({ sessionId }) => { + const opened = await openBoundedReplaySessionForE2E(sessionId); + + expect(mocks.activate).toHaveBeenCalledTimes(1); + expect(mocks.activate).toHaveBeenCalledWith(sessionId); + expect(mocks.open).toHaveBeenCalledTimes(1); + expect(mocks.open).toHaveBeenCalledWith(opened?.lease); + expect(mocks.startTurnEpisode).toHaveBeenCalledTimes(1); + expect(mocks.startTurnEpisode).toHaveBeenCalledWith( + sessionId, + "generation-1" + ); + expect(mocks.mergeTurnWindow).toHaveBeenCalledTimes(1); + expect(mocks.deactivate).not.toHaveBeenCalled(); + } + ); + + it("does not activate bounded replay for a native SDE session", async () => { + await expect( + openBoundedReplaySessionForE2E("sdeagent-native-e2e") + ).resolves.toBeNull(); + expect(mocks.activate).not.toHaveBeenCalled(); + expect(mocks.open).not.toHaveBeenCalled(); + expect(mocks.mergeTurnWindow).not.toHaveBeenCalled(); + }); + + it("keeps the managed CLI live EventStore projection while its binding is pending", async () => { + const liveEvents = [{ id: "live-user" }]; + mocks.open.mockResolvedValueOnce({ + cursor: { + sourceId: "managed_cli", + sessionId: "cliagent-pending", + generation: "pending", + revision: 0, + throughSequence: 0, + }, + events: [], + turnHeaders: [], + totalTurnCount: 0, + hasOlder: false, + watcherAvailable: true, + stats: { notReady: true }, + }); + mocks.getEvents.mockResolvedValueOnce(liveEvents); + + const opened = await openBoundedReplaySessionForE2E("cliagent-pending"); + + expect(opened?.events).toBe(liveEvents); + expect(mocks.getEvents).toHaveBeenCalledWith("cliagent-pending"); + }); + + it("releases the foreground lease when production open fails", async () => { + mocks.open.mockRejectedValueOnce(new Error("open failed")); + + await expect( + openBoundedReplaySessionForE2E("codexapp-open-failure") + ).rejects.toThrow("open failed"); + expect(mocks.deactivate).toHaveBeenCalledTimes(1); + expect(mocks.deactivate).toHaveBeenCalledWith( + expect.objectContaining({ sessionId: "codexapp-open-failure" }) + ); + }); +}); diff --git a/src/app/root/e2e/helpers/sessionHelpers/inspectChatState.ts b/src/app/root/e2e/helpers/sessionHelpers/inspectChatState.ts index 62726bf60..aee7e2b41 100644 --- a/src/app/root/e2e/helpers/sessionHelpers/inspectChatState.ts +++ b/src/app/root/e2e/helpers/sessionHelpers/inspectChatState.ts @@ -11,7 +11,13 @@ import { sortedEventsAtom, streamingDeltaContentAtom, } from "@src/engines/SessionCore/core/atoms/events"; +import { eventStoreProxy } from "@src/engines/SessionCore/core/store/EventStoreProxy"; import { chatEventsAtom } from "@src/engines/SessionCore/derived/chatEvents"; +import { getExternalReplayDebugStateForTest } from "@src/engines/SessionCore/sync/externalReplayTransport"; +import { + getAnchoredExternalReplayTurnIndices, + getExternalReplayTurnDebugStateForTest, +} from "@src/engines/SessionCore/sync/externalReplayTurnState"; import { isPendingCancelAtom, isSessionActiveAtom, @@ -19,6 +25,7 @@ import { sessionRuntimeStatusAtom, userInitiatedCancelAtom, } from "@src/store/session/cliSessionStatusAtom"; +import { externalReplayTurnSummariesAtomFamily } from "@src/store/session/externalReplayTurnSummariesAtom"; import { fileReviewMapAtom, pendingReviewCountAtom, @@ -55,6 +62,21 @@ export function createInspectChatStateHelper(store: E2EStore) { chatPanelMaximized: boolean; snapshotEventCount: number; snapshotChatEventCount: number; + proxySnapshotEventCount: number; + proxySnapshotChatEventCount: number; + proxySnapshotChatEventIds: string[]; + externalReplayTurnSummaryCount: number; + externalReplayTurnSummarySamples: Array<{ + turnIndex: number; + turnId: string; + renderedUserEventId: string | null; + userPreview: string; + }>; + externalReplayCompactTurnIndices: number[]; + externalReplayResidentTurnIndices: number[]; + externalReplayAnchoredTurnIndices: number[]; + externalReplayTransport: Json | null; + externalReplayTurnState: Json | null; chatEventCount: number; chatEventIds: string[]; runtimeStatus: string; @@ -155,6 +177,39 @@ export function createInspectChatStateHelper(store: E2EStore) { .filter((message) => message.priority === "now") .map(serializeQueuedMessage); const activeSessionId = store.get(activeSessionIdAtom); + const proxySnapshot = activeSessionId + ? eventStoreProxy.getLatestSessionSnapshot(activeSessionId) + : null; + const externalReplayTurnSummaries = activeSessionId + ? store.get(externalReplayTurnSummariesAtomFamily(activeSessionId)) + : []; + const externalReplayCompactTurnIndices = Object.keys( + externalReplayTurnSummaries + ) + .map(Number) + .filter(Number.isSafeInteger) + .sort((left, right) => left - right); + const externalReplayResidentTurnIndices = + externalReplayCompactTurnIndices.filter( + (turnIndex) => + externalReplayTurnSummaries[turnIndex]?.renderedUserEventId !== null + ); + const externalReplayAnchoredTurnIndices = [ + ...getAnchoredExternalReplayTurnIndices(externalReplayTurnSummaries), + ].sort((left, right) => left - right); + const summarySampleIndices = Array.from( + new Set( + [ + 0, + 1, + externalReplayTurnSummaries.length - 3, + externalReplayTurnSummaries.length - 2, + externalReplayTurnSummaries.length - 1, + ].filter( + (index) => index >= 0 && index < externalReplayTurnSummaries.length + ) + ) + ); const activeSession = activeSessionId ? (store .get(sessionsAtom) @@ -209,6 +264,33 @@ export function createInspectChatStateHelper(store: E2EStore) { chatPanelMaximized: store.get(chatPanelMaximizedAtom), snapshotEventCount: snapshot?.eventCount ?? 0, snapshotChatEventCount: snapshot?.chatEvents.length ?? 0, + proxySnapshotEventCount: proxySnapshot?.eventCount ?? 0, + proxySnapshotChatEventCount: proxySnapshot?.chatEvents.length ?? 0, + proxySnapshotChatEventIds: + proxySnapshot?.chatEvents.map((event) => event.id) ?? [], + externalReplayTurnSummaryCount: externalReplayTurnSummaries.length, + externalReplayTurnSummarySamples: summarySampleIndices.map( + (turnIndex) => { + const summary = externalReplayTurnSummaries[turnIndex]; + return { + turnIndex, + turnId: summary?.turnId ?? "", + renderedUserEventId: summary?.renderedUserEventId ?? null, + userPreview: summary?.userPreview ?? "", + }; + } + ), + externalReplayCompactTurnIndices, + externalReplayResidentTurnIndices, + externalReplayAnchoredTurnIndices, + externalReplayTransport: + (getExternalReplayDebugStateForTest() as unknown as Json | null) ?? + null, + externalReplayTurnState: activeSessionId + ? (getExternalReplayTurnDebugStateForTest( + activeSessionId + ) as unknown as Json) + : null, chatEventCount: chatEvents.length, chatEventIds: chatEvents.map((event) => event.id), runtimeStatus: store.get(sessionRuntimeStatusAtom), diff --git a/src/app/root/e2e/helpers/sessionHelpers/openBoundedReplaySession.ts b/src/app/root/e2e/helpers/sessionHelpers/openBoundedReplaySession.ts new file mode 100644 index 000000000..20ddfb2ac --- /dev/null +++ b/src/app/root/e2e/helpers/sessionHelpers/openBoundedReplaySession.ts @@ -0,0 +1,72 @@ +import { resolveExternalReplayTarget } from "@src/api/tauri/externalHistory/replay"; +import { eventStoreProxy } from "@src/engines/SessionCore/core/store/EventStoreProxy"; +import type { SessionEvent } from "@src/engines/SessionCore/core/types"; +import { + type ExternalReplaySessionLease, + activateExternalReplaySession, + deactivateExternalReplaySession, + openExternalReplaySession, +} from "@src/engines/SessionCore/sync/externalReplayTransport"; +import { + mergeExternalReplayTurnWindow, + startExternalReplayTurnEpisode, +} from "@src/engines/SessionCore/sync/externalReplayTurnState"; + +const OPEN_ATTEMPTS = 5; +const RETRY_DELAY_MS = 400; + +export interface OpenedE2EBoundedReplaySession { + events: SessionEvent[]; + lease: ExternalReplaySessionLease; +} + +/** + * Open the same foreground bounded-replay episode used by the product. + * + * This intentionally does not use the demand-driven prewarm RPC: the + * foreground open owns EventStore application, watcher creation and request + * epochs. The caller owns the returned lease and must deactivate it when the + * E2E session is switched, reset or fails to finish opening. + */ +export async function openBoundedReplaySessionForE2E( + sessionId: string +): Promise { + if (!resolveExternalReplayTarget(sessionId)) return null; + + const lease = activateExternalReplaySession(sessionId); + try { + for (let attempt = 1; attempt <= OPEN_ATTEMPTS; attempt++) { + const replay = await openExternalReplaySession(lease); + if (!replay) { + throw new Error( + `Bounded replay episode was superseded while opening ${sessionId}` + ); + } + + startExternalReplayTurnEpisode(sessionId, replay.cursor.generation); + mergeExternalReplayTurnWindow(sessionId, replay); + + // A newly launched managed CLI can temporarily have no native binding. + // Rust deliberately preserves its live EventStore in that state. + const events = replay.stats.notReady + ? await eventStoreProxy.getEvents(sessionId) + : replay.events; + if (!replay.stats.notReady || events.length > 0) { + return { events, lease }; + } + + if (attempt < OPEN_ATTEMPTS) { + await new Promise((resolve) => + window.setTimeout(resolve, RETRY_DELAY_MS) + ); + } else { + return { events, lease }; + } + } + + return { events: [], lease }; + } catch (error) { + deactivateExternalReplaySession(lease); + throw error; + } +} diff --git a/src/app/root/e2e/helpers/sessions.ts b/src/app/root/e2e/helpers/sessions.ts index 1a2215547..49edec719 100644 --- a/src/app/root/e2e/helpers/sessions.ts +++ b/src/app/root/e2e/helpers/sessions.ts @@ -8,7 +8,9 @@ import { rpc } from "@src/api/tauri/rpc"; import { normalizeAgentExecMode } from "@src/config/sessionCreatorConfig"; import { clearSessionAtom, + loadErrorAtom, loadSessionAtom, + loadStatusAtom, sessionIdAtom, } from "@src/engines/SessionCore"; import { resetTurnLifecycleForTests } from "@src/engines/SessionCore/control/turnLifecycle"; @@ -20,7 +22,6 @@ import { loadInitialTurnWindow, saveEvents, } from "@src/engines/SessionCore/storage/cacheAdapter"; -import { cliAdapter } from "@src/engines/SessionCore/sync/adapters"; import { getAdapterForSession } from "@src/engines/SessionCore/sync/types"; import { chatPanelTabsAtom, @@ -81,7 +82,6 @@ import { workstationLayoutAtom, } from "@src/store/workstation/tabs"; import { LAYOUT_STORAGE_KEY } from "@src/store/workstation/tabs/storage"; -import { isCliSession } from "@src/util/session/sessionDispatch"; import { asError } from "../result"; import type { E2EStore, Json, Result } from "../types"; @@ -428,7 +428,7 @@ export function createSessionHelpers(store: E2EStore) { const openSession = async ( sessionId: string - ): Promise> => { + ): Promise> => { try { if (!sessionId) { return { ok: false, error: "openSession: `sessionId` is required" }; @@ -461,66 +461,71 @@ export function createSessionHelpers(store: E2EStore) { sessionName, repoPath, }); - store.set(sessionIdAtom, sessionId); - store.set(sessionRuntimeStatusAtom, "idle"); - await eventStoreProxy.switchSession(sessionId); + const adapter = getAdapterForSession(sessionId); + if (adapter?.historyMode === "bounded-replay") { + // Exercise the same ownership graph as a real sidebar click. The + // SessionSyncProvider alone creates the foreground replay lease, + // opens the bounded window, publishes it to EventStore and releases + // it on navigation. The former helper opened a second lease before + // activating the UI session, so the product effect immediately + // superseded it and could replace a just-loaded older page. + store.set(sessionRuntimeStatusAtom, "idle"); + store.set(jumpToSessionAtom, { sessionId, sessionName, repoPath }); + await waitForSessionSurface(sessionId); - // CLI sessions: the authoritative post-restore history lives in - // `code_session_chunks`, read via the adapter. The session-persistence - // turn-index / events cache can survive a restore-to-checkpoint in an - // inconsistent state (a turn_count that no longer matches the truncated - // chunks, e.g. cursor-cli's double session_end inflating the count), so - // `loadInitialTurnWindow` may return a window that renders nothing even - // though stale rows are present. Always reload CLI history straight from - // the adapter (the chunk read can briefly lag the post-restore DB commit - // on a fresh page load, so retry a few times) rather than trusting the - // cached turn window. - let events: SessionEvent[] = []; - if (isCliSession(sessionId)) { - const ADAPTER_LOAD_ATTEMPTS = 5; - for (let attempt = 1; attempt <= ADAPTER_LOAD_ATTEMPTS; attempt++) { - const controller = new AbortController(); - const loaded = await cliAdapter.loadHistory( - sessionId, - controller.signal - ); - if (loaded.length > 0) { - events = loaded; - break; + const timeoutAt = Date.now() + 60_000; + while (Date.now() < timeoutAt) { + const status = store.get(loadStatusAtom); + if (status === "loaded" && store.get(sessionIdAtom) === sessionId) { + const events = await eventStoreProxy.getEvents(sessionId); + return { ok: true, sessionId, eventCount: events.length }; } - if (attempt < ADAPTER_LOAD_ATTEMPTS) { - await new Promise((resolve) => window.setTimeout(resolve, 400)); + if (status === "error") { + return { + ok: false, + error: + store.get(loadErrorAtom) ?? + `bounded replay failed to load ${sessionId}`, + }; } + await new Promise((resolve) => window.setTimeout(resolve, 50)); } - } else { - const initialWindow = await loadInitialTurnWindow(sessionId); - events = - initialWindow.turns.length > 0 - ? initialWindow.events - : await loadEvents(sessionId); - // Mirror the production reload guard (`handleCacheHit` in - // sessionSwitchOrchestrator): fall back to a fresh adapter load - // whenever the cached events are empty OR contain nothing renderable - // in chat, instead of only when the array is strictly length 0. - const cachedEventsRenderable = - events.length > 0 && events.some(isVisibleInChat); - if (!cachedEventsRenderable) { - const adapter = getAdapterForSession(sessionId); - if (adapter) { - const ADAPTER_LOAD_ATTEMPTS = 5; - for (let attempt = 1; attempt <= ADAPTER_LOAD_ATTEMPTS; attempt++) { - const controller = new AbortController(); - const loaded = await adapter.loadHistory( - sessionId, - controller.signal - ); - if (loaded.length > 0) { - events = loaded; - break; - } - if (attempt < ADAPTER_LOAD_ATTEMPTS) { - await new Promise((resolve) => window.setTimeout(resolve, 400)); - } + return { + ok: false, + error: `bounded replay did not finish loading ${sessionId}`, + }; + } + + store.set(sessionIdAtom, sessionId); + store.set(sessionRuntimeStatusAtom, "idle"); + await eventStoreProxy.switchSession(sessionId); + + const initialWindow = await loadInitialTurnWindow(sessionId); + let events = + initialWindow.turns.length > 0 + ? initialWindow.events + : await loadEvents(sessionId); + // Mirror the production reload guard (`handleCacheHit` in + // sessionSwitchOrchestrator): fall back to a fresh adapter load + // whenever the cached events are empty OR contain nothing renderable + // in chat, instead of only when the array is strictly length 0. + const cachedEventsRenderable = + events.length > 0 && events.some(isVisibleInChat); + if (!cachedEventsRenderable) { + if (adapter?.historyMode === "persisted-db") { + const ADAPTER_LOAD_ATTEMPTS = 5; + for (let attempt = 1; attempt <= ADAPTER_LOAD_ATTEMPTS; attempt++) { + const controller = new AbortController(); + const loaded = await adapter.loadHistory( + sessionId, + controller.signal + ); + if (loaded.length > 0) { + events = loaded; + break; + } + if (attempt < ADAPTER_LOAD_ATTEMPTS) { + await new Promise((resolve) => window.setTimeout(resolve, 400)); } } } @@ -542,7 +547,7 @@ export function createSessionHelpers(store: E2EStore) { await new Promise((resolve) => window.setTimeout(resolve, 150)); store.set(activeSessionIdAtom, sessionId); store.set(workstationActiveSessionIdAtom, sessionId); - return { ok: true, sessionId }; + return { ok: true, sessionId, eventCount: events.length }; } catch (err) { return asError(err); } diff --git a/src/app/root/e2e/types.ts b/src/app/root/e2e/types.ts index d65932b9c..7d399f5f6 100644 --- a/src/app/root/e2e/types.ts +++ b/src/app/root/e2e/types.ts @@ -488,7 +488,9 @@ export interface E2EHelpers { sessionId: string ) => Promise>; resetToNewSession: () => Promise<{ ok: true } | Err>; - openSession: (sessionId: string) => Promise>; + openSession: ( + sessionId: string + ) => Promise>; debugSessionSecuritySnapshot: ( sessionId: string ) => Promise>; diff --git a/src/components/SessionHoverCard/__tests__/useSessionTurnOverview.test.ts b/src/components/SessionHoverCard/__tests__/useSessionTurnOverview.test.ts new file mode 100644 index 000000000..0aa097a83 --- /dev/null +++ b/src/components/SessionHoverCard/__tests__/useSessionTurnOverview.test.ts @@ -0,0 +1,164 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import { + loadSessionTurnOverview, + rememberTurnOverview, +} from "../useSessionTurnOverview"; + +const mocks = vi.hoisted(() => ({ + resolveSecondary: vi.fn(), + queryWindowForTarget: vi.fn(), + foregroundOpen: vi.fn(), + loadTurnIndex: vi.fn(), +})); + +vi.mock("@src/api/tauri/externalHistory/replay", () => ({ + resolveSecondaryReplayTarget: mocks.resolveSecondary, + externalReplayQueryWindowForTarget: mocks.queryWindowForTarget, + externalReplayOpenWindow: mocks.foregroundOpen, +})); + +vi.mock("@src/engines/SessionCore/storage/cacheAdapter", () => ({ + loadTurnIndex: mocks.loadTurnIndex, +})); + +describe("external replay hover overview", () => { + beforeEach(() => { + vi.clearAllMocks(); + mocks.resolveSecondary.mockResolvedValue({ + sourceId: "codex_app", + sessionId: "codexapp-hover-pure", + }); + mocks.loadTurnIndex.mockResolvedValue([]); + }); + + it("queries one compact event without opening a foreground watcher/store", async () => { + mocks.queryWindowForTarget.mockResolvedValue({ + cursor: { + sourceId: "codex_app", + sessionId: "codexapp-hover-pure", + generation: "g1", + revision: 3, + throughSequence: 2, + }, + events: [], + turnHeaders: [], + totalEventCount: 50, + totalTurnCount: 12, + hasOlder: true, + watcherAvailable: false, + stats: { + parsedBytes: 0, + parsedRows: 0, + normalizedEvents: 0, + upsertedEvents: 0, + removedEvents: 0, + ipcBytes: 512, + notReady: false, + }, + }); + + await expect( + loadSessionTurnOverview("codexapp-hover-pure", []) + ).resolves.toEqual({ turnCount: 12, workedDurationMs: null }); + expect(mocks.queryWindowForTarget).toHaveBeenCalledWith({ + target: { + sourceId: "codex_app", + sessionId: "codexapp-hover-pure", + }, + limits: { maxTurns: 1, maxEvents: 1, maxIpcBytes: 128 * 1024 }, + }); + expect(mocks.loadTurnIndex).not.toHaveBeenCalled(); + expect(mocks.foregroundOpen).not.toHaveBeenCalled(); + }); + + it("does not let a session-only cache freeze an external turn count", async () => { + rememberTurnOverview("codexapp-hover-pure", { + turnCount: 1, + workedDurationMs: null, + }); + mocks.queryWindowForTarget.mockResolvedValue({ + cursor: { + sourceId: "codex_app", + sessionId: "codexapp-hover-pure", + generation: "g2", + revision: 9, + throughSequence: 20, + }, + events: [], + turnHeaders: [], + totalEventCount: 80, + totalTurnCount: 21, + hasOlder: true, + watcherAvailable: false, + stats: { + parsedBytes: 0, + parsedRows: 0, + normalizedEvents: 0, + upsertedEvents: 0, + removedEvents: 0, + ipcBytes: 512, + notReady: false, + }, + }); + + await expect( + loadSessionTurnOverview("codexapp-hover-pure", []) + ).resolves.toEqual({ turnCount: 21, workedDurationMs: null }); + expect(mocks.queryWindowForTarget).toHaveBeenCalledOnce(); + }); + + it("uses the verified secondary snapshot for a native collaboration fork", async () => { + const target = { + sourceId: "collaboration_snapshot", + sessionId: "agentsession-cloud-fork", + }; + mocks.resolveSecondary.mockResolvedValue(target); + mocks.queryWindowForTarget.mockResolvedValue({ + cursor: { + ...target, + generation: "fork-g1", + revision: 2, + throughSequence: 400, + }, + events: [], + turnHeaders: [], + totalEventCount: 2_000, + totalTurnCount: 350, + hasOlder: true, + watcherAvailable: false, + stats: { + parsedBytes: 0, + parsedRows: 0, + normalizedEvents: 0, + upsertedEvents: 0, + removedEvents: 0, + ipcBytes: 512, + notReady: false, + }, + }); + + await expect( + loadSessionTurnOverview("agentsession-cloud-fork", []) + ).resolves.toEqual({ turnCount: 350, workedDurationMs: null }); + expect(mocks.queryWindowForTarget).toHaveBeenCalledWith({ + target, + limits: { maxTurns: 1, maxEvents: 1, maxIpcBytes: 128 * 1024 }, + }); + expect(mocks.loadTurnIndex).not.toHaveBeenCalled(); + }); + + it("keeps ordinary native sessions on the persisted turn index", async () => { + mocks.resolveSecondary.mockResolvedValue(null); + mocks.loadTurnIndex.mockResolvedValue([ + { durationMs: 700 }, + { durationMs: 500 }, + ]); + + await expect( + loadSessionTurnOverview("sdeagent-native-hover", []) + ).resolves.toEqual({ turnCount: 2, workedDurationMs: 1_200 }); + expect(mocks.loadTurnIndex).toHaveBeenCalledWith("sdeagent-native-hover"); + expect(mocks.queryWindowForTarget).not.toHaveBeenCalled(); + }); +}); diff --git a/src/components/SessionHoverCard/useSessionTurnOverview.ts b/src/components/SessionHoverCard/useSessionTurnOverview.ts index b99bd6e61..fb15e6950 100644 --- a/src/components/SessionHoverCard/useSessionTurnOverview.ts +++ b/src/components/SessionHoverCard/useSessionTurnOverview.ts @@ -1,14 +1,18 @@ +import { type UnlistenFn, listen } from "@tauri-apps/api/event"; import { useAtomValue } from "jotai"; import { useEffect, useState } from "react"; +import type { ExternalReplayTurnSummary } from "@src/api/tauri/externalHistory"; import { - type CursorIdeTurnSummary, - cursorIdeInitialWindow, -} from "@src/api/tauri/externalHistory"; + EXTERNAL_REPLAY_INVALIDATED_EVENT, + type ExternalReplayInvalidation, + externalReplayQueryWindowForTarget, + resolveSecondaryReplayTarget, +} from "@src/api/tauri/externalHistory/replay"; +import { ExternalReplayInvalidationSchema } from "@src/api/tauri/rpc/schemas/externalReplay"; import { loadTurnIndex } from "@src/engines/SessionCore/storage/cacheAdapter"; import type { TurnSummary } from "@src/engines/SessionCore/storage/sqliteCache"; -import { cursorIdeTurnSummariesAtomFamily } from "@src/store/session/cursorIdeTurnSummariesAtom"; -import type { ActivityChunk } from "@src/types/session/session"; +import { externalReplayTurnSummariesAtomFamily } from "@src/store/session/externalReplayTurnSummariesAtom"; import { isCursorIdeSession } from "@src/util/session/sessionDispatch"; const MAX_TURN_OVERVIEW_CACHE_SIZE = 200; @@ -23,12 +27,6 @@ interface SessionTurnOverviewState { overview: SessionTurnOverview | null; } -interface SharedUnloadedTurnResult { - unloadedTurn?: { - durationMs?: unknown; - }; -} - const turnOverviewCache = new Map(); const inFlightOverviewLoads = new Map< string, @@ -46,27 +44,16 @@ export function rememberTurnOverview( turnOverviewCache.set(sessionId, overview); } -function getDurationFromChunk(chunk: ActivityChunk): number { - const result = chunk.result as SharedUnloadedTurnResult; - const durationMs = result.unloadedTurn?.durationMs; - return typeof durationMs === "number" && Number.isFinite(durationMs) - ? Math.max(0, durationMs) - : 0; -} - -function summarizeCursorIdeTurns( - turns: CursorIdeTurnSummary[] +function summarizeExternalReplayTurns( + turns: ExternalReplayTurnSummary[] ): SessionTurnOverview | null { if (turns.length === 0) return null; - const workedDurationMs = turns.reduce((total, turn) => { - const durationMs = turn.durationMs; - return typeof durationMs === "number" && Number.isFinite(durationMs) - ? total + Math.max(0, durationMs) - : total; - }, 0); return { turnCount: turns.length, - workedDurationMs: workedDurationMs > 0 ? workedDurationMs : null, + // Bounded replay deliberately does not materialize every turn header in + // the renderer. A partial duration sum would look authoritative but be + // wrong, so wait for a backend aggregate before displaying it. + workedDurationMs: null, }; } @@ -84,34 +71,40 @@ function summarizeIndexedTurns(turns: TurnSummary[]): SessionTurnOverview { }; } -async function loadSessionTurnOverview( +export async function loadSessionTurnOverview( sessionId: string, - cursorIdeTurnSummaries: CursorIdeTurnSummary[] + externalReplayTurnSummaries: ExternalReplayTurnSummary[] ): Promise { - const cachedOverview = turnOverviewCache.get(sessionId); - if (cachedOverview) return cachedOverview; - if (isCursorIdeSession(sessionId)) { - const summaryOverview = summarizeCursorIdeTurns(cursorIdeTurnSummaries); + const summaryOverview = summarizeExternalReplayTurns( + externalReplayTurnSummaries + ); if (summaryOverview) return summaryOverview; + } - const initialWindow = await cursorIdeInitialWindow({ - sessionId, - recentLimit: 1, + const replayTarget = await resolveSecondaryReplayTarget(sessionId); + if (replayTarget) { + // Hover for every external/managed CLI reads only the compact replay + // count. Verified collaboration forks use the same secondary replay + // capability so their inherited prefix never rebuilds/returns the full + // native turn index just to render a card count. + const window = await externalReplayQueryWindowForTarget({ + target: replayTarget, + limits: { + maxTurns: 1, + maxEvents: 1, + maxIpcBytes: 128 * 1024, + }, }); - const initialSummaryOverview = summarizeCursorIdeTurns(initialWindow.turns); - if (initialSummaryOverview) return initialSummaryOverview; - - const workedDurationMs = initialWindow.chunks.reduce( - (total, chunk) => total + getDurationFromChunk(chunk), - 0 - ); return { - turnCount: initialWindow.userBubbleCount, - workedDurationMs: workedDurationMs > 0 ? workedDurationMs : null, + turnCount: window.totalTurnCount, + workedDurationMs: null, }; } + const cachedOverview = turnOverviewCache.get(sessionId); + if (cachedOverview) return cachedOverview; + const turns = await loadTurnIndex(sessionId); if (turns.length === 0) return null; return summarizeIndexedTurns(turns); @@ -119,16 +112,18 @@ async function loadSessionTurnOverview( function loadSessionTurnOverviewCoalesced( sessionId: string, - cursorIdeTurnSummaries: CursorIdeTurnSummary[] + externalReplayTurnSummaries: ExternalReplayTurnSummary[] ): Promise { const inFlight = inFlightOverviewLoads.get(sessionId); if (inFlight) return inFlight; const work = loadSessionTurnOverview( sessionId, - cursorIdeTurnSummaries + externalReplayTurnSummaries ).finally(() => { - inFlightOverviewLoads.delete(sessionId); + if (inFlightOverviewLoads.get(sessionId) === work) { + inFlightOverviewLoads.delete(sessionId); + } }); inFlightOverviewLoads.set(sessionId, work); return work; @@ -137,8 +132,8 @@ function loadSessionTurnOverviewCoalesced( export function useSessionTurnOverview( sessionId: string ): SessionTurnOverview | null { - const cursorIdeTurnSummaries = useAtomValue( - cursorIdeTurnSummariesAtomFamily(sessionId) + const externalReplayTurnSummaries = useAtomValue( + externalReplayTurnSummariesAtomFamily(sessionId) ); const [overviewState, setOverviewState] = useState( () => ({ @@ -146,13 +141,47 @@ export function useSessionTurnOverview( overview: turnOverviewCache.get(sessionId) ?? null, }) ); + const [replayInvalidation, setReplayInvalidation] = useState(0); + + useEffect(() => { + let disposed = false; + let unlisten: UnlistenFn | null = null; + void (async () => { + const replayTarget = await resolveSecondaryReplayTarget(sessionId); + if (disposed || !replayTarget) return; + const release = await listen( + EXTERNAL_REPLAY_INVALIDATED_EVENT, + (event) => { + const parsed = ExternalReplayInvalidationSchema.safeParse( + event.payload + ); + if (!parsed.success || parsed.data.sessionId !== sessionId) return; + turnOverviewCache.delete(sessionId); + // Do not let an old compact query coalesce the refresh. Its + // component effect is cancelled below, and its guarded finally + // cannot delete the replacement request from the map. + inFlightOverviewLoads.delete(sessionId); + setReplayInvalidation((value) => value + 1); + } + ); + if (disposed) release(); + else unlisten = release; + })().catch(() => { + // Query-on-mount still provides fresh compact data when the desktop + // event bridge is unavailable (for example in web-only tests). + }); + return () => { + disposed = true; + unlisten?.(); + }; + }, [sessionId]); useEffect(() => { let cancelled = false; void loadSessionTurnOverviewCoalesced( sessionId, - cursorIdeTurnSummaries + externalReplayTurnSummaries ).then((nextOverview) => { if (cancelled) return; if (nextOverview) rememberTurnOverview(sessionId, nextOverview); @@ -162,7 +191,7 @@ export function useSessionTurnOverview( return () => { cancelled = true; }; - }, [cursorIdeTurnSummaries, sessionId]); + }, [externalReplayTurnSummaries, replayInvalidation, sessionId]); if (overviewState.sessionId === sessionId) return overviewState.overview; return turnOverviewCache.get(sessionId) ?? null; diff --git a/src/config/sessionAgentGroups.ts b/src/config/sessionAgentGroups.ts index d2e6e6780..764ec8006 100644 --- a/src/config/sessionAgentGroups.ts +++ b/src/config/sessionAgentGroups.ts @@ -40,6 +40,7 @@ export const SESSION_GROUP_ORDER: readonly SessionGroupKey[] = [ RUST_AGENT_TYPE.OS, RUST_AGENT_TYPE.SDE, RUST_AGENT_TYPE.WINGMAN, + RUST_AGENT_TYPE.CUSTOM, "human", "cli", ...IMPORTED_HISTORY_SOURCES.map((source) => source.listCategory), diff --git a/src/engines/ChatPanel/ChatHistory/TEST_CASES.md b/src/engines/ChatPanel/ChatHistory/TEST_CASES.md index d5056451c..fb62fded5 100644 --- a/src/engines/ChatPanel/ChatHistory/TEST_CASES.md +++ b/src/engines/ChatPanel/ChatHistory/TEST_CASES.md @@ -20,7 +20,7 @@ | 7 | Search bar opened; type a query. | `ChatSearchBar` highlights matching turns; non-matching items dimmed or filtered. | | 8 | Revert button clicked on a turn. | `RevertConfirmDialog` opens; confirming reverts session state. | | 9 | Agent is planning; planning indicator shown. | `usePlanningIndicator` returns `true`; planning spinner/indicator visible. | -| 10 | Cursor IDE session with turn summaries. | `cursorIdeTurnSummariesAtomFamily` data renders inline on matching turns. | +| 10 | External replay session with turn summaries. | `externalReplayTurnSummariesAtomFamily` data renders inline on matching turns. | | 11 | Disable pagination; open a long conversation. | A right-side conversation minimap shows at most 20 percentage-sampled markers; hovering previews a turn and clicking scrolls to it. | | 12 | Open a session containing several completed rounds. | Each round's final assistant message shows a visible timestamp and Copy button directly below the message; earlier assistant messages in the same round do not. | diff --git a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryList.tsx b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryList.tsx index 76580a0cb..f2180f436 100644 --- a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryList.tsx +++ b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryList.tsx @@ -54,6 +54,15 @@ export type { ChatHistoryListHandle } from "./ChatHistoryListTypes"; export { resolveActiveGroupPinState, resolveVisibleGroupIndices }; const STATIC_RENDER_ITEM_LIMIT = 24; +const PROGRAMMATIC_NAVIGATION_SIGNAL_MS = 500; +const HISTORY_BACKFILL_MIN_RUNWAY_PX = 32; + +export function isWithinHistoryBackfillRunway( + scrollTop: number, + clientHeight: number +): boolean { + return scrollTop <= Math.max(HISTORY_BACKFILL_MIN_RUNWAY_PX, clientHeight); +} // memo: parent (`ChatHistory/index.tsx`) re-renders on every chat event // (atom subscriptions, useDeferredValue ticks). All props are either @@ -64,7 +73,9 @@ const STATIC_RENDER_ITEM_LIMIT = 24; const ChatHistoryList: React.FC = memo( ({ flatItems, + groupKeys, groupCounts, + replayTurnIndices, turnIds, totalFlatItems, lastAssistantFlatIndexPerItem, @@ -79,6 +90,7 @@ const ChatHistoryList: React.FC = memo( getIsWpGeneWorking, getIsExploring, renderGroupHeader: renderGroupHeaderProp, + onAtStartStateChange, onAtBottomStateChange, onRangeChanged, onActiveGroupIndexChange, @@ -128,6 +140,21 @@ const ChatHistoryList: React.FC = memo( return group; }); }, [effectiveGroupCounts]); + const virtualGroupKeys = useMemo( + () => + virtualGroups.map((group) => { + const tailItem = + flatItems[group.startFlatIndex + Math.max(0, group.itemCount - 1)]; + return ( + groupKeys[group.groupIndex] ?? + turnIds[group.groupIndex] ?? + tailItem?.event?.id ?? + tailItem?.chunk_id ?? + `chat-group-${group.groupIndex}` + ); + }), + [flatItems, groupKeys, turnIds, virtualGroups] + ); const flatIndexToGroupIndex = useMemo(() => { const indexes: number[] = []; for (const group of virtualGroups) { @@ -143,29 +170,17 @@ const ChatHistoryList: React.FC = memo( getScrollElement: () => virtualScrollerRef.current, estimateSize: () => 360, overscan: 4, - getItemKey: (index) => { - const group = virtualGroups[index]; - if (!group) return `chat-group-${index}:0`; - const itemKeys = flatItems - .slice(group.startFlatIndex, group.startFlatIndex + group.itemCount) - .map((item) => { - const event = item.event; - const displayTextLength = event?.displayText?.length ?? 0; - return [ - item.chunk_id, - event?.displayStatus ?? "", - event?.activityStatus ?? "", - displayTextLength, - ].join(":"); - }) - .join("|"); - return `${index}:${group.itemCount}:${itemKeys}`; - }, + // A prepend changes every numerical group index. Provider/user ids (or + // the tail event for an anchorless partial turn) remain stable, allowing + // TanStack Virtual to reuse measurements instead of remounting every + // resident row after each bounded replay window. + getItemKey: (index) => virtualGroupKeys[index] ?? `chat-group-${index}`, }); const virtualItems = virtualizer.getVirtualItems(); const rowResizeObserverRef = useRef(null); const measuredRowHeightsRef = useRef(new WeakMap()); const observedRowsRef = useRef(new Set()); + const programmaticNavigationAtRef = useRef(0); const measureVirtualRow = useCallback( (node: HTMLDivElement | null) => { if (!node) return; @@ -222,13 +237,18 @@ const ChatHistoryList: React.FC = memo( () => ({ scrollToIndex: ({ index, behavior = "auto", align = "center" }) => { const groupIndex = flatIndexToGroupIndex[index] ?? 0; + programmaticNavigationAtRef.current = performance.now(); virtualizer.scrollToIndex(groupIndex, { align, behavior }); }, - scrollToGroup: ({ groupIndex, behavior = "smooth" }) => { + scrollToGroup: ({ groupIndex, behavior = "smooth", turnId = null }) => { + const currentTurnIndex = turnId ? turnIds.indexOf(turnId) : -1; + const resolvedGroupIndex = + currentTurnIndex >= 0 ? currentTurnIndex : groupIndex; const boundedGroupIndex = Math.max( 0, - Math.min(groupIndex, virtualGroups.length - 1) + Math.min(resolvedGroupIndex, virtualGroups.length - 1) ); + programmaticNavigationAtRef.current = performance.now(); const staticScrollRoot = staticScrollerRef?.current; const staticGroup = staticScrollRoot?.querySelector( `[data-chat-group-index="${boundedGroupIndex}"]` @@ -251,6 +271,7 @@ const ChatHistoryList: React.FC = memo( [ flatIndexToGroupIndex, staticScrollerRef, + turnIds, virtualGroups.length, virtualizer, ] @@ -385,13 +406,52 @@ const ChatHistoryList: React.FC = memo( onActiveGroupIndexChange, }); + useEffect(() => { + const frame = window.requestAnimationFrame(() => { + const scrollRoot = useStaticRendering + ? staticScrollerRef?.current + : virtualScrollerRef.current; + if (scrollRoot) { + onAtStartStateChange( + isWithinHistoryBackfillRunway( + scrollRoot.scrollTop, + scrollRoot.clientHeight + ), + scrollRoot.scrollHeight > scrollRoot.clientHeight + 1, + "layout" + ); + } + }); + return () => window.cancelAnimationFrame(frame); + }, [ + onAtStartStateChange, + staticScrollerRef, + useStaticRendering, + virtualListDataKey, + virtualScrollerRef, + ]); + if (useStaticRendering) { return (
{ const element = event.currentTarget; + const startSignalSource = + performance.now() - programmaticNavigationAtRef.current < + PROGRAMMATIC_NAVIGATION_SIGNAL_MS + ? "programmatic" + : "scroll"; + onAtStartStateChange( + isWithinHistoryBackfillRunway( + element.scrollTop, + element.clientHeight + ), + element.scrollHeight > element.clientHeight + 1, + startSignalSource + ); onAtBottomStateChange( isScrolledToContentBottom({ element, @@ -401,6 +461,36 @@ const ChatHistoryList: React.FC = memo( ); scheduleReportActiveGroupIndex(element); }} + onWheel={(event) => { + const element = event.currentTarget; + if (event.deltaY >= 0) { + if (event.deltaY > 0) { + onAtStartStateChange( + false, + element.scrollHeight > element.clientHeight + 1, + "wheel" + ); + } + return; + } + if ( + !isWithinHistoryBackfillRunway( + element.scrollTop, + element.clientHeight + ) + ) { + return; + } + // A wheel gesture at the physical top does not produce another + // `scroll` event. Surface the user's continued scroll-back intent + // so a request that overlapped the previous bounded page can queue + // exactly one successor instead of stalling until a manual nudge. + onAtStartStateChange( + true, + element.scrollHeight > element.clientHeight + 1, + "wheel" + ); + }} >
= memo( key={groupKey} className="relative" data-chat-group-index={groupIndex} + data-chat-group-key={groupKeys[groupIndex] ?? undefined} + data-replay-turn-index={ + replayTurnIndices[groupIndex] ?? undefined + } + data-chat-turn-id={turnIds[groupIndex] ?? undefined} >
@@ -468,9 +563,23 @@ const ChatHistoryList: React.FC = memo( ref={(node) => { virtualScrollerRef.current = node; }} + data-testid="chat-history-scroll-root" className="h-full w-full overflow-y-auto overscroll-contain pt-2 scrollbar-hide" onScroll={(event) => { const element = event.currentTarget; + const startSignalSource = + performance.now() - programmaticNavigationAtRef.current < + PROGRAMMATIC_NAVIGATION_SIGNAL_MS + ? "programmatic" + : "scroll"; + onAtStartStateChange( + isWithinHistoryBackfillRunway( + element.scrollTop, + element.clientHeight + ), + element.scrollHeight > element.clientHeight + 1, + startSignalSource + ); const isAtBottom = isScrolledToContentBottom({ element, footerSpacerHeight, @@ -480,6 +589,32 @@ const ChatHistoryList: React.FC = memo( scheduleReportActiveGroupIndex(element); if (isAtBottom) onEndReached(); }} + onWheel={(event) => { + const element = event.currentTarget; + if (event.deltaY >= 0) { + if (event.deltaY > 0) { + onAtStartStateChange( + false, + element.scrollHeight > element.clientHeight + 1, + "wheel" + ); + } + return; + } + if ( + !isWithinHistoryBackfillRunway( + element.scrollTop, + element.clientHeight + ) + ) { + return; + } + onAtStartStateChange( + true, + element.scrollHeight > element.clientHeight + 1, + "wheel" + ); + }} >
= memo( ref={measureVirtualRow} data-index={virtualItem.index} data-chat-group-index={group.groupIndex} + data-chat-group-key={groupKeys[group.groupIndex] ?? undefined} + data-replay-turn-index={ + replayTurnIndices[group.groupIndex] ?? undefined + } + data-chat-turn-id={turnIds[group.groupIndex] ?? undefined} className="absolute left-0 top-0 w-full" style={{ transform: `translateY(${virtualItem.start}px)`, @@ -507,10 +647,14 @@ const ChatHistoryList: React.FC = memo(
{Array.from({ length: group.itemCount }, (_, itemOffset) => { const flatIndex = group.startFlatIndex + itemOffset; + const item = flatItems[flatIndex]; return (
{renderGroupItem(flatIndex, group.groupIndex)}
diff --git a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListEquality.ts b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListEquality.ts index 43ee9a588..67a183774 100644 --- a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListEquality.ts +++ b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListEquality.ts @@ -175,7 +175,15 @@ export function sameChatHistoryListProps( CHAT_FOOTER_SPACER.UPDATE_THRESHOLD_PX; const checks: Array<[string, boolean]> = [ ["flatItems", sameFlatItems(previous.flatItems, next.flatItems)], + ["groupKeys", sameNullableStringArray(previous.groupKeys, next.groupKeys)], ["groupCounts", sameNumberArray(previous.groupCounts, next.groupCounts)], + [ + "replayTurnIndices", + sameNullableNumberArray( + previous.replayTurnIndices, + next.replayTurnIndices + ), + ], ["turnIds", sameNullableStringArray(previous.turnIds, next.turnIds)], ["totalFlatItems", previous.totalFlatItems === next.totalFlatItems], [ @@ -217,6 +225,10 @@ export function sameChatHistoryListProps( "renderGroupHeader", previous.renderGroupHeader === next.renderGroupHeader, ], + [ + "onAtStartStateChange", + previous.onAtStartStateChange === next.onAtStartStateChange, + ], [ "onAtBottomStateChange", previous.onAtBottomStateChange === next.onAtBottomStateChange, diff --git a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListTypes.ts b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListTypes.ts index 94b57e78a..82d457395 100644 --- a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListTypes.ts +++ b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryListTypes.ts @@ -16,6 +16,12 @@ import type { GroupHeaderRenderPart } from "../renderers/GroupHeaderRenderer"; export type EventSummary = NonNullable; +export type ChatHistoryStartSignalSource = + | "layout" + | "programmatic" + | "scroll" + | "wheel"; + export interface ChatHistoryListHandle { scrollToIndex: (options: { index: number; @@ -25,12 +31,17 @@ export interface ChatHistoryListHandle { scrollToGroup: (options: { groupIndex: number; behavior?: ScrollBehavior; + turnId?: string | null; }) => void; } export interface ChatHistoryListProps { flatItems: OptimizedChatItem[]; + /** Stable provider/native group identity used across history prepends. */ + groupKeys: (string | null)[]; groupCounts: number[]; + /** Provider turn index, exposed read-only for rendered replay assertions. */ + replayTurnIndices: (number | null)[]; turnIds: (string | null)[]; totalFlatItems: number; lastAssistantFlatIndexPerItem: (number | null)[]; @@ -57,6 +68,11 @@ export interface ChatHistoryListProps { groupIndex: number, renderPart?: GroupHeaderRenderPart ) => React.ReactNode; + onAtStartStateChange: ( + atStart: boolean, + canScroll: boolean, + source: ChatHistoryStartSignalSource + ) => void; onAtBottomStateChange: (atBottom: boolean) => void; onRangeChanged: (range: { startIndex: number; endIndex: number }) => void; onActiveGroupIndexChange?: ( diff --git a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryView.tsx b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryView.tsx index 27c850d0f..d713c7417 100644 --- a/src/engines/ChatPanel/ChatHistory/components/ChatHistoryView.tsx +++ b/src/engines/ChatPanel/ChatHistory/components/ChatHistoryView.tsx @@ -129,6 +129,7 @@ const ChatHistoryView: React.FC = ({ currentTurnPageTimeLabel, defaultTurnCollapsed, displayFlatItems, + displayGroupKeys, displayGroupCounts, displayGroupHeaders, displayGroupMeta, @@ -175,6 +176,7 @@ const ChatHistoryView: React.FC = ({ const { conversationMinimapScrolling, footerSpacerHeight, + handleAtStartStateChange, handleChatListScrollStateChange, handleRangeChanged, handleTurnPageEndReached, @@ -207,6 +209,10 @@ const ChatHistoryView: React.FC = ({ () => isExploringRef.current ?? false, [isExploringRef] ); + const displayReplayTurnIndices = useMemo( + () => displayGroupMeta.map((meta) => meta.replayTurnIndex), + [displayGroupMeta] + ); const renderGroupHeader = useGroupHeaderRenderer({ displaySourceGroupIndices, sourceGroupCount: groupCounts.length, @@ -361,6 +367,8 @@ const ChatHistoryView: React.FC = ({ !turnPageListOpen && !agentOrgOverviewOpen && ( = ({ isScrolling={conversationMinimapScrolling} labelVariant={groupChatEnabled ? "agents" : "agent"} onNavigate={handleConversationMinimapNavigate} + onReplayNavigate={handleConversationHistorySelect} onHistoryToggle={handleConversationHistoryToggle} /> )} @@ -381,6 +390,7 @@ const ChatHistoryView: React.FC = ({ ? turnPaginationReady : pages.length > 0) && ( = ({ planningIndicatorEnabled={planningIndicatorEnabled} onPlanningIndicatorCount={handlePlanningIndicatorCount} flatItems={displayFlatItems} + groupKeys={displayGroupKeys} groupCounts={displayGroupCounts} + replayTurnIndices={displayReplayTurnIndices} turnIds={displayTurnIds} totalFlatItems={displayTotalFlatItems} lastAssistantFlatIndexPerItem={ @@ -462,6 +474,7 @@ const ChatHistoryView: React.FC = ({ ? renderNoGroupHeader : renderGroupHeader } + onAtStartStateChange={handleAtStartStateChange} onAtBottomStateChange={handleChatListScrollStateChange} onRangeChanged={handleRangeChanged} onActiveGroupIndexChange={handleActiveGroupIndexChange} diff --git a/src/engines/ChatPanel/ChatHistory/components/ConversationMinimap.tsx b/src/engines/ChatPanel/ChatHistory/components/ConversationMinimap.tsx index e4882fccd..8090d7364 100644 --- a/src/engines/ChatPanel/ChatHistory/components/ConversationMinimap.tsx +++ b/src/engines/ChatPanel/ChatHistory/components/ConversationMinimap.tsx @@ -10,6 +10,8 @@ import { TabBarTrailingIconButton } from "@src/modules/WorkStation/shared"; import { isAssistantMessageEvent } from "../chatItemPipeline/dedup"; import type { OptimizedChatItem } from "../chatItemPipeline/types"; import type { ChatGroupMeta } from "../hooks/useChatGroups"; +import type { UseChatTurnPaginationReturn } from "../hooks/useChatTurnPagination"; +import { useVisibleExternalReplayTurnMetadata } from "../hooks/useVisibleExternalReplayTurnMetadata"; import { getRoundPreviewText } from "../utils/turnPageFormatting"; import { getTurnTimingLabels } from "../utils/turnTimingFormatting"; @@ -25,6 +27,26 @@ export function getConversationPreviewPositionClass( return "right-full mr-3 @[640px]/chatbody:mr-1"; } +export function shouldShowFloatingConversationMinimap({ + isAtBottom, + isPointerOver, + isScrolling, + previewMarkerIndex, +}: { + isAtBottom: boolean; + isPointerOver: boolean; + isScrolling: boolean; + previewMarkerIndex: number | null; +}): boolean { + // On narrow panes the navigator may rest while the user is at the newest + // content, but it must remain available anywhere in historical content. + // Otherwise the 1.2 s scroll-idle timer removes the only bounded jump path + // back to newer provider Rounds. + return ( + !isAtBottom || isScrolling || isPointerOver || previewMarkerIndex !== null + ); +} + export function sampleConversationGroupIndices( groupIndices: readonly number[], maxMarkers = MAX_CONVERSATION_MINIMAP_MARKERS @@ -40,6 +62,48 @@ export function sampleConversationGroupIndices( }); } +export function sampleConversationReplayTurnIndices( + totalTurnCount: number, + maxMarkers = MAX_CONVERSATION_MINIMAP_MARKERS +): number[] { + if (totalTurnCount <= 0 || maxMarkers <= 0) return []; + if (totalTurnCount <= maxMarkers) { + return Array.from({ length: totalTurnCount }, (_, turnIndex) => turnIndex); + } + if (maxMarkers === 1) return [totalTurnCount - 1]; + + const lastTurnIndex = totalTurnCount - 1; + return Array.from({ length: maxMarkers }, (_, markerIndex) => + Math.round((markerIndex / (maxMarkers - 1)) * lastTurnIndex) + ); +} + +export function resolveActiveConversationReplayTurnIndex( + activeTurnIndex: number, + totalTurnCount: number, + isAtBottom: boolean +): number | null { + if (totalTurnCount <= 0) return null; + if (isAtBottom) return totalTurnCount - 1; + return Math.min(Math.max(0, activeTurnIndex), totalTurnCount - 1); +} + +export function getAdjacentConversationReplayTurnIndex( + activeTurnIndex: number, + totalTurnCount: number, + direction: -1 | 1, + isAtBottom: boolean +): number | null { + const current = resolveActiveConversationReplayTurnIndex( + activeTurnIndex, + totalTurnCount, + isAtBottom + ); + if (current === null) return null; + const adjacent = current + direction; + return adjacent >= 0 && adjacent < totalTurnCount ? adjacent : null; +} + export function findNearestConversationMarker( markerGroupIndices: readonly number[], activeGroupIndex: number @@ -99,6 +163,28 @@ export function getNavigableConversationGroupIndices( ); } +export function getConversationTurnPosition( + groupIndex: number, + navigableGroupIndices: readonly number[], + meta: ChatGroupMeta | undefined +): { current: number; total: number } { + if ( + meta?.replayTurnIndex !== null && + meta?.replayTurnIndex !== undefined && + meta.replayTotalTurnCount !== null && + meta.replayTotalTurnCount !== undefined + ) { + return { + current: meta.replayTurnIndex + 1, + total: meta.replayTotalTurnCount, + }; + } + return { + current: Math.max(1, navigableGroupIndices.indexOf(groupIndex) + 1), + total: navigableGroupIndices.length, + }; +} + export function getAdjacentConversationGroupIndex( groupIndices: readonly number[], activeGroupIndex: number, @@ -157,6 +243,8 @@ function buildAssistantPreviews( } interface ConversationMinimapProps { + sessionId: string | null; + pages: UseChatTurnPaginationReturn["pages"]; groupHeaders: readonly (OptimizedChatItem | null)[]; groupMeta: readonly ChatGroupMeta[]; groupCounts: readonly number[]; @@ -168,11 +256,14 @@ interface ConversationMinimapProps { isScrolling: boolean; labelVariant?: "agent" | "agents"; onNavigate: (groupIndex: number) => void; + onReplayNavigate: (turnIndex: number) => void; onHistoryToggle: () => void; } const ConversationMinimap: React.FC = memo( ({ + sessionId, + pages, groupHeaders, groupMeta, groupCounts, @@ -184,11 +275,12 @@ const ConversationMinimap: React.FC = memo( isScrolling, labelVariant = "agent", onNavigate, + onReplayNavigate, onHistoryToggle, }) => { const { t } = useTranslation(); const tooltipId = useId(); - const [previewGroupIndex, setPreviewGroupIndex] = useState( + const [previewMarkerIndex, setPreviewMarkerIndex] = useState( null ); const [isPointerOver, setIsPointerOver] = useState(false); @@ -200,52 +292,140 @@ const ConversationMinimap: React.FC = memo( () => sampleConversationGroupIndices(navigableGroupIndices), [navigableGroupIndices] ); + const replayTurnCount = + pages.length > 0 && pages[0]?.replayTurnSummary ? pages.length : 0; + const replayMarkerTurnIndices = useMemo( + () => sampleConversationReplayTurnIndices(replayTurnCount), + [replayTurnCount] + ); + const compactReplaySummaries = useVisibleExternalReplayTurnMetadata({ + sessionId: replayTurnCount > 0 ? sessionId : null, + pages, + visiblePageIndices: replayMarkerTurnIndices, + }); + const useReplayCatalog = replayTurnCount > 0; + const markerIndices = useReplayCatalog + ? replayMarkerTurnIndices + : markerGroupIndices; const assistantPreviews = useMemo( () => buildAssistantPreviews(flatItems, groupCounts), [flatItems, groupCounts] ); - const activeMarkerGroupIndex = resolveActiveConversationMarker( - markerGroupIndices, - activeGroupIndex, - isAtBottom - ); - const highlightedMarkerGroupIndices = useMemo( - () => - new Set( + const activeReplayTurnIndex = + groupMeta[activeGroupIndex]?.replayTurnIndex ?? replayTurnCount - 1; + const activeMarkerIndex = useReplayCatalog + ? findNearestConversationMarker( + replayMarkerTurnIndices, + resolveActiveConversationReplayTurnIndex( + activeReplayTurnIndex, + replayTurnCount, + isAtBottom + ) ?? 0 + ) + : resolveActiveConversationMarker( + markerGroupIndices, + activeGroupIndex, + isAtBottom + ); + const highlightedMarkerIndices = useMemo(() => { + if (!useReplayCatalog) { + return new Set( resolveHighlightedConversationMarkers( markerGroupIndices, visibleGroupIndices, activeGroupIndex, isAtBottom ) - ), - [activeGroupIndex, isAtBottom, markerGroupIndices, visibleGroupIndices] - ); - const previewMarkerPosition = - previewGroupIndex === null - ? -1 - : navigableGroupIndices.indexOf(previewGroupIndex); + ); + } + const visibleReplayTurnIndices = visibleGroupIndices.flatMap( + (groupIndex) => { + const turnIndex = groupMeta[groupIndex]?.replayTurnIndex; + return turnIndex === null || turnIndex === undefined + ? [] + : [turnIndex]; + } + ); + return new Set( + resolveHighlightedConversationMarkers( + replayMarkerTurnIndices, + visibleReplayTurnIndices, + activeReplayTurnIndex, + isAtBottom + ) + ); + }, [ + activeGroupIndex, + activeReplayTurnIndex, + groupMeta, + isAtBottom, + markerGroupIndices, + replayMarkerTurnIndices, + useReplayCatalog, + visibleGroupIndices, + ]); const previewSampledMarkerIndex = - previewGroupIndex === null + previewMarkerIndex === null ? -1 - : markerGroupIndices.indexOf(previewGroupIndex); + : markerIndices.indexOf(previewMarkerIndex); + const previewReplayPage = + useReplayCatalog && previewMarkerIndex !== null + ? pages[previewMarkerIndex] + : undefined; + const previewReplaySummary = previewReplayPage?.replayTurnSummary + ?.userPreview + ? previewReplayPage.replayTurnSummary + : previewMarkerIndex === null + ? undefined + : (compactReplaySummaries.get(previewMarkerIndex) ?? + previewReplayPage?.replayTurnSummary); + const previewGroupIndex = + previewMarkerIndex === null + ? null + : useReplayCatalog + ? previewReplayPage?.replayBodyLoaded + ? previewReplayPage.startGroupIndex + : null + : previewMarkerIndex; const previewHeader = previewGroupIndex === null ? null : groupHeaders[previewGroupIndex]; - const previewTitle = getUserPreview(previewHeader); + const previewTitle = useReplayCatalog + ? getRoundPreviewText(previewReplaySummary?.userPreview) + : getUserPreview(previewHeader); const previewResponse = previewGroupIndex === null ? "" : (assistantPreviews[previewGroupIndex] ?? ""); const previewMeta = previewGroupIndex === null ? undefined : groupMeta[previewGroupIndex]; + const previewTurnPosition = + previewMarkerIndex === null + ? null + : useReplayCatalog + ? { + current: previewMarkerIndex + 1, + total: replayTurnCount, + } + : getConversationTurnPosition( + previewMarkerIndex, + navigableGroupIndices, + previewMeta + ); + const previewStartedAtMs = previewReplaySummary?.startedAt + ? Date.parse(previewReplaySummary.startedAt) + : (previewMeta?.startMs ?? null); + const previewEndedAtMs = previewReplaySummary?.endedAt + ? Date.parse(previewReplaySummary.endedAt) + : (previewMeta?.endMs ?? null); const previewTiming = getTurnTimingLabels( - previewMeta?.durationMs ?? 0, - previewMeta?.startMs ?? null, - previewMeta?.endMs ?? null + previewReplaySummary?.durationMs ?? previewMeta?.durationMs ?? 0, + Number.isFinite(previewStartedAtMs) ? previewStartedAtMs : null, + Number.isFinite(previewEndedAtMs) ? previewEndedAtMs : null ); const showTiming = - previewMeta !== undefined && - (previewMeta.durationMs > 0 || previewTiming.showRange); + (previewMeta !== undefined || previewReplaySummary !== undefined) && + ((previewReplaySummary?.durationMs ?? previewMeta?.durationMs ?? 0) > 0 || + previewTiming.showRange); const durationLabel = t( labelVariant === "agents" ? "sessions:tools.turnCollapse.agentsWorkedFor" @@ -258,31 +438,52 @@ const ConversationMinimap: React.FC = memo( end: previewTiming.endClock, }) : ""; - const previewFallback = - previewMarkerPosition >= 0 - ? t("common:pagination.round", { - current: previewMarkerPosition + 1, - }) - : ""; - const showFloatingMinimap = - isScrolling || isPointerOver || previewGroupIndex !== null; + const previewFallback = previewTurnPosition + ? t("common:pagination.round", { + current: previewTurnPosition.current, + }) + : ""; + const showFloatingMinimap = shouldShowFloatingConversationMinimap({ + isAtBottom, + isPointerOver, + isScrolling, + previewMarkerIndex, + }); const previewPositionClass = getConversationPreviewPositionClass(chatPanelPosition); - const showHoverControls = isPointerOver || previewGroupIndex !== null; - const previousGroupIndex = getAdjacentConversationGroupIndex( - navigableGroupIndices, - activeGroupIndex, - -1, - isAtBottom - ); - const nextGroupIndex = getAdjacentConversationGroupIndex( - navigableGroupIndices, - activeGroupIndex, - 1, - isAtBottom - ); + const showHoverControls = isPointerOver || previewMarkerIndex !== null; + const previousMarkerIndex = useReplayCatalog + ? getAdjacentConversationReplayTurnIndex( + activeReplayTurnIndex, + replayTurnCount, + -1, + isAtBottom + ) + : getAdjacentConversationGroupIndex( + navigableGroupIndices, + activeGroupIndex, + -1, + isAtBottom + ); + const nextMarkerIndex = useReplayCatalog + ? getAdjacentConversationReplayTurnIndex( + activeReplayTurnIndex, + replayTurnCount, + 1, + isAtBottom + ) + : getAdjacentConversationGroupIndex( + navigableGroupIndices, + activeGroupIndex, + 1, + isAtBottom + ); + const navigateToMarker = (markerIndex: number): void => { + if (useReplayCatalog) onReplayNavigate(markerIndex); + else onNavigate(markerIndex); + }; - if (markerGroupIndices.length < 2) return null; + if (markerIndices.length < 2) return null; return (