feat(agent-manager): add incremental conversation tailing

docs/ai/design/2026-08-14-feature-console-incremental-tailing.md

@@ -0,0 +1,84 @@
+---
+phase: design
+title: Agent Console Incremental Conversation Tailing
+description: Shared asynchronous tail API, incremental JSONL state, and adapter-specific efficient readers
+---
+
+# Design
+
+## Architecture
+
+```mermaid
+flowchart LR
+  H[useAgentConversation] --> M[AgentManager async tail API]
+  M --> A{Adapter tail capability}
+  A -->|Codex| J[Incremental JSONL cache]
+  A -->|OpenCode| S[SQL newest-row query]
+  A -->|other formats| F[Async full-parse fallback]
+  J --> R[ConversationTailResult]
+  S --> R
+  F --> R
+  R -->|request token still current| H
+```
+
+## Public Contract
+
+Add tail options and a result envelope alongside `ConversationMessage`:
+
+- `ConversationTailOptions`: `verbose`, `limit`.
+- `ConversationReadStats`: bytes and complete records processed for this request, cache-hit flag, and reset reason.
+- `ConversationTailResult`: newest messages plus stats.
+- `AgentAdapter.getConversationTail?`: optional optimized async adapter implementation.
+- `AgentManager.getConversationTail(type, path, options)`: the single console entry point. It delegates to an optimized method or the safe async fallback and always applies the requested tail bound.
+
+The existing synchronous method stays intact for compatibility. The new manager method owns fallback selection so UI code never chooses between architectures.
+
+## Incremental JSONL Cache
+
+Each LRU entry is keyed by adapter/parser namespace, absolute session reference, verbosity, and tail limit. It stores:
+
+- file identity (`dev`, `ino`) and last observed size/mtime;
+- next byte offset;
+- raw incomplete final-record bytes;
+- adapter reducer state and bounded output messages;
+- diagnostics accumulated for the current request.
+
+Reads use `fs.promises.open`, `stat`, and positional reads. On first load, identity change, or `size < offset`, state resets and reading begins at zero. An unchanged identity/size/mtime returns cached messages without opening the data range. Complete newline-delimited records are decoded and parsed independently. Empty lines are ignored; malformed complete records increment `parseErrors`; the remaining suffix is retained until a newline arrives.
+
+The cache holds 50 sessions and refreshes LRU order on access. Eviction discards the entire state, so a later access performs a correct full rebuild.
+
+## Codex Reducer
+
+Codex retains the exact existing line-to-message conversion. Reducer state additionally tracks response-item mirror keys and mirror metadata for retained event messages:
+
+- response item first: record its key and emit it; a later mirrored event is skipped;
+- event first: emit it provisionally; a later mirrored response removes the retained event and emits the response in its actual order;
+- entries without turn IDs remain independent.
+
+The response-key set is retained for the cache lifetime because a later event can refer to an earlier response. Output messages are bounded to the requested tail, while dedup state preserves semantics across appends.
+
+## OpenCode
+
+OpenCode implements the async API directly with SQL ordering newest-first and `LIMIT`, filtering to displayable part types for non-verbose preview. Rows are reversed before mapping so visible chronological ordering matches `getConversation()`. No file-stat cache is applied to encoded database references; SQLite is the source of truth.
+
+## Safe Fallback
+
+Adapters without an optimized tail method retain exact `getConversation()` semantics. The manager defers that compatibility parse out of the initiating render/effect stack, slices only after parsing, and caches unchanged real files. Gemini, the monolithic JSON adapter, provides an optimized worker-thread implementation so its read and `JSON.parse` do not block Ink. Claude, Copilot, Grok, and Pi continue through the awaited compatibility path in this change; adopting the reusable JSONL reducer is an explicit follow-up. This is intentionally a migration bridge, not a second API.
+
+## Concurrency and UI
+
+`useAgentConversation` awaits the manager API. Every request captures a monotonically increasing token and the selected session identity. Results/errors are committed only when mounted and still current. Poll ticks do not start a second read while one is active; a later selection invalidates the previous request. Cached messages are shown immediately, the 150 ms selection debounce and 3 s polling fallback remain, and `PREVIEW_TAIL` remains 20.
+
+## Alternatives Considered
+
+- Reparse asynchronously on the main thread: rejected because `Promise`/`setImmediate` changes scheduling but CPU parsing still freezes Ink.
+- Replace every synchronous adapter method: rejected as an unnecessary breaking change for command and channel callers.
+- Put a separate JSONL cache in the hook: rejected because it duplicates adapter parsing semantics and cannot correctly preserve Codex deduplication.
+- Build adapter-specific UI readers: rejected because it creates competing architectures.
+
+## Risks and Mitigations
+
+- File rewritten in place between polls: identity and size regression trigger reset; replacement/rotation changes inode. Tests cover both reset paths.
+- Unbounded session state: LRU bounds session count and message arrays are tail-bounded; Codex mirror keys are the only file-lifetime semantic index.
+- Worker/fallback failure: return a rejected async result, preserve previously rendered messages, and surface the existing parse-error state.
+- Adapter drift: parity tests compare async tail output with existing synchronous semantics.

docs/ai/implementation/2026-08-14-feature-console-incremental-tailing.md

@@ -0,0 +1,72 @@
+---
+phase: implementation
+title: Agent Console Incremental Conversation Tailing
+description: Implementation record for async preview reads and incremental JSONL caching
+---
+
+# Implementation Record
+
+## Status
+
+Implemented and validated; publication remains.
+
+## Baseline
+
+- Fixture: Codex JSONL, 86,682,730 bytes (82.7 MiB), 468 visible messages.
+- Existing synchronous parser, five warm runs: 250.5, 261.8, 292.0, 304.4, 307.2 ms; median 292.0 ms on 2026-08-14.
+- User-provided measured hotspot: 336.6 ms for the same size class.
+
+## Intended Changed Surfaces
+
+- `packages/agent-manager`: shared tail types/cache, manager API, Codex incremental reducer, OpenCode limited reader, fallback, exports, and tests.
+- `packages/cli`: awaited hook integration, stale request rejection, cache/polling behavior, and tests.
+
+## Decisions
+
+- Preserve the synchronous adapter contract for existing callers.
+- Give the console one manager-level async API; optimized and fallback paths remain hidden below it.
+- Measure bytes and complete records processed in tests and benchmark output.
+
+## Implemented Surfaces
+
+- `AgentAdapter` exports async tail options/result/stats and an optional optimized method; `AgentManager.getConversationTail()` is the single UI entry point.
+- `JsonlConversationTailCache` performs serialized positional reads and maintains identity, byte offset, incomplete bytes, reducer state, deterministic diagnostics, and a 50-entry LRU.
+- `CodexAdapter` uses the cache and retains response-item mirror keys so mirrored event records remain deduplicated even when the pair straddles polls.
+- `OpenCodeAdapter` filters displayable parts, orders newest-first, applies SQL `LIMIT ?`, then reverses mapped results into chronological order.
+- `GeminiCliAdapter` reads and parses monolithic JSON in a worker thread and caches unchanged results.
+- `useAgentConversation` awaits the manager API, prevents overlapping reads, invalidates stale selection tokens, keeps immediate display caching, preserves the 20-message default, and retains 3-second polling.
+- A checked-in `benchmark:conversation-tail` command copies the supplied fixture to a temporary directory before appending, leaving the source untouched.
+
+## Edge Cases
+
+- Partial final JSONL records remain buffered until newline completion.
+- Malformed complete lines are counted and skipped; later records continue.
+- Inode changes, size regression, and same-size in-place rewrites reset state.
+- Missing files evict state and return a missing reset result.
+- Per-key reads serialize to protect offsets from overlapping requests.
+- LRU eviction drops complete parser state and causes a correct rebuild on return.
+
+## Adapter Support and Follow-ups
+
+- Optimized now: Codex (incremental JSONL), OpenCode (limited SQLite), Gemini (off-thread monolithic JSON).
+- Compatibility fallback now: Claude, Copilot, Grok, Pi. These preserve their existing `getConversation()` semantics and unchanged-file caching through the shared async manager API, but a changed file still receives a deferred full parse.
+- Follow-up: migrate Claude, Copilot, Grok, and Pi to `JsonlConversationTailCache` with adapter-specific reducers and parity tests. No second hook or adapter API is needed.
+
+## Benchmark
+
+82.7 MiB Codex fixture (86,682,730 bytes, 11,747 complete records, 468 visible legacy messages):
+
+- Legacy full parse: five runs 300.5, 350.9, 276.2, 313.1, 276.4 ms; median 300.5 ms. User-reported prior hotspot: 336.6 ms.
+- Incremental initial load: 241.4 ms, 86,682,730 bytes, 11,747 records.
+- One appended record: 0.246 ms, 130 bytes, 1 record.
+- Unchanged refresh: 0.029 ms, 0 bytes, 0 records, cache hit.
+
+## Validation
+
+- Focused new/changed agent-manager tests: 5 files, 104 tests passed.
+- Full agent-manager: 27 files, 526 tests passed sequentially; the default fixed 5-second print-agent integration timeout was exceeded during loaded parallel runs, so the unrelated process-inspection integration was validated separately with a 30-second allowance.
+- Full CLI after rebasing onto current `main`: 81 files, 970 tests passed.
+- Agent-manager lint: exit 0. CLI lint: exit 0 with five pre-existing unused-catch warnings outside touched files.
+- Monorepo build: all 6 projects passed.
+- Feature lint and `git diff --check`: exit 0.
+- Regression proof: removing same-size rewrite detection made its deterministic test fail with stale content; restoring it passed.

docs/ai/planning/2026-08-14-feature-console-incremental-tailing.md

@@ -0,0 +1,37 @@
+---
+phase: planning
+title: Agent Console Incremental Conversation Tailing
+description: Ordered implementation plan for async and incremental conversation preview reads
+---
+
+# Implementation Plan
+
+## 1. Baseline and contracts
+
+- [x] Confirm clean feature worktree and measure the existing Codex full parse.
+- [x] Review adapter and hook semantics and choose the shared API architecture.
+- [x] Add public tail result/options/stats types and manager delegation.
+
+## 2. Deterministic red tests
+
+- [x] Add reusable JSONL cache tests for initial bytes, append-only bytes, unchanged hits, partial records, malformed records, truncate, replacement, and LRU eviction.
+- [x] Add Codex tests for incremental append and mirrored-message deduplication.
+- [x] Add OpenCode limited-query coverage.
+- [x] Add hook tests for awaited results, stale selection rejection, 20-message slicing, unchanged cache behavior, and polling fallback.
+- [x] Add synthetic large-fixture benchmark/tests that assert processed bytes/records rather than timing.
+
+## 3. Implementation
+
+- [x] Implement the reusable async JSONL cache and Codex reducer.
+- [x] Implement manager async delegation and safe monolithic fallback.
+- [x] Implement OpenCode storage-native tail query.
+- [x] Convert `useAgentConversation` to awaited requests with stale-result protection.
+- [x] Document optimized adapters and fallback follow-ups.
+
+## 4. Validation and publication
+
+- [x] Run focused and full agent-manager and CLI tests.
+- [x] Run package/repository lint and builds.
+- [x] Run repeatable full-read versus appended-read benchmark.
+- [x] Review design alignment and final diffs.
+- [ ] Commit conventionally, rebase on `origin/main`, push `feature-console-incremental-tailing`, and open a PR targeting `main` without merging.

docs/ai/requirements/2026-08-14-feature-console-incremental-tailing.md

@@ -0,0 +1,44 @@
+---
+phase: requirements
+title: Agent Console Incremental Conversation Tailing
+description: Keep console conversation previews responsive by reading only appended session data when formats allow
+---
+
+# Requirements & Problem Understanding
+
+## Problem Statement
+
+`useAgentConversation` currently calls synchronous adapter parsers whenever a selected session file changes. A measured 86,682,730-byte (82.7 MiB) Codex JSONL session requires hundreds of milliseconds for one full reread and parse, blocking Ink input and rendering on every poll that observes a write.
+
+## Goals
+
+- Add one asynchronous conversation-tail API used by the console preview.
+- Keep the newest 20-message default and adapter-specific conversation semantics.
+- For append-friendly JSONL, cache file identity, byte offset, incomplete record bytes, parser state, and recent messages so later polls read only appended bytes.
+- Handle unchanged files, append, partial final lines, malformed complete records, truncation, replacement/rotation, missing files, and bounded cache eviction deterministically.
+- Preserve Codex response-item/event-message mirrored-message deduplication across incremental reads.
+- Let adapters provide more efficient storage-native implementations, including a limited OpenCode SQL query.
+- Keep monolithic formats and unsupported incremental formats off the Ink event loop through a safe asynchronous fallback.
+- Ignore stale asynchronous results after selection changes or overlapping polls.
+- Preserve polling when filesystem watching is unavailable or unreliable.
+
+## Constraints and Acceptance Criteria
+
+- Existing synchronous `getConversation()` behavior and callers remain compatible.
+- The async API returns read diagnostics suitable for deterministic tests and benchmarks (`bytesRead`, `recordsProcessed`, cache/reset information); tests must not depend on wall-clock thresholds.
+- A completed malformed JSONL record is skipped and counted; an incomplete final record is buffered without being reported as malformed.
+- File identity changes or size regression reset parser state and rebuild from byte zero.
+- Cache capacity is bounded and least-recently-used entries are evicted.
+- Initial parsing may read the complete file; an append-only refresh must read only the appended byte range.
+- Console state is updated only by the newest request for the currently selected agent.
+- Focused and full agent-manager and CLI tests, lint, builds, and a repeatable before/after benchmark must pass before publication.
+
+## Scope Decision
+
+The shared async API, reusable JSONL tail cache, Codex integration, OpenCode limited query, console integration, and safe fallback are required now. Additional adapters may adopt the reusable incremental reducer in follow-up changes if preserving their exact stateful semantics would make this change unsafe; they must still work through the shared async API rather than through a second UI architecture.
+
+## Non-goals
+
+- Changing visible conversation content, verbose rendering, or non-console command output.
+- Replacing adapter detection/session discovery.
+- Merging the resulting pull request.

docs/ai/testing/2026-08-14-feature-console-incremental-tailing.md

@@ -0,0 +1,41 @@
+---
+phase: testing
+title: Agent Console Incremental Conversation Tailing - Testing Strategy
+description: Deterministic correctness, regression, and benchmark coverage for async incremental preview reads
+---
+
+# Testing Strategy
+
+## TDD Coverage
+
+- [x] Initial JSONL load reports exact fixture bytes and complete record count.
+- [x] Unchanged load processes zero bytes and zero records.
+- [x] Append load processes exactly appended bytes/records.
+- [x] Partial final record is buffered and completed by a later append.
+- [x] Malformed complete record is counted and skipped without poisoning later records.
+- [x] Truncation and identity replacement rebuild state from byte zero, including same-size in-place rewrites.
+- [x] LRU eviction forces a rebuild when the evicted path returns.
+- [x] Synthetic large fixture proves append work is independent of prior file size using byte/record assertions.
+- [x] Codex async output preserves legacy roles/content/order and mirrored-message deduplication, including mirrors split across reads.
+- [x] OpenCode returns the newest requested displayable rows in chronological order using a limited query.
+- [x] Gemini preserves monolithic adapter semantics while reading and parsing off the Ink event loop.
+- [x] Hook ignores stale result/error completions after selection changes and keeps the newest 20 messages.
+- [x] Hook serves unchanged cached data and continues interval polling when no watch event exists.
+
+## Validation Commands
+
+- Focused Vitest files during each red/green/refactor cycle.
+- Full `packages/agent-manager` and `packages/cli` test suites.
+- Package lint/typecheck/build plus repository lint/build.
+- `npx ai-devkit@latest lint --feature console-incremental-tailing`.
+- Repeatable benchmark against the 82.7 MiB Codex fixture, reporting full-load and append-refresh work/time.
+
+## Evidence
+
+- Focused agent-manager feature set: 5 files / 104 tests passed.
+- Hook/cache test: 14 tests passed.
+- Full agent-manager: 27 files / 526 tests passed sequentially with a 30-second allowance for the process-inspection integration.
+- Full CLI after rebasing onto current `main`: 81 files / 970 tests passed.
+- Agent-manager and CLI lint exited 0; CLI reported five pre-existing warnings outside touched files.
+- Six-project monorepo build and feature-doc lint exited 0.
+- Benchmark append refresh processed exactly 130 bytes / 1 record versus 86,682,730 bytes / 11,747 records on initial load.

packages/agent-manager/package.json

@@ -19,6 +19,7 @@
     "test:coverage": "vitest run --coverage",
     "lint": "eslint src --ext .ts",
     "typecheck": "tsc --noEmit",
+    "benchmark:conversation-tail": "node scripts/benchmark-codex-conversation-tail.mjs",
     "clean": "rm -rf dist"
   },
   "keywords": [