feat(intake): stream session transcripts past the 512MB string ceiling

[drewstone]

The gap

parseCodeAgentJsonl takes the whole transcript as one string. V8 cannot build a string longer than 536,870,888 characters, so any session above that size throws ERR_STRING_TOO_LONG and no part of it can be ingested. This is total loss, not degraded reading.

The largest real Codex rollout in the local corpus is 696,493,387 bytes (664.2MB) and is invisible to every trace analysis today. The biggest sessions are exactly the long-horizon runs a 24-hour research program most needs analyzed.

OLD  parseCodeAgentJsonl(readFileSync)  file=664.2MB (696493387 bytes)
  THREW ERR_STRING_TOO_LONG: Cannot create a string longer than 0x1fffffe8 characters

The change

• streamCodeAgentJsonlFile(path) — async generator over a createReadStream, yielding one CodeAgentJsonlLine per non-blank line. Only the unterminated tail is retained, so live memory is bounded by the longest single line, not by file size.
• parseCodeAgentJsonlFile(path) — returns the existing { entries, malformedLines } shape from a file path, so a caller that holds every entry keeps working above the ceiling.
• parseCodeAgentJsonl(jsonl) keeps its exact signature and behavior.

Both paths call one readCodeAgentJsonlLine helper. Blank-line skipping, malformed-line counting, and entry parsing are shared code, not a copy, so they cannot drift apart.

Why not node:readline

node:readline breaks on a bare \r; split('\n') does not. The same file therefore reports a different malformed-line count on the two paths:

readline lines: ["a","b","{\"ok\":1}","{\"two\":2}"]
split lines:    ["a\rb","{\"ok\":1}\r","{\"two\":2}",""]

The stream breaks on \n only. A test pins this, and it fails against a readline implementation (see below).

Memory proof

The bounded consumer reads the 664.2MB rollout under a hard 128MB heap cap:

heap cap	result	entries	malformed	peak RSS	wall
default (4288MB)	ok	90,170	1	827.7MB	4.0s
256MB	ok	90,170	1	404.5MB	5.6s
128MB	ok	90,170	1	345.9MB	7.2s
64MB	OOM	—	—	—	—

The 64MB failure is the tsx + TypeScript-import baseline, not the parser: the same script OOMs identically at 64MB on a 2MB file. Peak RSS under the default heap is a lazy-GC high-water mark, not the live set — the hard-cap runs are the real bound.

parseCodeAgentJsonlFile on the same file: 90,170 entries, peak RSS 1547.2MB, 4.9s. It clears the string ceiling but still holds every entry; consume the generator directly when memory must stay flat. The doc comment says so.

Equivalence proof

Three real Codex sessions, per-entry SHA-256 over JSON.stringify (per entry rather than one giant string, so the comparison cannot hit the ceiling it is testing):

file	size	entries	malformed	digests match
rollout-2026-07-22T20-33-54	189.4MB	85,430	0	yes
rollout-2026-07-15T09-14-33	164.2MB	110,666	2	yes
rollout-2026-08-01T02-22-50	158.0MB	39,342	0	yes

file=164.2MB (172135764 bytes)
  STRING  entries=110666 malformed=2 sha256=59923fe39520d9493bd92e7a0007d47ef30b93ab80fa2487e3f1f3e267513a92
  STREAM  entries=110666 malformed=2 sha256=59923fe39520d9493bd92e7a0007d47ef30b93ab80fa2487e3f1f3e267513a92
  IDENTICAL=true

Tests

11 new cases in tests/contract-code-agent-intake.test.ts, all asserting the file path equals the string path: the committed codex-exec-0.144.1.jsonl fixture, malformed-line parity, no trailing newline, CRLF, blank and whitespace-only lines, empty file, all-malformed content, bare \r, chunk-spanning entries, and line numbering with early consumer exit.

Two are load-bearing, proven by mutation:

• Replacing the splitter with node:readline → the bare-\r case fails (expected 2 to be 1).
• Dropping the unterminated-tail retention → the chunk-boundary case fails (expected [] to have a length of 8), and so does "no trailing newline".

Verify chain

Run on this branch, rebased onto origin/main at 8093968. Every step exit 0, no pre-existing failures encountered.

step	result
pnpm lint	Checked 726 files, no fixes applied
pnpm run check:model-ids	gate valid
pnpm typecheck	clean
pnpm typecheck:examples	clean
pnpm typecheck:scripts	clean
pnpm test	5245 passed, 3 skipped, 376 files
pnpm build	complete, OpenAPI spec written
pnpm run verify:package	exit 0

Built artifact re-checked, not just the source:

parseCodeAgentJsonl:       function
parseCodeAgentJsonlFile:   function
streamCodeAgentJsonlFile:  function

[tangletools]

✅ Auto-approved drewstone PR — 6d0573fc

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-15T05:23:57Z}