From 334b6d56b6f612e8bd76be132e9150bbfe2e65ad Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Sun, 9 Aug 2026 09:53:26 -0400 Subject: [PATCH 01/14] =?UTF-8?q?=E2=9C=A8=20Address=20a=20root=20document?= =?UTF-8?q?'s=20sections=20as=20targets?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A root document now catalogs its own addressable static headings, resolves one selector to exactly one of them, and projects itself down to the preamble, each ancestor's own content, and that section's subtree before anything expands. Heading discovery parses a masked copy of the body, with the boundary scanner's top-level component spans blanked to spaces of the same length. Remark ends the HTML block it infers for a component at a blank line, so a component child's `#` line surfaces as a root heading without the mask; offsets, lines, and everything outside those spans are untouched, so the mask changes what is seen and never where anything is. Projection retains original source ranges and scans each one under its own origin instead of concatenating and rescanning, so skipped source cannot renumber what follows it and a retained element keeps its expansion ID. The exact resolved target, never the caller's glob, is what the durable root import records. A replay guard resolves the current selector against the recorded content and requires the recorded exact target, in the check phase so a completed journal cannot answer for a section it never ran. `xmd targets`, targeted `xmd run`, and the targeted workflow definition are the later layers of #412 and remain unbuilt. --- architecture.md | 37 + packages/core/mod.ts | 16 +- packages/core/src/definition.ts | 199 +++++- packages/core/src/document-targets.ts | 607 ++++++++++++++++ packages/core/src/execute.ts | 150 +++- packages/core/src/inspect.ts | 31 +- packages/core/src/root-source.ts | 83 ++- packages/core/src/scanner.ts | 31 +- .../tests/document-target-execution.test.ts | 498 ++++++++++++++ packages/core/tests/document-targets.test.ts | 646 ++++++++++++++++++ specs/executable-mdx-spec.md | 227 +++++- 11 files changed, 2473 insertions(+), 52 deletions(-) create mode 100644 packages/core/src/document-targets.ts create mode 100644 packages/core/tests/document-target-execution.test.ts create mode 100644 packages/core/tests/document-targets.test.ts diff --git a/architecture.md b/architecture.md index 294bb3ea..44f32212 100644 --- a/architecture.md +++ b/architecture.md @@ -31,6 +31,7 @@ Existing documents and code get aligned to this section retroactively. | definition base | the Git revision supplied to choose a workflow definition's pinned commit | | Repository base | the optional Git revision from which one named Workspace Repository initializes its primary checkout | | pinned commit | the commit obtained by resolving a base once; it remains the workflow run's starting repository state even as the run creates descendant commits | +| document target | an addressable static heading in a root document's own Markdown flow, named by the canonical path of heading labels that reaches it; selecting one executes the preamble, each ancestor's own content, and that heading's complete subtree | | expansion | one logical evaluation of an authored executable element within a document execution | | expansion ID | a deterministic identifier for one logical expansion; restoring or retrying that expansion preserves the ID, while a distinct evaluation requested by the document receives another | | Git capability | the contextual interface through which workflow infrastructure queries the Git repository associated with the current working directory | @@ -603,6 +604,33 @@ interpreter; graceful Worker shutdown alone is insufficient. Worker Shell exposes no native executable or host PATH and is not described as POSIX or native Bash. +## Document targets + +A root document addresses its own sections. The outline is discovered from the +document's static Markdown alone: only headings in the root flow are targets, +and a heading is addressable only when its text is statically rendered — a +heading generated inside a component, or one carrying an interpolation, has no +stable address, and neither does anything beneath it. Discovery therefore +parses a copy in which the scanner's top-level component spans are blanked, +because a Markdown parser reading raw XMD cannot tell a component's children +from the root flow. + +Selection resolves exactly once, before the document expands and before any +authored effect runs. A selector may glob, but it must name exactly one catalog +entry: naming none and naming several are both failures, and two sections that +canonicalize to the same path stay two entries so the ambiguity is reported +rather than resolved arbitrarily. + +The selector and the target it resolves to are different things, and only one +of them is identity. A selector is invocation input — it describes what a +caller asked for, and two callers may spell the same request differently. The +**exact resolved target** is what ran, so it is what a document execution +records durably, what a targeted workflow definition carries, and what a resumed +run is checked against. A caller's glob is never recorded and never re-resolved +against a newer checkout; a resumed run re-resolves the current selector against +the *recorded* content and refuses to continue unless it still names the +recorded target. + ## Expansion identity Core describes the executable element currently being expanded: @@ -639,6 +667,14 @@ JavaScript object identity. document execution receives expansion identity without installing workflow middleware. +Selecting a document target does not disturb any of this. Projection retains +the original source ranges and scans each one under its own origin, so a +retained element keeps the offset and line it was authored at, and with them +its expansion ID. Two runs of the same document under different targets +therefore agree on the ID of every element they both retain, and may share IDs +without sharing effects — run identity and workflow-definition identity are +what tell those runs apart. + ## Two layers Error handling has two layers: @@ -992,6 +1028,7 @@ Status is measured against main. | `` / `printErrors(fn)` | prints failures | built on main | | `` region `output` mode | an undecided error fails the document execution | built on main | | `Expansion` / `getExpansion()` | describes the current logical element expansion | built on main | +| document targets | catalogs a root document's addressable static headings, resolves one selector to one exact target, and projects the document to it before expansion | built on the #412 stack; `xmd targets`, targeted `xmd run`, and the targeted workflow definition are unbuilt | | `useWorkflow()` / `getWorkflowRun()` | associates one document execution with a workflow run | built on main | | `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main | | workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; public workflow execution is unbuilt | diff --git a/packages/core/mod.ts b/packages/core/mod.ts index d1c66473..332f96ac 100644 --- a/packages/core/mod.ts +++ b/packages/core/mod.ts @@ -130,8 +130,20 @@ export type { ExecutionApi, DocumentExecution, } from "./src/execute.ts"; -export { INLINE_SOURCE_PATH, inlineSource, rootSourcePath } from "./src/root-source.ts"; -export type { InlineRootDocument, RootDocumentSource } from "./src/root-source.ts"; +export { + fileSource, + formatDocumentReference, + INLINE_SOURCE_PATH, + inlineSource, + rootSourcePath, +} from "./src/root-source.ts"; +export type { + FileRootDocument, + InlineRootDocument, + RootDocumentSource, +} from "./src/root-source.ts"; +export { DocumentTargetError } from "./src/document-targets.ts"; +export type { DocumentTargetErrorKind } from "./src/document-targets.ts"; export { inspectComponent, inspectDocument } from "./src/inspect.ts"; export type { ComponentInfo, diff --git a/packages/core/src/definition.ts b/packages/core/src/definition.ts index f3796d96..f24457b8 100644 --- a/packages/core/src/definition.ts +++ b/packages/core/src/definition.ts @@ -1,8 +1,10 @@ import type { Operation } from "effection"; -import type { ComponentDefinition } from "./types.ts"; +import type { ComponentDefinition, Segment } from "./types.ts"; import { parseFrontmatter } from "./frontmatter.ts"; import { compilePropsSchema, compileReturnsSchema } from "./validate.ts"; -import { scanSegments } from "./scanner.ts"; +import { scanComponentSpans, scanSegments } from "./scanner.ts"; +import { outlineDocument, retainedRanges, selectTarget } from "./document-targets.ts"; +import type { DocumentOutline } from "./document-targets.ts"; import matter from "gray-matter"; @@ -15,56 +17,187 @@ export function isFunctionComponentPath(path: string): boolean { return path.endsWith(".ts"); } +/** A document's frontmatter data, its markdown body, and where the body sits. */ +interface ParsedSource { + data: Record; + content: string; + baseOffset: number; + baseLine: number; +} + /** - * Parse markdown source into a component definition. Execution and - * inspection share this so their frontmatter and schema behavior cannot - * drift: both compile the props and return schemas, so a malformed schema - * fails the same way whether the document runs or is only described. + * Split frontmatter from the markdown body without reading either. + * + * The markdown body is a verbatim suffix of the raw file, so the body start + * is computed by length — never by content search, which could false-match + * body text repeated inside frontmatter. The invariant check turns any + * gray-matter normalization surprise into a loud error instead of silently + * wrong source positions. */ -export function* parseMarkdownDefinition( - name: string, - path: string, - content: string, -): Operation { +function parseSource(path: string, content: string): ParsedSource { const parsed = matter(content); - const { meta, props, returns } = parseFrontmatter(parsed.data); - yield* compilePropsSchema(props); - if (returns !== undefined) { - yield* compileReturnsSchema(returns); - } - // The markdown body is a verbatim suffix of the raw file, so the body start - // is computed by length — never by content search, which could false-match - // body text repeated inside frontmatter. The invariant check turns any - // gray-matter normalization surprise into a loud error instead of silently - // wrong source positions. - const bodyStart = content.length - parsed.content.length; - if (content.slice(bodyStart) !== parsed.content) { + const baseOffset = content.length - parsed.content.length; + if (content.slice(baseOffset) !== parsed.content) { throw new Error(`frontmatter parse did not preserve the markdown body verbatim: ${path}`); } let baseLine = 1; - for (let i = 0; i < bodyStart; i++) { + for (let i = 0; i < baseOffset; i++) { if (content[i] === "\n") { baseLine++; } } - const bodySegments = scanSegments(parsed.content, { - path, - baseOffset: bodyStart, - baseLine, - }); + return { data: parsed.data, content: parsed.content, baseOffset, baseLine }; +} +/** The static heading structure a document's body offers as targets. */ +function documentOutline(path: string, content: string): DocumentOutline { + const body = parseSource(path, content).content; + return outlineDocument(body, scanComponentSpans(body)); +} + +/** + * The exact canonical target a selector names in this document's content. + * + * Synchronous and free of effects, so the resolution that decides *what* runs + * happens before anything runs — including inside the durable operation that + * records the root, and inside a replay guard reading recorded content. + */ +export function resolveDocumentTarget(path: string, content: string, selector: string): string { + return selectTarget(documentOutline(path, content), selector).target; +} + +interface CompiledFrontmatter { + meta: Record; + props: ComponentDefinition["props"]; + returns: ComponentDefinition["returns"]; +} + +function* compileFrontmatter(data: Record): Operation { + const { meta, props, returns } = parseFrontmatter(data); + yield* compilePropsSchema(props); + if (returns !== undefined) { + yield* compileReturnsSchema(returns); + } + return { meta, props, returns }; +} + +function buildDefinition( + name: string, + path: string, + frontmatter: CompiledFrontmatter, + bodySegments: Segment[], +): ComponentDefinition { // `returns` stays absent in text mode: absence is what distinguishes a text // component from one that explicitly declares a string return. const definition: ComponentDefinition = { kind: "markdown", name, path, - meta, - props, + meta: frontmatter.meta, + props: frontmatter.props, bodySegments, }; - if (returns !== undefined) { - definition.returns = returns; + if (frontmatter.returns !== undefined) { + definition.returns = frontmatter.returns; } return definition; } + +/** + * Parse markdown source into a component definition. Execution and + * inspection share this so their frontmatter and schema behavior cannot + * drift: both compile the props and return schemas, so a malformed schema + * fails the same way whether the document runs or is only described. + */ +export function* parseMarkdownDefinition( + name: string, + path: string, + content: string, +): Operation { + const body = parseSource(path, content); + const frontmatter = yield* compileFrontmatter(body.data); + return buildDefinition( + name, + path, + frontmatter, + scanSegments(body.content, { path, baseOffset: body.baseOffset, baseLine: body.baseLine }), + ); +} + +/** A root document as parsed: what it declares, and what it addresses. */ +export interface ParsedRootDocument { + definition: ComponentDefinition; + /** Canonical encoded target fragments in document order, duplicates kept. */ + targets: readonly string[]; + /** The exact canonical target selected, when one was requested. */ + target?: string; +} + +/** + * Parse a root document, projecting it to one target when a selector asks for + * one. + * + * Selection happens here, before any segment exists, so a selector that names + * nothing or names several sections fails with nothing expanded. Without a + * selector the whole body is scanned exactly as an ordinary markdown component + * is. + * + * A projection scans each retained range on its own, with the origin that range + * has in the original file, rather than scanning a concatenated string. Skipped + * source therefore cannot renumber what follows it: a retained element keeps + * the offset and line it was authored at, and with them its expansion ID. + */ +export function* parseRootMarkdownDefinition( + name: string, + path: string, + content: string, + selector?: string, +): Operation { + const body = parseSource(path, content); + const frontmatter = yield* compileFrontmatter(body.data); + const outline = outlineDocument(body.content, scanComponentSpans(body.content)); + + if (selector === undefined) { + const bodySegments = scanSegments(body.content, { + path, + baseOffset: body.baseOffset, + baseLine: body.baseLine, + }); + return { + definition: buildDefinition(name, path, frontmatter, bodySegments), + targets: outline.targets, + }; + } + + const entry = selectTarget(outline, selector); + const newlines = newlineCounts(body.content); + const bodySegments: Segment[] = []; + for (const range of retainedRanges(outline, entry)) { + bodySegments.push( + ...scanSegments(body.content.slice(range.start, range.end), { + path, + baseOffset: body.baseOffset + range.start, + baseLine: body.baseLine + newlines[range.start]!, + }), + ); + } + return { + definition: buildDefinition(name, path, frontmatter, bodySegments), + targets: outline.targets, + target: entry.target, + }; +} + +/** How many newlines precede each offset, so a retained range knows its line. */ +function newlineCounts(body: string): number[] { + const counts = new Array(body.length + 1); + let seen = 0; + for (let i = 0; i < body.length; i++) { + counts[i] = seen; + if (body[i] === "\n") { + seen++; + } + } + counts[body.length] = seen; + return counts; +} diff --git a/packages/core/src/document-targets.ts b/packages/core/src/document-targets.ts new file mode 100644 index 00000000..8b3912bb --- /dev/null +++ b/packages/core/src/document-targets.ts @@ -0,0 +1,607 @@ +/** + * Document targets (spec §5.4). + * + * A target is an addressable static heading in a root document's Markdown flow. + * Selecting one projects the document down to the preamble, the direct content + * of every ancestor needed to reach it, and its complete subtree — so a section + * of a document runs on its own without its siblings. + * + * Two properties shape everything here. + * + * The outline is discovered from *static* Markdown only. Component children can + * hold text that looks like a heading, and Remark cannot tell the difference: a + * blank line inside component children ends its HTML block and the child heading + * surfaces as a root heading. Discovery therefore parses a masked copy of the + * body, where every top-level component span is replaced by spaces of the same + * length. Offsets, lines, and everything outside those spans are untouched, so + * the mask changes what is *seen*, never where anything *is*. + * + * Projection retains original source ranges rather than a rebuilt document. Each + * retained range is scanned with its own origin, so every retained element keeps + * the offset and line it was authored at — which is what keeps expansion + * identifiers equal between a full run and a targeted one. + */ + +import { remark } from "remark"; +import { toString as mdastToString } from "mdast-util-to-string"; + +import type { ComponentSpan } from "./scanner.ts"; + +/** A half-open slice of the original document body. */ +export interface SourceRange { + readonly start: number; + readonly end: number; +} + +/** One catalog entry: an addressable heading and the path that reaches it. */ +export interface DocumentTarget { + /** The canonical encoded target fragment, without a leading `#`. */ + readonly target: string; + /** The decoded, normalized labels the fragment encodes. */ + readonly labels: readonly string[]; + /** Which heading in the outline this entry addresses. */ + readonly heading: number; +} + +interface OutlineHeading { + readonly depth: number; + readonly start: number; + readonly end: number; + readonly parent: number | undefined; + readonly addressable: boolean; + readonly label: string; +} + +/** The static heading structure of one document body, and what it addresses. */ +export interface DocumentOutline { + readonly headings: readonly OutlineHeading[]; + readonly entries: readonly DocumentTarget[]; + /** Canonical encoded fragments in source order, duplicates retained. */ + readonly targets: readonly string[]; + /** Where the preamble ends: the first outermost heading, or the body end. */ + readonly preambleEnd: number; + readonly bodyLength: number; +} + +/** Why a requested target did not resolve to exactly one catalog entry. */ +export type DocumentTargetErrorKind = "invalid-selector" | "no-match" | "multiple-matches"; + +const KIND_WORDING: ReadonlyMap = new Map([ + ["invalid-selector", "is not a valid document target selector"], + ["no-match", "matches no document target"], + ["multiple-matches", "matches more than one document target"], +]); + +/** + * A requested document target that does not name exactly one section. + * + * An ordinary invocation failure: the caller asked for something the document + * does not offer, and nothing durable or contained is involved. It is raised + * before the document expands, so a run that cannot decide what to execute + * executes nothing. + * + * Everything it carries is rebuilt and frozen here. The selector arrives from a + * command line and the catalog from a parser, and neither object belongs to a + * failure that outlives them. Every reference in the message is canonically + * encoded, so a heading holding a control character cannot reach a diagnostic + * literally. + */ +export class DocumentTargetError extends Error { + readonly kind: DocumentTargetErrorKind; + /** The selector fragment as it was requested, still encoded. */ + readonly selector: string; + /** Canonical encoded targets the selector matched; empty unless ambiguous. */ + readonly matches: readonly string[]; + /** Every canonical encoded target the document offers. */ + readonly available: readonly string[]; + + constructor( + kind: DocumentTargetErrorKind, + selector: string, + matches: readonly string[], + available: readonly string[], + ) { + const listed = kind === "multiple-matches" ? matches : available; + const heading = kind === "multiple-matches" ? "Matched targets:" : "Available targets:"; + super( + `${JSON.stringify(selector)} ${KIND_WORDING.get(kind)}.\n` + + (listed.length === 0 + ? "The document has no targets." + : `${heading}\n${listed.map((target) => ` ${target}`).join("\n")}`), + ); + this.name = "DocumentTargetError"; + this.kind = kind; + this.selector = selector; + this.matches = Object.freeze([...matches]); + this.available = Object.freeze([...available]); + } +} + +const UNRESERVED = /^[A-Za-z0-9\-._~]$/; +const HEX = /^[0-9A-Fa-f]$/; + +const ENCODER = new TextEncoder(); + +function encodeCharacter(character: string): string { + let encoded = ""; + for (const byte of ENCODER.encode(character)) { + encoded += `%${byte.toString(16).toUpperCase().padStart(2, "0")}`; + } + return encoded; +} + +/** + * Percent-encode one canonical label. Everything outside RFC 3986's unreserved + * set is escaped, so `/`, `*`, `#`, and `%` inside a heading cannot be read as + * hierarchy or operator syntax. + */ +export function encodeTargetLabel(label: string): string { + let encoded = ""; + for (const character of label) { + encoded += UNRESERVED.test(character) ? character : encodeCharacter(character); + } + return encoded; +} + +/** + * Percent-encode a decoded filesystem path. Separators survive as raw `/`; a + * `/` that is part of a filename cannot be told apart from one afterwards, so + * this is a formatter for paths the caller already holds, not a round trip. + */ +export function encodeDocumentPath(path: string): string { + let encoded = ""; + for (const character of path) { + encoded += + character === "/" || UNRESERVED.test(character) ? character : encodeCharacter(character); + } + return encoded; +} + +/** + * Decode one percent-encoded chunk, or `undefined` when it is not decodable. + * + * Malformed escapes, byte sequences that are not UTF-8, and NUL are all + * refused rather than repaired: a selector that cannot be read exactly is not a + * selector this can match against. `+` is an ordinary character — this is URI + * path syntax, not a form encoding. + */ +export function decodePercentEncoded(text: string): string | undefined { + const characters = Array.from(text); + const bytes: number[] = []; + for (let index = 0; index < characters.length; index++) { + const character = characters[index]!; + if (character !== "%") { + for (const byte of ENCODER.encode(character)) { + bytes.push(byte); + } + continue; + } + const high = characters[index + 1]; + const low = characters[index + 2]; + if (high === undefined || low === undefined || !HEX.test(high) || !HEX.test(low)) { + return undefined; + } + bytes.push(Number.parseInt(`${high}${low}`, 16)); + index += 2; + } + try { + const decoded = new TextDecoder("utf-8", { fatal: true }).decode(new Uint8Array(bytes)); + return decoded.includes("\u0000") ? undefined : decoded; + } catch { + return undefined; + } +} + +/** + * The canonical form of rendered heading text: NFC, every run of Unicode + * whitespace collapsed to one ASCII space, trimmed, case preserved. + */ +export function normalizeLabel(text: string): string { + return text.normalize("NFC").replace(/\s+/gu, " ").trim(); +} + +/** + * Whether a fragment is already an exact canonical target: raw `/` between + * nonempty levels, every level percent-encoded exactly as this module encodes + * it, and no wildcard operator anywhere. + */ +export function isCanonicalTarget(target: string): boolean { + if (target.length === 0) { + return false; + } + return target.split("/").every((level) => { + if (level.length === 0 || level.includes("*")) { + return false; + } + const decoded = decodePercentEncoded(level); + if (decoded === undefined || decoded.length === 0) { + return false; + } + return encodeTargetLabel(decoded) === level; + }); +} + +type LevelPart = + | { readonly kind: "literal"; readonly text: string } + | { readonly kind: "wildcard" }; + +type SelectorLevel = + | { readonly kind: "recursive" } + | { readonly kind: "label"; readonly parts: readonly LevelPart[] }; + +/** + * Parse a target selector into levels, or `undefined` when the syntax is not a + * selector at all. + * + * Raw `/` separates levels and raw `*` is an operator, so the split happens + * before decoding: `%2F` stays a slash inside one label and `%2A` stays a + * literal asterisk. Only the literal chunks between operators are decoded. + */ +function parseSelector(selector: string): readonly SelectorLevel[] | undefined { + if (selector.length === 0 || selector.startsWith("/") || selector.endsWith("/")) { + return undefined; + } + const levels: SelectorLevel[] = []; + for (const raw of selector.split("/")) { + if (raw.length === 0) { + return undefined; + } + if (raw === "**") { + levels.push({ kind: "recursive" }); + continue; + } + const chunks: string[] = []; + for (const chunk of raw.split("*")) { + const decoded = decodePercentEncoded(chunk); + if (decoded === undefined) { + return undefined; + } + chunks.push(decoded.normalize("NFC").replace(/\s+/gu, " ")); + } + // Only the outer edges are trimmed: whitespace beside a wildcard is part of + // what the author asked to match, while the whole level is compared against + // an already-trimmed label. + chunks[0] = chunks[0]!.trimStart(); + chunks[chunks.length - 1] = chunks[chunks.length - 1]!.trimEnd(); + + const parts: LevelPart[] = []; + for (const [index, chunk] of chunks.entries()) { + if (index > 0 && parts[parts.length - 1]?.kind !== "wildcard") { + parts.push({ kind: "wildcard" }); + } + if (chunk.length > 0) { + parts.push({ kind: "literal", text: chunk }); + } + } + levels.push({ kind: "label", parts }); + } + return levels; +} + +/** + * Whether one level's parts match one label, by code point. + * + * A reachability sweep rather than backtracking: each part advances a set of + * positions the label could have been consumed to, so a selector holding many + * wildcards costs the product of its size and the label's, never an exponential + * search. + */ +function matchLabel(parts: readonly LevelPart[], label: readonly string[]): boolean { + let reachable = new Array(label.length + 1).fill(false); + reachable[0] = true; + for (const part of parts) { + const next = new Array(label.length + 1).fill(false); + if (part.kind === "wildcard") { + let open = false; + for (let index = 0; index <= label.length; index++) { + open ||= reachable[index]!; + next[index] = open; + } + } else { + const literal = Array.from(part.text); + for (let index = 0; index + literal.length <= label.length; index++) { + if (!reachable[index]) { + continue; + } + if (literal.every((character, offset) => label[index + offset] === character)) { + next[index + literal.length] = true; + } + } + } + reachable = next; + } + return reachable[label.length]!; +} + +/** Whether a parsed selector matches a canonical label path. */ +function matchPath(levels: readonly SelectorLevel[], path: readonly string[]): boolean { + const characters = path.map((label) => Array.from(label)); + let reachable = new Array(path.length + 1).fill(false); + reachable[0] = true; + for (const level of levels) { + const next = new Array(path.length + 1).fill(false); + if (level.kind === "recursive") { + let open = false; + for (let index = 0; index <= path.length; index++) { + open ||= reachable[index]!; + next[index] = open; + } + } else { + for (let index = 0; index < path.length; index++) { + if (reachable[index] && matchLabel(level.parts, characters[index]!)) { + next[index + 1] = true; + } + } + } + reachable = next; + } + return reachable[path.length]!; +} + +/** + * The one catalog entry a selector names. + * + * Zero matches and several matches are both failures, and both are decided + * here — before the document expands — so an ambiguous request never runs half + * a document to discover it was ambiguous. Duplicate canonical paths stay + * duplicate entries, which is what makes that ambiguity observable at all. + */ +export function selectTarget(outline: DocumentOutline, selector: string): DocumentTarget { + const levels = parseSelector(selector); + if (levels === undefined) { + throw new DocumentTargetError("invalid-selector", selector, [], outline.targets); + } + const matched = outline.entries.filter((entry) => matchPath(levels, entry.labels)); + const first = matched[0]; + if (first === undefined) { + throw new DocumentTargetError("no-match", selector, [], outline.targets); + } + if (matched.length > 1) { + throw new DocumentTargetError( + "multiple-matches", + selector, + matched.map((entry) => entry.target), + outline.targets, + ); + } + return first; +} + +/** + * The interpolation forms a heading's own source may not contain. + * + * A heading whose text is computed is not a stable address, so it is not one. + * `\{` escapes an interpolation back into literal text, which stays static and + * stays addressable. + */ +const INTERPOLATION = + /(\\?)\{(?:(?:meta|props)\.[^}]+|[A-Za-z_$][A-Za-z0-9_$]*(?:\.[A-Za-z_$][A-Za-z0-9_$]*)*)\}/g; + +function hasUnescapedInterpolation(source: string): boolean { + for (const match of source.matchAll(INTERPOLATION)) { + if (match[1] !== "\\") { + return true; + } + } + return false; +} + +/** + * A copy of the body with every top-level component span blanked. + * + * Length, newline positions, and every offset are preserved, so a heading found + * in the mask sits at the same place in the original. + */ +function maskComponents(body: string, spans: readonly ComponentSpan[]): string { + if (spans.length === 0) { + return body; + } + let masked = ""; + let cursor = 0; + for (const span of spans) { + masked += body.slice(cursor, span.start); + masked += body.slice(span.start, span.end).replace(/[^\n]/g, " "); + cursor = span.end; + } + return masked + body.slice(cursor); +} + +interface RawHeading { + readonly depth: number; + readonly start: number; + readonly end: number; + readonly text: string; +} + +function rootHeadings(masked: string): RawHeading[] { + const headings: RawHeading[] = []; + for (const child of remark().parse(masked).children) { + if (child.type !== "heading") { + continue; + } + const start = child.position?.start.offset; + const end = child.position?.end.offset; + if (start === undefined || end === undefined) { + continue; + } + headings.push({ + depth: child.depth, + start, + end, + // Read from the masked tree, which is the original text for every heading + // that does not overlap a component span — and one that does is refused + // below, so no label is ever built from blanked source. + text: mdastToString(child, { includeHtml: false, includeImageAlt: true }), + }); + } + return headings; +} + +function overlapsComponent(heading: RawHeading, spans: readonly ComponentSpan[]): boolean { + return spans.some((span) => span.start < heading.end && heading.start < span.end); +} + +/** + * Discover the outline of one document body and the targets it addresses. + * + * The hierarchy is the standard outline stack — a heading's parent is the + * nearest preceding heading with a smaller depth — so skipped depths are + * ordinary. "Outermost" is the smallest depth present, not `h1`. + * + * A single outermost heading is the document's title: it is retained in every + * projection and takes no level in any path, which is why a document that opens + * with one title still addresses its sections by their own names. + */ +export function outlineDocument(body: string, spans: readonly ComponentSpan[]): DocumentOutline { + const raw = rootHeadings(maskComponents(body, spans)); + if (raw.length === 0) { + return { + headings: [], + entries: [], + targets: [], + preambleEnd: body.length, + bodyLength: body.length, + }; + } + + const outermostDepth = Math.min(...raw.map((heading) => heading.depth)); + const outermost = raw.filter((heading) => heading.depth === outermostDepth); + const titleIndex = outermost.length === 1 ? raw.indexOf(outermost[0]!) : undefined; + + const headings: OutlineHeading[] = []; + const stack: number[] = []; + for (const [index, heading] of raw.entries()) { + while (stack.length > 0 && raw[stack[stack.length - 1]!]!.depth >= heading.depth) { + stack.pop(); + } + const label = normalizeLabel(heading.text); + headings.push({ + depth: heading.depth, + start: heading.start, + end: heading.end, + parent: stack[stack.length - 1], + addressable: + label.length > 0 && + !overlapsComponent(heading, spans) && + !hasUnescapedInterpolation(body.slice(heading.start, heading.end)), + label, + }); + stack.push(index); + } + + const entries: DocumentTarget[] = []; + for (let index = 0; index < headings.length; index++) { + const labels = pathLabels(headings, index, titleIndex); + if (labels !== undefined) { + entries.push({ + target: labels.map(encodeTargetLabel).join("/"), + labels, + heading: index, + }); + } + } + + return { + headings, + entries, + targets: entries.map((entry) => entry.target), + preambleEnd: outermost[0]!.start, + bodyLength: body.length, + }; +} + +/** + * The canonical path a heading is addressed by, or `undefined` when it has + * none. + * + * Every level has to be addressable: a heading under one whose text is not + * static cannot be named, so its subtree is unreachable. The sole title is not + * a level, which is what lets a static section under a computed title stay + * addressable. + */ +function pathLabels( + headings: readonly OutlineHeading[], + index: number, + titleIndex: number | undefined, +): readonly string[] | undefined { + if (index === titleIndex) { + return undefined; + } + const labels: string[] = []; + for (let current: number | undefined = index; current !== undefined; ) { + if (current !== titleIndex) { + if (!headings[current]!.addressable) { + return undefined; + } + labels.unshift(headings[current]!.label); + } + current = headings[current]!.parent; + } + return labels.length === 0 ? undefined : labels; +} + +/** Where a heading's own subtree ends: the next heading at its depth or above. */ +function subtreeEnd(outline: DocumentOutline, index: number): number { + const depth = outline.headings[index]!.depth; + for (let next = index + 1; next < outline.headings.length; next++) { + if (outline.headings[next]!.depth <= depth) { + return outline.headings[next]!.start; + } + } + return outline.bodyLength; +} + +/** An ancestor's own content: its heading through its first child heading. */ +function directPrefixEnd(outline: DocumentOutline, index: number): number { + const child = outline.headings[index + 1]; + if (child !== undefined && child.depth > outline.headings[index]!.depth) { + return child.start; + } + return subtreeEnd(outline, index); +} + +/** + * The original source ranges a target retains: the preamble, each ancestor's + * own content, and the selected subtree. + * + * The ranges are returned in source order and never overlap, so scanning them + * in turn reproduces authored positions exactly. Sibling subtrees fall in the + * gaps between them and are never scanned, which is what keeps their + * components, resources, and code blocks from running at all. + */ +export function retainedRanges( + outline: DocumentOutline, + entry: DocumentTarget, +): readonly SourceRange[] { + const ancestors: number[] = []; + for ( + let current = outline.headings[entry.heading]!.parent; + current !== undefined; + current = outline.headings[current]!.parent + ) { + ancestors.unshift(current); + } + + const ranges: SourceRange[] = [{ start: 0, end: outline.preambleEnd }]; + for (const ancestor of ancestors) { + ranges.push({ + start: outline.headings[ancestor]!.start, + end: directPrefixEnd(outline, ancestor), + }); + } + ranges.push({ + start: outline.headings[entry.heading]!.start, + end: subtreeEnd(outline, entry.heading), + }); + + const retained: SourceRange[] = []; + let consumed = 0; + for (const range of ranges) { + const start = Math.max(range.start, consumed); + if (start < range.end) { + retained.push({ start, end: range.end }); + consumed = range.end; + } + } + return retained; +} diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 8426f37f..d15ae2ce 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -17,7 +17,10 @@ import { durableRun, createDurableOperation, ephemeral, + ReplayGuard, + StaleInputError, type DurableStream, + type Yield, } from "@executablemd/durable-streams"; import { exec, readTextFile, cwd } from "@executablemd/runtime"; import { cwd as processCwd } from "@effectionx/fs"; @@ -35,7 +38,7 @@ import type { ReturnsSchema, Segment, } from "./types.ts"; -import { parseJson, parseJsonObject } from "./json.ts"; +import { isJsonObject, parseJson, parseJsonObject } from "./json.ts"; import { compilePropsSchema, compileReturnsSchema, @@ -43,7 +46,12 @@ import { validateProps, } from "./validate.ts"; import { useParseCompiler } from "./components/parse-schema.ts"; -import { isFunctionComponentPath, parseMarkdownDefinition } from "./definition.ts"; +import { + isFunctionComponentPath, + parseMarkdownDefinition, + parseRootMarkdownDefinition, + resolveDocumentTarget, +} from "./definition.ts"; import { parseReturnsDeclaration } from "./frontmatter.ts"; import { expandSegments, @@ -129,7 +137,7 @@ export type ExecuteOptions = RootDocumentSource & ExecuteSettings; * implementation up again in the scope that is running now. */ type DurableSelection = - | { kind: "repository"; path: string; content: string } + | { kind: "repository"; path: string; content: string; target?: string } | { kind: "registered"; origin: string; reserved: boolean }; function* durableImportComponent( @@ -145,10 +153,21 @@ function* durableImportComponent( // Inside the durable operation, so the journal holds the root's identity // and its text: a replay restores both without reading anything, whether // the source was a file or supplied. + // + // The selector resolves here too, against the text this operation is + // about to record, so the exact target the run executed is part of the + // record rather than something a later read has to rediscover. Only the + // exact target is recorded — a glob describes what the caller asked + // for, not what ran. + const path = rootSourcePath(root); + const content = yield* readRootSource(root); + const target = + root.target === undefined ? undefined : resolveDocumentTarget(path, content, root.target); return { kind: "repository", - path: rootSourcePath(root), - content: yield* readRootSource(root), + path, + content, + ...(target === undefined ? {} : { target }), }; } @@ -193,7 +212,7 @@ function* durableImportComponent( return found.definition; } - const { path, content } = selection; + const { path, content, target } = selection; // Function component: .ts file — import() the module if (isFunctionComponentPath(path)) { @@ -238,7 +257,13 @@ function* durableImportComponent( return definition; } - // Markdown component: parse at runtime — deterministic from content + // Markdown component: parse at runtime — deterministic from content. + // A recorded target projects the recorded content, so a resumed run executes + // the same section from the same text the first run recorded, whatever the + // file on disk says now. + if (target !== undefined) { + return (yield* ephemeral(parseRootMarkdownDefinition(name, path, content, target))).definition; + } return yield* ephemeral(parseMarkdownDefinition(name, path, content)); } @@ -246,6 +271,113 @@ function isFunctionComponent(value: unknown): value is FunctionComponent { return typeof value === "function"; } +/** The recorded root import this event is, when it is one that can be read. */ +function recordedRootImport(event: Yield): { content: string; target?: string } | undefined { + if ( + event.description.type !== "import_component" || + event.description.name !== "__root__" || + event.result.status !== "ok" + ) { + return undefined; + } + const record = event.result.value; + if (!isJsonObject(record)) { + return undefined; + } + const content = record["content"]; + const target = record["target"]; + if (typeof content !== "string" || (target !== undefined && typeof target !== "string")) { + return undefined; + } + return target === undefined ? { content } : { content, target }; +} + +/** + * Refuse to replay a run that was recorded against a different section. + * + * Only `type` and `name` decide whether a journal entry matches, and the root + * import's name is the same for every target — so without this, resuming with a + * different selector would restore the recorded content and then project a + * section the recorded run never executed. + * + * The current selector is resolved against the *recorded* content, so a glob + * that still names the same section replays and a glob that now names another + * one does not. A selector that has become invalid or ambiguous against that + * content is refused for the same reason: nothing here may guess which section + * a resumed run meant. + * + * This validates in the check phase rather than the decide phase because + * `durableRun` reuses a recorded root Close before any effect is replayed. A + * decision made later would never run for a completed journal, which is exactly + * the run whose recorded target must still be the one being asked for. + * + * A `StaleInputError`, so it propagates as a durability failure rather than + * being printed into the document. + */ +function refuseChangedRootTarget(root: RootDocumentSource): Operation { + return ReplayGuard.around({ + *check([event], next) { + const recorded = recordedRootImport(event); + if (recorded === undefined) { + return yield* next(event); + } + const requested = resolveRecordedTarget(root, recorded.content); + const compatible = + requested.kind === "whole" + ? recorded.target === undefined + : requested.kind === "exact" && requested.target === recorded.target; + if (!compatible) { + const stale = new StaleInputError( + `the recorded root document import ran ${describeRecorded(recorded.target)}, and this ` + + `run asks for ${describeRequested(requested)}. Re-run the document from the ` + + "start rather than resuming from a journal that recorded another section.", + { coroutineId: event.coroutineId, description: event.description }, + ); + if (requested.kind === "unresolved") { + stale.cause = requested.failure; + } + throw stale; + } + return yield* next(event); + }, + }); +} + +/** What this run's selector names in the recorded content. */ +type RequestedTarget = + | { kind: "whole" } + | { kind: "exact"; target: string } + | { kind: "unresolved"; failure: unknown }; + +function resolveRecordedTarget(root: RootDocumentSource, content: string): RequestedTarget { + if (root.target === undefined) { + return { kind: "whole" }; + } + try { + return { + kind: "exact", + target: resolveDocumentTarget(rootSourcePath(root), content, root.target), + }; + } catch (failure) { + return { kind: "unresolved", failure }; + } +} + +function describeRecorded(target: string | undefined): string { + return target === undefined ? "the whole document" : `the target ${JSON.stringify(target)}`; +} + +function describeRequested(requested: RequestedTarget): string { + switch (requested.kind) { + case "whole": + return "the whole document"; + case "exact": + return `the target ${JSON.stringify(requested.target)}`; + case "unresolved": + return "a target that recorded content no longer names exactly once"; + } +} + const execFactory: ModifierFactory = (_params) => (_args, _next) => (function* () { const context = yield* useCodeBlock(); @@ -864,6 +996,10 @@ function* executeDocument(options: ExecuteOptions): Operation { at: "min" }, ); + // Installed before the durable run, so the check phase sees the recorded + // root import before `durableRun` can reuse a recorded Close. + yield* refuseChangedRootTarget(root); + // The policy is selected here — before the durable run and before any // document, frontmatter, prop, component, or eval code exists — so the // root component import is already behind the gate. What comes back is diff --git a/packages/core/src/inspect.ts b/packages/core/src/inspect.ts index bf3617f9..941669c4 100644 --- a/packages/core/src/inspect.ts +++ b/packages/core/src/inspect.ts @@ -2,7 +2,11 @@ import type { Operation } from "effection"; import { readTextFile } from "@executablemd/runtime"; import type { ComponentOrigin, PropsSchema, ReturnsSchema } from "./types.ts"; -import { isFunctionComponentPath, parseMarkdownDefinition } from "./definition.ts"; +import { + isFunctionComponentPath, + parseMarkdownDefinition, + parseRootMarkdownDefinition, +} from "./definition.ts"; import { Component } from "./component-api.ts"; import { selectComponent } from "./components/select.ts"; import { readRootSource, rootSourcePath } from "./root-source.ts"; @@ -38,6 +42,22 @@ export interface DocumentInfo { * the default, so the mode — not the schema — tells the two apart. */ returnMode: "text" | "value"; + + /** + * Every target the document addresses, as canonical encoded fragments without + * the document path or a leading `#`, in document order. + * + * Duplicates are retained: two sections that canonicalize to the same path + * are an ambiguity a caller can see rather than one a selector resolves + * arbitrarily. + */ + readonly targets: readonly string[]; + + /** + * The exact canonical target the requested selector resolved to. Present only + * when a target was requested and resolved; it is never the caller's glob. + */ + readonly target?: string; } /** @@ -47,6 +67,10 @@ export interface DocumentInfo { * validation as execution, but never expands the document, evaluates a * code block, imports a body component, starts an agent, or creates a * journal — so describing a document is always free of its effects. + * + * Target discovery and selection happen here too. A requested selector that + * names no section, or several, fails as a `DocumentTargetError` — before + * anything is expanded, and without a journal ever existing. */ export function* inspectDocument(options: InspectOptions): Operation { const path = rootSourcePath(options); @@ -58,7 +82,8 @@ export function* inspectDocument(options: InspectOptions): Operation` identity. */ -export function inlineSource(source: string): InlineRootDocument { - return { path: INLINE_SOURCE_PATH, source }; +export function inlineSource( + source: string, + options?: { readonly target?: string }, +): InlineRootDocument { + const target = options?.target; + return { + path: INLINE_SOURCE_PATH, + source, + ...(target === undefined ? {} : { target }), + }; +} + +/** + * A file root document from a URI-style document reference. + * + * The reference is `#`, split at the + * first raw `#`. The path is percent-decoded; the fragment is not, because + * `%2F` has to stay distinguishable from the raw `/` that separates target + * levels — the selector parser splits the hierarchy and operator syntax first + * and decodes only the literal chunks between them. + * + * A reference that cannot be read fails with a `TypeError` carrying nothing but + * fixed wording: the input is a command-line argument, and echoing it back + * would put arbitrary bytes into a diagnostic. + * + * A filename containing `#` is written `%23`, and one containing a literal + * `%HH` sequence is written `%25HH`. + */ +export function fileSource(reference: string): FileRootDocument { + const fragment = reference.indexOf("#"); + const encodedPath = fragment === -1 ? reference : reference.slice(0, fragment); + const path = decodePercentEncoded(encodedPath); + if (path === undefined || path.length === 0) { + throw new TypeError(INVALID_REFERENCE); + } + return fragment === -1 ? { path } : { path, target: reference.slice(fragment + 1) }; +} + +/** + * The canonical reference for a document, and optionally one exact target + * inside it. + * + * The path arrives decoded and is encoded here; the target arrives already + * canonical — `DocumentInfo.target`, or a stored workflow definition's — and is + * validated rather than encoded again, so a canonical `%2F` is never turned + * into `%252F`. Making an authored glob canonical is the selector parser's job, + * not this one's. + * + * This is the one formatter diagnostics, command output, and workflow handoff + * use, so a reference printed by one of them is a reference the others accept. + */ +export function formatDocumentReference(path: string, target?: string): string { + if (path.length === 0) { + throw new TypeError(INVALID_REFERENCE); + } + if (target === undefined) { + return encodeDocumentPath(path); + } + if (!isCanonicalTarget(target)) { + throw new TypeError(INVALID_REFERENCE); + } + return `${encodeDocumentPath(path)}#${target}`; } /** The identity printed errors and source positions report for this root. */ diff --git a/packages/core/src/scanner.ts b/packages/core/src/scanner.ts index 71e83bef..6f07ff48 100644 --- a/packages/core/src/scanner.ts +++ b/packages/core/src/scanner.ts @@ -112,6 +112,19 @@ function positionAt(index: PositionIndex, offset: number): SourcePosition { }; } +/** + * Where one top-level component invocation begins and ends in the scanned text. + * + * Half-open, relative to the text handed to the scanner. Internal: this is the + * scanner's own boundary decision, recorded so heading discovery can blank the + * regions this scanner owns before a Markdown parser looks at them. It is not a + * public authority over what a component is. + */ +export interface ComponentSpan { + readonly start: number; + readonly end: number; +} + /** * Scan raw markdown text into segments. * @@ -120,8 +133,16 @@ function positionAt(index: PositionIndex, offset: number): SourcePosition { * * When `origin` is provided, component invocations carry `position` values * expressed in the original file's coordinates. + * + * When `spans` is provided, each top-level component invocation records its + * source span into it, in source order. Collecting them changes nothing about + * the segments produced. */ -export function scanSegments(text: string, origin?: SourceOrigin): Segment[] { +export function scanSegments( + text: string, + origin?: SourceOrigin, + spans?: ComponentSpan[], +): Segment[] { const index: PositionIndex = { origin, lineStarts: computeLineStarts(text) }; const segments: Segment[] = []; let pos = 0; @@ -184,6 +205,7 @@ export function scanSegments(text: string, origin?: SourceOrigin): Segment[] { pushText(segments, text.slice(textStart, pos)); } segments.push(component.segment); + spans?.push({ start: pos, end: component.end }); pos = component.end; textStart = pos; continue; @@ -201,6 +223,13 @@ export function scanSegments(text: string, origin?: SourceOrigin): Segment[] { return segments; } +/** The source spans of the top-level component invocations in `text`. */ +export function scanComponentSpans(text: string): ComponentSpan[] { + const spans: ComponentSpan[] = []; + scanSegments(text, undefined, spans); + return spans; +} + interface FenceOpen { fenceChar: string; fenceLen: number; diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts new file mode 100644 index 00000000..0236bbde --- /dev/null +++ b/packages/core/tests/document-target-execution.test.ts @@ -0,0 +1,498 @@ +/** + * Tier TX — targeted document execution and replay (spec §5.4, §6.11). + * + * A projected document is not a rendering exercise: what it must prove is that + * a skipped sibling *did not run*, that a retained element kept the identity it + * has in a full run, and that a journal recorded against one section cannot be + * resumed as another. + * + * Every "did not run" assertion is made from a component that records its own + * invocation, not from absent text — text can be absent because it rendered + * empty. Every identity assertion reads the expansion ID the engine derived, + * not a position that merely looks unchanged. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped, until } from "effection"; +import type { Operation } from "effection"; +import { rm, writeTextFile } from "@effectionx/fs"; +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import { StaleInputError } from "@executablemd/durable-streams"; +import { API, useHostFiles } from "@executablemd/runtime"; + +import { collect } from "../src/collect.ts"; +import { execute } from "../src/execute.ts"; +import { inspectDocument } from "../src/inspect.ts"; +import { getExpansion } from "../src/expansion.ts"; +import { registerComponents } from "../src/components/registration.ts"; +import { DocumentTargetError } from "../src/document-targets.ts"; +import { fileSource, formatDocumentReference, inlineSource } from "../src/root-source.ts"; +import type { RootDocumentSource } from "../src/root-source.ts"; +import { asText } from "./helpers.ts"; + +/** What every `` in a run reported, in the order it expanded. */ +interface Probes { + names: string[]; + ids: string[]; +} + +/** + * `` — proof of expansion. + * + * A component that records its own invocation and its expansion ID. Absent + * output would not distinguish "skipped" from "rendered nothing"; an absent + * entry here can only mean the element never expanded. + */ +function* useProbes(seen: Probes): Operation { + yield* registerComponents([ + { + name: "Probe", + origin: "tier-tx", + props: { type: "object", properties: { name: { type: "string" } } }, + *fn(props) { + const name = props["name"]; + seen.names.push(typeof name === "string" ? name : "?"); + seen.ids.push((yield* getExpansion()).id); + return `[${typeof name === "string" ? name : "?"}]`; + }, + }, + ]); +} + +/** A directory the contextual cwd points at, removed when the test ends. */ +function* useWorkspace(files: Record): Operation { + const root = yield* until(mkdtemp(join(tmpdir(), "xmd-targets-"))); + yield* ensure(() => rm(root, { recursive: true, force: true })); + for (const [name, content] of Object.entries(files)) { + yield* writeTextFile(join(root, name), content); + } + yield* API.Env.around( + { + *cwd() { + return root; + }, + }, + { at: "min" }, + ); + yield* useHostFiles(); + return root; +} + +function rootImports(stream: InMemoryStream) { + return stream + .snapshot() + .flatMap((event) => + event.type === "yield" && + event.description.type === "import_component" && + event.description.name === "__root__" + ? [event] + : [], + ); +} + +function closes(stream: InMemoryStream) { + return stream.snapshot().filter((event) => event.type === "close"); +} + +/** Run a document and report both its text and what expanded. */ +function run(root: RootDocumentSource, stream: InMemoryStream, seen: Probes): Operation { + return scoped(function* () { + yield* useProbes(seen); + return asText(yield* collect(yield* execute({ ...root, stream }))); + }); +} + +/** The failure a run produced, refusing to pass a success off as one. */ +function* failure( + root: RootDocumentSource, + stream: InMemoryStream, + seen: Probes = { names: [], ids: [] }, +): Operation { + try { + yield* run(root, stream, seen); + } catch (error) { + return error; + } + throw new Error("the run completed instead of failing"); +} + +const SECTIONS = [ + 'preamble ', + "", + "# Title", + "", + 'title content ', + "", + "## Alpha", + "", + 'alpha content ', + "", + "### Inner", + "", + 'inner content ', + "", + "## Beta", + "", + 'beta content ', + "", + "```sh exec", + "echo beta-ran", + "```", + "", +].join("\n"); + +describe("Tier TX — targeted execution", () => { + it("TX1: only the preamble, the ancestors, and the subtree expand", function* () { + const seen: Probes = { names: [], ids: [] }; + const text = yield* run( + inlineSource(SECTIONS, { target: "Alpha/Inner" }), + new InMemoryStream(), + seen, + ); + + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + expect(text).toContain("# Title"); + expect(text).toContain("## Alpha"); + expect(text).toContain("### Inner"); + expect(text).not.toContain("## Beta"); + }); + + it("TX2: a skipped sibling's components and code blocks never run", function* () { + const seen: Probes = { names: [], ids: [] }; + yield* scoped(function* () { + yield* API.Process.around({ + *exec([options], _next) { + throw new Error(`a skipped code block ran: ${JSON.stringify(options.command)}`); + }, + }); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), new InMemoryStream(), seen); + }); + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + }); + + it("TX3: selecting a non-leaf expands every descendant", function* () { + const seen: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), new InMemoryStream(), seen); + expect(seen.names).toContain("inner"); + }); + + it("TX4: a retained element keeps the expansion ID it has in a full run", function* () { + const whole: Probes = { names: [], ids: [] }; + const targeted: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS), new InMemoryStream(), whole); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), new InMemoryStream(), targeted); + + const idOf = (probes: Probes, name: string) => probes.ids[probes.names.indexOf(name)]; + expect(targeted.names).toEqual(["pre", "title", "beta"]); + expect(idOf(targeted, "beta")).toBe(idOf(whole, "beta")); + expect(idOf(targeted, "pre")).toBe(idOf(whole, "pre")); + }); + + /** + * The identifier is derived from position, not from what ran. Two targets + * that retain the same element therefore agree with each other and with the + * full run — and seeding identity with the target string would break all + * three at once. + */ + it("TX5: two different targets agree on a shared retained element", function* () { + const alpha: Probes = { names: [], ids: [] }; + const beta: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), new InMemoryStream(), alpha); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), new InMemoryStream(), beta); + expect(beta.ids[beta.names.indexOf("title")]).toBe(alpha.ids[alpha.names.indexOf("title")]); + }); + + it("TX6: a file root and an inline root behave identically", function* () { + const workspace = yield* useWorkspace({ "doc.md": SECTIONS }); + const fromFile: Probes = { names: [], ids: [] }; + const fromText: Probes = { names: [], ids: [] }; + const fileText = yield* run( + fileSource(formatDocumentReference(join(workspace, "doc.md"), "Beta")), + new InMemoryStream(), + fromFile, + ); + const inlineText = yield* run( + inlineSource(SECTIONS, { target: "Beta" }), + new InMemoryStream(), + fromText, + ); + expect(fromFile.names).toEqual(fromText.names); + expect(fileText).toBe(inlineText); + }); + + it("TX7: root props and frontmatter apply to the projected body", function* () { + const body = [ + "---", + "title: Doc", + "props:", + " who:", + " type: string", + "---", + "", + "# {meta.title}", + "", + "## Greeting", + "", + "hello {props.who}", + "", + "## Skipped", + "", + "skipped {props.who}", + "", + ].join("\n"); + const text = yield* scoped(function* () { + const execution = yield* execute({ + ...inlineSource(body, { target: "Greeting" }), + stream: new InMemoryStream(), + props: { who: "world" }, + }); + return asText(yield* collect(execution)); + }); + expect(text).toContain("# Doc"); + expect(text).toContain("hello world"); + expect(text).not.toContain("skipped"); + }); + + it("TX8: a value root returns from the projected body", function* () { + const body = [ + "---", + "returns:", + " type: object", + " properties:", + " picked:", + " type: string", + "---", + "", + "# Title", + "", + "## Kept", + "", + '', + "", + ].join("\n"); + const value = yield* collect( + yield* execute({ + ...inlineSource(body, { target: "Kept" }), + stream: new InMemoryStream(), + }), + ); + expect(value).toEqual({ picked: "kept" }); + }); + + it("TX9: `` in the projected body selects what is emitted", function* () { + const body = [ + "# Title", + "", + "title text", + "", + "## Kept", + "", + "", + "chosen", + "", + "", + "not chosen", + "", + ].join("\n"); + const text = asText( + yield* collect( + yield* execute({ + ...inlineSource(body, { target: "Kept" }), + stream: new InMemoryStream(), + }), + ), + ); + expect(text.trim()).toBe("chosen"); + }); + + /** + * Structural preflight applies to the projected body, so a violation the + * caller did not select is not a violation of what runs. + */ + it("TX10: an invalid structure in a skipped sibling is irrelevant", function* () { + const body = [ + "# Title", + "", + "## Kept", + "", + 'kept body ', + "", + "## Broken", + "", + '', + "", + ].join("\n"); + const seen: Probes = { names: [], ids: [] }; + const text = yield* run(inlineSource(body, { target: "Kept" }), new InMemoryStream(), seen); + expect(seen.names).toEqual(["kept"]); + expect(text).toContain("kept body"); + expect(text).not.toContain(" requires"); + }); + + it("TX11: an invalid structure in the retained range fails before any effect", function* () { + const body = [ + "# Title", + "", + '', + "", + "## Kept", + "", + '', + "", + ].join("\n"); + const seen: Probes = { names: [], ids: [] }; + const text = yield* run(inlineSource(body, { target: "Kept" }), new InMemoryStream(), seen); + expect(seen.names).toEqual([]); + expect(text).toContain(" requires"); + }); + + /** + * Resolution sits inside the durable root import, so a target failure travels + * out of `execute()` the way every failure crossing that boundary does: by + * name and message, its class left behind with the journal round trip. The + * typed `DocumentTargetError` is what `inspectDocument()` reports, and + * inspection is where a host resolves a selector before running anything. + */ + it("TX12: a target that resolves to nothing runs no authored effect", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream, seen); + expect((error as Error).name).toBe("DocumentTargetError"); + expect((error as Error).message).toContain("matches no document target"); + expect(seen.names).toEqual([]); + // The root import is the only effect the journal saw, and it failed. + expect(rootImports(stream).map((event) => event.result.status)).toEqual(["err"]); + expect(stream.snapshot().filter((event) => event.type === "yield").length).toBe(1); + }); + + it("TX13: an ambiguous target runs no authored effect", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream, seen); + expect((error as Error).message).toContain("matches more than one document target"); + expect(seen.names).toEqual([]); + expect(stream.snapshot().filter((event) => event.type === "yield").length).toBe(1); + }); + + it("TX14: inspection resolves a target without expanding a component", function* () { + const seen: Probes = { names: [], ids: [] }; + const info = yield* scoped(function* () { + yield* useProbes(seen); + return yield* inspectDocument(inlineSource(SECTIONS, { target: "**/Inner" })); + }); + expect(info.target).toBe("Alpha/Inner"); + expect(seen.names).toEqual([]); + }); +}); + +describe("Tier TX — targeted replay", () => { + it("TX15: the journal records the exact target, never the glob", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS, { target: "**/I*" }), stream, seen); + + const imports = rootImports(stream); + expect(imports.length).toBe(1); + expect(imports[0]).toMatchObject({ + result: { status: "ok", value: { target: "Alpha/Inner" } }, + }); + }); + + it("TX16: an untargeted run records no target member at all", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + const recorded = rootImports(stream)[0]; + expect(recorded?.result.status).toBe("ok"); + const value = recorded?.result.status === "ok" ? recorded.result.value : undefined; + expect(value !== null && typeof value === "object" && "target" in value).toBe(false); + }); + + it("TX17: a different selector naming the same section replays", function* () { + const stream = new InMemoryStream(); + const first: Probes = { names: [], ids: [] }; + const golden = yield* run(inlineSource(SECTIONS, { target: "Alpha/Inner" }), stream, first); + + const second: Probes = { names: [], ids: [] }; + const replayed = yield* run(inlineSource(SECTIONS, { target: "**/I*" }), stream, second); + + expect(replayed).toBe(golden); + expect(rootImports(stream).length).toBe(1); + }); + + /** + * The reuse this has to beat is the root Close, which `durableRun` honours + * before any effect is replayed. Validating in the decide phase alone would + * leave a completed journal answering for a section it never ran. + */ + it("TX18: a different exact target refuses to reuse a completed journal", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { names: [], ids: [] }); + expect(closes(stream).length).toBeGreaterThan(0); + + const error = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect(error).toBeInstanceOf(StaleInputError); + expect((error as Error).message).toContain("Alpha"); + expect((error as Error).message).toContain("Beta"); + }); + + it("TX19: an untargeted request refuses a targeted journal", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), stream, { names: [], ids: [] }); + + const error = yield* failure(inlineSource(SECTIONS), stream); + expect(error).toBeInstanceOf(StaleInputError); + expect((error as Error).message).toContain("the whole document"); + }); + + it("TX20: a targeted request refuses an untargeted journal", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + + const error = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect(error).toBeInstanceOf(StaleInputError); + }); + + it("TX21: a selector the recorded content no longer resolves fails stale", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { names: [], ids: [] }); + + // "**" is ambiguous against the recorded content, so this run cannot show + // that it means the recorded section. + const error = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream); + expect(error).toBeInstanceOf(StaleInputError); + expect((error as Error).cause).toBeInstanceOf(DocumentTargetError); + }); + + it("TX22: an untargeted journal still replays for an untargeted run", function* () { + const stream = new InMemoryStream(); + const golden = yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + const replayed = yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + expect(replayed).toBe(golden); + expect(rootImports(stream).length).toBe(1); + }); + + /** + * A replay projects the text the journal holds. Rewriting the file between + * runs would change which section a re-resolved selector names if the current + * copy were consulted; it does not, so the replayed output is the first run's. + */ + it("TX23: replay projects the recorded content, not the file on disk", function* () { + const workspace = yield* useWorkspace({ "doc.md": SECTIONS }); + const reference = formatDocumentReference(join(workspace, "doc.md"), "Beta"); + const stream = new InMemoryStream(); + const golden = yield* run(fileSource(reference), stream, { names: [], ids: [] }); + + yield* writeTextFile( + join(workspace, "doc.md"), + ["# Title", "", "## Beta", "", "rewritten beta", ""].join("\n"), + ); + + const replayed = yield* run(fileSource(reference), stream, { names: [], ids: [] }); + expect(replayed).toBe(golden); + expect(replayed).not.toContain("rewritten beta"); + }); +}); diff --git a/packages/core/tests/document-targets.test.ts b/packages/core/tests/document-targets.test.ts new file mode 100644 index 00000000..92cedfc0 --- /dev/null +++ b/packages/core/tests/document-targets.test.ts @@ -0,0 +1,646 @@ +/** + * Tier DT — document targets (spec §5.4). + * + * A target is an addressable static heading, and selecting one runs the + * preamble, each ancestor's own content, and that heading's subtree. These + * assert the three properties the feature stands on. + * + * **Discovery cannot see inside a component.** The masked parse is not a + * refinement of a Remark parse — a component child holding a blank line and a + * `#` line surfaces as a root heading without it, so DT13 fails outright + * against raw Remark discovery. + * + * **A selector resolves exactly once, before anything runs.** Zero matches and + * several matches are both failures, and duplicate canonical paths stay + * duplicates so the ambiguity is visible rather than silently resolved. + * + * **Projection retains original ranges.** The assertions read the projected + * source and the scanned positions rather than a rendering, because the + * position is what an expansion identifier is derived from. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Operation } from "effection"; + +import { + DocumentTargetError, + encodeTargetLabel, + isCanonicalTarget, + normalizeLabel, + outlineDocument, + retainedRanges, + selectTarget, +} from "../src/document-targets.ts"; +import { scanComponentSpans } from "../src/scanner.ts"; +import { parseRootMarkdownDefinition } from "../src/definition.ts"; +import { fileSource, formatDocumentReference, inlineSource } from "../src/root-source.ts"; +import { inspectDocument } from "../src/inspect.ts"; + +function outline(body: string) { + return outlineDocument(body, scanComponentSpans(body)); +} + +function catalog(body: string): readonly string[] { + return outline(body).targets; +} + +function project(body: string, selector: string): string { + const found = outline(body); + const entry = selectTarget(found, selector); + return retainedRanges(found, entry) + .map((range) => body.slice(range.start, range.end)) + .join(""); +} + +/** The failure a selector produced, refusing to pass a success off as one. */ +function refusal(body: string, selector: string): DocumentTargetError { + try { + selectTarget(outline(body), selector); + } catch (error) { + if (error instanceof DocumentTargetError) { + return error; + } + throw error; + } + throw new Error(`${selector} resolved instead of failing`); +} + +const SECTIONS = [ + "preamble", + "", + "# Title", + "", + "intro", + "", + "## Test", + "", + "test intro", + "", + "### Node", + "", + "node body", + "", + "### Bun", + "", + "bun body", + "", + "## Other", + "", + "other body", + "", +].join("\n"); + +describe("Tier DT — document target catalog", () => { + it("DT1: catalogs ATX headings in source order under a sole title", function* () { + expect(catalog(SECTIONS)).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + }); + + it("DT2: a Setext heading is an ordinary outline heading", function* () { + const body = ["Title", "=====", "", "Section", "-------", "", "body", ""].join("\n"); + expect(catalog(body)).toEqual(["Section"]); + }); + + it("DT3: a skipped depth still nests, and the depth itself is not the path", function* () { + const body = ["# Title", "", "#### Deep", "", "body", ""].join("\n"); + expect(catalog(body)).toEqual(["Deep"]); + }); + + it("DT4: the outermost depth is the smallest present, not h1", function* () { + const body = ["## A", "", "### A1", "", "## B", ""].join("\n"); + expect(catalog(body)).toEqual(["A", "A/A1", "B"]); + }); + + it("DT5: several outermost headings all take a path level", function* () { + const body = ["# A", "", "## A1", "", "# B", "", "## B1", ""].join("\n"); + expect(catalog(body)).toEqual(["A", "A/A1", "B", "B/B1"]); + }); + + it("DT6: matching is case sensitive", function* () { + const body = ["# Title", "", "## Test", ""].join("\n"); + expect(refusal(body, "test").kind).toBe("no-match"); + expect(selectTarget(outline(body), "Test").target).toBe("Test"); + }); + + it("DT7: a label is the statically rendered text, formatting removed", function* () { + const body = [ + "# Title", + "", + "## **Bold** and _italic_", + "", + "## A [link](https://example.test/x) here", + "", + "## Inline `code` text", + "", + "## Alt ![a picture](img.png) text", + "", + "## Tagged text", + "", + ].join("\n"); + expect(catalog(body)).toEqual([ + "Bold%20and%20italic", + "A%20link%20here", + "Inline%20code%20text", + "Alt%20a%20picture%20text", + "Tagged%20text", + ]); + }); + + it("DT8: NFC-equivalent spellings are one label, and Unicode space collapses", function* () { + const decomposed = ["# Title", "", "## Café   name", ""].join("\n"); + expect(catalog(decomposed)).toEqual([encodeTargetLabel("Café name")]); + expect(normalizeLabel("Café   name")).toBe("Café name"); + // The precomposed spelling addresses the decomposed heading. + expect(selectTarget(outline(decomposed), encodeTargetLabel("Café name")).labels).toEqual([ + "Café name", + ]); + }); + + it("DT9: a heading that renders no text is not addressable", function* () { + const body = ["# Title", "", "##", "", "body", "", "## Real", ""].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT10: reserved characters are percent-encoded, never left as syntax", function* () { + const body = [ + "# Title", + "", + "## a/b", + "", + "## 100% done", + "", + "## C\\# sharp", + "", + "## star \\* here", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["a%2Fb", "100%25%20done", "C%23%20sharp", "star%20%2A%20here"]); + // `%2F` addresses one label containing a slash; a raw `/` would be hierarchy. + expect(selectTarget(outline(body), "a%2Fb").labels).toEqual(["a/b"]); + expect(refusal(body, "a/b").kind).toBe("no-match"); + // `%2A` is a literal asterisk; a raw `*` is the operator. + expect(selectTarget(outline(body), "star%20%2A%20here").labels).toEqual(["star * here"]); + }); + + it("DT11: duplicate canonical paths stay duplicate entries", function* () { + const body = ["# Title", "", "## Same", "", "one", "", "## Same", "", "two", ""].join("\n"); + expect(catalog(body)).toEqual(["Same", "Same"]); + const ambiguous = refusal(body, "Same"); + expect(ambiguous.kind).toBe("multiple-matches"); + expect(ambiguous.matches).toEqual(["Same", "Same"]); + }); + + it("DT12: only root-flow headings count", function* () { + const body = [ + "# Title", + "", + "> # Quoted", + "", + "- # Listed", + "", + "```md", + "# Fenced", + "```", + "", + "```sh exec", + "# Executed", + "```", + "", + "
", + "# Raw html child", + "
", + "", + "## Real", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + /** + * The regression that decides the parser boundary. + * + * Remark ends an HTML block at a blank line, so a component child holding one + * puts every following `#` line at the root of the tree. Discovery therefore + * parses a masked copy in which the component's whole span is blanked. Remove + * the mask and `Inner` appears here. + */ + it("DT13: a component child's apparent headings are never targets", function* () { + const body = [ + "# Title", + "", + "", + "", + "# Inner", + "", + "some text", + "", + "## Inner two", + "", + "", + "", + "## Real", + "", + "real body", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT14: a heading overlapping component syntax is not addressable", function* () { + const body = ["# Title", "", "## Head tail", "", "## Real", ""].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT15: an interpolated heading is not addressable, and blocks its subtree", function* () { + const body = [ + "# Title", + "", + "## {meta.name}", + "", + "### Under computed", + "", + "## {binding}", + "", + "## {props.a.b}", + "", + "## Real", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT16: escaped interpolation is static text and stays addressable", function* () { + const body = ["# Title", "", "## \\{meta.name\\}", "", "body", ""].join("\n"); + expect(catalog(body)).toEqual(["%7Bmeta.name%7D"]); + expect(selectTarget(outline(body), "%7Bmeta.name%7D").labels).toEqual(["{meta.name}"]); + }); + + /** + * The title is not a path level, so it is not a level that has to be + * addressable either — which is the whole reason the exception exists. + */ + it("DT17: a computed sole title still leaves its sections addressable", function* () { + const body = ["# {meta.title}", "", "## Real", "", "### Deeper", ""].join("\n"); + expect(catalog(body)).toEqual(["Real", "Real/Deeper"]); + }); + + it("DT18: a document with no heading has an empty catalog", function* () { + expect(catalog("just prose\n")).toEqual([]); + expect(refusal("just prose\n", "Anything").available).toEqual([]); + }); + + it("DT19: a sole title is itself no target", function* () { + expect(catalog("# Only\n\nbody\n")).toEqual([]); + }); +}); + +describe("Tier DT — target selectors", () => { + it("DT20: a literal selector matches one whole label", function* () { + expect(selectTarget(outline(SECTIONS), "Test/Node").labels).toEqual(["Test", "Node"]); + expect(refusal(SECTIONS, "Nod").kind).toBe("no-match"); + }); + + it("DT21: `*` matches within one level, in any position, more than once", function* () { + expect(selectTarget(outline(SECTIONS), "Test/N*").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "Test/*ode").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "Test/N*d*").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "*ther").target).toBe("Other"); + // One `*` never crosses a level boundary. + expect(refusal(SECTIONS, "*Node").kind).toBe("no-match"); + }); + + it("DT22: `**` matches zero or more complete levels", function* () { + expect(selectTarget(outline(SECTIONS), "**/Node").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "**/Other").target).toBe("Other"); + expect(selectTarget(outline(SECTIONS), "Other/**").target).toBe("Other"); + expect(selectTarget(outline(SECTIONS), "**/Bun/**").target).toBe("Test/Bun"); + }); + + it("DT23: a selector must name exactly one entry", function* () { + expect(refusal(SECTIONS, "**").kind).toBe("multiple-matches"); + expect(refusal(SECTIONS, "**").matches).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + expect(refusal(SECTIONS, "Missing").kind).toBe("no-match"); + expect(refusal(SECTIONS, "Missing").matches).toEqual([]); + expect(refusal(SECTIONS, "Missing").available).toEqual([ + "Test", + "Test/Node", + "Test/Bun", + "Other", + ]); + }); + + it("DT24: malformed selector syntax is refused as syntax", function* () { + for (const selector of ["", "/Test", "Test/", "Test//Node", "%zz", "Test/%2"]) { + expect(refusal(SECTIONS, selector).kind).toBe("invalid-selector"); + } + }); + + it("DT25: percent decoding is URI path decoding — `+` is a plus", function* () { + const body = ["# Title", "", "## a+b", "", "## a b", ""].join("\n"); + expect(catalog(body)).toEqual(["a%2Bb", "a%20b"]); + expect(selectTarget(outline(body), "a+b").labels).toEqual(["a+b"]); + expect(selectTarget(outline(body), "a%2Bb").labels).toEqual(["a+b"]); + expect(selectTarget(outline(body), "a%20b").labels).toEqual(["a b"]); + }); + + it("DT26: a malformed or NUL-bearing escape never decodes", function* () { + expect(refusal(SECTIONS, "%00").kind).toBe("invalid-selector"); + // A lone continuation byte is not UTF-8. + expect(refusal(SECTIONS, "%80").kind).toBe("invalid-selector"); + }); + + /** + * A backtracking matcher answers this in exponential time; the reachability + * sweep answers it in the product of the two lengths. A regression to + * backtracking does not fail this assertion — it never reaches it. + */ + it("DT27: a wildcard-dense selector against a long label terminates", function* () { + const label = "a".repeat(120); + const body = ["# Title", "", `## ${label}`, ""].join("\n"); + const selector = `${"*a".repeat(30)}*b`; + expect(refusal(body, selector).kind).toBe("no-match"); + expect(selectTarget(outline(body), `${"*a".repeat(30)}*`).labels).toEqual([label]); + }); + + it("DT28: whitespace beside a wildcard is matched; only the outer edges trim", function* () { + const spaced = ["# Title", "", "## alpha beta gamma", ""].join("\n"); + const joined = ["# Title", "", "## alphabetagamma", ""].join("\n"); + expect(selectTarget(outline(spaced), "alpha%20*%20gamma").labels).toEqual(["alpha beta gamma"]); + // The spaces around the wildcard are part of what was asked for. + expect(refusal(joined, "alpha%20*%20gamma").kind).toBe("no-match"); + // The level's own outer whitespace is not, so a padded selector still lands. + expect(selectTarget(outline(spaced), "%20alpha*gamma%20").labels).toEqual(["alpha beta gamma"]); + }); +}); + +describe("Tier DT — canonical references", () => { + it("DT29: a reference splits at the first raw `#`", function* () { + expect(fileSource("README.md")).toEqual({ path: "README.md" }); + expect(fileSource("README.md#Test/Node")).toEqual({ + path: "README.md", + target: "Test/Node", + }); + // A `#` inside the filename is written `%23`; the fragment keeps its own. + expect(fileSource("odd%23name.md#A%23B")).toEqual({ + path: "odd#name.md", + target: "A%23B", + }); + }); + + it("DT30: a path keeps its separators and decodes its escapes", function* () { + expect(fileSource("docs/sub%20dir/a.md").path).toBe("docs/sub dir/a.md"); + // A literal `%HH` in a filename is spelled `%25HH`. + expect(fileSource("lit%2520.md").path).toBe("lit%20.md"); + }); + + it("DT31: an unreadable reference says only that", function* () { + for (const reference of ["", "#Test", "a%zz.md", "a%00b.md"]) { + let caught: unknown; + try { + fileSource(reference); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(TypeError); + expect((caught as Error).message).toBe("Invalid document reference"); + expect(Object.hasOwn(caught as Error, "cause")).toBe(false); + } + }); + + it("DT32: formatting encodes the path and validates an exact target", function* () { + expect(formatDocumentReference("README.md")).toBe("README.md"); + expect(formatDocumentReference("docs/a b.md")).toBe("docs/a%20b.md"); + expect(formatDocumentReference("odd#name.md", "A%23B")).toBe("odd%23name.md#A%23B"); + // Already canonical: encoded once, never twice. + expect(formatDocumentReference("a.md", "a%2Fb")).toBe("a.md#a%2Fb"); + }); + + it("DT33: formatting refuses anything that is not an exact canonical target", function* () { + for (const target of ["", "Test/", "/Test", "Test/*", "**", "a/b*c", "a b", "a%2fb"]) { + let caught: unknown; + try { + formatDocumentReference("a.md", target); + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe("Invalid document reference"); + } + expect(isCanonicalTarget("Test/Node")).toBe(true); + expect(isCanonicalTarget("Test/%2A")).toBe(true); + expect(isCanonicalTarget("Test/*")).toBe(false); + }); +}); + +describe("Tier DT — projection", () => { + it("DT34: preamble, ancestor content, and the whole selected subtree", function* () { + expect(project(SECTIONS, "Test/Node")).toBe( + [ + "preamble", + "", + "# Title", + "", + "intro", + "", + "## Test", + "", + "test intro", + "", + "### Node", + "", + "node body", + "", + "", + ].join("\n"), + ); + }); + + it("DT35: selecting a non-leaf keeps every descendant", function* () { + const projected = project(SECTIONS, "Test"); + expect(projected).toContain("### Node"); + expect(projected).toContain("### Bun"); + expect(projected).not.toContain("## Other"); + expect(projected).not.toContain("other body"); + }); + + it("DT36: sibling subtrees are absent, earlier and later alike", function* () { + const projected = project(SECTIONS, "Test/Bun"); + expect(projected).toContain("bun body"); + expect(projected).not.toContain("node body"); + expect(projected).not.toContain("### Node"); + expect(projected).not.toContain("other body"); + }); + + it("DT37: retained headings and the sole title stay in the projection", function* () { + const projected = project(SECTIONS, "Test/Node"); + expect(projected).toContain("# Title"); + expect(projected).toContain("intro"); + expect(projected).toContain("## Test"); + expect(projected).toContain("test intro"); + }); + + it("DT38: with several outermost headings none is retained by default", function* () { + const body = ["pre", "", "# A", "", "a body", "", "# B", "", "b body", ""].join("\n"); + expect(project(body, "B")).toBe(["pre", "", "# B", "", "b body", ""].join("\n")); + }); + + it("DT39: an ancestor keeps only its own content, not an earlier sibling's", function* () { + const body = [ + "# Title", + "", + "title content", + "", + "## First", + "", + "first content", + "", + "## Second", + "", + "second content", + "", + ].join("\n"); + expect(project(body, "Second")).toBe( + ["# Title", "", "title content", "", "## Second", "", "second content", ""].join("\n"), + ); + }); +}); + +describe("Tier DT — projected parsing", () => { + function* parsed(body: string, selector?: string) { + return yield* parseRootMarkdownDefinition("__root__", "doc.md", body, selector); + } + + it("DT40: a retained element keeps the offset and line it was authored at", function* () { + const body = [ + "# Title", + "", + "## Skipped", + "", + "x".repeat(400), + "", + "## Kept", + "", + "", + "", + ].join("\n"); + + const whole = yield* parsed(body); + const targeted = yield* parsed(body, "Kept"); + const positionOf = (definition: { bodySegments: readonly unknown[] }) => + definition.bodySegments + .flatMap((segment) => + typeof segment === "object" && + segment !== null && + "type" in segment && + segment.type === "component" + ? [segment] + : [], + ) + .map((segment) => (segment as { position?: unknown }).position); + + expect(positionOf(whole.definition)).toEqual(positionOf(targeted.definition)); + expect(targeted.target).toBe("Kept"); + }); + + it("DT41: CRLF source keeps its original offsets and lines too", function* () { + const body = [ + "# Title", + "", + "## Skipped", + "", + "skipped body", + "", + "## Kept", + "", + "", + "", + ].join("\r\n"); + const whole = yield* parsed(body); + const targeted = yield* parsed(body, "Kept"); + const componentsOf = (segments: readonly unknown[]) => + segments.flatMap((segment) => + typeof segment === "object" && + segment !== null && + "type" in segment && + segment.type === "component" + ? [segment as { position?: { offset: number; line: number } }] + : [], + ); + expect(componentsOf(targeted.definition.bodySegments)[0]?.position).toEqual( + componentsOf(whole.definition.bodySegments)[0]?.position, + ); + }); + + it("DT42: frontmatter, props, and the return mode survive projection", function* () { + const body = [ + "---", + "title: Doc", + "props:", + " name:", + " type: string", + "returns:", + " type: object", + "---", + "", + "# Title", + "", + "## Kept", + "", + "kept", + "", + ].join("\n"); + const targeted = yield* parsed(body, "Kept"); + expect(targeted.definition.meta).toEqual({ title: "Doc" }); + expect(targeted.definition.props).toMatchObject({ properties: { name: { type: "string" } } }); + expect(targeted.definition.returns).toMatchObject({ type: "object" }); + expect(targeted.targets).toEqual(["Kept"]); + }); + + it("DT43: the untargeted parse still scans the whole body", function* () { + const whole = yield* parsed(SECTIONS); + expect(whole.target).toBe(undefined); + expect(whole.targets).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + const text = whole.definition.bodySegments + .map((segment) => (segment.type === "text" ? segment.content : "")) + .join(""); + expect(text).toBe(SECTIONS); + }); +}); + +describe("Tier DT — inspection", () => { + it("DT44: inspection reports the catalog without selecting anything", function* (): Operation { + const info = yield* inspectDocument(inlineSource(SECTIONS)); + expect(info.targets).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + expect(info.target).toBe(undefined); + }); + + it("DT45: inspection resolves a glob to the exact canonical target", function* (): Operation { + const info = yield* inspectDocument(inlineSource(SECTIONS, { target: "**/N*" })); + expect(info.target).toBe("Test/Node"); + expect(info.targets).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + }); + + it("DT46: an unresolvable target fails inspection", function* (): Operation { + let caught: unknown; + try { + yield* inspectDocument(inlineSource(SECTIONS, { target: "Nope" })); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(DocumentTargetError); + expect((caught as DocumentTargetError).kind).toBe("no-match"); + expect((caught as DocumentTargetError).selector).toBe("Nope"); + }); + + it("DT47: the error's data is frozen and rebuilt, not the parser's arrays", function* () { + const error = refusal(SECTIONS, "**"); + expect(Object.isFrozen(error.matches)).toBe(true); + expect(Object.isFrozen(error.available)).toBe(true); + expect(error.matches).not.toBe(outline(SECTIONS).targets); + // Encoded throughout, so a control character in a heading cannot reach a + // diagnostic literally. + expect(error.message).toContain('"**"'); + for (const line of error.message.split("\n").slice(1)) { + expect(line).not.toMatch(/[\u0000-\u001F]/); + } + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index aa991474..1c7ef380 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2351,9 +2351,16 @@ operation, with the read it leads to. What the journal holds is serializable: a repository selection records the chosen path and its content, and a registration records its origin, never its function. +The root's selection carries one more member. A targeted root resolves its +selector here too, against the text this operation is about to record, and +records the **exact** target it resolved to — so the section the run executed is +part of the record rather than something a later read rediscovers (§5.4). An +untargeted root records no `target` member, which is what keeps journals written +before targets existed readable. + ```typescript type DurableSelection = - | { kind: "repository"; path: string; content: string } + | { kind: "repository"; path: string; content: string; target?: string } | { kind: "registered"; origin: string; reserved: boolean }; function* durableImportComponent( @@ -2606,6 +2613,210 @@ schema, an invalid value, a body error, and a failure raised after `` all complete `Err`, and body text emitted before the failure remains only on the output stream. +#### Document targets + +A root document addresses its own sections. A **document target** is an +addressable static heading in the document's root Markdown flow, named by the +canonical path of heading labels that reaches it. Selecting one executes: + +1. the document preamble; +2. the direct content of every ancestor needed to reach the target; and +3. the selected heading's complete subtree. + +Sibling subtrees do not execute. Retained headings stay in the projected body, +so the projection reads as a document rather than as an excerpt. + +##### Which headings are targets + +Only root-level Markdown heading nodes form the outline. A heading inside a +block quote, a list, a fenced block, raw HTML, or component children is not one. + +Heading discovery does not parse raw XMD with a Markdown parser. A component's +children are ordinary text to that parser, and a blank line among them ends the +HTML block it inferred, which surfaces a child heading as a root heading. +Discovery instead parses a copy of the body in which the boundary scanner's +top-level component spans are replaced by spaces of the same length. Newline +positions, offsets, and everything outside those spans are unchanged, so a +heading found in the masked copy sits where it sits in the original, and the +original supplies its text and its source. + +A heading's parent is the nearest preceding heading with a smaller depth. +Skipped depths are ordinary. **Outermost** means the smallest heading depth +present in the root flow, which need not be `h1`. + +When the document has exactly one outermost heading, that heading is the +document **title**: it takes no level in any target path, it is no target +itself, and its heading and direct content are retained in every projection +beneath it. When the document has more than one outermost heading, each of them +takes a path level. A document with no addressable heading has an empty +catalog. + +A heading is **not addressable** when its own source overlaps executable +component syntax, or contains an unescaped Executable MDX interpolation — +`{meta.key}`, `{props.key}`, `{binding}`, and the dotted forms of each. Escaped +interpolation (`\{meta.key\}`) is literal static text and stays addressable. A +heading that renders no text is not addressable. An unaddressable heading +required as a path level makes its whole subtree unaddressable; because the sole +title is not a path level, static sections beneath a computed title remain +addressable. + +##### Labels and canonical encoding + +A label is the statically rendered Markdown text of the heading: formatting and +link destinations are removed, while visible text, inline-code text, and image +alternative text are retained. The result is normalized to NFC, every run of +Unicode whitespace collapses to one ASCII space, leading and trailing +whitespace is trimmed, and case is preserved. There are no generated slugs, +suffixes, case folding, or punctuation removal. + +A canonical target is the sequence of labels from the target's outermost +addressable ancestor to the target, each percent-encoded and joined with raw +`/`. Encoding leaves the RFC 3986 unreserved characters (`A-Z a-z 0-9 - . _ ~`) +alone and escapes everything else as uppercase UTF-8 hexadecimal, so a `/`, +`*`, `#`, or `%` inside a heading becomes `%2F`, `%2A`, `%23`, or `%25` and +cannot be read as syntax. + +The catalog is in source order and retains duplicates: two sections whose +canonical paths are equal stay two entries, so the ambiguity is observable. + +##### Selectors + +A document reference is: + +```text +# +``` + +The first raw `#` separates the two. Raw `/` separates target levels and raw +`*` and `**` are operators; the selector is split on those before its literal +chunks are percent-decoded, which is what keeps `%2F` a slash inside one label +and `%2A` a literal asterisk. Decoding is URI path decoding: `+` is a plus, not +a space. Malformed escapes, byte sequences that are not UTF-8, NUL, a leading +or trailing slash, and an empty level are all refused. Matching is +case-sensitive. + +- A literal level matches one canonical label exactly, after decoding and label + normalization. +- `*` within a level matches zero or more characters of that one label, and may + appear more than once. +- A level that is exactly `**` matches zero or more complete path levels. + +There is no `?`, character class, brace, or backslash dialect. Within a +wildcard level only the literal chunks are decoded and normalized; whitespace +beside a wildcard is part of what the selector asked for, and only the beginning +of the first chunk and the end of the last are trimmed. Matching compares +Unicode code points and completes in time bounded by the product of the pattern +and label sizes. + +A selector must resolve to exactly one catalog entry. Zero matches and several +matches both fail. Diagnostics report canonical encoded references, so a +duplicate canonical path is reported as an ambiguity rather than resolved. + +##### Projection + +Source ranges are defined against the original, unprojected body: + +- the **preamble** runs from the body start to immediately before the first + outermost heading; +- an **ancestor's direct content** runs from its heading start to its first + child heading's start, or to its subtree end when it has no child heading; + and +- the **selected subtree** runs from the selected heading's start to the next + heading of equal or smaller depth, or to the body end. + +The projected body is the preamble, each retained ancestor's direct content in +order, and the selected subtree. For a sole outermost title, the title is the +first retained ancestor even though it takes no level in the path. + +Each retained range is scanned separately, under the origin that range has in +the original file — its path, its offset, and its line. The ranges are not +concatenated and rescanned: skipped source must not renumber what follows it, +because a retained element's source position is what its expansion identifier +is derived from. A retained element therefore carries the same expansion ID in +a targeted run as in a full one, and two targets that retain it agree with each +other. The target string takes no part in expansion identity; a run's own +identity is what distinguishes the effects of two target runs. + +Frontmatter, root props, `returns`, the return mode, and `` behavior are +unchanged and apply to the projected body. Structural validation applies to the +projected body too: an invalid skipped sibling is irrelevant, while an invalid +retained range fails before any authored effect in the projection runs. + +##### Failure timing and durable identity + +Selection happens before the body expands. An invalid, unmatched, or ambiguous +selector runs no authored document effect. + +The live root import records the **exact canonical target**, never the caller's +selector. An untargeted import records no target member at all, so journals +written before targets existed stay readable by untargeted runs. + +A replay guard validates the target before the recorded run is reused. It parses +the recorded root content, resolves the current selector against *that* content, +and requires the result to equal the recorded exact target; the recorded content +is then what the projection is taken from. A different selector naming the same +section replays. A different exact target, a targeted request against an +untargeted record, an untargeted request against a targeted record, and a +selector the recorded content no longer resolves are all stale input (§6.11). +The check runs before a completed run's recorded terminal result can be reused, +so a finished journal cannot answer for a section it never ran. + +##### Naming a root document + +`@executablemd/core` exposes the shared shapes: + +```ts +interface FileRootDocument { + readonly path: string; + readonly source?: undefined; + readonly target?: string; +} + +interface InlineRootDocument { + readonly path: ""; + readonly source: string; + readonly target?: string; +} + +type RootDocumentSource = FileRootDocument | InlineRootDocument; + +function fileSource(reference: string): FileRootDocument; +function inlineSource(source: string, options?: { readonly target?: string }): InlineRootDocument; +function formatDocumentReference(path: string, target?: string): string; +``` + +`fileSource()` splits a document reference at the first raw `#`, percent-decodes +the path portion, and stores the fragment — still encoded — as `target`. It does +not decode the fragment as one string, because `%2F` must stay distinguishable +from a level separator. An empty path, a malformed escape, a byte sequence that +is not UTF-8, and NUL each fail with a cause-free `TypeError` whose message is +exactly `Invalid document reference`; the input is a command-line argument, and +echoing it back would put arbitrary bytes into a diagnostic. A filename +containing `#` is written `%23`, and one containing a literal `%HH` sequence is +written `%25HH`. + +`formatDocumentReference()` takes a decoded path and, optionally, an +already-canonical exact target. It encodes the path, validates the target rather +than encoding it again, and joins them with `#`. It is the one formatter +diagnostics, command output, and workflow handoff use. Making an authored glob +canonical is the selector parser's work, not this function's. + +Existing programmatic `{ path }` values and `inlineSource(source)` remain valid +and untargeted. + +An unresolvable target raises `DocumentTargetError`, whose `kind` is +`invalid-selector`, `no-match`, or `multiple-matches`. It carries the requested +`selector` as it arrived, the canonical encoded `matches` (empty except for +`multiple-matches`), and every canonical encoded `available` target. Its data is +rebuilt and frozen at the boundary, and its message quotes the selector as JSON +and lists canonical encoded references, so a heading holding a control character +cannot reach a diagnostic literally. It is an ordinary invocation failure, not a +durability or `API.Files` failure. Because target resolution sits inside the +durable root import, a failure reaching a caller through `execute()` arrives by +name and message like every other failure crossing that boundary; the typed +error is what `inspectDocument()` reports, and inspection is where a host +resolves a selector before running anything. + ### 5.5 The Component Api Expansion's context-dependent operations are exposed through one public @@ -5551,6 +5762,9 @@ workflow and returns a `DocumentExecution` handle. Options: - the root document source — either `path`, the path to the root markdown document, or an inline document built with `inlineSource(text)`, which carries the supplied text together with its `` identity +- `target?` — a document target selector, still encoded, resolved against the + root before its body expands (§5.4). `fileSource(reference)` builds a file + root and its selector from one document reference - `stream` — the durable stream that journals the run - `props?` — JSON values supplied to the root document (default: `{}`) - `componentDirs?` — component search directories (default: @@ -5675,9 +5889,18 @@ what it declares — without executing the document or creating a journal: - `returnMode` — `"text"` or `"value"`. An explicit `returns: { type: string }` produces the same effective schema as the default, so the mode is what tells the two apart. +- `targets` — every document target the root addresses, as canonical encoded + fragments without the document path or a leading `#`, in document order, + duplicates retained (§5.4). +- `target` — the exact canonical target the requested selector resolved to. + Present only when a target was requested and resolved, and never the caller's + glob. An invalid return schema fails inspection exactly as it fails execution: both -load the definition through the same path. +load the definition through the same path. So does an unresolvable target: +inspection discovers and selects targets without expanding the document, +evaluating a code block, importing a body component, or creating a journal, so +a host resolves a selector to one exact target before anything runs. `DocumentExecution` is an `Operation>`: `yield* execution` completes with `Ok(value)` on success and `Err(error)` on document, From 8c5f3ec52e10faf798bf0aa708ea942b82871f73 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Sun, 9 Aug 2026 12:09:33 -0400 Subject: [PATCH 02/14] =?UTF-8?q?=F0=9F=90=9B=20Hold=20a=20resumed=20run?= =?UTF-8?q?=20to=20the=20selection=20its=20journal=20recorded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A root import whose selector matched nothing failed the effect, so the journal kept only a serialized message and the replay guard delegated past every `err` result. A completed journal written by `Missing` then answered a later request for a section the document really has, with the old `Missing` error. A failed selection is an observation of the document, so it is recorded as one: its kind, the requested selector, the matches, and the catalog. The guard compares whole selection outcomes rather than target strings, reproduces a recorded failure from that record before the recorded Close can be reused, and reports any difference as stale input carrying no foreign object. `DocumentTargetError` now carries frozen, namespaced-tagged data and is recognized structurally, so inspection, a live run, and a replayed run all raise the same error with the same fields across separately loaded copies. A level is canonical only when decoding, label normalization, and canonical re-encoding reproduce it exactly, which refuses NFD, tabs, uncollapsed and edge whitespace, lowercase escapes, empty levels, raw operators, and a raw `#`. `formatDocumentReference()` only formats what `fileSource()` reads back. The recorded selector is sanitized invocation metadata: architecture.md now states that the exact canonical target is definition identity and a caller glob never substitutes for it. --- architecture.md | 31 +- packages/core/mod.ts | 9 +- packages/core/src/definition.ts | 21 +- packages/core/src/document-targets.ts | 377 ++++++++++++++++-- packages/core/src/execute.ts | 238 +++++++---- packages/core/src/root-source.ts | 13 +- .../tests/document-target-execution.test.ts | 94 ++++- packages/core/tests/document-targets.test.ts | 237 ++++++++++- specs/executable-mdx-spec.md | 153 +++++-- 9 files changed, 985 insertions(+), 188 deletions(-) diff --git a/architecture.md b/architecture.md index 44f32212..c174baf2 100644 --- a/architecture.md +++ b/architecture.md @@ -24,7 +24,7 @@ Existing documents and code get aligned to this section retroactively. | middleware | applied by the lexical structure, used by runtime execution | | workflow run | a workflow being carried out with its progress and outcome recorded durably; document executions perform its work, while ongoing effects remain scoped to the document execution in which they run | | document execution | one evaluation of a root document initiated through `execute()`, producing one output stream and one completion result while reading and appending a durable journal; its ongoing effects belong to the Effection scope in which the evaluation runs | -| workflow definition | what a workflow run is a run of: a versioned descriptor naming an immutable object — its format and object ID — together with the repository-relative path of the root document inside it. A repository locator is not part of it, and it is distinct from every Repository created inside the run's Workspace | +| workflow definition | what a workflow run is a run of: a versioned descriptor naming an immutable object — its format and object ID — together with the repository-relative path of the root document inside it, and the exact canonical document target when one is selected. A repository locator is not part of it, and it is distinct from every Repository created inside the run's Workspace | | retrieval metadata | replaceable, credential-free information about where a workflow definition can be fetched from now; it takes no part in run identity and is reauthorized by the host before use | | stop reason | why a workflow run or a document execution stopped: a categorical host code, or a reference to an already-filtered journal event | | run ID | an opaque stable public identifier generated by the host or selected by an authorized caller; it associates the run's durable records and effects, remains unchanged for the life of the run, and has no semantics beyond equality and lifecycle addressing | @@ -621,15 +621,26 @@ entry: naming none and naming several are both failures, and two sections that canonicalize to the same path stay two entries so the ambiguity is reported rather than resolved arbitrarily. -The selector and the target it resolves to are different things, and only one -of them is identity. A selector is invocation input — it describes what a -caller asked for, and two callers may spell the same request differently. The -**exact resolved target** is what ran, so it is what a document execution -records durably, what a targeted workflow definition carries, and what a resumed -run is checked against. A caller's glob is never recorded and never re-resolved -against a newer checkout; a resumed run re-resolves the current selector against -the *recorded* content and refuses to continue unless it still names the -recorded target. +The selector and the target it resolves to are different things, and only one of +them is identity. + +**The exact canonical target is definition identity.** It is what ran, so it is +what a document execution records durably, what a targeted workflow definition +carries, and what a resumed run is checked against. + +**A caller's glob is non-authoritative invocation metadata.** It describes what +a caller asked for — two callers may spell one request differently — and it +never substitutes for the exact target: it does not occupy the recorded +exact-target field, it never enters a workflow definition, and it is never +re-resolved against a newer checkout to decide what a resumed run means. A glob +is retained in exactly one place, a failed selection's structural record, and +only so that an ordinary failed execution can be reproduced. + +A resumed run re-resolves the current selector against the *recorded* content +and refuses to continue unless the outcome is the one recorded. A failed +selection is an outcome too, and is recorded and compared as one — otherwise a +journal left by a selector that matched nothing would answer a later request for +a section that does exist. ## Expansion identity diff --git a/packages/core/mod.ts b/packages/core/mod.ts index 332f96ac..45af4b8a 100644 --- a/packages/core/mod.ts +++ b/packages/core/mod.ts @@ -142,8 +142,13 @@ export type { InlineRootDocument, RootDocumentSource, } from "./src/root-source.ts"; -export { DocumentTargetError } from "./src/document-targets.ts"; -export type { DocumentTargetErrorKind } from "./src/document-targets.ts"; +export { + asDocumentTargetError, + DocumentTargetError, + isDocumentTargetError, + parseDocumentTargetFailure, +} from "./src/document-targets.ts"; +export type { DocumentTargetErrorKind, DocumentTargetFailure } from "./src/document-targets.ts"; export { inspectComponent, inspectDocument } from "./src/inspect.ts"; export type { ComponentInfo, diff --git a/packages/core/src/definition.ts b/packages/core/src/definition.ts index f24457b8..388d7f49 100644 --- a/packages/core/src/definition.ts +++ b/packages/core/src/definition.ts @@ -1,9 +1,10 @@ -import type { Operation } from "effection"; +import { Ok } from "effection"; +import type { Operation, Result } from "effection"; import type { ComponentDefinition, Segment } from "./types.ts"; import { parseFrontmatter } from "./frontmatter.ts"; import { compilePropsSchema, compileReturnsSchema } from "./validate.ts"; import { scanComponentSpans, scanSegments } from "./scanner.ts"; -import { outlineDocument, retainedRanges, selectTarget } from "./document-targets.ts"; +import { findTarget, outlineDocument, retainedRanges, selectTarget } from "./document-targets.ts"; import type { DocumentOutline } from "./document-targets.ts"; import matter from "gray-matter"; @@ -56,14 +57,22 @@ function documentOutline(path: string, content: string): DocumentOutline { } /** - * The exact canonical target a selector names in this document's content. + * The exact canonical target a selector names in this document's content, or + * the failure describing why it names none. * * Synchronous and free of effects, so the resolution that decides *what* runs * happens before anything runs — including inside the durable operation that - * records the root, and inside a replay guard reading recorded content. + * records the root, and inside a replay guard reading recorded content. The + * outcome comes back rather than being thrown because both of those callers + * record it as data before anyone reports it. */ -export function resolveDocumentTarget(path: string, content: string, selector: string): string { - return selectTarget(documentOutline(path, content), selector).target; +export function resolveDocumentTarget( + path: string, + content: string, + selector: string, +): Result { + const found = findTarget(documentOutline(path, content), selector); + return found.ok ? Ok(found.value.target) : found; } interface CompiledFrontmatter { diff --git a/packages/core/src/document-targets.ts b/packages/core/src/document-targets.ts index 8b3912bb..9329aab3 100644 --- a/packages/core/src/document-targets.ts +++ b/packages/core/src/document-targets.ts @@ -22,6 +22,8 @@ * identifiers equal between a full run and a targeted one. */ +import { Err, Ok } from "effection"; +import type { Result } from "effection"; import { remark } from "remark"; import { toString as mdastToString } from "mdast-util-to-string"; @@ -66,6 +68,12 @@ export interface DocumentOutline { /** Why a requested target did not resolve to exactly one catalog entry. */ export type DocumentTargetErrorKind = "invalid-selector" | "no-match" | "multiple-matches"; +const KINDS: readonly DocumentTargetErrorKind[] = [ + "invalid-selector", + "no-match", + "multiple-matches", +]; + const KIND_WORDING: ReadonlyMap = new Map([ ["invalid-selector", "is not a valid document target selector"], ["no-match", "matches no document target"], @@ -73,20 +81,25 @@ const KIND_WORDING: ReadonlyMap = new Map([ ]); /** - * A requested document target that does not name exactly one section. + * The structural tag a document-target failure carries. * - * An ordinary invocation failure: the caller asked for something the document - * does not offer, and nothing durable or contained is involved. It is raised - * before the document expands, so a run that cannot decide what to execute - * executes nothing. + * Namespaced and stable, because it is the whole recognition mechanism. Two + * loaded copies of this package are two classes, so `instanceof` answers false + * between them; a failure built by one copy has to be recognized by the other + * on exactly the same terms as one built here (AGENTS.md rule 15). + */ +const DOCUMENT_TARGET_FAILURE = "executablemd.document-target-failure"; + +/** + * Why one requested selector did not name exactly one section, as data. * - * Everything it carries is rebuilt and frozen here. The selector arrives from a - * command line and the catalog from a parser, and neither object belongs to a - * failure that outlives them. Every reference in the message is canonically - * encoded, so a heading holding a control character cannot reach a diagnostic - * literally. + * Frozen and rebuilt from validated parts wherever it crosses a boundary. The + * selector is retained because reproducing an ordinary failed execution needs + * to say what was asked for — it is sanitized invocation metadata, never + * identity, and it never stands in for an exact target. */ -export class DocumentTargetError extends Error { +export interface DocumentTargetFailure { + readonly type: typeof DOCUMENT_TARGET_FAILURE; readonly kind: DocumentTargetErrorKind; /** The selector fragment as it was requested, still encoded. */ readonly selector: string; @@ -94,29 +107,287 @@ export class DocumentTargetError extends Error { readonly matches: readonly string[]; /** Every canonical encoded target the document offers. */ readonly available: readonly string[]; +} - constructor( - kind: DocumentTargetErrorKind, - selector: string, - matches: readonly string[], - available: readonly string[], - ) { - const listed = kind === "multiple-matches" ? matches : available; - const heading = kind === "multiple-matches" ? "Matched targets:" : "Available targets:"; - super( - `${JSON.stringify(selector)} ${KIND_WORDING.get(kind)}.\n` + - (listed.length === 0 - ? "The document has no targets." - : `${heading}\n${listed.map((target) => ` ${target}`).join("\n")}`), - ); +/** The fields a failure carries, without the tag that authenticates them. */ +interface TargetFailureFields { + kind: DocumentTargetErrorKind; + selector: string; + matches: string[]; + available: string[]; +} + +function attempt(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** Whether this is an Error, without trusting its prototype chain. */ +function isError(value: unknown): value is Error { + return attempt(() => value instanceof Error) === true; +} + +/** One property, read through a trap that may refuse or fail. */ +function property(target: object, name: string): unknown { + return attempt(() => Reflect.get(target, name)); +} + +function stringList(value: unknown): string[] | undefined { + return attempt(() => { + if (!Array.isArray(value)) { + return undefined; + } + const items: string[] = []; + for (let index = 0; index < value.length; index++) { + const item = index in value ? value[index] : undefined; + if (typeof item !== "string") { + return undefined; + } + items.push(item); + } + return items; + }); +} + +/** + * Read a candidate's failure fields, rebuilding every one of them. + * + * Total: an unreadable property, a missing one, a kind outside the closed set, + * a sparse or non-string list, and matches on a kind that has none are all "not + * this shape" rather than a throw. Nothing the candidate owns is retained — the + * arrays that come back are new. + */ +function targetFailureFields(value: unknown): TargetFailureFields | undefined { + return attempt(() => { + if (!isRecord(value)) { + return undefined; + } + const kind = KINDS.find((candidate) => candidate === property(value, "kind")); + const selector = property(value, "selector"); + const matches = stringList(property(value, "matches")); + const available = stringList(property(value, "available")); + if (kind === undefined || typeof selector !== "string") { + return undefined; + } + if (matches === undefined || available === undefined) { + return undefined; + } + // `matches` is the ambiguity list and nothing else; a populated one under + // any other kind is not the closed shape this contract describes. + if (kind !== "multiple-matches" && matches.length > 0) { + return undefined; + } + return { kind, selector, matches, available }; + }); +} + +/** Freeze validated fields into the failure data an error carries. */ +function sealFailure(fields: TargetFailureFields): DocumentTargetFailure { + return Object.freeze({ + type: DOCUMENT_TARGET_FAILURE, + kind: fields.kind, + selector: fields.selector, + matches: Object.freeze([...fields.matches]), + available: Object.freeze([...fields.available]), + }); +} + +/** + * The failure data this value carries, if it carries valid, tagged, frozen + * data. + * + * Every field is checked, the member count with them, and that the object is + * frozen: extra keys are not the shape this contract describes, and a mutable + * one is not the shape a constructor here produces. + */ +export function parseDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined { + return attempt(() => { + if (!isRecord(value) || property(value, "type") !== DOCUMENT_TARGET_FAILURE) { + return undefined; + } + if (attempt(() => Object.isFrozen(value)) !== true) { + return undefined; + } + if (attempt(() => Object.keys(value).length) !== 5) { + return undefined; + } + const fields = targetFailureFields(value); + return fields === undefined ? undefined : sealFailure(fields); + }); +} + +/** + * The failure a journal record describes, rebuilt and sealed. + * + * The record is untagged — its place inside a recorded root-import selection is + * what identifies it — so this validates the fields and supplies the tag, + * rather than requiring a tag the journal never held. + */ +export function recordedDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined { + const fields = targetFailureFields(value); + return fields === undefined ? undefined : sealFailure(fields); +} + +/** + * The one diagnostic a failure carries, derived from its data alone. + * + * Recognition compares against this, so the message cannot disagree with the + * fields. Every reference is canonically encoded and the selector is JSON + * quoted, so a heading holding a control character cannot reach a diagnostic + * literally. + */ +function documentTargetMessage(failure: DocumentTargetFailure): string { + const ambiguous = failure.kind === "multiple-matches"; + const listed = ambiguous ? failure.matches : failure.available; + const heading = ambiguous ? "Matched targets:" : "Available targets:"; + return ( + `${JSON.stringify(failure.selector)} ${KIND_WORDING.get(failure.kind)}.\n` + + (listed.length === 0 + ? "The document has no targets." + : `${heading}\n${listed.map((target) => ` ${target}`).join("\n")}`) + ); +} + +/** + * Everything a constructor here puts on the Error itself, and nothing else. + * + * `message` and `stack` are non-enumerable own properties of every Error, so + * what remains enumerable is exactly what this constructor assigned. + */ +const FAILURE_MEMBERS: readonly string[] = ["data", "name"]; + +function hasOnlyContractMembers(error: Error): boolean { + const keys = attempt(() => [...Object.keys(error)].sort()); + if (keys === undefined || keys.length !== FAILURE_MEMBERS.length) { + return false; + } + if (!keys.every((key, index) => key === FAILURE_MEMBERS[index])) { + return false; + } + const payload = attempt(() => + Object.getOwnPropertySymbols(error).filter( + (symbol) => Object.getOwnPropertyDescriptor(error, symbol)?.enumerable === true, + ), + ); + return payload !== undefined && payload.length === 0; +} + +/** + * A requested document target that does not name exactly one section. + * + * An ordinary invocation failure: the caller asked for something the document + * does not offer, and nothing durable or contained is involved. It is raised + * before the document expands, so a run that cannot decide what to execute + * executes nothing. + * + * Its data is the contract; the message is derived from it. Construct one from + * validated data — `documentTargetError()` — rather than from parts, so a + * failure rebuilt at a journal boundary is indistinguishable from the one the + * live run raised. + */ +export class DocumentTargetError extends Error { + readonly data: DocumentTargetFailure; + + constructor(data: DocumentTargetFailure) { + super(documentTargetMessage(data)); this.name = "DocumentTargetError"; - this.kind = kind; - this.selector = selector; - this.matches = Object.freeze([...matches]); - this.available = Object.freeze([...available]); + this.data = data; } } +/** Build the failure this selector produced, from parts this module owns. */ +export function documentTargetFailure( + kind: DocumentTargetErrorKind, + selector: string, + matches: readonly string[], + available: readonly string[], +): DocumentTargetFailure { + return sealFailure({ + kind, + selector, + matches: [...matches], + available: [...available], + }); +} + +/** + * Rebuild the error a failure describes. + * + * The one constructor call outside this module's own selection path, so a + * replayed failure and a live one are the same object shape carrying the same + * fields — a caller cannot tell which run raised it, and does not have to. + */ +export function documentTargetError(data: DocumentTargetFailure): DocumentTargetError { + return new DocumentTargetError(data); +} + +/** + * Whether this failure satisfies the whole contract, not merely the tag. + * + * Structural throughout, so a failure constructed by a separately loaded copy + * of this package is recognized on exactly the same terms as one constructed + * here. The name is checked rather than the class for the same reason: a second + * copy's constructor is a different function producing the same name. + * + * Stricter than `parseDocumentTargetFailure` because recognition hands the + * object onward: the message has to be the one its own data derives, there can + * be no cause, and no member beyond the contract — otherwise a candidate could + * carry a path, a foreign object, or a second message past this boundary under + * a recognized tag. + */ +export function isDocumentTargetError(error: unknown): error is DocumentTargetError { + return ( + attempt(() => { + if (!isError(error)) { + return false; + } + const data = parseDocumentTargetFailure(property(error, "data")); + if (data === undefined) { + return false; + } + if (property(error, "name") !== "DocumentTargetError") { + return false; + } + if (property(error, "message") !== documentTargetMessage(data)) { + return false; + } + if (property(error, "cause") !== undefined) { + return false; + } + return hasOnlyContractMembers(error); + }) === true + ); +} + +/** The document-target failure this error is, by identity. */ +export function asDocumentTargetError(error: unknown): DocumentTargetError | undefined { + return isDocumentTargetError(error) ? error : undefined; +} + +/** Whether two failures describe the same selection outcome, field by field. */ +export function sameDocumentTargetFailure( + left: DocumentTargetFailure, + right: DocumentTargetFailure, +): boolean { + return ( + left.kind === right.kind && + left.selector === right.selector && + sameList(left.matches, right.matches) && + sameList(left.available, right.available) + ); +} + +function sameList(left: readonly string[], right: readonly string[]): boolean { + return left.length === right.length && left.every((item, index) => item === right[index]); +} + const UNRESERVED = /^[A-Za-z0-9\-._~]$/; const HEX = /^[0-9A-Fa-f]$/; @@ -201,23 +472,26 @@ export function normalizeLabel(text: string): string { } /** - * Whether a fragment is already an exact canonical target: raw `/` between - * nonempty levels, every level percent-encoded exactly as this module encodes - * it, and no wildcard operator anywhere. + * Whether a fragment is already an exact canonical target. + * + * A level is canonical only when decoding it, normalizing the label, and + * re-encoding that label reproduce the level byte for byte. Requiring the whole + * round trip is what makes this total: it rejects a wildcard operator, an empty + * level, a lowercase escape, a raw `#`, an NFD spelling, a tab, and leading, + * trailing, or uncollapsed whitespace without naming any of them, because none + * of them is what this module would have written. */ export function isCanonicalTarget(target: string): boolean { if (target.length === 0) { return false; } return target.split("/").every((level) => { - if (level.length === 0 || level.includes("*")) { - return false; - } const decoded = decodePercentEncoded(level); if (decoded === undefined || decoded.length === 0) { return false; } - return encodeTargetLabel(decoded) === level; + const label = normalizeLabel(decoded); + return label === decoded && encodeTargetLabel(label) === level; }); } @@ -241,6 +515,12 @@ function parseSelector(selector: string): readonly SelectorLevel[] | undefined { if (selector.length === 0 || selector.startsWith("/") || selector.endsWith("/")) { return undefined; } + // A raw `#` is the reference's own delimiter, so it never reaches a selector + // by the supported route and cannot be written back into one. `%23` addresses + // a heading that really contains it. + if (selector.includes("#")) { + return undefined; + } const levels: SelectorLevel[] = []; for (const raw of selector.split("/")) { if (raw.length === 0) { @@ -347,24 +627,37 @@ function matchPath(levels: readonly SelectorLevel[], path: readonly string[]): b * duplicate entries, which is what makes that ambiguity observable at all. */ export function selectTarget(outline: DocumentOutline, selector: string): DocumentTarget { + const found = findTarget(outline, selector); + if (found.ok) { + return found.value; + } + throw found.error; +} + +/** The entry a selector names, or the failure describing why it names none. */ +export function findTarget(outline: DocumentOutline, selector: string): Result { + const fail = ( + kind: DocumentTargetErrorKind, + matches: readonly string[], + ): Result => + Err(documentTargetError(documentTargetFailure(kind, selector, matches, outline.targets))); + const levels = parseSelector(selector); if (levels === undefined) { - throw new DocumentTargetError("invalid-selector", selector, [], outline.targets); + return fail("invalid-selector", []); } const matched = outline.entries.filter((entry) => matchPath(levels, entry.labels)); const first = matched[0]; if (first === undefined) { - throw new DocumentTargetError("no-match", selector, [], outline.targets); + return fail("no-match", []); } if (matched.length > 1) { - throw new DocumentTargetError( + return fail( "multiple-matches", - selector, matched.map((entry) => entry.target), - outline.targets, ); } - return first; + return Ok(first); } /** diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index d15ae2ce..1e6bcd83 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -52,6 +52,14 @@ import { parseRootMarkdownDefinition, resolveDocumentTarget, } from "./definition.ts"; +import { + asDocumentTargetError, + documentTargetError, + documentTargetFailure, + recordedDocumentTargetFailure, + sameDocumentTargetFailure, +} from "./document-targets.ts"; +import type { DocumentTargetFailure } from "./document-targets.ts"; import { parseReturnsDeclaration } from "./frontmatter.ts"; import { expandSegments, @@ -138,8 +146,39 @@ export type ExecuteOptions = RootDocumentSource & ExecuteSettings; */ type DurableSelection = | { kind: "repository"; path: string; content: string; target?: string } + | { kind: "target-failure"; path: string; content: string; failure: TargetFailureRecord } | { kind: "registered"; origin: string; reserved: boolean }; +/** + * A selection that named no single section, as the journal holds it. + * + * A failed selection is an observation of the document, not an accident: the + * text was read, and it does not offer what was asked for. Recording it as data + * — rather than letting the effect fail and keeping only a serialized message — + * is what lets a resumed run tell "the same request, failing the same way" from + * "a different request the recorded run never made", and what lets the failure + * be rebuilt with its fields intact instead of reduced to prose. + * + * `selector` is sanitized invocation metadata. It is never identity: it does + * not occupy the exact-target field, and it never reaches a workflow + * definition. + */ +type TargetFailureRecord = { + kind: string; + selector: string; + matches: string[]; + available: string[]; +}; + +function targetFailureRecord(failure: DocumentTargetFailure): TargetFailureRecord { + return { + kind: failure.kind, + selector: failure.selector, + matches: [...failure.matches], + available: [...failure.available], + }; +} + function* durableImportComponent( name: string, root: RootDocumentSource | undefined, @@ -161,13 +200,22 @@ function* durableImportComponent( // for, not what ran. const path = rootSourcePath(root); const content = yield* readRootSource(root); - const target = - root.target === undefined ? undefined : resolveDocumentTarget(path, content, root.target); + if (root.target === undefined) { + return { kind: "repository", path, content }; + } + const resolved = resolveDocumentTarget(path, content, root.target); + if (resolved.ok) { + return { kind: "repository", path, content, target: resolved.value }; + } + const failure = asDocumentTargetError(resolved.error); + if (failure === undefined) { + throw resolved.error; + } return { - kind: "repository", + kind: "target-failure", path, content, - ...(target === undefined ? {} : { target }), + failure: targetFailureRecord(failure.data), }; } @@ -196,6 +244,20 @@ function* durableImportComponent( }, )) as DurableSelection; + // Rebuilt here rather than carried out of the durable operation, so a replayed + // failed selection and a live one raise the same error with the same fields. + // Parsed rather than trusted: the record is journal data. + if (selection.kind === "target-failure") { + const failure = recordedDocumentTargetFailure(selection.failure); + if (failure === undefined) { + throw new Error( + "The recorded root document import describes a failed target selection this version " + + "cannot read.", + ); + } + throw documentTargetError(failure); + } + if (selection.kind === "registered") { // The function was never journaled. Find the implementation the recorded // origin names in the registry this run has; refusing when it is gone is @@ -271,8 +333,24 @@ function isFunctionComponent(value: unknown): value is FunctionComponent { return typeof value === "function"; } +/** + * What one run's selector decided: the whole document, one exact section, or a + * failure that named none. + * + * Selection is compared as an outcome rather than as a target string, because a + * failed selection is an outcome too. Without the third case a journal written + * by one selector that matched nothing would answer a later request for a + * section that does exist. + */ +type SelectionOutcome = + | { kind: "whole" } + | { kind: "exact"; target: string } + | { kind: "failed"; failure: DocumentTargetFailure }; + /** The recorded root import this event is, when it is one that can be read. */ -function recordedRootImport(event: Yield): { content: string; target?: string } | undefined { +function recordedRootImport( + event: Yield, +): { content: string; selection: SelectionOutcome } | undefined { if ( event.description.type !== "import_component" || event.description.name !== "__root__" || @@ -285,99 +363,111 @@ function recordedRootImport(event: Yield): { content: string; target?: string } return undefined; } const content = record["content"]; - const target = record["target"]; - if (typeof content !== "string" || (target !== undefined && typeof target !== "string")) { + if (typeof content !== "string") { return undefined; } - return target === undefined ? { content } : { content, target }; + if (record["kind"] === "target-failure") { + const failure = recordedDocumentTargetFailure(record["failure"]); + return failure === undefined ? undefined : { content, selection: { kind: "failed", failure } }; + } + const target = record["target"]; + if (target === undefined) { + return { content, selection: { kind: "whole" } }; + } + return typeof target === "string" ? { content, selection: { kind: "exact", target } } : undefined; +} + +/** What this run's selector decides against the content the journal recorded. */ +function requestedSelection(root: RootDocumentSource, content: string): SelectionOutcome { + if (root.target === undefined) { + return { kind: "whole" }; + } + const resolved = resolveDocumentTarget(rootSourcePath(root), content, root.target); + if (resolved.ok) { + return { kind: "exact", target: resolved.value }; + } + const failure = asDocumentTargetError(resolved.error); + // A failure this module did not build is not a selection outcome that can be + // compared, so it cannot be shown compatible with anything. + return failure === undefined + ? { kind: "failed", failure: documentTargetFailure("invalid-selector", root.target, [], []) } + : { kind: "failed", failure: failure.data }; +} + +function sameSelection(recorded: SelectionOutcome, requested: SelectionOutcome): boolean { + if (recorded.kind === "whole" || requested.kind === "whole") { + return recorded.kind === requested.kind; + } + if (recorded.kind === "exact" || requested.kind === "exact") { + return ( + recorded.kind === "exact" && + requested.kind === "exact" && + recorded.target === requested.target + ); + } + return sameDocumentTargetFailure(recorded.failure, requested.failure); +} + +function describeSelection(selection: SelectionOutcome): string { + switch (selection.kind) { + case "whole": + return "the whole document"; + case "exact": + return `the target ${JSON.stringify(selection.target)}`; + case "failed": + return `a selector that names no single target (${selection.failure.kind})`; + } } /** - * Refuse to replay a run that was recorded against a different section. + * Hold a resumed run to the selection its journal recorded. * * Only `type` and `name` decide whether a journal entry matches, and the root - * import's name is the same for every target — so without this, resuming with a - * different selector would restore the recorded content and then project a - * section the recorded run never executed. + * import's name is the same for every selector — so without this, resuming with + * a different one would restore the recorded content and then project a section + * the recorded run never executed, or restore a recorded selection failure as + * the answer to a request that would have succeeded. * - * The current selector is resolved against the *recorded* content, so a glob - * that still names the same section replays and a glob that now names another - * one does not. A selector that has become invalid or ambiguous against that - * content is refused for the same reason: nothing here may guess which section - * a resumed run meant. + * The current selector is resolved against the *recorded* content, so what is + * compared is what each run decided, not what each caller typed: a different + * glob naming the same section replays, and so does the same failing selector, + * while any difference in outcome is stale input. * - * This validates in the check phase rather than the decide phase because - * `durableRun` reuses a recorded root Close before any effect is replayed. A - * decision made later would never run for a completed journal, which is exactly - * the run whose recorded target must still be the one being asked for. + * A recorded failed selection is reproduced here, not delegated. Nothing later + * would reproduce it with its fields intact — `durableRun` reuses a recorded + * root Close before any effect is replayed, and that path restores a + * deserialized error — so a recorded failure is rebuilt from its structural + * record and raised before that reuse. Either way no authored effect runs. * - * A `StaleInputError`, so it propagates as a durability failure rather than - * being printed into the document. + * This is also why validation is in the check phase rather than the decide + * phase: a decision made during replay never runs for a completed journal, + * which is exactly the run whose recorded selection must still be the one being + * asked for. */ -function refuseChangedRootTarget(root: RootDocumentSource): Operation { +function holdRootSelection(root: RootDocumentSource): Operation { return ReplayGuard.around({ *check([event], next) { const recorded = recordedRootImport(event); if (recorded === undefined) { return yield* next(event); } - const requested = resolveRecordedTarget(root, recorded.content); - const compatible = - requested.kind === "whole" - ? recorded.target === undefined - : requested.kind === "exact" && requested.target === recorded.target; - if (!compatible) { - const stale = new StaleInputError( - `the recorded root document import ran ${describeRecorded(recorded.target)}, and this ` + - `run asks for ${describeRequested(requested)}. Re-run the document from the ` + - "start rather than resuming from a journal that recorded another section.", + const requested = requestedSelection(root, recorded.content); + if (!sameSelection(recorded.selection, requested)) { + throw new StaleInputError( + `the recorded root document import ran ${describeSelection(recorded.selection)}, and ` + + `this run asks for ${describeSelection(requested)}. Re-run the document from the ` + + "start rather than resuming from a journal that recorded another selection.", { coroutineId: event.coroutineId, description: event.description }, ); - if (requested.kind === "unresolved") { - stale.cause = requested.failure; - } - throw stale; + } + if (recorded.selection.kind === "failed") { + throw documentTargetError(recorded.selection.failure); } return yield* next(event); }, }); } -/** What this run's selector names in the recorded content. */ -type RequestedTarget = - | { kind: "whole" } - | { kind: "exact"; target: string } - | { kind: "unresolved"; failure: unknown }; - -function resolveRecordedTarget(root: RootDocumentSource, content: string): RequestedTarget { - if (root.target === undefined) { - return { kind: "whole" }; - } - try { - return { - kind: "exact", - target: resolveDocumentTarget(rootSourcePath(root), content, root.target), - }; - } catch (failure) { - return { kind: "unresolved", failure }; - } -} - -function describeRecorded(target: string | undefined): string { - return target === undefined ? "the whole document" : `the target ${JSON.stringify(target)}`; -} - -function describeRequested(requested: RequestedTarget): string { - switch (requested.kind) { - case "whole": - return "the whole document"; - case "exact": - return `the target ${JSON.stringify(requested.target)}`; - case "unresolved": - return "a target that recorded content no longer names exactly once"; - } -} - const execFactory: ModifierFactory = (_params) => (_args, _next) => (function* () { const context = yield* useCodeBlock(); @@ -998,7 +1088,7 @@ function* executeDocument(options: ExecuteOptions): Operation // Installed before the durable run, so the check phase sees the recorded // root import before `durableRun` can reuse a recorded Close. - yield* refuseChangedRootTarget(root); + yield* holdRootSelection(root); // The policy is selected here — before the durable run and before any // document, frontmatter, prop, component, or eval code exists — so the diff --git a/packages/core/src/root-source.ts b/packages/core/src/root-source.ts index 15001889..51503a12 100644 --- a/packages/core/src/root-source.ts +++ b/packages/core/src/root-source.ts @@ -87,13 +87,22 @@ export function formatDocumentReference(path: string, target?: string): string { if (path.length === 0) { throw new TypeError(INVALID_REFERENCE); } + // The round trip is the rule, not a sample of it: a path only formats when + // decoding what this would write reproduces it exactly. NUL, which the + // decoder refuses, and an unpaired surrogate, which encodes lossily to the + // replacement character, both fail here rather than producing a reference + // that names a different file than the one asked about. + const encoded = encodeDocumentPath(path); + if (decodePercentEncoded(encoded) !== path) { + throw new TypeError(INVALID_REFERENCE); + } if (target === undefined) { - return encodeDocumentPath(path); + return encoded; } if (!isCanonicalTarget(target)) { throw new TypeError(INVALID_REFERENCE); } - return `${encodeDocumentPath(path)}#${target}`; + return `${encoded}#${target}`; } /** The identity printed errors and source positions report for this root. */ diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 0236bbde..8df04fdf 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -29,7 +29,11 @@ import { execute } from "../src/execute.ts"; import { inspectDocument } from "../src/inspect.ts"; import { getExpansion } from "../src/expansion.ts"; import { registerComponents } from "../src/components/registration.ts"; -import { DocumentTargetError } from "../src/document-targets.ts"; +import { + asDocumentTargetError, + DocumentTargetError, + isDocumentTargetError, +} from "../src/document-targets.ts"; import { fileSource, formatDocumentReference, inlineSource } from "../src/root-source.ts"; import type { RootDocumentSource } from "../src/root-source.ts"; import { asText } from "./helpers.ts"; @@ -361,11 +365,16 @@ describe("Tier TX — targeted execution", () => { const stream = new InMemoryStream(); const seen: Probes = { names: [], ids: [] }; const error = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream, seen); - expect((error as Error).name).toBe("DocumentTargetError"); - expect((error as Error).message).toContain("matches no document target"); + expect(isDocumentTargetError(error)).toBe(true); + expect(asDocumentTargetError(error)?.data).toMatchObject({ + kind: "no-match", + selector: "Missing", + matches: [], + }); expect(seen.names).toEqual([]); - // The root import is the only effect the journal saw, and it failed. - expect(rootImports(stream).map((event) => event.result.status)).toEqual(["err"]); + // The selection was recorded as an observation, and the effect succeeded: + // what failed is the document, deterministically, from that record. + expect(rootImports(stream).map((event) => event.result.status)).toEqual(["ok"]); expect(stream.snapshot().filter((event) => event.type === "yield").length).toBe(1); }); @@ -373,7 +382,7 @@ describe("Tier TX — targeted execution", () => { const stream = new InMemoryStream(); const seen: Probes = { names: [], ids: [] }; const error = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream, seen); - expect((error as Error).message).toContain("matches more than one document target"); + expect(asDocumentTargetError(error)?.data.kind).toBe("multiple-matches"); expect(seen.names).toEqual([]); expect(stream.snapshot().filter((event) => event.type === "yield").length).toBe(1); }); @@ -464,7 +473,78 @@ describe("Tier TX — targeted replay", () => { // that it means the recorded section. const error = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream); expect(error).toBeInstanceOf(StaleInputError); - expect((error as Error).cause).toBeInstanceOf(DocumentTargetError); + // Stale input is what this is, and it carries nothing else: the guard + // retains no failure object from the selection it could not match. + expect((error as Error).cause).toBe(undefined); + expect(isDocumentTargetError(error)).toBe(false); + }); + + /** + * The defect this stack shipped first, and the reason a failed selection is + * recorded structurally rather than left to the effect's own failure. + * + * `Missing` matched nothing, so the run failed and `durableRun` closed the + * root. A later request for `Good` — which the document really offers — was + * then answered with the recorded `Missing` error, because the guard + * delegated past every `err` result and the completed Close short-circuited + * everything after it. + */ + it("TX24: a journal from a failed selector never answers a valid one", function* () { + const stream = new InMemoryStream(); + const first = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + expect(asDocumentTargetError(first)?.data.selector).toBe("Missing"); + + const second = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect(second).toBeInstanceOf(StaleInputError); + expect((second as Error).message).not.toContain("Missing"); + }); + + it("TX25: the same failing selector replays its own recorded failure", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + const first = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + const replayed = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream, seen); + + expect(isDocumentTargetError(replayed)).toBe(true); + expect(asDocumentTargetError(replayed)?.data).toEqual(asDocumentTargetError(first)?.data); + expect(seen.names).toEqual([]); + // One recorded import, and it is the first run's. + expect(rootImports(stream).length).toBe(1); + }); + + it("TX26: one failed selection never answers for another kind of failure", function* () { + const stream = new InMemoryStream(); + yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + + // Ambiguous rather than unmatched: a different outcome, not a different + // spelling of the same one. + const ambiguous = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream); + expect(ambiguous).toBeInstanceOf(StaleInputError); + + // Invalid syntax rather than unmatched. + const invalid = yield* failure(inlineSource(SECTIONS, { target: "/bad" }), stream); + expect(invalid).toBeInstanceOf(StaleInputError); + + // A different selector that also matches nothing is still a different + // request, and the recorded failure describes the one that was made. + const other = yield* failure(inlineSource(SECTIONS, { target: "AlsoMissing" }), stream); + expect(other).toBeInstanceOf(StaleInputError); + }); + + it("TX27: live and replayed selection failures are the same structural error", function* () { + const stream = new InMemoryStream(); + const live = yield* failure(inlineSource(SECTIONS, { target: "**/N*/Deep" }), stream); + const replayed = yield* failure(inlineSource(SECTIONS, { target: "**/N*/Deep" }), stream); + + for (const error of [live, replayed]) { + expect(isDocumentTargetError(error)).toBe(true); + expect(asDocumentTargetError(error)?.data.selector).toBe("**/N*/Deep"); + expect(asDocumentTargetError(error)?.data.available).toEqual([ + "Alpha", + "Alpha/Inner", + "Beta", + ]); + } }); it("TX22: an untargeted journal still replays for an untargeted run", function* () { diff --git a/packages/core/tests/document-targets.test.ts b/packages/core/tests/document-targets.test.ts index 92cedfc0..9905c7ba 100644 --- a/packages/core/tests/document-targets.test.ts +++ b/packages/core/tests/document-targets.test.ts @@ -24,10 +24,13 @@ import { expect } from "@executablemd/test-support/expect"; import type { Operation } from "effection"; import { + asDocumentTargetError, DocumentTargetError, encodeTargetLabel, isCanonicalTarget, + isDocumentTargetError, normalizeLabel, + parseDocumentTargetFailure, outlineDocument, retainedRanges, selectTarget, @@ -118,7 +121,7 @@ describe("Tier DT — document target catalog", () => { it("DT6: matching is case sensitive", function* () { const body = ["# Title", "", "## Test", ""].join("\n"); - expect(refusal(body, "test").kind).toBe("no-match"); + expect(refusal(body, "test").data.kind).toBe("no-match"); expect(selectTarget(outline(body), "Test").target).toBe("Test"); }); @@ -177,7 +180,7 @@ describe("Tier DT — document target catalog", () => { expect(catalog(body)).toEqual(["a%2Fb", "100%25%20done", "C%23%20sharp", "star%20%2A%20here"]); // `%2F` addresses one label containing a slash; a raw `/` would be hierarchy. expect(selectTarget(outline(body), "a%2Fb").labels).toEqual(["a/b"]); - expect(refusal(body, "a/b").kind).toBe("no-match"); + expect(refusal(body, "a/b").data.kind).toBe("no-match"); // `%2A` is a literal asterisk; a raw `*` is the operator. expect(selectTarget(outline(body), "star%20%2A%20here").labels).toEqual(["star * here"]); }); @@ -186,8 +189,8 @@ describe("Tier DT — document target catalog", () => { const body = ["# Title", "", "## Same", "", "one", "", "## Same", "", "two", ""].join("\n"); expect(catalog(body)).toEqual(["Same", "Same"]); const ambiguous = refusal(body, "Same"); - expect(ambiguous.kind).toBe("multiple-matches"); - expect(ambiguous.matches).toEqual(["Same", "Same"]); + expect(ambiguous.data.kind).toBe("multiple-matches"); + expect(ambiguous.data.matches).toEqual(["Same", "Same"]); }); it("DT12: only root-flow headings count", function* () { @@ -286,7 +289,7 @@ describe("Tier DT — document target catalog", () => { it("DT18: a document with no heading has an empty catalog", function* () { expect(catalog("just prose\n")).toEqual([]); - expect(refusal("just prose\n", "Anything").available).toEqual([]); + expect(refusal("just prose\n", "Anything").data.available).toEqual([]); }); it("DT19: a sole title is itself no target", function* () { @@ -297,7 +300,7 @@ describe("Tier DT — document target catalog", () => { describe("Tier DT — target selectors", () => { it("DT20: a literal selector matches one whole label", function* () { expect(selectTarget(outline(SECTIONS), "Test/Node").labels).toEqual(["Test", "Node"]); - expect(refusal(SECTIONS, "Nod").kind).toBe("no-match"); + expect(refusal(SECTIONS, "Nod").data.kind).toBe("no-match"); }); it("DT21: `*` matches within one level, in any position, more than once", function* () { @@ -306,7 +309,7 @@ describe("Tier DT — target selectors", () => { expect(selectTarget(outline(SECTIONS), "Test/N*d*").target).toBe("Test/Node"); expect(selectTarget(outline(SECTIONS), "*ther").target).toBe("Other"); // One `*` never crosses a level boundary. - expect(refusal(SECTIONS, "*Node").kind).toBe("no-match"); + expect(refusal(SECTIONS, "*Node").data.kind).toBe("no-match"); }); it("DT22: `**` matches zero or more complete levels", function* () { @@ -317,11 +320,16 @@ describe("Tier DT — target selectors", () => { }); it("DT23: a selector must name exactly one entry", function* () { - expect(refusal(SECTIONS, "**").kind).toBe("multiple-matches"); - expect(refusal(SECTIONS, "**").matches).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); - expect(refusal(SECTIONS, "Missing").kind).toBe("no-match"); - expect(refusal(SECTIONS, "Missing").matches).toEqual([]); - expect(refusal(SECTIONS, "Missing").available).toEqual([ + expect(refusal(SECTIONS, "**").data.kind).toBe("multiple-matches"); + expect(refusal(SECTIONS, "**").data.matches).toEqual([ + "Test", + "Test/Node", + "Test/Bun", + "Other", + ]); + expect(refusal(SECTIONS, "Missing").data.kind).toBe("no-match"); + expect(refusal(SECTIONS, "Missing").data.matches).toEqual([]); + expect(refusal(SECTIONS, "Missing").data.available).toEqual([ "Test", "Test/Node", "Test/Bun", @@ -331,7 +339,7 @@ describe("Tier DT — target selectors", () => { it("DT24: malformed selector syntax is refused as syntax", function* () { for (const selector of ["", "/Test", "Test/", "Test//Node", "%zz", "Test/%2"]) { - expect(refusal(SECTIONS, selector).kind).toBe("invalid-selector"); + expect(refusal(SECTIONS, selector).data.kind).toBe("invalid-selector"); } }); @@ -344,9 +352,9 @@ describe("Tier DT — target selectors", () => { }); it("DT26: a malformed or NUL-bearing escape never decodes", function* () { - expect(refusal(SECTIONS, "%00").kind).toBe("invalid-selector"); + expect(refusal(SECTIONS, "%00").data.kind).toBe("invalid-selector"); // A lone continuation byte is not UTF-8. - expect(refusal(SECTIONS, "%80").kind).toBe("invalid-selector"); + expect(refusal(SECTIONS, "%80").data.kind).toBe("invalid-selector"); }); /** @@ -358,7 +366,7 @@ describe("Tier DT — target selectors", () => { const label = "a".repeat(120); const body = ["# Title", "", `## ${label}`, ""].join("\n"); const selector = `${"*a".repeat(30)}*b`; - expect(refusal(body, selector).kind).toBe("no-match"); + expect(refusal(body, selector).data.kind).toBe("no-match"); expect(selectTarget(outline(body), `${"*a".repeat(30)}*`).labels).toEqual([label]); }); @@ -367,7 +375,7 @@ describe("Tier DT — target selectors", () => { const joined = ["# Title", "", "## alphabetagamma", ""].join("\n"); expect(selectTarget(outline(spaced), "alpha%20*%20gamma").labels).toEqual(["alpha beta gamma"]); // The spaces around the wildcard are part of what was asked for. - expect(refusal(joined, "alpha%20*%20gamma").kind).toBe("no-match"); + expect(refusal(joined, "alpha%20*%20gamma").data.kind).toBe("no-match"); // The level's own outer whitespace is not, so a padded selector still lands. expect(selectTarget(outline(spaced), "%20alpha*gamma%20").labels).toEqual(["alpha beta gamma"]); }); @@ -429,6 +437,79 @@ describe("Tier DT — canonical references", () => { expect(isCanonicalTarget("Test/%2A")).toBe(true); expect(isCanonicalTarget("Test/*")).toBe(false); }); + + /** + * Canonical means "exactly what the encoder would have written". Anything + * that decodes to a label needing normalization is a spelling of a target, + * not the target — accepting one would let two spellings of one section + * become two workflow-definition identities. + */ + it("DT48: a level is canonical only through the whole round trip", function* () { + for (const canonical of ["Caf%C3%A9", "a%20b", "A%2FB", "%2A", "%23", "a%2Bb", "Test/Node"]) { + expect(isCanonicalTarget(canonical)).toBe(true); + expect(formatDocumentReference("a.md", canonical)).toBe(`a.md#${canonical}`); + } + const rejected = [ + "Cafe%CC%81", // NFD — normalization would change it + "a%09b", // a tab is not an ASCII space + "a%20%20b", // uncollapsed whitespace + "%20a", // leading whitespace + "a%20", // trailing whitespace + "a%2fb", // lowercase escape + "A#B", // a raw `#` is the reference delimiter + "a//b", // an empty level + "a*b", // a raw wildcard operator + "%00", // NUL + ]; + for (const target of rejected) { + expect(isCanonicalTarget(target)).toBe(false); + } + }); + + it("DT49: a raw `#` is never a literal selector character, but `%23` is", function* () { + const body = ["# Title", "", "## A#B", "", "## Real", ""].join("\n"); + expect(catalog(body)).toEqual(["A%23B", "Real"]); + expect(selectTarget(outline(body), "A%23B").labels).toEqual(["A#B"]); + expect(refusal(body, "A#B").data.kind).toBe("invalid-selector"); + }); + + /** + * The formatter may only produce references the parser reads back. Sampling + * the rule would miss the two ways encoding loses information, so the + * implementation checks the round trip itself and these pin both losses. + */ + it("DT50: formatting refuses a path it could not encode losslessly", function* () { + // NUL, which the decoder refuses outright, and both halves of a broken + // surrogate pair, which encode lossily to the replacement character. + for (const path of ["a\u0000b.md", "lone\uD800.md", "trail\uDC00.md"]) { + let caught: unknown; + try { + formatDocumentReference(path); + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe("Invalid document reference"); + } + }); + + it("DT51: every formatted reference parses back to what it named", function* () { + const paths = [ + "README.md", + "docs/sub dir/a.md", + "odd#name.md", + "lit%20.md", + "café/ü.md", + "star*.md", + "a+b.md", + ]; + for (const path of paths) { + expect(fileSource(formatDocumentReference(path))).toEqual({ path }); + expect(fileSource(formatDocumentReference(path, "A%2FB"))).toEqual({ + path, + target: "A%2FB", + }); + } + }); }); describe("Tier DT — projection", () => { @@ -627,15 +708,15 @@ describe("Tier DT — inspection", () => { caught = error; } expect(caught).toBeInstanceOf(DocumentTargetError); - expect((caught as DocumentTargetError).kind).toBe("no-match"); - expect((caught as DocumentTargetError).selector).toBe("Nope"); + expect((caught as DocumentTargetError).data.kind).toBe("no-match"); + expect((caught as DocumentTargetError).data.selector).toBe("Nope"); }); it("DT47: the error's data is frozen and rebuilt, not the parser's arrays", function* () { const error = refusal(SECTIONS, "**"); - expect(Object.isFrozen(error.matches)).toBe(true); - expect(Object.isFrozen(error.available)).toBe(true); - expect(error.matches).not.toBe(outline(SECTIONS).targets); + expect(Object.isFrozen(error.data.matches)).toBe(true); + expect(Object.isFrozen(error.data.available)).toBe(true); + expect(error.data.matches).not.toBe(outline(SECTIONS).targets); // Encoded throughout, so a control character in a heading cannot reach a // diagnostic literally. expect(error.message).toContain('"**"'); @@ -644,3 +725,115 @@ describe("Tier DT — inspection", () => { } }); }); + +/** + * Recognition is the whole contract, so it is tested as one. + * + * A second loaded copy of this package is a different class producing the same + * name and the same tagged data, and it must be recognized on exactly the same + * terms. Everything else — a candidate carrying payload, a mutable data object, + * a message that disagrees with its own fields, a property that refuses to be + * read — must be refused, because recognition hands the object onward by + * identity and whatever it carries travels with it. + */ +describe("Tier DT — structural recognition", () => { + const FAILURE = Object.freeze({ + type: "executablemd.document-target-failure", + kind: "no-match", + selector: "Missing", + matches: Object.freeze([]), + available: Object.freeze(["Alpha"]), + }); + + const MESSAGE = '"Missing" matches no document target.\nAvailable targets:\n Alpha'; + + /** What a separately loaded copy of this module produces: same shape, own class. */ + function foreignError(): Error { + class DocumentTargetError extends Error { + readonly data = FAILURE; + constructor() { + super(MESSAGE); + this.name = "DocumentTargetError"; + } + } + return new DocumentTargetError(); + } + + it("DT52: a failure from another loaded copy is recognized", function* () { + const foreign = foreignError(); + expect(foreign instanceof DocumentTargetError).toBe(false); + expect(isDocumentTargetError(foreign)).toBe(true); + expect(asDocumentTargetError(foreign)?.data.kind).toBe("no-match"); + // Rebuilt, not adopted: the arrays a caller reads are this module's. + expect(parseDocumentTargetFailure(FAILURE)?.available).not.toBe(FAILURE.available); + }); + + it("DT53: this module's own failure is recognized", function* () { + expect(isDocumentTargetError(refusal(SECTIONS, "Missing"))).toBe(true); + }); + + it("DT54: every hostile or unreadable candidate is refused", function* () { + const withData = (data: unknown): Error => { + const error = new Error(MESSAGE); + error.name = "DocumentTargetError"; + Object.assign(error, { data }); + return error; + }; + const mutate = (change: Record) => Object.freeze({ ...FAILURE, ...change }); + + const hostile: unknown[] = [ + undefined, + null, + "a string", + new Error(MESSAGE), + // Untagged, wrongly tagged, unfrozen, and over- or under-populated data. + withData({ ...FAILURE }), + withData(mutate({ type: "other.tag" })), + withData(mutate({ kind: "made-up" })), + withData(Object.freeze({ ...FAILURE, extra: 1 })), + withData(Object.freeze({ type: FAILURE.type, kind: "no-match", selector: "Missing" })), + // A list holding something that is not a canonical reference. + withData(mutate({ available: Object.freeze([1]) })), + // `matches` populated under a kind that has none. + withData(mutate({ matches: Object.freeze(["Alpha"]) })), + // A property that refuses to answer. + withData( + Object.freeze( + Object.defineProperties( + { type: FAILURE.type, kind: "no-match", matches: [], available: [] }, + { + selector: { + get() { + throw new Error("hostile"); + }, + enumerable: true, + }, + }, + ), + ), + ), + ]; + for (const candidate of hostile) { + expect(isDocumentTargetError(candidate)).toBe(false); + expect(asDocumentTargetError(candidate)).toBe(undefined); + } + }); + + it("DT55: a recognized failure carries no cause and no extra payload", function* () { + const withCause = new Error(MESSAGE); + withCause.name = "DocumentTargetError"; + Object.assign(withCause, { data: FAILURE, cause: new Error("foreign") }); + expect(isDocumentTargetError(withCause)).toBe(false); + + const withPayload = new Error(MESSAGE); + withPayload.name = "DocumentTargetError"; + Object.assign(withPayload, { data: FAILURE, path: "/etc/passwd" }); + expect(isDocumentTargetError(withPayload)).toBe(false); + + // A message that does not derive from the data it claims. + const wrongMessage = new Error("something else"); + wrongMessage.name = "DocumentTargetError"; + Object.assign(wrongMessage, { data: FAILURE }); + expect(isDocumentTargetError(wrongMessage)).toBe(false); + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 1c7ef380..54cd2210 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2676,6 +2676,13 @@ alone and escapes everything else as uppercase UTF-8 hexadecimal, so a `/`, `*`, `#`, or `%` inside a heading becomes `%2F`, `%2A`, `%23`, or `%25` and cannot be read as syntax. +A fragment is an **exact** canonical target only when every level survives the +whole round trip: decoding it, normalizing the label, and re-encoding that label +reproduce the level byte for byte. That one rule rejects a wildcard operator, an +empty level, a lowercase escape, a raw `#`, an NFD spelling, a tab, and leading, +trailing, or uncollapsed whitespace, because none of them is what the encoder +writes. Two spellings of one section are therefore never two identities. + The catalog is in source order and retains duplicates: two sections whose canonical paths are equal stay two entries, so the ambiguity is observable. @@ -2692,8 +2699,9 @@ The first raw `#` separates the two. Raw `/` separates target levels and raw chunks are percent-decoded, which is what keeps `%2F` a slash inside one label and `%2A` a literal asterisk. Decoding is URI path decoding: `+` is a plus, not a space. Malformed escapes, byte sequences that are not UTF-8, NUL, a leading -or trailing slash, and an empty level are all refused. Matching is -case-sensitive. +or trailing slash, an empty level, and a raw `#` — the reference's own +delimiter, written `%23` when a heading really contains one — are all refused. +Matching is case-sensitive. - A literal level matches one canonical label exactly, after decoding and label normalization. @@ -2751,15 +2759,17 @@ The live root import records the **exact canonical target**, never the caller's selector. An untargeted import records no target member at all, so journals written before targets existed stay readable by untargeted runs. -A replay guard validates the target before the recorded run is reused. It parses -the recorded root content, resolves the current selector against *that* content, -and requires the result to equal the recorded exact target; the recorded content -is then what the projection is taken from. A different selector naming the same -section replays. A different exact target, a targeted request against an -untargeted record, an untargeted request against a targeted record, and a -selector the recorded content no longer resolves are all stale input (§6.11). -The check runs before a completed run's recorded terminal result can be reused, -so a finished journal cannot answer for a section it never ran. +A replay guard validates the selection before the recorded run is reused. It +resolves the current selector against the *recorded* content and requires the +same selection outcome; the recorded content is then what the projection is +taken from. A different selector naming the same section replays, and so does +the same failing selector. A different exact target, a targeted request against +an untargeted record, an untargeted request against a targeted record, and any +difference in a failed selection are all stale input (§6.11). Stale input is +reported as itself: the guard retains no failure object from the selection it +could not match. The check runs before a completed run's recorded terminal +result can be reused, so a finished journal cannot answer for a selection it +never made. ##### Naming a root document @@ -2801,21 +2811,73 @@ than encoding it again, and joins them with `#`. It is the one formatter diagnostics, command output, and workflow handoff use. Making an authored glob canonical is the selector parser's work, not this function's. +It only formats what `fileSource()` reads back: the encoded path is decoded +again and must reproduce the path exactly. NUL, which the decoder refuses, and +an unpaired surrogate, which encodes lossily to the replacement character, are +therefore rejected rather than turned into a reference naming a different file. + Existing programmatic `{ path }` values and `inlineSource(source)` remain valid and untargeted. -An unresolvable target raises `DocumentTargetError`, whose `kind` is -`invalid-selector`, `no-match`, or `multiple-matches`. It carries the requested -`selector` as it arrived, the canonical encoded `matches` (empty except for -`multiple-matches`), and every canonical encoded `available` target. Its data is -rebuilt and frozen at the boundary, and its message quotes the selector as JSON -and lists canonical encoded references, so a heading holding a control character -cannot reach a diagnostic literally. It is an ordinary invocation failure, not a -durability or `API.Files` failure. Because target resolution sits inside the -durable root import, a failure reaching a caller through `execute()` arrives by -name and message like every other failure crossing that boundary; the typed -error is what `inspectDocument()` reports, and inspection is where a host -resolves a selector before running anything. +An unresolvable target raises `DocumentTargetError`. It is an ordinary +invocation failure, not a durability or `API.Files` failure, and it is the same +error on every public path: `inspectDocument()`, a live `execute()`, and a +replayed `execute()` all raise one carrying the same fields. + +```ts +interface DocumentTargetFailure { + readonly type: "executablemd.document-target-failure"; + readonly kind: "invalid-selector" | "no-match" | "multiple-matches"; + readonly selector: string; + readonly matches: readonly string[]; + readonly available: readonly string[]; +} + +class DocumentTargetError extends Error { + readonly data: DocumentTargetFailure; +} + +function isDocumentTargetError(error: unknown): error is DocumentTargetError; +function asDocumentTargetError(error: unknown): DocumentTargetError | undefined; +function parseDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined; +``` + +`selector` is the fragment as it arrived. `matches` is the ambiguity list and is +empty for every other kind; `available` is the whole catalog. The message is +derived from the data, quotes the selector as JSON, and lists canonical encoded +references, so a heading holding a control character cannot reach a diagnostic +literally. + +Recognition is structural and total. The data carries a stable namespaced tag, +so a failure built by a separately loaded copy of the package is recognized on +the same terms as one built locally — `instanceof` cannot answer that question +across two copies. Recognition also requires the name, the message its own data +derives, frozen data with exactly the described members, no cause, and no other +enumerable member: a recognized failure is handed onward by identity, so a +candidate carrying a path or a foreign object is refused rather than adopted. +The data is rebuilt from validated parts wherever it crosses a boundary, so +nothing a candidate owns is retained. + +##### A failed selection is recorded, not merely failed + +A selection that names no single section is an observation of the document: the +text was read, and it does not offer what was asked for. The root import records +that outcome structurally — its kind, the requested selector, the matches, and +the catalog — and the failure is then rebuilt from that record and raised. + +Recording it is what makes a resumed run correct. A journal is matched by effect +type and name alone, so without the record a run whose selector matched nothing +would leave a completed journal that answers a later request for a section that +does exist. The replay guard therefore compares whole selection outcomes — the +whole document, one exact target, or one failure — rather than target strings, +and reproduces a recorded failure with its fields intact before the recorded +terminal result can be reused. The same failing selector replays its own +failure; any difference in outcome, including a different selector that fails +the same way, is stale input. No authored effect runs in either case. + +The recorded selector is sanitized invocation metadata, retained only so an +ordinary failed execution can be reproduced. It never occupies the exact-target +field and never reaches a workflow definition. ### 5.5 The Component Api @@ -7322,6 +7384,51 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | WRR10/WRR10b | Outer rollback cache coherence | Failure and cancellation after an uncommitted removal and negative lookup roll back and invalidate both authoritative DOFS caches | | WRR11 | Historical file size | Every historical file entry's declared size agrees with its retained DOFS manifest during read-only recognition | +### Tier DT — Document target catalog, selectors, and projection + +| # | Test | Verify | +|---|------|--------| +| DT1–DT5 | Outline | ATX and Setext headings catalog in source order; a skipped depth still nests; the outermost depth is the smallest present; a sole outermost heading is the title and several are path levels | +| DT6 | Case | Matching is case-sensitive | +| DT7–DT9 | Labels | Formatting, link destinations, inline code, image alt text and passive HTML tags reduce to statically rendered text; a heading rendering no text is unaddressable | +| DT8 | Normalization | NFC-equivalent spellings are one label and Unicode whitespace collapses | +| DT10 | Encoding | `/`, `%`, `#` and `*` in a heading encode to `%2F`, `%25`, `%23` and `%2A` and never read as syntax | +| DT11 | Duplicates | Two sections with one canonical path stay two entries and report as ambiguous | +| DT12 | Nested flow | Headings in block quotes, lists, fences, exec fences and raw HTML are not targets | +| DT13 | Component children | A component child holding blank lines and `#` lines contributes no target — the regression that kills raw Remark discovery | +| DT14–DT17 | Addressability | A heading overlapping component syntax or carrying an interpolation is unaddressable and blocks its subtree; escaped interpolation stays static; a computed sole title still leaves its sections addressable | +| DT18/DT19 | Empty catalog | A document with no heading addresses nothing, and a sole title is no target | +| DT20–DT22 | Matching | Literal levels, embedded `*`, and `**` across zero or more levels | +| DT23 | Exactly one | Zero matches and several matches both fail, reporting matches and the catalog | +| DT24–DT26 | Selector syntax | Empty, leading/trailing slash, empty level, malformed escape, NUL and non-UTF-8 are refused; `+` is a plus | +| DT27 | Termination | A wildcard-dense selector against a long label completes without exponential search | +| DT28 | Wildcard whitespace | Whitespace beside a wildcard is matched; only the level's outer edges trim | +| DT29–DT32 | References | A reference splits at the first raw `#`; a path keeps separators and decodes escapes; an unreadable reference says only `Invalid document reference`, cause-free; formatting encodes the path and validates an exact target | +| DT33/DT48 | Canonical exactness | A level is canonical only through decode, normalize and re-encode — NFD, tabs, uncollapsed or edge whitespace, lowercase escapes, empty levels and raw operators are refused | +| DT49 | Raw `#` | A raw `#` is never a literal selector character; `%23` addresses a heading containing one | +| DT50/DT51 | Formatter totality | A path that cannot encode losslessly — NUL, an unpaired surrogate — is refused, and every formatted reference parses back to what it named | +| DT34–DT39 | Projection | Preamble, ancestor direct content and the selected subtree are retained; siblings are absent; a non-leaf keeps its descendants; a sole title stays | +| DT40–DT43 | Positions | A retained element keeps its authored offset and line, CRLF included; frontmatter, props and return mode survive; the untargeted parse still scans the whole body | +| DT44–DT47 | Inspection | The catalog is reported without selecting; a glob resolves to the exact target; an unresolvable target fails inspection; the failure's data is frozen and rebuilt | +| DT52–DT55 | Recognition | A failure from a separately loaded copy is recognized; hostile, unreadable, mutable, over-populated, cause-bearing and payload-bearing candidates are all refused | + +### Tier TX — Targeted execution and replay + +| # | Test | Verify | +|---|------|--------| +| TX1–TX3 | Selection | Only the preamble, ancestors and subtree expand; a skipped sibling's components and code blocks never run; a non-leaf expands its descendants | +| TX4/TX5 | Identity | A retained element keeps the expansion ID it has in a full run, and two targets retaining it agree | +| TX6 | Sources | A file root and an inline root behave identically | +| TX7–TX9 | Root values | Root props, frontmatter interpolation, a declared `returns`, and `` apply to the projected body | +| TX10/TX11 | Structure | An invalid skipped sibling is irrelevant; an invalid retained range fails before any authored effect | +| TX12–TX14 | Failure timing | An unmatched or ambiguous target runs no authored effect and is structurally recognizable; inspection resolves without expanding a component | +| TX15/TX16 | Recording | The journal records the exact target, never the glob, and an untargeted run records no target member | +| TX17 | Compatibility | A different selector naming the same section replays | +| TX18–TX21 | Staleness | A different exact target, a targeted request against an untargeted journal, the reverse, and a selector the recorded content no longer resolves all fail stale before completed-Close reuse, carrying no foreign cause | +| TX22/TX23 | Recorded content | An untargeted journal replays untargeted; replay projects the recorded content, not the file on disk | +| TX24 | Failed selection | A journal from a selector that matched nothing never answers a later valid one | +| TX25–TX27 | Failed replay | The same failing selector replays its own recorded failure with no authored effect; a different failure kind or a different selector is stale; live and replayed failures are the same structural error | + ### Tier SL — Own-scope context updates | # | Test | Verify | From 5ae5bc503ab6c62629b5c7ff32d8a2371df0b17a Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Sun, 9 Aug 2026 14:47:50 -0400 Subject: [PATCH 03/14] =?UTF-8?q?=F0=9F=94=92=20Fail=20closed=20on=20a=20r?= =?UTF-8?q?ecorded=20root=20selection=20this=20version=20cannot=20read?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Not the root import" and "the root import, malformed" were one absent value, so a corrupted record fell through to the recorded terminal result and replayed an outcome the record no longer describes. The recorded root import is now a closed protocol: a repository selection with an optional exact canonical target, or a failed selection with an exact failure record. An unknown kind, a missing, unreadable, mistyped or extra member, and a noncanonical target are malformed, and malformed fails with one fixed cause-free diagnostic before the recorded Close can be reused — delegating nothing, executing nothing, appending nothing. The record carries the content it was taken from, so the selection is verified against that content rather than merely parsed: a recorded target must resolve to itself, and a recorded failure must be the failure that selector produces. `asDocumentTargetError()` returns a fresh local error built from reconstructed data instead of the candidate. An ordinary invocation failure has no fail-stop reason to preserve identity, and returning the candidate hands on whatever it owns — a list its owner can still rewrite, a revocable Proxy, a prototype with accessors. The data contract is closed to exactly five members with no symbol or non-enumerable extras, every list entry must be an exact canonical target, and the fields must describe an outcome selection could have reached. --- architecture.md | 7 + packages/core/src/document-targets.ts | 224 +++++++++++---- packages/core/src/execute.ts | 121 ++++++-- .../tests/document-target-execution.test.ts | 155 ++++++++++ packages/core/tests/document-targets.test.ts | 268 +++++++++++++----- specs/executable-mdx-spec.md | 73 ++++- 6 files changed, 683 insertions(+), 165 deletions(-) diff --git a/architecture.md b/architecture.md index c174baf2..d91f7b12 100644 --- a/architecture.md +++ b/architecture.md @@ -642,6 +642,13 @@ selection is an outcome too, and is recorded and compared as one — otherwise a journal left by a selector that matched nothing would answer a later request for a section that does exist. +A recorded selection is a closed protocol, and a record that does not satisfy it +is refused rather than delegated. "This event is not the root import" and "the +root import, malformed" are different answers: one continues, the other fails +before the recorded terminal result can be reused, without executing authored +work or appending history. The record carries the content it was taken from, so +the selection is verified against that content rather than merely parsed. + ## Expansion identity Core describes the executable element currently being expanded: diff --git a/packages/core/src/document-targets.ts b/packages/core/src/document-targets.ts index 9329aab3..333f4861 100644 --- a/packages/core/src/document-targets.ts +++ b/packages/core/src/document-targets.ts @@ -156,13 +156,73 @@ function stringList(value: unknown): string[] | undefined { }); } +/** + * The canonical labels a canonical target encodes. + * + * Only called on an entry already proven canonical, so decoding cannot fail; + * the guard is here because this reads data a candidate supplied. + */ +function targetLabels(target: string): string[] | undefined { + const labels: string[] = []; + for (const level of target.split("/")) { + const decoded = decodePercentEncoded(level); + if (decoded === undefined) { + return undefined; + } + labels.push(decoded); + } + return labels; +} + +/** A dense list of canonical encoded targets, copied out of the candidate. */ +function targetList(value: unknown): string[] | undefined { + const items = stringList(value); + if (items === undefined) { + return undefined; + } + return items.every((item) => isCanonicalTarget(item)) ? items : undefined; +} + +/** + * Whether these fields describe an outcome selection could actually have + * reached. + * + * The check re-derives the outcome rather than trusting the three fields to + * agree: the selector is parsed, matched against the catalog the candidate + * supplied, and the result compared with the matches it claims. A record whose + * kind, selector, matches, and catalog cannot all be true at once is refused, + * so an inconsistent journal or a hand-built candidate cannot describe a + * selection that never happened. + */ +function consistentOutcome(fields: TargetFailureFields): boolean { + const levels = parseSelector(fields.selector); + if (fields.kind === "invalid-selector") { + return levels === undefined && fields.matches.length === 0; + } + if (levels === undefined) { + return false; + } + const derived = fields.available.filter((target) => { + const labels = targetLabels(target); + return labels !== undefined && matchPath(levels, labels); + }); + if (fields.kind === "no-match") { + return derived.length === 0 && fields.matches.length === 0; + } + return derived.length > 1 && sameList(derived, fields.matches); +} + /** * Read a candidate's failure fields, rebuilding every one of them. * * Total: an unreadable property, a missing one, a kind outside the closed set, - * a sparse or non-string list, and matches on a kind that has none are all "not - * this shape" rather than a throw. Nothing the candidate owns is retained — the - * arrays that come back are new. + * a sparse list, an entry that is not a canonical encoded target, and a + * combination of kind, selector, matches and catalog that no selection could + * have produced are all "not this shape" rather than a throw. + * + * Nothing the candidate owns is retained. Every entry is read once and copied + * into a fresh array, so a list that is mutated afterwards — or reached through + * a Proxy that is later revoked — cannot change what comes back. */ function targetFailureFields(value: unknown): TargetFailureFields | undefined { return attempt(() => { @@ -171,20 +231,16 @@ function targetFailureFields(value: unknown): TargetFailureFields | undefined { } const kind = KINDS.find((candidate) => candidate === property(value, "kind")); const selector = property(value, "selector"); - const matches = stringList(property(value, "matches")); - const available = stringList(property(value, "available")); + const matches = targetList(property(value, "matches")); + const available = targetList(property(value, "available")); if (kind === undefined || typeof selector !== "string") { return undefined; } if (matches === undefined || available === undefined) { return undefined; } - // `matches` is the ambiguity list and nothing else; a populated one under - // any other kind is not the closed shape this contract describes. - if (kind !== "multiple-matches" && matches.length > 0) { - return undefined; - } - return { kind, selector, matches, available }; + const fields = { kind, selector, matches, available }; + return consistentOutcome(fields) ? fields : undefined; }); } @@ -199,23 +255,49 @@ function sealFailure(fields: TargetFailureFields): DocumentTargetFailure { }); } +/** Exactly the members failure data describes, and nothing else. */ +const FAILURE_DATA_MEMBERS: readonly string[] = [ + "available", + "kind", + "matches", + "selector", + "type", +]; + +/** + * Whether this object's own members are exactly the contract's. + * + * Own *property names* rather than enumerable keys, and own symbols with them: + * a non-enumerable extra is still payload the contract does not describe, and a + * symbol-keyed one survives spreading and `Object.assign`, which are exactly the + * mechanisms a consumer uses to pass data on. + */ +function hasOnlyDataMembers(value: object): boolean { + const names = attempt(() => [...Object.getOwnPropertyNames(value)].sort()); + if (names === undefined || names.length !== FAILURE_DATA_MEMBERS.length) { + return false; + } + if (!names.every((name, index) => name === FAILURE_DATA_MEMBERS[index])) { + return false; + } + return attempt(() => Object.getOwnPropertySymbols(value).length) === 0; +} + /** - * The failure data this value carries, if it carries valid, tagged, frozen - * data. + * The failure data this value carries, if it carries valid, tagged data. * - * Every field is checked, the member count with them, and that the object is - * frozen: extra keys are not the shape this contract describes, and a mutable - * one is not the shape a constructor here produces. + * Every field is checked, the exact member set with them, and the outcome the + * fields describe is re-derived rather than trusted. What comes back is built + * here from validated parts: the candidate's own arrays are never adopted, so a + * nested list that is mutable, mutated later, or reached through a revocable + * Proxy cannot reach a caller. */ export function parseDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined { return attempt(() => { if (!isRecord(value) || property(value, "type") !== DOCUMENT_TARGET_FAILURE) { return undefined; } - if (attempt(() => Object.isFrozen(value)) !== true) { - return undefined; - } - if (attempt(() => Object.keys(value).length) !== 5) { + if (!hasOnlyDataMembers(value)) { return undefined; } const fields = targetFailureFields(value); @@ -223,16 +305,33 @@ export function parseDocumentTargetFailure(value: unknown): DocumentTargetFailur }); } +/** Exactly the members a journal's failure record holds: the data, untagged. */ +const RECORD_MEMBERS: readonly string[] = ["available", "kind", "matches", "selector"]; + /** * The failure a journal record describes, rebuilt and sealed. * * The record is untagged — its place inside a recorded root-import selection is * what identifies it — so this validates the fields and supplies the tag, - * rather than requiring a tag the journal never held. + * rather than requiring a tag the journal never held. The member set is closed + * all the same: an extra key is data this contract does not describe, and a + * journal is not a place to carry undescribed data forward from. */ export function recordedDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined { - const fields = targetFailureFields(value); - return fields === undefined ? undefined : sealFailure(fields); + return attempt(() => { + if (!isRecord(value)) { + return undefined; + } + const names = attempt(() => [...Object.getOwnPropertyNames(value)].sort()); + if (names === undefined || names.length !== RECORD_MEMBERS.length) { + return undefined; + } + if (!names.every((name, index) => name === RECORD_MEMBERS[index])) { + return undefined; + } + const fields = targetFailureFields(value); + return fields === undefined ? undefined : sealFailure(fields); + }); } /** @@ -329,46 +428,61 @@ export function documentTargetError(data: DocumentTargetFailure): DocumentTarget } /** - * Whether this failure satisfies the whole contract, not merely the tag. + * The validated failure data this error carries, if it satisfies the whole + * contract rather than merely the tag. * * Structural throughout, so a failure constructed by a separately loaded copy - * of this package is recognized on exactly the same terms as one constructed - * here. The name is checked rather than the class for the same reason: a second + * of this package is read on exactly the same terms as one constructed here. + * The name is checked rather than the class for the same reason: a second * copy's constructor is a different function producing the same name. * - * Stricter than `parseDocumentTargetFailure` because recognition hands the - * object onward: the message has to be the one its own data derives, there can - * be no cause, and no member beyond the contract — otherwise a candidate could - * carry a path, a foreign object, or a second message past this boundary under - * a recognized tag. + * The shell is checked as strictly as the data. The message has to be the one + * the data derives, so a diagnostic cannot disagree with the fields it claims; + * there can be no cause and no enumerable member beyond the contract, so a + * candidate cannot carry a path, a foreign object, or a second message under a + * recognized tag. */ -export function isDocumentTargetError(error: unknown): error is DocumentTargetError { - return ( - attempt(() => { - if (!isError(error)) { - return false; - } - const data = parseDocumentTargetFailure(property(error, "data")); - if (data === undefined) { - return false; - } - if (property(error, "name") !== "DocumentTargetError") { - return false; - } - if (property(error, "message") !== documentTargetMessage(data)) { - return false; - } - if (property(error, "cause") !== undefined) { - return false; - } - return hasOnlyContractMembers(error); - }) === true - ); +function readDocumentTargetError(error: unknown): DocumentTargetFailure | undefined { + return attempt(() => { + if (!isError(error)) { + return undefined; + } + const data = parseDocumentTargetFailure(property(error, "data")); + if (data === undefined) { + return undefined; + } + if (property(error, "name") !== "DocumentTargetError") { + return undefined; + } + if (property(error, "message") !== documentTargetMessage(data)) { + return undefined; + } + if (property(error, "cause") !== undefined) { + return undefined; + } + return hasOnlyContractMembers(error) ? data : undefined; + }); } -/** The document-target failure this error is, by identity. */ +/** Whether this failure satisfies the whole document-target contract. */ +export function isDocumentTargetError(error: unknown): boolean { + return readDocumentTargetError(error) !== undefined; +} + +/** + * The document-target failure this error describes, as a local error. + * + * A fresh error built from reconstructed data, never the candidate itself. + * Nothing here has a fail-stop's reason to preserve object identity — this is + * an ordinary invocation failure, and what a caller needs is the outcome, not + * the instance that reported it. Returning the candidate would hand on whatever + * it owns: nested arrays somebody else can still mutate, a revocable Proxy, a + * prototype with accessors. Rebuilding costs one allocation and removes all of + * it, so the result stays readable however the original is treated afterwards. + */ export function asDocumentTargetError(error: unknown): DocumentTargetError | undefined { - return isDocumentTargetError(error) ? error : undefined; + const data = readDocumentTargetError(error); + return data === undefined ? undefined : new DocumentTargetError(data); } /** Whether two failures describe the same selection outcome, field by field. */ diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 1e6bcd83..3263ecf6 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -56,6 +56,7 @@ import { asDocumentTargetError, documentTargetError, documentTargetFailure, + isCanonicalTarget, recordedDocumentTargetFailure, sameDocumentTargetFailure, } from "./document-targets.ts"; @@ -250,10 +251,7 @@ function* durableImportComponent( if (selection.kind === "target-failure") { const failure = recordedDocumentTargetFailure(selection.failure); if (failure === undefined) { - throw new Error( - "The recorded root document import describes a failed target selection this version " + - "cannot read.", - ); + throw new Error(UNREADABLE_ROOT_RECORD); } throw documentTargetError(failure); } @@ -347,34 +345,101 @@ type SelectionOutcome = | { kind: "exact"; target: string } | { kind: "failed"; failure: DocumentTargetFailure }; -/** The recorded root import this event is, when it is one that can be read. */ -function recordedRootImport( - event: Yield, -): { content: string; selection: SelectionOutcome } | undefined { - if ( - event.description.type !== "import_component" || - event.description.name !== "__root__" || - event.result.status !== "ok" - ) { - return undefined; +/** + * What the fixed diagnostic says when a recorded root import cannot be read, + * and all it says. + * + * Cause-free: the record is journal data, and quoting it back would put + * whatever it holds into a diagnostic. + */ +const UNREADABLE_ROOT_RECORD = "The recorded root document import cannot be read by this version."; + +/** + * What a recorded event turned out to be. + * + * "Not the root import" and "the root import, malformed" are deliberately + * different answers. Collapsing them into one absent value is what would let a + * corrupted record fall through to the recorded terminal result, which is the + * failure this distinction exists to prevent. + */ +type RootImportRecord = + | { kind: "unrelated" } + | { kind: "malformed" } + | { kind: "read"; content: string; selection: SelectionOutcome }; + +const UNRELATED: RootImportRecord = { kind: "unrelated" }; +const MALFORMED: RootImportRecord = { kind: "malformed" }; + +/** + * Parse a recorded root import as a closed protocol. + * + * Two selection shapes are supported and nothing else: a repository selection + * with an optional canonical target, and a failed selection with an exact + * failure record. An unknown kind, a missing or mistyped member, an extra + * member, a noncanonical target, and failure data that no selection could have + * produced are each malformed rather than absent. + * + * A result that is not `ok` is left alone. A root import can fail for reasons + * that have nothing to do with selection — an unreadable file — and those + * recorded failures are not this protocol's to interpret. + */ +function recordedRootImport(event: Yield): RootImportRecord { + if (event.description.type !== "import_component" || event.description.name !== "__root__") { + return UNRELATED; + } + if (event.result.status !== "ok") { + return UNRELATED; } const record = event.result.value; if (!isJsonObject(record)) { - return undefined; + return MALFORMED; } const content = record["content"]; - if (typeof content !== "string") { - return undefined; + const path = record["path"]; + if (typeof content !== "string" || typeof path !== "string") { + return MALFORMED; } - if (record["kind"] === "target-failure") { - const failure = recordedDocumentTargetFailure(record["failure"]); - return failure === undefined ? undefined : { content, selection: { kind: "failed", failure } }; + const kind = record["kind"]; + const members = Object.keys(record).length; + + if (kind === "repository") { + const target = record["target"]; + if (target === undefined) { + return members === 3 ? { kind: "read", content, selection: { kind: "whole" } } : MALFORMED; + } + if (members !== 4 || typeof target !== "string" || !isCanonicalTarget(target)) { + return MALFORMED; + } + // The recorded content is here, so the target is verified against it rather + // than merely parsed: a well-formed target the recorded document does not + // offer describes a selection that never happened. + const resolved = resolveDocumentTarget(path, content, target); + if (!resolved.ok || resolved.value !== target) { + return MALFORMED; + } + return { kind: "read", content, selection: { kind: "exact", target } }; } - const target = record["target"]; - if (target === undefined) { - return { content, selection: { kind: "whole" } }; + + if (kind === "target-failure") { + const failure = recordedDocumentTargetFailure(record["failure"]); + if (members !== 4 || failure === undefined) { + return MALFORMED; + } + // Same standard for a failure: the recorded selector must fail against the + // recorded content in exactly the way the record claims. That verifies the + // catalog and the matches too, which no amount of shape checking could. + const rederived = resolveDocumentTarget(path, content, failure.selector); + if (rederived.ok) { + return MALFORMED; + } + const actual = asDocumentTargetError(rederived.error); + if (actual === undefined || !sameDocumentTargetFailure(actual.data, failure)) { + return MALFORMED; + } + return { kind: "read", content, selection: { kind: "failed", failure } }; } - return typeof target === "string" ? { content, selection: { kind: "exact", target } } : undefined; + + return MALFORMED; } /** What this run's selector decides against the content the journal recorded. */ @@ -448,9 +513,15 @@ function holdRootSelection(root: RootDocumentSource): Operation { return ReplayGuard.around({ *check([event], next) { const recorded = recordedRootImport(event); - if (recorded === undefined) { + if (recorded.kind === "unrelated") { return yield* next(event); } + // Refused here, so a corrupted record can never reach the recorded + // terminal result. Nothing is delegated, nothing is executed, and no + // history is appended. + if (recorded.kind === "malformed") { + throw new Error(UNREADABLE_ROOT_RECORD); + } const requested = requestedSelection(root, recorded.content); if (!sameSelection(recorded.selection, requested)) { throw new StaleInputError( diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 8df04fdf..3b529b70 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -34,6 +34,7 @@ import { DocumentTargetError, isDocumentTargetError, } from "../src/document-targets.ts"; +import { isJsonObject, parseJson } from "../src/json.ts"; import { fileSource, formatDocumentReference, inlineSource } from "../src/root-source.ts"; import type { RootDocumentSource } from "../src/root-source.ts"; import { asText } from "./helpers.ts"; @@ -576,3 +577,157 @@ describe("Tier TX — targeted replay", () => { expect(replayed).not.toContain("rewritten beta"); }); }); + +/** + * Tier TX — a corrupted root-import record fails closed. + * + * "Not the root import" and "the root import, malformed" have to be different + * answers. A boundary that returns one absent value for both delegates a + * corrupted record onward, and `durableRun` then reuses the recorded terminal + * result — replaying a failure or a success the record no longer describes. + * + * Every case here starts from a *valid* completed journal and corrupts only the + * recorded selection, so what is being measured is the parse and nothing else. + * Each is resumed twice: once with the selector that produced the journal, once + * with a selector that would succeed against a healthy record. Both must be + * refused with the fixed diagnostic, and neither may expand anything or append + * history. + */ +describe("Tier TX — malformed recorded selections", () => { + const UNREADABLE = "The recorded root document import cannot be read by this version."; + + /** A completed journal whose root import recorded a failed selection. */ + function* failedJournal(): Operation { + const stream = new InMemoryStream(); + yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + return stream; + } + + /** Rewrite the recorded root-import selection, keeping everything else. */ + function* corrupt( + stream: InMemoryStream, + change: (record: Record) => Record, + ): Operation { + const corrupted = new InMemoryStream(); + for (const event of stream.snapshot()) { + const record = + event.type === "yield" && + event.description.name === "__root__" && + event.result.status === "ok" + ? event.result.value + : undefined; + if (event.type === "yield" && isJsonObject(record)) { + yield* corrupted.append({ + ...event, + result: { status: "ok", value: parseJson(change({ ...record })) }, + }); + continue; + } + yield* corrupted.append(event); + } + return corrupted; + } + + /** Every way a corrupted record must be refused, resumed both ways. */ + function* refuses( + change: (record: Record) => Record, + ): Operation { + const healthy = yield* failedJournal(); + const before = (yield* corrupt(healthy, change)).snapshot().length; + + for (const target of ["Missing", "Beta"]) { + const stream = yield* corrupt(healthy, change); + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target }), stream, seen); + + expect((error as Error).message).toBe(UNREADABLE); + expect((error as Error).cause).toBe(undefined); + // Not the recorded failure, and not a document-target failure at all. + expect(isDocumentTargetError(error)).toBe(false); + expect((error as Error).message).not.toContain("Missing"); + // Nothing expanded, and no history was appended on top of the corruption. + expect(seen.names).toEqual([]); + expect(stream.snapshot().length).toBe(before); + } + } + + it("TX28: a missing or non-array `available` is refused", function* () { + yield* refuses((record) => ({ + ...record, + failure: omit(record["failure"], "available"), + })); + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), available: "Beta" }, + })); + }); + + it("TX29: an unknown selection kind is refused", function* () { + yield* refuses((record) => ({ ...record, kind: "something-else" })); + yield* refuses((record) => omit(record, "kind")); + }); + + it("TX30: extra data in the record or the failure is refused", function* () { + yield* refuses((record) => ({ ...record, extra: "payload" })); + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), extra: "payload" }, + })); + }); + + it("TX31: a noncanonical target entry is refused", function* () { + for (const available of [["../../etc/passwd"], ["Beta "], ["a%2fb"]]) { + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), available }, + })); + } + // The same rule on a successful repository selection's own target. + const healthy = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), healthy, { names: [], ids: [] }); + const stream = yield* corrupt(healthy, (record) => ({ ...record, target: "beta" })); + const error = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect((error as Error).message).toBe(UNREADABLE); + }); + + it("TX32: semantically inconsistent kind and matches are refused", function* () { + // `no-match` carrying matches, and a selector that really does match. + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), matches: ["Beta"] }, + })); + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), selector: "Beta" }, + })); + // `multiple-matches` with a single match. + yield* refuses((record) => ({ + ...record, + failure: { + ...asRecord(record["failure"]), + kind: "multiple-matches", + selector: "Beta", + matches: ["Beta"], + }, + })); + }); + + it("TX33: a valid record still replays, so the refusals are not vacuous", function* () { + const stream = yield* failedJournal(); + const seen: Probes = { names: [], ids: [] }; + const replayed = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream, seen); + expect(isDocumentTargetError(replayed)).toBe(true); + expect((replayed as Error).message).not.toBe(UNREADABLE); + expect(seen.names).toEqual([]); + }); +}); + +function asRecord(value: unknown): Record { + return typeof value === "object" && value !== null ? { ...(value as object) } : {}; +} + +function omit(value: unknown, key: string): Record { + const record = asRecord(value); + delete record[key]; + return record; +} diff --git a/packages/core/tests/document-targets.test.ts b/packages/core/tests/document-targets.test.ts index 9905c7ba..77d66f7d 100644 --- a/packages/core/tests/document-targets.test.ts +++ b/packages/core/tests/document-targets.test.ts @@ -729,28 +729,45 @@ describe("Tier DT — inspection", () => { /** * Recognition is the whole contract, so it is tested as one. * - * A second loaded copy of this package is a different class producing the same - * name and the same tagged data, and it must be recognized on exactly the same - * terms. Everything else — a candidate carrying payload, a mutable data object, - * a message that disagrees with its own fields, a property that refuses to be - * read — must be refused, because recognition hands the object onward by - * identity and whatever it carries travels with it. + * The boundary is closed and it reconstructs: a candidate is validated field by + * field and a *fresh local* error is built from the result. Nothing the + * candidate owns is handed on, which is why these mutate and revoke the + * originals afterwards and assert the answer is unchanged. + * + * A separately loaded copy of this package is a different class producing the + * same name and the same tagged data, and must be read on exactly the same + * terms. Everything else — payload, a list that is not a catalog, fields that no + * selection could have produced — must be refused. */ describe("Tier DT — structural recognition", () => { - const FAILURE = Object.freeze({ - type: "executablemd.document-target-failure", - kind: "no-match", - selector: "Missing", - matches: Object.freeze([]), - available: Object.freeze(["Alpha"]), - }); + const CATALOG = ["# T", "", "## Alpha", "", "## Beta", ""].join("\n"); + + /** Genuine failure data, so the fixtures cannot drift from the real thing. */ + const GENUINE = refusal(CATALOG, "Missing").data; + const MESSAGE = refusal(CATALOG, "Missing").message; + + function data(change: Record = {}): Record { + return { + type: "executablemd.document-target-failure", + kind: GENUINE.kind, + selector: GENUINE.selector, + matches: [...GENUINE.matches], + available: [...GENUINE.available], + ...change, + }; + } - const MESSAGE = '"Missing" matches no document target.\nAvailable targets:\n Alpha'; + function shell(payload: unknown, message = MESSAGE): Error { + const error = new Error(message); + error.name = "DocumentTargetError"; + Object.assign(error, { data: payload }); + return error; + } - /** What a separately loaded copy of this module produces: same shape, own class. */ + /** What a separately loaded copy produces: same shape, its own class. */ function foreignError(): Error { class DocumentTargetError extends Error { - readonly data = FAILURE; + readonly data = Object.freeze(data()); constructor() { super(MESSAGE); this.name = "DocumentTargetError"; @@ -759,81 +776,182 @@ describe("Tier DT — structural recognition", () => { return new DocumentTargetError(); } - it("DT52: a failure from another loaded copy is recognized", function* () { + it("DT52: a failure from another loaded copy is read on the same terms", function* () { const foreign = foreignError(); expect(foreign instanceof DocumentTargetError).toBe(false); expect(isDocumentTargetError(foreign)).toBe(true); expect(asDocumentTargetError(foreign)?.data.kind).toBe("no-match"); - // Rebuilt, not adopted: the arrays a caller reads are this module's. - expect(parseDocumentTargetFailure(FAILURE)?.available).not.toBe(FAILURE.available); }); it("DT53: this module's own failure is recognized", function* () { - expect(isDocumentTargetError(refusal(SECTIONS, "Missing"))).toBe(true); + const own = refusal(CATALOG, "Missing"); + expect(isDocumentTargetError(own)).toBe(true); + expect(asDocumentTargetError(own)?.data).toEqual(own.data); }); - it("DT54: every hostile or unreadable candidate is refused", function* () { - const withData = (data: unknown): Error => { - const error = new Error(MESSAGE); - error.name = "DocumentTargetError"; - Object.assign(error, { data }); - return error; - }; - const mutate = (change: Record) => Object.freeze({ ...FAILURE, ...change }); - - const hostile: unknown[] = [ - undefined, - null, - "a string", - new Error(MESSAGE), - // Untagged, wrongly tagged, unfrozen, and over- or under-populated data. - withData({ ...FAILURE }), - withData(mutate({ type: "other.tag" })), - withData(mutate({ kind: "made-up" })), - withData(Object.freeze({ ...FAILURE, extra: 1 })), - withData(Object.freeze({ type: FAILURE.type, kind: "no-match", selector: "Missing" })), - // A list holding something that is not a canonical reference. - withData(mutate({ available: Object.freeze([1]) })), - // `matches` populated under a kind that has none. - withData(mutate({ matches: Object.freeze(["Alpha"]) })), - // A property that refuses to answer. - withData( - Object.freeze( - Object.defineProperties( - { type: FAILURE.type, kind: "no-match", matches: [], available: [] }, - { - selector: { - get() { - throw new Error("hostile"); - }, - enumerable: true, - }, - }, - ), - ), - ), + /** + * The reconstruction claim, made where it can fail. A boundary that returned + * the candidate would pass every field assertion above and still hand a + * caller arrays somebody else can rewrite. + */ + it("DT54: recognition returns a fresh local error, never the candidate", function* () { + const foreign = foreignError(); + const safe = asDocumentTargetError(foreign); + + expect(safe).not.toBe(foreign); + expect(safe).toBeInstanceOf(DocumentTargetError); + const original = foreign as unknown as { data: { available: unknown } }; + expect(safe?.data).not.toBe(original.data); + expect(safe?.data.available).not.toBe(original.data.available); + expect(Object.isFrozen(safe?.data)).toBe(true); + expect(Object.isFrozen(safe?.data.available)).toBe(true); + }); + + it("DT55: a mutable nested list is copied, and later mutation changes nothing", function* () { + const available = ["Alpha", "Beta"]; + // Frozen outer data around a list its owner can still rewrite. + const candidate = shell(Object.freeze(data({ available }))); + const safe = asDocumentTargetError(candidate); + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + + available.push("Injected"); + available[0] = "Rewritten"; + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + expect(safe?.message).toBe(MESSAGE); + }); + + it("DT56: a revoked Proxy cannot reach through a result already built", function* () { + const revocable = Proxy.revocable(["Alpha", "Beta"], {}); + const candidate = shell(Object.freeze(data({ available: revocable.proxy }))); + const safe = asDocumentTargetError(candidate); + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + + revocable.revoke(); + // Reading the result must not touch the revoked original. + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + expect(safe?.message).toBe(MESSAGE); + expect(String(safe)).toContain("Alpha"); + // A candidate whose Proxy is already revoked is simply refused. + expect(isDocumentTargetError(candidate)).toBe(false); + expect(asDocumentTargetError(candidate)).toBe(undefined); + }); + + it("DT57: data-level extras are refused, enumerable, hidden, or symbol-keyed", function* () { + const enumerable = shell(Object.freeze(data({ extra: "payload" }))); + expect(isDocumentTargetError(enumerable)).toBe(false); + + const hidden = Object.freeze( + Object.defineProperty(data(), "extra", { value: "/etc/passwd", enumerable: false }), + ); + expect(isDocumentTargetError(shell(hidden))).toBe(false); + + const symbolic = Object.freeze( + Object.defineProperty(data(), Symbol.for("payload"), { + value: "/etc/passwd", + enumerable: true, + }), + ); + expect(isDocumentTargetError(shell(symbolic))).toBe(false); + }); + + it("DT58: a list entry that is not a canonical target is refused", function* () { + const rejected: unknown[][] = [ + ["../../etc/passwd"], + ["Alpha/../Beta"], + ["Alpha Beta"], + ["Alpha\u0009Beta"], + ["AlphaBeta"], + ["a%2fb"], + ["Alpha", "Alpha "], + [1], + [null], + // A sparse list is not a dense one. + Object.assign(Array.from({ length: 2 }) as unknown[], { 0: "Alpha" }), ]; - for (const candidate of hostile) { - expect(isDocumentTargetError(candidate)).toBe(false); - expect(asDocumentTargetError(candidate)).toBe(undefined); + for (const available of rejected) { + expect(isDocumentTargetError(shell(Object.freeze(data({ available }))))).toBe(false); } }); - it("DT55: a recognized failure carries no cause and no extra payload", function* () { - const withCause = new Error(MESSAGE); - withCause.name = "DocumentTargetError"; - Object.assign(withCause, { data: FAILURE, cause: new Error("foreign") }); + it("DT59: fields no selection could have produced are refused", function* () { + const inconsistent: Record[] = [ + // `no-match` whose selector really does match the catalog. + data({ kind: "no-match", selector: "Alpha", matches: [] }), + // `no-match` carrying matches. + data({ kind: "no-match", matches: ["Alpha"] }), + // `multiple-matches` with one match. + data({ kind: "multiple-matches", selector: "Alpha", matches: ["Alpha"] }), + // `multiple-matches` claiming a match outside the catalog. + data({ kind: "multiple-matches", selector: "**", matches: ["Alpha", "Gamma"] }), + // `invalid-selector` whose selector parses perfectly well. + data({ kind: "invalid-selector", selector: "Alpha" }), + // A kind outside the closed set. + data({ kind: "made-up" }), + // A missing member. + (() => { + const partial = data(); + delete partial["available"]; + return partial; + })(), + // A member of the wrong type. + data({ available: "Alpha" }), + data({ selector: 7 }), + ]; + for (const candidate of inconsistent) { + expect(isDocumentTargetError(shell(Object.freeze(candidate)))).toBe(false); + } + }); + + it("DT60: the Error shell is closed too", function* () { + const withCause = shell(Object.freeze(data())); + Object.assign(withCause, { cause: new Error("foreign") }); expect(isDocumentTargetError(withCause)).toBe(false); - const withPayload = new Error(MESSAGE); - withPayload.name = "DocumentTargetError"; - Object.assign(withPayload, { data: FAILURE, path: "/etc/passwd" }); + const withPayload = shell(Object.freeze(data())); + Object.assign(withPayload, { path: "/etc/passwd" }); expect(isDocumentTargetError(withPayload)).toBe(false); - // A message that does not derive from the data it claims. - const wrongMessage = new Error("something else"); - wrongMessage.name = "DocumentTargetError"; - Object.assign(wrongMessage, { data: FAILURE }); - expect(isDocumentTargetError(wrongMessage)).toBe(false); + // A diagnostic that does not derive from the data it claims. + expect(isDocumentTargetError(shell(Object.freeze(data()), "something else"))).toBe(false); + + for (const candidate of [undefined, null, "a string", new Error(MESSAGE), {}]) { + expect(isDocumentTargetError(candidate)).toBe(false); + expect(asDocumentTargetError(candidate)).toBe(undefined); + } + }); + + /** + * The payload question asked the way a consumer would ask it: after the + * boundary, is any of it still reachable by the ordinary means of passing an + * error on? + */ + it("DT61: no planted payload survives the boundary", function* () { + const planted = Object.freeze( + Object.defineProperty(data(), Symbol.for("secret"), { + value: "s3cret", + enumerable: true, + }), + ); + // Refused outright, so nothing to survive. + expect(asDocumentTargetError(shell(planted))).toBe(undefined); + + // And for a candidate that is accepted, the result carries only the + // contract: no extra own member, string or symbol, on the data or the error. + const safe = asDocumentTargetError(foreignError()); + expect(safe).toBeDefined(); + expect(Object.getOwnPropertyNames(safe?.data ?? {}).sort()).toEqual([ + "available", + "kind", + "matches", + "selector", + "type", + ]); + expect(Object.getOwnPropertySymbols(safe?.data ?? {})).toEqual([]); + expect(Object.keys(safe ?? {}).sort()).toEqual(["data", "name"]); + expect(Object.getOwnPropertySymbols(safe ?? {})).toEqual([]); + expect(JSON.stringify({ ...safe })).not.toContain("s3cret"); + expect(String(safe)).toBe(`DocumentTargetError: ${MESSAGE}`); + // A journal round trip rebuilds the same data from the same fields. + expect(parseDocumentTargetFailure(JSON.parse(JSON.stringify(safe?.data)))).toEqual(safe?.data); }); }); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 54cd2210..be300c4e 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2848,15 +2848,35 @@ derived from the data, quotes the selector as JSON, and lists canonical encoded references, so a heading holding a control character cannot reach a diagnostic literally. -Recognition is structural and total. The data carries a stable namespaced tag, -so a failure built by a separately loaded copy of the package is recognized on -the same terms as one built locally — `instanceof` cannot answer that question -across two copies. Recognition also requires the name, the message its own data -derives, frozen data with exactly the described members, no cause, and no other -enumerable member: a recognized failure is handed onward by identity, so a -candidate carrying a path or a foreign object is refused rather than adopted. -The data is rebuilt from validated parts wherever it crosses a boundary, so -nothing a candidate owns is retained. +Recognition is structural, total, and reconstructing. The data carries a stable +namespaced tag, so a failure built by a separately loaded copy of the package is +read on the same terms as one built locally — `instanceof` cannot answer that +question across two copies. + +`asDocumentTargetError()` never returns the candidate. It validates every field +and builds a **fresh local error** from the result, so nothing the candidate +owns is handed on: a nested list stays correct after its owner mutates it, and a +list reached through a revocable Proxy stays readable after the Proxy is +revoked. This is an ordinary invocation failure with no fail-stop reason to +preserve object identity, so rebuilding costs one allocation and removes every +way payload could travel. + +A candidate is read only when all of this holds: + +- the data carries exactly `type`, `kind`, `selector`, `matches`, and + `available`, with no other own member — enumerable, non-enumerable, or + symbol-keyed; +- `matches` and `available` are dense lists whose every entry is an exact + canonical target; +- `matches` is empty for `invalid-selector` and `no-match`, and holds more than + one entry for `multiple-matches`; and +- the fields describe an outcome selection could have reached: the selector is + parsed and matched against the catalog the data supplies, and the result must + be the matches it claims. + +The Error shell is checked as strictly: the fixed name, the message its own data +derives, no cause, and no enumerable member beyond the contract. Diagnostics are +derived from the reconstructed canonical data alone. ##### A failed selection is recorded, not merely failed @@ -2879,6 +2899,34 @@ The recorded selector is sanitized invocation metadata, retained only so an ordinary failed execution can be reproduced. It never occupies the exact-target field and never reaches a workflow definition. +##### A recorded root selection is a closed protocol + +The recorded root import is parsed as a closed protocol with exactly two +supported shapes: a repository selection carrying the path, the content, and an +optional exact canonical target; and a failed selection carrying the path, the +content, and an exact failure record. An unknown kind, a missing or unreadable +member, a member of the wrong type, an extra member, a noncanonical target, and +a failure record that is not exactly this contract are each **malformed**. + +Malformed is not the same answer as "this event is not the root import". +Collapsing the two is what would let a corrupted record fall through to the +recorded terminal result, replaying an outcome the record no longer describes. + +Because the record carries the content it was taken from, the selection is +verified against it rather than merely parsed: a recorded exact target must +still resolve to itself in the recorded content, and a recorded failure must be +exactly the failure the recorded selector produces against that content. A +catalog, a match list, or a kind that the recorded document contradicts is +therefore malformed too. + +A malformed record fails before the recorded terminal result can be reused, with +one fixed, cause-free diagnostic. It never delegates, never replays the recorded +terminal error, never executes authored work, and never appends new history. + +A root import whose recorded result is not `ok` is left alone: a root can fail +for reasons that are not about selection, and those failures are not this +protocol's to interpret. + ### 5.5 The Component Api Expansion's context-dependent operations are exposed through one public @@ -7410,7 +7458,10 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | DT34–DT39 | Projection | Preamble, ancestor direct content and the selected subtree are retained; siblings are absent; a non-leaf keeps its descendants; a sole title stays | | DT40–DT43 | Positions | A retained element keeps its authored offset and line, CRLF included; frontmatter, props and return mode survive; the untargeted parse still scans the whole body | | DT44–DT47 | Inspection | The catalog is reported without selecting; a glob resolves to the exact target; an unresolvable target fails inspection; the failure's data is frozen and rebuilt | -| DT52–DT55 | Recognition | A failure from a separately loaded copy is recognized; hostile, unreadable, mutable, over-populated, cause-bearing and payload-bearing candidates are all refused | +| DT52/DT53 | Recognition | A failure from a separately loaded copy, and one built here, are read on the same terms | +| DT54–DT56 | Reconstruction | The result is a fresh local error, never the candidate; a mutable nested list is copied and later mutation changes nothing; a revoked Proxy cannot reach through a result already built | +| DT57–DT59 | Closed data | Enumerable, non-enumerable and symbol-keyed extras, entries that are not canonical targets, sparse lists, and fields no selection could have produced are all refused | +| DT60/DT61 | Closed shell | A cause, an enumerable payload, and a message that does not derive from its data are refused; no planted payload survives stringification, spreading, symbol enumeration, or a journal round trip | ### Tier TX — Targeted execution and replay @@ -7428,6 +7479,8 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX22/TX23 | Recorded content | An untargeted journal replays untargeted; replay projects the recorded content, not the file on disk | | TX24 | Failed selection | A journal from a selector that matched nothing never answers a later valid one | | TX25–TX27 | Failed replay | The same failing selector replays its own recorded failure with no authored effect; a different failure kind or a different selector is stale; live and replayed failures are the same structural error | +| TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a noncanonical or unresolvable target, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | +| TX33 | Not vacuous | An uncorrupted record still replays its recorded failure | ### Tier SL — Own-scope context updates From 1e9e058b39ab86159290455c93e1618f0f7191e3 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 02:06:50 -0400 Subject: [PATCH 04/14] =?UTF-8?q?=F0=9F=94=92=20Make=20reading=20a=20recor?= =?UTF-8?q?ded=20root=20selection=20total?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reading a recognized successful root import could still throw: `matter()` rejects invalid frontmatter, a recorded property may be an accessor that refuses, and a record may come from a Proxy that will not enumerate. Each escaped carrying a parser message, a path, or whatever the record planted — from the one boundary whose job is to refuse. Reading is now total over journal-provided values. The record is parsed into this run's own copy before any member is read, the recorded content is parsed as part of reading the record for every shape, and every remaining throw becomes the same fixed cause-free diagnostic. The boundary is synchronous throughout, so it can swallow no cancellation and no durability failure. `isDocumentTargetError()` is documented as returning `boolean` rather than narrowing. A type predicate would claim the candidate is the safe value, which is exactly what it is not: the safe value is what `asDocumentTargetError()` reconstructs. DT58 changed failure data while keeping the derived message, so recognition failed on the message and the row proved nothing about list validation. The data-level rows now go at `parseDocumentTargetFailure()` directly. `.` and `..` are legal heading labels, so `../../etc/passwd` is a canonical heading path rather than filesystem authority; DT59 states that, and TX31 is renamed to say what actually refuses it — the catalog the recorded document derives. --- architecture.md | 4 +- packages/core/src/definition.ts | 2 +- packages/core/src/execute.ts | 89 +++++++--- .../tests/document-target-execution.test.ts | 157 +++++++++++++++++- packages/core/tests/document-targets.test.ts | 55 ++++-- specs/executable-mdx-spec.md | 31 +++- 6 files changed, 297 insertions(+), 41 deletions(-) diff --git a/architecture.md b/architecture.md index d91f7b12..f4ebde6d 100644 --- a/architecture.md +++ b/architecture.md @@ -647,7 +647,9 @@ is refused rather than delegated. "This event is not the root import" and "the root import, malformed" are different answers: one continues, the other fails before the recorded terminal result can be reused, without executing authored work or appending history. The record carries the content it was taken from, so -the selection is verified against that content rather than merely parsed. +the selection is verified against that content rather than merely parsed, and +reading it is total — every way journal-controlled data can refuse to be read is +the same refusal, carrying nothing the record supplied. ## Expansion identity diff --git a/packages/core/src/definition.ts b/packages/core/src/definition.ts index 388d7f49..30f94aed 100644 --- a/packages/core/src/definition.ts +++ b/packages/core/src/definition.ts @@ -51,7 +51,7 @@ function parseSource(path: string, content: string): ParsedSource { } /** The static heading structure a document's body offers as targets. */ -function documentOutline(path: string, content: string): DocumentOutline { +export function documentOutline(path: string, content: string): DocumentOutline { const body = parseSource(path, content).content; return outlineDocument(body, scanComponentSpans(body)); } diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 3263ecf6..32bfe136 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -47,6 +47,7 @@ import { } from "./validate.ts"; import { useParseCompiler } from "./components/parse-schema.ts"; import { + documentOutline, isFunctionComponentPath, parseMarkdownDefinition, parseRootMarkdownDefinition, @@ -56,11 +57,12 @@ import { asDocumentTargetError, documentTargetError, documentTargetFailure, + findTarget, isCanonicalTarget, recordedDocumentTargetFailure, sameDocumentTargetFailure, } from "./document-targets.ts"; -import type { DocumentTargetFailure } from "./document-targets.ts"; +import type { DocumentOutline, DocumentTargetFailure } from "./document-targets.ts"; import { parseReturnsDeclaration } from "./frontmatter.ts"; import { expandSegments, @@ -365,11 +367,32 @@ const UNREADABLE_ROOT_RECORD = "The recorded root document import cannot be read type RootImportRecord = | { kind: "unrelated" } | { kind: "malformed" } - | { kind: "read"; content: string; selection: SelectionOutcome }; + | { kind: "read"; outline: DocumentOutline; selection: SelectionOutcome }; const UNRELATED: RootImportRecord = { kind: "unrelated" }; const MALFORMED: RootImportRecord = { kind: "malformed" }; +/** + * Read a value that may refuse to be read. + * + * Every value this boundary touches comes from the journal, and a journal is + * data: a property may be an accessor that throws, a key list may come from a + * Proxy that refuses, and content may be markdown whose frontmatter no parser + * accepts. None of those is a failure of this run — they are ways of saying the + * record cannot be read — so none of them may travel as an error of its own. + * + * Synchronous throughout, so nothing an Effection scope owns passes through + * here: this cannot swallow a cancellation or a durability failure, because + * neither can arise inside a synchronous parse. + */ +function attempt(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + /** * Parse a recorded root import as a closed protocol. * @@ -384,13 +407,27 @@ const MALFORMED: RootImportRecord = { kind: "malformed" }; * recorded failures are not this protocol's to interpret. */ function recordedRootImport(event: Yield): RootImportRecord { - if (event.description.type !== "import_component" || event.description.name !== "__root__") { - return UNRELATED; - } - if (event.result.status !== "ok") { + const recorded = attempt(() => { + if (event.description.type !== "import_component" || event.description.name !== "__root__") { + return undefined; + } + return event.result.status === "ok" ? { value: event.result.value } : undefined; + }); + if (recorded === undefined) { return UNRELATED; } - const record = event.result.value; + // Recognized and successful, so from here every way of failing to read it is + // the same answer. `attempt` covers the throwing ways; `readRootSelection` + // returns MALFORMED for the rest. + return attempt(() => readRootSelection(recorded.value)) ?? MALFORMED; +} + +function readRootSelection(value: unknown): RootImportRecord { + // Parsed rather than read in place. `parseJson` walks every property once and + // rebuilds the record, so a trap that throws or a value that is not JSON is + // discovered here — and every read below is of this run's own copy rather + // than of an object the journal still controls. + const record = parseJson(value); if (!isJsonObject(record)) { return MALFORMED; } @@ -401,11 +438,16 @@ function recordedRootImport(event: Yield): RootImportRecord { } const kind = record["kind"]; const members = Object.keys(record).length; + // Parsing the recorded content is part of reading the record, for every + // shape. It is what the verification below compares against, and doing it + // here means a later read of the same content cannot be the first to + // discover that it does not parse. + const outline = documentOutline(path, content); if (kind === "repository") { const target = record["target"]; if (target === undefined) { - return members === 3 ? { kind: "read", content, selection: { kind: "whole" } } : MALFORMED; + return members === 3 ? { kind: "read", outline, selection: { kind: "whole" } } : MALFORMED; } if (members !== 4 || typeof target !== "string" || !isCanonicalTarget(target)) { return MALFORMED; @@ -413,11 +455,11 @@ function recordedRootImport(event: Yield): RootImportRecord { // The recorded content is here, so the target is verified against it rather // than merely parsed: a well-formed target the recorded document does not // offer describes a selection that never happened. - const resolved = resolveDocumentTarget(path, content, target); - if (!resolved.ok || resolved.value !== target) { + const resolved = findTarget(outline, target); + if (!resolved.ok || resolved.value.target !== target) { return MALFORMED; } - return { kind: "read", content, selection: { kind: "exact", target } }; + return { kind: "read", outline, selection: { kind: "exact", target } }; } if (kind === "target-failure") { @@ -428,7 +470,7 @@ function recordedRootImport(event: Yield): RootImportRecord { // Same standard for a failure: the recorded selector must fail against the // recorded content in exactly the way the record claims. That verifies the // catalog and the matches too, which no amount of shape checking could. - const rederived = resolveDocumentTarget(path, content, failure.selector); + const rederived = findTarget(outline, failure.selector); if (rederived.ok) { return MALFORMED; } @@ -436,22 +478,29 @@ function recordedRootImport(event: Yield): RootImportRecord { if (actual === undefined || !sameDocumentTargetFailure(actual.data, failure)) { return MALFORMED; } - return { kind: "read", content, selection: { kind: "failed", failure } }; + return { kind: "read", outline, selection: { kind: "failed", failure } }; } return MALFORMED; } -/** What this run's selector decides against the content the journal recorded. */ -function requestedSelection(root: RootDocumentSource, content: string): SelectionOutcome { +/** + * What this run's selector decides against the outline the journal recorded. + * + * Takes the outline the record already produced rather than the content, so + * this cannot be the call that discovers unparseable recorded markdown — that + * discovery belongs to reading the record, where it is malformed rather than an + * error of this run's own. + */ +function requestedSelection(root: RootDocumentSource, outline: DocumentOutline): SelectionOutcome { if (root.target === undefined) { return { kind: "whole" }; } - const resolved = resolveDocumentTarget(rootSourcePath(root), content, root.target); - if (resolved.ok) { - return { kind: "exact", target: resolved.value }; + const found = findTarget(outline, root.target); + if (found.ok) { + return { kind: "exact", target: found.value.target }; } - const failure = asDocumentTargetError(resolved.error); + const failure = asDocumentTargetError(found.error); // A failure this module did not build is not a selection outcome that can be // compared, so it cannot be shown compatible with anything. return failure === undefined @@ -522,7 +571,7 @@ function holdRootSelection(root: RootDocumentSource): Operation { if (recorded.kind === "malformed") { throw new Error(UNREADABLE_ROOT_RECORD); } - const requested = requestedSelection(root, recorded.content); + const requested = requestedSelection(root, recorded.outline); if (!sameSelection(recorded.selection, requested)) { throw new StaleInputError( `the recorded root document import ran ${describeSelection(recorded.selection)}, and ` + diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 3b529b70..c7ca27a2 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -22,6 +22,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { InMemoryStream } from "@executablemd/durable-streams"; import { StaleInputError } from "@executablemd/durable-streams"; +import type { DurableEvent, DurableStream } from "@executablemd/durable-streams"; import { API, useHostFiles } from "@executablemd/runtime"; import { collect } from "../src/collect.ts"; @@ -37,6 +38,7 @@ import { import { isJsonObject, parseJson } from "../src/json.ts"; import { fileSource, formatDocumentReference, inlineSource } from "../src/root-source.ts"; import type { RootDocumentSource } from "../src/root-source.ts"; +import type { Json } from "../src/types.ts"; import { asText } from "./helpers.ts"; /** What every `` in a run reported, in the order it expanded. */ @@ -593,8 +595,11 @@ describe("Tier TX — targeted replay", () => { * refused with the fixed diagnostic, and neither may expand anything or append * history. */ +/** The one thing an unreadable recorded root import says. */ +const UNREADABLE_RECORD = "The recorded root document import cannot be read by this version."; + describe("Tier TX — malformed recorded selections", () => { - const UNREADABLE = "The recorded root document import cannot be read by this version."; + const UNREADABLE = UNREADABLE_RECORD; /** A completed journal whose root import recorded a failed selection. */ function* failedJournal(): Operation { @@ -675,14 +680,23 @@ describe("Tier TX — malformed recorded selections", () => { })); }); - it("TX31: a noncanonical target entry is refused", function* () { + /** + * Two different facts, and only the second belongs to this protocol. + * + * `Beta ` and `a%2fb` are not canonical encodings, so the data parser refuses + * them (DT58). `../../etc/passwd` *is* a canonical four-level heading path + * (DT59) — what refuses it here is that the recorded document's derived + * catalog does not contain it. Shape checking alone would accept it. + */ + it("TX31: a catalog the recorded document does not derive is refused", function* () { for (const available of [["../../etc/passwd"], ["Beta "], ["a%2fb"]]) { yield* refuses((record) => ({ ...record, failure: { ...asRecord(record["failure"]), available }, })); } - // The same rule on a successful repository selection's own target. + // The same rule on a successful repository selection's own target: `beta` + // is a perfectly canonical target that this document simply does not have. const healthy = new InMemoryStream(); yield* run(inlineSource(SECTIONS, { target: "Beta" }), healthy, { names: [], ids: [] }); const stream = yield* corrupt(healthy, (record) => ({ ...record, target: "beta" })); @@ -731,3 +745,140 @@ function omit(value: unknown, key: string): Record { delete record[key]; return record; } + +/** + * Tier TX — reading a recorded root selection is total. + * + * The record is journal data, and journal data has ways of refusing to be read + * that are not defects of this run: a property may be an accessor that throws, + * a key list may come from a Proxy that refuses, and recorded markdown may have + * frontmatter no parser accepts. Each must become the one fixed answer, not an + * error of its own — an exception escaping here would carry a parser message, a + * path, or whatever a hostile record planted, and would do it from a boundary + * whose whole job is to refuse. + * + * The seam is the stream, not the record. `InMemoryStream` structured-clones on + * append, so nothing hostile can be *stored* in one; what a run actually + * consumes is whatever `readAll()` hands back. `PlantedStream` is therefore a + * stream that answers reads with a substituted root-import result, which is the + * shape a damaged or hostile backend really has. + * + * Each row plants a distinctive value and asserts it reaches nothing. + */ +const PLANTED = "pl4nted-s3cret"; + +/** A stream that answers reads with a substituted recorded root import. */ +class PlantedStream implements DurableStream { + readonly appended: DurableEvent[] = []; + + constructor( + private readonly events: readonly DurableEvent[], + private readonly planted: unknown, + ) {} + + // deno-lint-ignore require-yield + *readAll(): Operation { + return this.events.map((event) => + event.type === "yield" && event.description.name === "__root__" + ? // The one cast in this file. `Result.value` is typed `Json`, and the + // point of these rows is to put something there that is not — which + // is what a damaged backend does and what the boundary must survive. + { ...event, result: { status: "ok", value: this.planted as Json } } + : event, + ); + } + + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + this.appended.push(event); + } +} + +describe("Tier TX — unreadable recorded selections", () => { + /** A completed journal, then a stream that answers reads with `planted`. */ + function* plantedStream(planted: unknown): Operation { + const healthy = new InMemoryStream(); + yield* failure(inlineSource(SECTIONS, { target: "Missing" }), healthy); + return new PlantedStream(healthy.snapshot(), planted); + } + + /** Resume a planted journal both ways and hold every refusal to the contract. */ + function* refusesTotally(planted: unknown): Operation { + for (const target of ["Missing", "Beta"]) { + const stream = yield* plantedStream(planted); + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* useProbes(seen); + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target }), stream })); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect(isDocumentTargetError(error)).toBe(false); + // Nothing the record planted escapes, by any route a consumer would use. + const rendered = `${String(error)} ${(error as Error).stack ?? ""} ${JSON.stringify({ + ...(error as object), + })}`; + expect(rendered).not.toContain(PLANTED); + expect(rendered).not.toContain("Missing"); + // Nothing expanded, and nothing was appended on top of the record. + expect(seen.names).toEqual([]); + expect(stream.appended).toEqual([]); + } + } + + it("TX34: recorded content whose frontmatter no parser accepts is malformed", function* () { + // Valid JSON, valid string, unparseable document: without a total boundary + // the YAML parser's own failure escapes, carrying the planted text with it. + yield* refusesTotally({ + kind: "target-failure", + path: `${PLANTED}.md`, + content: `---\n: [unbalanced\n ${PLANTED}\n---\n\n# T\n`, + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, + }); + }); + + it("TX35: an unreadable member is malformed, and says nothing about itself", function* () { + const record: Record = { + kind: "target-failure", + path: "doc.md", + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, + }; + Object.defineProperty(record, "content", { + enumerable: true, + get() { + throw new Error(`accessor refused: ${PLANTED}`); + }, + }); + yield* refusesTotally(record); + }); + + it("TX36: a record that refuses key enumeration is malformed", function* () { + yield* refusesTotally( + new Proxy( + { + kind: "target-failure", + path: "doc.md", + content: "# T\n\n## Beta\n", + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, + }, + { + ownKeys() { + throw new Error(`enumeration refused: ${PLANTED}`); + }, + }, + ), + ); + }); + + it("TX37: a value that is not a record at all is malformed", function* () { + for (const planted of [`a string ${PLANTED}`, 7, null, [PLANTED]]) { + yield* refusesTotally(planted); + } + }); +}); diff --git a/packages/core/tests/document-targets.test.ts b/packages/core/tests/document-targets.test.ts index 77d66f7d..7b58cee4 100644 --- a/packages/core/tests/document-targets.test.ts +++ b/packages/core/tests/document-targets.test.ts @@ -854,26 +854,61 @@ describe("Tier DT — structural recognition", () => { expect(isDocumentTargetError(shell(symbolic))).toBe(false); }); - it("DT58: a list entry that is not a canonical target is refused", function* () { + /** + * Tested through the data parser, not through an Error shell. + * + * A shell carries a message derived from its data, so changing the data + * without rebuilding the message makes recognition fail on the message — + * which would make every case here green whether or not list validation + * exists. Going straight at `parseDocumentTargetFailure` keeps each case + * load-bearing. + */ + it("DT58: a list entry that is not an encoded canonical target is refused", function* () { const rejected: unknown[][] = [ - ["../../etc/passwd"], - ["Alpha/../Beta"], + // A raw space, tab or no-break space is not how a label is encoded. ["Alpha Beta"], ["Alpha\u0009Beta"], - ["AlphaBeta"], + ["Alpha\u00A0Beta"], + // Edge whitespace survives no round trip. + ["Alpha "], + // A lowercase escape is not what the encoder writes. ["a%2fb"], - ["Alpha", "Alpha "], + // NUL never decodes. + ["%00"], + // Not strings at all. [1], [null], // A sparse list is not a dense one. Object.assign(Array.from({ length: 2 }) as unknown[], { 0: "Alpha" }), ]; for (const available of rejected) { - expect(isDocumentTargetError(shell(Object.freeze(data({ available }))))).toBe(false); + expect(parseDocumentTargetFailure(Object.freeze(data({ available })))).toBe(undefined); + } + // The control: the same shape with a valid catalog parses. + expect(parseDocumentTargetFailure(Object.freeze(data()))).toBeDefined(); + }); + + /** + * `.` and `..` are ordinary heading labels — a document may really have a + * section called `..` — so `../../etc/passwd` is a well-formed four-level + * canonical *heading path*, never filesystem authority, and nothing here + * resolves it against a filesystem. + * + * Structural parsing therefore accepts it when the rest of the failure is + * consistent. What refuses it is the journal protocol, which compares the + * record against the catalog the recorded document derives — see TX31. + */ + it("DT59: a dotted heading path is canonical, and parses when consistent", function* () { + for (const path of ["../../etc/passwd", "Alpha/../Beta", "..", "."]) { + expect(isCanonicalTarget(path)).toBe(true); } + const parsed = parseDocumentTargetFailure( + Object.freeze(data({ available: ["../../etc/passwd", "Alpha/../Beta"] })), + ); + expect(parsed?.available).toEqual(["../../etc/passwd", "Alpha/../Beta"]); }); - it("DT59: fields no selection could have produced are refused", function* () { + it("DT60: fields no selection could have produced are refused", function* () { const inconsistent: Record[] = [ // `no-match` whose selector really does match the catalog. data({ kind: "no-match", selector: "Alpha", matches: [] }), @@ -898,11 +933,11 @@ describe("Tier DT — structural recognition", () => { data({ selector: 7 }), ]; for (const candidate of inconsistent) { - expect(isDocumentTargetError(shell(Object.freeze(candidate)))).toBe(false); + expect(parseDocumentTargetFailure(Object.freeze(candidate))).toBe(undefined); } }); - it("DT60: the Error shell is closed too", function* () { + it("DT61: the Error shell is closed too", function* () { const withCause = shell(Object.freeze(data())); Object.assign(withCause, { cause: new Error("foreign") }); expect(isDocumentTargetError(withCause)).toBe(false); @@ -925,7 +960,7 @@ describe("Tier DT — structural recognition", () => { * boundary, is any of it still reachable by the ordinary means of passing an * error on? */ - it("DT61: no planted payload survives the boundary", function* () { + it("DT62: no planted payload survives the boundary", function* () { const planted = Object.freeze( Object.defineProperty(data(), Symbol.for("secret"), { value: "s3cret", diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index be300c4e..f37b3e62 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2837,7 +2837,7 @@ class DocumentTargetError extends Error { readonly data: DocumentTargetFailure; } -function isDocumentTargetError(error: unknown): error is DocumentTargetError; +function isDocumentTargetError(error: unknown): boolean; function asDocumentTargetError(error: unknown): DocumentTargetError | undefined; function parseDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined; ``` @@ -2861,6 +2861,12 @@ revoked. This is an ordinary invocation failure with no fail-stop reason to preserve object identity, so rebuilding costs one allocation and removes every way payload could travel. +`isDocumentTargetError()` answers `boolean` and deliberately does not narrow. +A type predicate would say the candidate *is* the safe value, which is the one +thing it is not: the safe value is what `asDocumentTargetError()` builds. A +caller asking only "was this a target failure?" uses the predicate; a caller +that needs the typed error uses `asDocumentTargetError()` and reads its `data`. + A candidate is read only when all of this holds: - the data carries exactly `type`, `kind`, `selector`, `matches`, and @@ -2919,9 +2925,18 @@ exactly the failure the recorded selector produces against that content. A catalog, a match list, or a kind that the recorded document contradicts is therefore malformed too. +Reading a record is **total**. Every way a recognized successful record can fail +to be read is the same answer: an unreadable property, a key list a Proxy +refuses to produce, recorded markdown whose frontmatter no parser accepts, and +any other exception raised while reading or verifying it are all malformed. The +boundary is synchronous throughout, so it can swallow no cancellation and no +durability failure — neither arises inside a synchronous parse. + A malformed record fails before the recorded terminal result can be reused, with -one fixed, cause-free diagnostic. It never delegates, never replays the recorded -terminal error, never executes authored work, and never appends new history. +one fixed, cause-free diagnostic and nothing else: no journal text, parser +message, path, selector, YAML fragment, or accessor message reaches it. It never +delegates, never replays the recorded terminal error, never executes authored +work, and never appends new history. A root import whose recorded result is not `ok` is left alone: a root can fail for reasons that are not about selection, and those failures are not this @@ -7460,8 +7475,11 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | DT44–DT47 | Inspection | The catalog is reported without selecting; a glob resolves to the exact target; an unresolvable target fails inspection; the failure's data is frozen and rebuilt | | DT52/DT53 | Recognition | A failure from a separately loaded copy, and one built here, are read on the same terms | | DT54–DT56 | Reconstruction | The result is a fresh local error, never the candidate; a mutable nested list is copied and later mutation changes nothing; a revoked Proxy cannot reach through a result already built | -| DT57–DT59 | Closed data | Enumerable, non-enumerable and symbol-keyed extras, entries that are not canonical targets, sparse lists, and fields no selection could have produced are all refused | -| DT60/DT61 | Closed shell | A cause, an enumerable payload, and a message that does not derive from its data are refused; no planted payload survives stringification, spreading, symbol enumeration, or a journal round trip | +| DT57 | Closed data | Enumerable, non-enumerable and symbol-keyed extras are refused | +| DT58 | Canonical lists | Raw spaces, tabs, no-break spaces, edge whitespace, lowercase escapes, NUL, non-string entries and sparse lists are refused — asserted against the data parser, so the derived message cannot mask the check | +| DT59 | Dotted heading paths | `.` and `..` are legal heading labels, so `../../etc/passwd` and `Alpha/../Beta` are canonical heading paths, never filesystem authority; structural parsing accepts them when the rest of the failure is consistent | +| DT60 | Semantic outcome | Fields no selection could have produced — a `no-match` whose selector matches, a single-match ambiguity, a match outside the catalog, an `invalid-selector` that parses — are refused | +| DT61/DT62 | Closed shell | A cause, an enumerable payload, and a message that does not derive from its data are refused; no planted payload survives stringification, spreading, symbol enumeration, or a journal round trip | ### Tier TX — Targeted execution and replay @@ -7479,8 +7497,9 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX22/TX23 | Recorded content | An untargeted journal replays untargeted; replay projects the recorded content, not the file on disk | | TX24 | Failed selection | A journal from a selector that matched nothing never answers a later valid one | | TX25–TX27 | Failed replay | The same failing selector replays its own recorded failure with no authored effect; a different failure kind or a different selector is stale; live and replayed failures are the same structural error | -| TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a noncanonical or unresolvable target, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | +| TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a catalog or target the recorded document does not derive, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | | TX33 | Not vacuous | An uncorrupted record still replays its recorded failure | +| TX34–TX37 | Totality | Recorded markdown whose frontmatter no parser accepts, an unreadable member, a record refusing key enumeration, and a value that is not a record at all each become the one fixed diagnostic — cause-free, carrying no planted value, expanding nothing and appending nothing | ### Tier SL — Own-scope context updates From 4dd6fca52626c1b7a54131506a7fb6449df5a732 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 02:30:07 -0400 Subject: [PATCH 05/14] =?UTF-8?q?=F0=9F=94=92=20Tell=20an=20unreadable=20r?= =?UTF-8?q?oot=20result=20apart=20from=20an=20unrelated=20event?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recognition and reading the settled value shared one `attempt()` whose absent answer meant "unrelated". A recorded root import whose `result.value` accessor refused was therefore not the root import at all: the guard delegated, and `durableRun` reused the recorded terminal result — so resuming with a section that really exists was answered with the failure of a request nobody made. Identification and reading are now separate, and absence and refusal are distinct answers. An event that will not say what it is stays unrelated. Once it is the root import, an absent, unreadable or invalid result or settlement is malformed, except the ordinary failed settlement, which selection knows nothing about; a successful result whose value is absent or unreadable is malformed too. The boundary begins where the durable protocol hands the event over. A `result` envelope that cannot be read at all fails inside `durableRun`'s own replay indexing, before any guard runs, so no diagnostic of this protocol applies there — TX40 states that and pins what still holds: the recorded terminal result is not reused, nothing authored runs, nothing is appended. --- packages/core/src/execute.ts | 62 +++++- .../tests/document-target-execution.test.ts | 197 +++++++++++++++--- specs/executable-mdx-spec.md | 34 ++- 3 files changed, 243 insertions(+), 50 deletions(-) diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 32bfe136..6b6af8ce 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -406,20 +406,60 @@ function attempt(read: () => T): T | undefined { * that have nothing to do with selection — an unreadable file — and those * recorded failures are not this protocol's to interpret. */ +/** + * A value the journal refused to produce. + * + * Distinct from `undefined`, which is an ordinary absent value. Reading a + * member and finding nothing there, and reading a member that will not say what + * is there, are different facts about a record, and one of them is a refusal: + * conflating them is how "the root import will not say what it settled to" + * became "this is not the root import" and fell through to terminal-result + * reuse. + */ +const UNREADABLE: unique symbol = Symbol("unreadable"); + +/** One read of journal-controlled data: its value, or a refusal. */ +function read(get: () => T): T | typeof UNREADABLE { + try { + return get(); + } catch { + return UNREADABLE; + } +} + +/** The settlements the protocol recognizes as an ordinary failed root import. */ +const SETTLED_FAILURES: readonly string[] = ["err", "cancelled"]; + function recordedRootImport(event: Yield): RootImportRecord { - const recorded = attempt(() => { - if (event.description.type !== "import_component" || event.description.name !== "__root__") { - return undefined; - } - return event.result.status === "ok" ? { value: event.result.value } : undefined; - }); - if (recorded === undefined) { + // Identification first. An event that will not say what it is cannot be + // claimed as the root import, so it stays unrelated. + const description = read(() => event.description); + if (description === UNREADABLE) { return UNRELATED; } - // Recognized and successful, so from here every way of failing to read it is - // the same answer. `attempt` covers the throwing ways; `readRootSelection` - // returns MALFORMED for the rest. - return attempt(() => readRootSelection(recorded.value)) ?? MALFORMED; + const type = read(() => description.type); + const name = read(() => description.name); + if (type !== "import_component" || name !== "__root__") { + return UNRELATED; + } + + // Identified. From here the event owes this protocol an answer, and every way + // of not giving one is malformed — except the ordinary failed settlement, + // which is a root import that failed for reasons selection knows nothing + // about. + const result = read(() => event.result); + if (result === UNREADABLE || typeof result !== "object" || result === null) { + return MALFORMED; + } + const status = read(() => result.status); + if (status !== "ok") { + return typeof status === "string" && SETTLED_FAILURES.includes(status) ? UNRELATED : MALFORMED; + } + const value = read(() => ("value" in result ? result.value : undefined)); + if (value === UNREADABLE || value === undefined) { + return MALFORMED; + } + return attempt(() => readRootSelection(value)) ?? MALFORMED; } function readRootSelection(value: unknown): RootImportRecord { diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index c7ca27a2..a82a7f8d 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -767,24 +767,26 @@ function omit(value: unknown, key: string): Record { */ const PLANTED = "pl4nted-s3cret"; -/** A stream that answers reads with a substituted recorded root import. */ +/** + * A stream that answers reads with a substituted recorded root import. + * + * The substitution is a function of the healthy event, so a row can plant at + * any depth: inside a normally readable `result.value`, or on the envelope + * itself — `result`, `result.status`, `result.value` — which is where a damaged + * backend's unreadability actually lives. + */ class PlantedStream implements DurableStream { readonly appended: DurableEvent[] = []; constructor( private readonly events: readonly DurableEvent[], - private readonly planted: unknown, + private readonly plant: (event: DurableEvent) => DurableEvent, ) {} // deno-lint-ignore require-yield *readAll(): Operation { return this.events.map((event) => - event.type === "yield" && event.description.name === "__root__" - ? // The one cast in this file. `Result.value` is typed `Json`, and the - // point of these rows is to put something there that is not — which - // is what a damaged backend does and what the boundary must survive. - { ...event, result: { status: "ok", value: this.planted as Json } } - : event, + event.type === "yield" && event.description.name === "__root__" ? this.plant(event) : event, ); } @@ -794,18 +796,59 @@ class PlantedStream implements DurableStream { } } +/** + * Substitute the recorded root-import result value. + * + * `Result.value` is typed `Json`, and the point of these rows is to put + * something there that is not — which is what a damaged backend does and what + * the boundary has to survive. The casts in this file exist for that reason and + * no other. + */ +function plantValue(value: unknown): (event: DurableEvent) => DurableEvent { + return (event) => ({ ...event, result: { status: "ok", value: value as Json } }); +} + +/** Replace part of the recorded root-import envelope with a refusing accessor. */ +function plantEnvelope( + member: "result" | "status" | "value", +): (event: DurableEvent) => DurableEvent { + const refuse = () => { + throw new Error(`envelope refused: ${PLANTED}`); + }; + return (event) => { + if (member === "result") { + const planted: Record = { ...event }; + Object.defineProperty(planted, "result", { enumerable: true, get: refuse }); + return planted as unknown as DurableEvent; + } + const result: Record = + member === "status" + ? { value: { kind: "repository", path: "doc.md", content: SECTIONS } } + : { status: "ok" }; + Object.defineProperty(result, member, { enumerable: true, get: refuse }); + return { ...event, result: result as unknown as DurableEvent["result"] }; + }; +} + describe("Tier TX — unreadable recorded selections", () => { - /** A completed journal, then a stream that answers reads with `planted`. */ - function* plantedStream(planted: unknown): Operation { + /** A completed journal, then a stream that answers reads with `plant`. */ + function* plantedStream(plant: (event: DurableEvent) => DurableEvent): Operation { const healthy = new InMemoryStream(); yield* failure(inlineSource(SECTIONS, { target: "Missing" }), healthy); - return new PlantedStream(healthy.snapshot(), planted); + return new PlantedStream(healthy.snapshot(), plant); } - /** Resume a planted journal both ways and hold every refusal to the contract. */ - function* refusesTotally(planted: unknown): Operation { + /** + * Resume a planted journal both ways and hold every refusal to the contract. + * + * Both directions matter and they fail differently when the boundary is + * wrong: the original `Missing` selector would replay the recorded failure, + * and the different, genuinely valid `Beta` selector would replay that same + * `Missing` failure — an outcome for a request nobody made. + */ + function* refusesTotally(plant: (event: DurableEvent) => DurableEvent): Operation { for (const target of ["Missing", "Beta"]) { - const stream = yield* plantedStream(planted); + const stream = yield* plantedStream(plant); const seen: Probes = { names: [], ids: [] }; const error = yield* scoped(function* () { yield* useProbes(seen); @@ -819,6 +862,7 @@ describe("Tier TX — unreadable recorded selections", () => { expect((error as Error).message).toBe(UNREADABLE_RECORD); expect((error as Error).cause).toBe(undefined); + // Never the recorded failure, for either request. expect(isDocumentTargetError(error)).toBe(false); // Nothing the record planted escapes, by any route a consumer would use. const rendered = `${String(error)} ${(error as Error).stack ?? ""} ${JSON.stringify({ @@ -835,12 +879,14 @@ describe("Tier TX — unreadable recorded selections", () => { it("TX34: recorded content whose frontmatter no parser accepts is malformed", function* () { // Valid JSON, valid string, unparseable document: without a total boundary // the YAML parser's own failure escapes, carrying the planted text with it. - yield* refusesTotally({ - kind: "target-failure", - path: `${PLANTED}.md`, - content: `---\n: [unbalanced\n ${PLANTED}\n---\n\n# T\n`, - failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, - }); + yield* refusesTotally( + plantValue({ + kind: "target-failure", + path: `${PLANTED}.md`, + content: `---\n: [unbalanced\n ${PLANTED}\n---\n\n# T\n`, + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, + }), + ); }); it("TX35: an unreadable member is malformed, and says nothing about itself", function* () { @@ -855,30 +901,115 @@ describe("Tier TX — unreadable recorded selections", () => { throw new Error(`accessor refused: ${PLANTED}`); }, }); - yield* refusesTotally(record); + yield* refusesTotally(plantValue(record)); }); it("TX36: a record that refuses key enumeration is malformed", function* () { yield* refusesTotally( - new Proxy( - { - kind: "target-failure", - path: "doc.md", - content: "# T\n\n## Beta\n", - failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, - }, - { - ownKeys() { - throw new Error(`enumeration refused: ${PLANTED}`); + plantValue( + new Proxy( + { + kind: "target-failure", + path: "doc.md", + content: "# T\n\n## Beta\n", + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, }, - }, + { + ownKeys() { + throw new Error(`enumeration refused: ${PLANTED}`); + }, + }, + ), ), ); }); it("TX37: a value that is not a record at all is malformed", function* () { for (const planted of [`a string ${PLANTED}`, 7, null, [PLANTED]]) { - yield* refusesTotally(planted); + yield* refusesTotally(plantValue(planted)); } }); + + /** + * The envelope, not its contents. + * + * Recognizing the event and reading its settled value are different questions, + * and answering both with one absent value conflates "this is not the root + * import" with "the root import will not say what it settled to". The second + * then delegates, and `durableRun` reuses the recorded terminal result — so a + * request for a section that really exists is answered with the failure of a + * request nobody made. + */ + it("TX38: a successful root result whose value refuses to be read is malformed", function* () { + yield* refusesTotally(plantEnvelope("value")); + }); + + it("TX39: a root result whose settlement refuses to be read is malformed", function* () { + yield* refusesTotally(plantEnvelope("status")); + }); + + /** + * Where this boundary begins, stated rather than assumed. + * + * `result` is the protocol envelope, and `durableRun` reads it while building + * its replay index — before any guard's check phase. A stream that will not + * produce it therefore fails inside the durable protocol's own read, carrying + * that read's error, and no fixed diagnostic of this protocol's applies. + * + * What still has to hold is the safety property: the recorded terminal result + * is not reused, nothing authored runs, and nothing is appended. Those are + * asserted here; the diagnostic is not, because claiming it would describe a + * refusal this package never performed. + */ + it("TX40: an unreadable result envelope fails before this boundary, reusing nothing", function* () { + const stream = yield* plantedStream(plantEnvelope("result")); + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* useProbes(seen); + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream })); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + + // Not the recorded failure: no terminal result was reused for a request + // nobody made. + expect(isDocumentTargetError(error)).toBe(false); + expect((error as Error).message).not.toContain("matches no document target"); + expect(seen.names).toEqual([]); + expect(stream.appended).toEqual([]); + }); + + it("TX41: a successful root result with no value at all is malformed", function* () { + yield* refusesTotally((event) => ({ ...event, result: { status: "ok" } })); + }); + + /** + * The other side of the classification: an ordinary failed settlement on the + * root import is recognized and left alone, because a root can fail for + * reasons that are not about selection. + */ + it("TX42: an ordinary failed root settlement is not this protocol's to refuse", function* () { + const stream = yield* plantedStream((event) => ({ + ...event, + result: { status: "err", error: { message: "the file went away" } }, + })); + const error = yield* scoped(function* () { + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream })); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + // Not refused by this protocol. Because the guard leaves the event alone, + // the completed journal's terminal result is reused as it always is — and + // that path deserializes, so what arrives is the recorded outcome's message + // rather than a reconstructed typed failure. That is the ordinary durable + // behavior an unclaimed event should get. + expect((error as Error).message).not.toBe(UNREADABLE_RECORD); + expect((error as Error).message).toContain("matches no document target"); + }); }); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index f37b3e62..a5e0e99b 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2925,13 +2925,32 @@ exactly the failure the recorded selector produces against that content. A catalog, a match list, or a kind that the recorded document contradicts is therefore malformed too. -Reading a record is **total**. Every way a recognized successful record can fail -to be read is the same answer: an unreadable property, a key list a Proxy -refuses to produce, recorded markdown whose frontmatter no parser accepts, and -any other exception raised while reading or verifying it are all malformed. The -boundary is synchronous throughout, so it can swallow no cancellation and no +Reading a record is **total**, and identification is separate from reading. An +event that will not say what it is stays unrelated — it cannot be claimed as the +root import. Once it *is* identified as the root import, every way of not +producing a selection is malformed: + +- an absent, unreadable, or invalid result or settlement, except the ordinary + failed settlement, which is a root import that failed for reasons selection + knows nothing about and stays unrelated; +- a successful result whose value is absent or unreadable; +- an unreadable property, a key list a Proxy refuses to produce, recorded + markdown whose frontmatter no parser accepts, and any other exception raised + while reading or verifying the record. + +Absence and refusal are represented distinctly. Using one value for both is what +turns "the root import will not say what it settled to" into "this is not the +root import", which delegates and lets the recorded terminal result be reused +for a request nobody made. + +The boundary is synchronous throughout, so it can swallow no cancellation and no durability failure — neither arises inside a synchronous parse. +It begins where the durable protocol hands the event over. A `result` envelope +that cannot be read at all fails inside `durableRun`'s own replay indexing, +before any guard runs; no diagnostic of this protocol applies there, and what +still holds is that the recorded terminal result is not reused. + A malformed record fails before the recorded terminal result can be reused, with one fixed, cause-free diagnostic and nothing else: no journal text, parser message, path, selector, YAML fragment, or accessor message reaches it. It never @@ -7499,7 +7518,10 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX25–TX27 | Failed replay | The same failing selector replays its own recorded failure with no authored effect; a different failure kind or a different selector is stale; live and replayed failures are the same structural error | | TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a catalog or target the recorded document does not derive, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | | TX33 | Not vacuous | An uncorrupted record still replays its recorded failure | -| TX34–TX37 | Totality | Recorded markdown whose frontmatter no parser accepts, an unreadable member, a record refusing key enumeration, and a value that is not a record at all each become the one fixed diagnostic — cause-free, carrying no planted value, expanding nothing and appending nothing | +| TX34–TX37 | Totality, inside the value | Recorded markdown whose frontmatter no parser accepts, an unreadable member, a record refusing key enumeration, and a value that is not a record at all each become the one fixed diagnostic — cause-free, carrying no planted value, expanding nothing and appending nothing | +| TX38/TX39/TX41 | Totality, on the envelope | A successful result whose value refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated, for the original failing selector and for a different selector that would otherwise succeed | +| TX40 | Where the boundary begins | An unreadable `result` envelope fails in the durable protocol's own replay indexing, ahead of any guard; the recorded terminal result is still not reused, nothing authored runs, and nothing is appended | +| TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | ### Tier SL — Own-scope context updates From 37e9585de316d83d085de3c466fa7acb31dccb19 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 02:45:06 -0400 Subject: [PATCH 06/14] =?UTF-8?q?=F0=9F=94=92=20Index=20a=20retained=20Yie?= =?UTF-8?q?ld's=20identity=20without=20reading=20its=20result?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ReplayIndex` read every Yield's result while building itself, before any replay guard's check phase. A stream that would not produce one therefore failed inside construction and carried its own error out past every refusal — which is how an unreadable root-import envelope escaped the total, sanitized journal-reading contract as a raw accessor message. Indexing now reads identity, which is what indexing is for, and leaves the result alone until a consumer asks. A guard that would refuse an event gets its chance first, so an unreadable envelope reaches the existing malformed classification and the one fixed cause-free diagnostic. The read happens once and is kept, so a source that answers differently on a second read cannot change what replay already used. TX40 drops its exception and joins TX38, TX39 and TX41 under the full contract: the fixed diagnostic, no recorded terminal result reused, no planted text in message, stack, stringification, spreading or cause, nothing expanded and nothing appended, for the recorded selector and for a different valid one. The ordering property itself is pinned in the durable-streams suite. --- .../tests/document-target-execution.test.ts | 38 ++------- packages/durable-streams/replay-index.ts | 46 ++++++++-- .../specs/protocol-specification.md | 15 +++- .../tests/replay-index.test.ts | 85 +++++++++++++++++++ specs/executable-mdx-spec.md | 13 +-- 5 files changed, 155 insertions(+), 42 deletions(-) diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index a82a7f8d..f5998db6 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -949,37 +949,17 @@ describe("Tier TX — unreadable recorded selections", () => { }); /** - * Where this boundary begins, stated rather than assumed. + * The envelope itself, which used to be an exception. * - * `result` is the protocol envelope, and `durableRun` reads it while building - * its replay index — before any guard's check phase. A stream that will not - * produce it therefore fails inside the durable protocol's own read, carrying - * that read's error, and no fixed diagnostic of this protocol's applies. - * - * What still has to hold is the safety property: the recorded terminal result - * is not reused, nothing authored runs, and nothing is appended. Those are - * asserted here; the diagnostic is not, because claiming it would describe a - * refusal this package never performed. + * `ReplayIndex` read every Yield's result while building itself — before any + * guard's check phase — so a stream that would not produce the envelope + * failed inside indexing and carried its own error out past every refusal. + * Indexing now reads identity and leaves the result alone until a consumer + * asks, which puts this case back inside the same sanitized refusal as the + * rest. The ordering property itself is pinned in the durable-streams suite. */ - it("TX40: an unreadable result envelope fails before this boundary, reusing nothing", function* () { - const stream = yield* plantedStream(plantEnvelope("result")); - const seen: Probes = { names: [], ids: [] }; - const error = yield* scoped(function* () { - yield* useProbes(seen); - try { - yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream })); - } catch (caught) { - return caught; - } - throw new Error("the run completed instead of failing"); - }); - - // Not the recorded failure: no terminal result was reused for a request - // nobody made. - expect(isDocumentTargetError(error)).toBe(false); - expect((error as Error).message).not.toContain("matches no document target"); - expect(seen.names).toEqual([]); - expect(stream.appended).toEqual([]); + it("TX40: a root event whose result refuses to be read is malformed", function* () { + yield* refusesTotally(plantEnvelope("result")); }); it("TX41: a successful root result with no value at all is malformed", function* () { diff --git a/packages/durable-streams/replay-index.ts b/packages/durable-streams/replay-index.ts index 0d8e0e95..6db71b2f 100644 --- a/packages/durable-streams/replay-index.ts +++ b/packages/durable-streams/replay-index.ts @@ -5,13 +5,52 @@ * to Close events. See spec §4.1. */ -import type { Close, CoroutineId, DurableEvent, EffectDescription, Result } from "./types.ts"; +import type { + Close, + CoroutineId, + DurableEvent, + EffectDescription, + Result, + Yield, +} from "./types.ts"; export interface YieldEntry { description: EffectDescription; result: Result; } +/** + * One retained Yield whose result has not been read yet. + * + * Indexing reads a Yield's identity, because that is what indexing is for. It + * deliberately does not read the *result*: a replay guard's check phase runs + * after the index is built and before anything is replayed, and a guard that + * would have refused an event must get to refuse it before the stream is asked + * to produce what that event settled to. Reading eagerly took that chance away + * — a backend that could not produce a result failed during construction, + * carrying its own error out past every guard. + * + * The read happens once and is kept, so a source that answers differently on a + * second read cannot change what replay already used. + */ +class RetainedYield implements YieldEntry { + readonly description: EffectDescription; + private event: Yield; + private settled: { result: Result } | undefined; + + constructor(event: Yield) { + this.event = event; + this.description = event.description; + } + + get result(): Result { + if (this.settled === undefined) { + this.settled = { result: this.event.result }; + } + return this.settled.result; + } +} + export class ReplayIndex { private yields = new Map(); private cursors = new Map(); @@ -29,10 +68,7 @@ export class ReplayIndex { list = []; this.yields.set(event.coroutineId, list); } - list.push({ - description: event.description, - result: event.result, - }); + list.push(new RetainedYield(event)); } if (event.type === "close") { this.closes.set(event.coroutineId, event); diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 8892010a..6d953ffc 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -297,7 +297,17 @@ replay runs, even for children spawned during teardown. The replay index is a derived, in-memory structure built from the stream on startup. It provides per-coroutine cursored access to yield events -and keyed access to close events: +and keyed access to close events. + +Indexing reads each Yield's **identity** and not its result. A replay guard's +check phase runs after the index is built and before anything is replayed, so a +guard that would refuse an event has to get that chance before the stream is +asked to produce what the event settled to — an eager read hands a backend that +cannot produce a result the ability to fail past every guard, carrying its own +error. The result is read when a consumer asks for it, and read once, so a +source that answers differently on a second read cannot change what replay +already used. + ```typescript class ReplayIndex { @@ -312,7 +322,8 @@ class ReplayIndex { for (const event of events) { if (event.type === "yield") { const list = this.yields.get(event.coroutineId) ?? []; - list.push({ description: event.description, result: event.result }); + // Identity now; the result when a consumer asks. See below. + list.push(new RetainedYield(event)); this.yields.set(event.coroutineId, list); } if (event.type === "close") { diff --git a/packages/durable-streams/tests/replay-index.test.ts b/packages/durable-streams/tests/replay-index.test.ts index f9b15dc3..b046d999 100644 --- a/packages/durable-streams/tests/replay-index.test.ts +++ b/packages/durable-streams/tests/replay-index.test.ts @@ -298,3 +298,88 @@ describe("ReplayIndex", () => { }); }); }); + +/** + * Ordering — indexing reads identity, never a Yield's result. + * + * A replay guard's check phase runs after the index is built and before + * anything is replayed, so a guard that would refuse an event has to get its + * chance before the stream is asked what that event settled to. Reading + * eagerly took that chance away: a backend that could not produce a result + * failed inside construction, carrying its own error out past every guard. + * + * These are unit-level on purpose. The consequence for a document run is + * covered in `@executablemd/core`'s Tier TX; what belongs here is the ordering + * property itself. + */ +describe("ReplayIndex — result access ordering", () => { + /** A Yield whose result refuses to be read, counting attempts. */ + function refusingYield(coroutineId: string, reads: { count: number }): DurableEvent { + const event: Record = { + type: "yield", + coroutineId, + description: { type: "call", name: "work" }, + }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + reads.count++; + throw new Error("the backend will not produce this result"); + }, + }); + return event as unknown as DurableEvent; + } + + it("builds an index over a Yield whose result cannot be read", function* () { + const reads = { count: 0 }; + const index = new ReplayIndex([refusingYield("root", reads)]); + + // Construction completed, and it never asked. + expect(reads.count).toBe(0); + // Identity is indexed, because indexing is what identity is for. + expect(index.yieldCount("root")).toBe(1); + expect(index.peekYield("root")?.description).toEqual({ type: "call", name: "work" }); + expect(index.hasAnyUnconsumedYields()).toBe(true); + // Still never asked: peeking is not consuming. + expect(reads.count).toBe(0); + }); + + it("reads the result only when a consumer asks for it", function* () { + const reads = { count: 0 }; + const index = new ReplayIndex([refusingYield("root", reads)]); + const entry = index.peekYield("root"); + expect(entry).toBeDefined(); + + let caught: unknown; + try { + void entry?.result; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe("the backend will not produce this result"); + expect(reads.count).toBe(1); + }); + + it("reads a result once, so a changing source cannot change replay", function* () { + let answers = 0; + const event: Record = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + answers++; + return { status: "ok", value: answers }; + }, + }); + + const index = new ReplayIndex([event as unknown as DurableEvent]); + const entry = index.peekYield("root"); + expect(entry?.result).toEqual({ status: "ok", value: 1 }); + expect(entry?.result).toEqual({ status: "ok", value: 1 }); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: 1 }); + expect(answers).toBe(1); + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index a5e0e99b..d6ff7cfc 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2946,10 +2946,12 @@ for a request nobody made. The boundary is synchronous throughout, so it can swallow no cancellation and no durability failure — neither arises inside a synchronous parse. -It begins where the durable protocol hands the event over. A `result` envelope -that cannot be read at all fails inside `durableRun`'s own replay indexing, -before any guard runs; no diagnostic of this protocol applies there, and what -still holds is that the recorded terminal result is not reused. +It covers the whole envelope, including a `result` that cannot be read at all. +That requires replay indexing to read a Yield's *identity* without reading what +it settled to: a guard's check phase runs after the index is built, so a guard +that would refuse an event has to get its chance before the stream is asked to +produce that event's result. `ReplayIndex` therefore defers the read until a +consumer asks, and reads it once. A malformed record fails before the recorded terminal result can be reused, with one fixed, cause-free diagnostic and nothing else: no journal text, parser @@ -7519,8 +7521,7 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a catalog or target the recorded document does not derive, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | | TX33 | Not vacuous | An uncorrupted record still replays its recorded failure | | TX34–TX37 | Totality, inside the value | Recorded markdown whose frontmatter no parser accepts, an unreadable member, a record refusing key enumeration, and a value that is not a record at all each become the one fixed diagnostic — cause-free, carrying no planted value, expanding nothing and appending nothing | -| TX38/TX39/TX41 | Totality, on the envelope | A successful result whose value refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated, for the original failing selector and for a different selector that would otherwise succeed | -| TX40 | Where the boundary begins | An unreadable `result` envelope fails in the durable protocol's own replay indexing, ahead of any guard; the recorded terminal result is still not reused, nothing authored runs, and nothing is appended | +| TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | | TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | ### Tier SL — Own-scope context updates From b789219ac72a50df9c2e512692afd4046915e927 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 03:05:58 -0400 Subject: [PATCH 07/14] =?UTF-8?q?=F0=9F=94=92=20Let=20a=20guard=20and=20re?= =?UTF-8?q?play=20read=20one=20settled=20result,=20not=20two?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ReplayIndex` memoized a result only for accesses through its own entry, while the check phase was still handed the stream's own events. A guard and the replay consumer therefore performed separate reads, and a source that answered differently between them decided one thing for validation and another for execution. Both answers can be entirely valid. Resuming a partial journal with target Alpha, an accessor returning a real Alpha selection and then a real Beta selection had the guard approve Alpha, the run execute Beta, and a successful Close published for a section nobody asked for — exact-target identity broken with nothing malformed anywhere. Each retained Yield now owns one cell for what it settled to, and the check phase is handed those retained Yields rather than the stream's events, so both phases observe one value and the stream is consulted at most once. The result is snapshotted as a frozen copy of its own members, so a settled `value` is read once rather than re-read per access, and a read that threw is remembered and re-raised rather than retried. The unit rows could not have caught this: they exercise the index alone. The new rows are a durableRun integration regression and a core regression built from two real recorded selections. --- .../tests/document-target-execution.test.ts | 90 +++++++++++++++++++ packages/durable-streams/replay-index.ts | 52 +++++++++-- packages/durable-streams/run.ts | 16 ++-- .../specs/protocol-specification.md | 18 +++- .../tests/replay-guard.test.ts | 78 +++++++++++++++- specs/executable-mdx-spec.md | 13 ++- 6 files changed, 249 insertions(+), 18 deletions(-) diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index f5998db6..21126437 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -993,3 +993,93 @@ describe("Tier TX — unreadable recorded selections", () => { expect((error as Error).message).toContain("matches no document target"); }); }); + +/** + * Tier TX — the guard and replay read one settled value, not two. + * + * A recorded result is validated in the check phase and consumed during replay. + * If those are two reads of the stream's own event, a source that answers + * differently between them decides one thing for the guard and another for + * execution — the guard approves the section it was shown, and the run executes + * the section it was handed. Exact-target identity says the section that ran is + * the section the record names, and two reads make that unenforceable. + * + * Both answers here are *valid*: two real recorded selections from two real + * runs. Nothing is malformed, nothing is refused, and the only thing separating + * a correct run from a wrong one is that the value is read once. + */ +describe("Tier TX — one read across check and replay", () => { + /** The recorded root-import value of a real run against `target`. */ + function* recordedSelection(target: string): Operation { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), stream, { names: [], ids: [] }); + const recorded = stream + .snapshot() + .find((event) => event.type === "yield" && event.description.name === "__root__"); + if (recorded === undefined || recorded.result.status !== "ok") { + throw new Error("no recorded root import"); + } + const value = recorded.result.value; + if (value === undefined) { + throw new Error("recorded root import carried no value"); + } + return value; + } + + /** + * A partial journal: the recorded root import and nothing after it. + * + * Without a Close there is no completed-run shortcut, so the run replays the + * import and then continues live — which is the shape that lets the guard and + * the replay consumer both reach the same retained event. + */ + function* partialJournal(target: string): Operation { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), stream, { names: [], ids: [] }); + return stream + .snapshot() + .filter((event) => event.type === "yield" && event.description.name === "__root__"); + } + + it("TX43: a source that answers twice decides nothing twice", function* () { + const alpha = yield* recordedSelection("Alpha"); + const beta = yield* recordedSelection("Beta"); + const events = yield* partialJournal("Alpha"); + const answers: Json[] = [alpha, beta]; + const reads = { count: 0 }; + + const stream = new PlantedStream(events, (event) => { + const result: Record = { status: "ok" }; + Object.defineProperty(result, "value", { + enumerable: true, + get() { + const answer = answers[Math.min(reads.count, answers.length - 1)]!; + reads.count++; + return answer; + }, + }); + return { ...event, result: result as unknown as DurableEvent["result"] }; + }); + + const seen: Probes = { names: [], ids: [] }; + const text = yield* scoped(function* () { + yield* useProbes(seen); + return asText( + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream })), + ); + }); + + // The guard approved Alpha, so Alpha is what ran. + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + expect(text).toContain("### Inner"); + expect(text).not.toContain("## Beta"); + // And the record was consulted once, so there was never a second answer to + // disagree with the first. + expect(reads.count).toBe(1); + // What the run published describes the section the guard approved. + const close = stream.appended.find((event) => event.type === "close"); + expect(close).toBeDefined(); + expect(JSON.stringify(close)).toContain("alpha content"); + expect(JSON.stringify(close)).not.toContain("beta content"); + }); +}); diff --git a/packages/durable-streams/replay-index.ts b/packages/durable-streams/replay-index.ts index 6db71b2f..28b1be64 100644 --- a/packages/durable-streams/replay-index.ts +++ b/packages/durable-streams/replay-index.ts @@ -19,8 +19,13 @@ export interface YieldEntry { result: Result; } +/** What one retained Yield's single read of the stream produced. */ +type Settled = + | { readonly kind: "result"; readonly result: Result } + | { readonly kind: "refusal"; readonly refusal: unknown }; + /** - * One retained Yield whose result has not been read yet. + * One retained Yield, and the one cell that owns what it settled to. * * Indexing reads a Yield's identity, because that is what indexing is for. It * deliberately does not read the *result*: a replay guard's check phase runs @@ -30,22 +35,43 @@ export interface YieldEntry { * — a backend that could not produce a result failed during construction, * carrying its own error out past every guard. * - * The read happens once and is kept, so a source that answers differently on a - * second read cannot change what replay already used. + * The cell spans the whole replay lifecycle, not one accessor. A guard + * validates a retained result and a replay consumer then uses it, and those + * must be the same value: a source answering differently between the two would + * have the guard approve one thing and execution perform another, which no + * amount of validation downstream can detect. This is therefore the object the + * check phase is handed as well as the one replay reads from, and the stream's + * event is consulted at most once for either. + * + * The snapshot is a frozen copy of the result's own members, so the members a + * consumer reads — a settled `value` above all — are read from the stream + * exactly once rather than re-read per access. Both outcomes are kept: a + * refusal is remembered and re-raised rather than retried, so a source cannot + * refuse the guard and then answer replay. */ class RetainedYield implements YieldEntry { + readonly type = "yield" as const; + readonly coroutineId: CoroutineId; readonly description: EffectDescription; private event: Yield; - private settled: { result: Result } | undefined; + private settled: Settled | undefined; constructor(event: Yield) { this.event = event; + this.coroutineId = event.coroutineId; this.description = event.description; } get result(): Result { if (this.settled === undefined) { - this.settled = { result: this.event.result }; + try { + this.settled = { kind: "result", result: Object.freeze({ ...this.event.result }) }; + } catch (refusal) { + this.settled = { kind: "refusal", refusal }; + } + } + if (this.settled.kind === "refusal") { + throw this.settled.refusal; } return this.settled.result; } @@ -53,6 +79,8 @@ class RetainedYield implements YieldEntry { export class ReplayIndex { private yields = new Map(); + /** Every retained Yield in stream order, each owning its own result cell. */ + private retained: RetainedYield[] = []; private cursors = new Map(); private closes = new Map(); /** Coroutines where replay has been disabled (run-live mode). */ @@ -68,7 +96,9 @@ export class ReplayIndex { list = []; this.yields.set(event.coroutineId, list); } - list.push(new RetainedYield(event)); + const entry = new RetainedYield(event); + this.retained.push(entry); + list.push(entry); } if (event.type === "close") { this.closes.set(event.coroutineId, event); @@ -76,6 +106,16 @@ export class ReplayIndex { } } + /** + * Every retained Yield in stream order, as the events a check phase sees. + * + * The same objects the replay path consumes, so a guard and a later consumer + * observe one settled result rather than two reads of the stream. + */ + retainedYields(): Yield[] { + return [...this.retained]; + } + /** * Disable replay for a coroutine (run-live mode). * diff --git a/packages/durable-streams/run.ts b/packages/durable-streams/run.ts index 141ab862..0514dda0 100644 --- a/packages/durable-streams/run.ts +++ b/packages/durable-streams/run.ts @@ -36,13 +36,17 @@ function unalignedReplay(replayIndex: ReplayIndex, coroutineId: string) { * phase to gather observations (hash files, check timestamps) and cache * results for the decide phase. * + * The events come from the index rather than from the stream, so a guard reads + * the same retained result the replay path will use. Handing over the stream's + * own events instead would make validation and consumption two separate reads, + * and a source that answered differently between them could have a guard + * approve one result while execution used another. + * * See replay-guard-spec.md §5.5. */ -function* runCheckPhase(events: DurableEvent[], scope: Scope): Operation { - for (const event of events) { - if (event.type === "yield") { - yield* ReplayGuard.invoke(scope, "check", [event]); - } +function* runCheckPhase(replayIndex: ReplayIndex, scope: Scope): Operation { + for (const event of replayIndex.retainedYields()) { + yield* ReplayGuard.invoke(scope, "check", [event]); } } @@ -104,7 +108,7 @@ export function* durableRun( // files, make network requests) to gather observations for the decide // phase. The check loop iterates all Yield events in journal order. // See replay-guard-spec.md §5.5. - yield* runCheckPhase(events, scope); + yield* runCheckPhase(replayIndex, scope); // If the root coroutine already has a Close event in the journal, // the workflow completed in a previous run. Return the stored result diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 6d953ffc..8359818c 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -304,9 +304,21 @@ check phase runs after the index is built and before anything is replayed, so a guard that would refuse an event has to get that chance before the stream is asked to produce what the event settled to — an eager read hands a backend that cannot produce a result the ability to fail past every guard, carrying its own -error. The result is read when a consumer asks for it, and read once, so a -source that answers differently on a second read cannot change what replay -already used. +error. + +Each retained Yield owns one cell for what it settled to, and **that cell spans +the whole replay lifecycle**: the check phase is handed the retained Yields +rather than the stream's own events, so a guard that validates a result and the +replay path that later consumes it observe the same value. The stream's event is +consulted at most once. Reading once across phases is the property that makes +validation meaningful — with two reads, a source answering differently between +them has a guard approve one result while execution uses another, and nothing +downstream can detect the substitution. + +Both outcomes of that read are kept. A result is snapshotted as a frozen copy of +its own members, so a settled `value` is read from the stream exactly once +rather than re-read per access; a read that threw is remembered and re-raised +rather than retried, so a source cannot refuse the guard and then answer replay. ```typescript diff --git a/packages/durable-streams/tests/replay-guard.test.ts b/packages/durable-streams/tests/replay-guard.test.ts index 4baf074c..a6fba6f4 100644 --- a/packages/durable-streams/tests/replay-guard.test.ts +++ b/packages/durable-streams/tests/replay-guard.test.ts @@ -11,11 +11,15 @@ */ import { describe, it } from "@executablemd/test-support/bdd"; -import { run } from "effection"; +import { run, scoped } from "effection"; import { expect } from "@executablemd/test-support/expect"; +import type { Operation } from "effection"; import { + createDurableOperation, type DurableEvent, + type DurableStream, InMemoryStream, + type Json, ReplayGuard, type ReplayOutcome, StaleInputError, @@ -636,3 +640,75 @@ describe("replay guard", () => { } }); }); + +/** + * durableRun — a guard and the replay path read one settled result. + * + * The unit rows above prove repeated access through one `YieldEntry` is cached. + * They cannot prove the property that matters, which spans two phases: a guard + * validates a retained result in the check phase, and the replay path consumes + * it afterwards. If those are two reads of the stream's own event, a source + * answering differently between them has the guard approve one value while + * execution uses another, and nothing downstream can detect it. + * + * The journal here is partial — no Close — so replay consumes the retained + * Yield and then continues live, which is the shape that reaches both phases. + */ +describe("durableRun — one retained result across check and replay", () => { + it("hands the check phase and replay the same first answer", function* () { + const answers: Json[] = ["first", "second"]; + const reads = { count: 0 }; + + const event: Record = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + }; + const result: Record = { status: "ok" }; + Object.defineProperty(result, "value", { + enumerable: true, + get() { + const answer = answers[Math.min(reads.count, answers.length - 1)]; + reads.count++; + return answer; + }, + }); + event["result"] = result; + + const stream = new InMemoryStream(); + // Appended directly rather than through `append`, which clones and would + // resolve the accessor before the run ever starts. + const events = [event as unknown as DurableEvent]; + const reading: DurableStream = { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + append: (durable: DurableEvent) => stream.append(durable), + }; + + const checked: unknown[] = []; + const replayed = yield* scoped(function* () { + yield* ReplayGuard.around({ + *check([yielded], next) { + checked.push(yielded.result.status === "ok" ? yielded.result.value : undefined); + return yield* next(yielded); + }, + }); + return yield* durableRun( + function* () { + return (yield createDurableOperation({ type: "call", name: "work" }, function* () { + throw new Error("replay must not re-execute this effect"); + })) as Json; + }, + { stream: reading }, + ); + }); + + // The guard saw the first answer, replay used the first answer, and the + // stream was asked exactly once. + expect(checked).toEqual(["first"]); + expect(replayed).toBe("first"); + expect(reads.count).toBe(1); + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index d6ff7cfc..bc02ec16 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2950,8 +2950,16 @@ It covers the whole envelope, including a `result` that cannot be read at all. That requires replay indexing to read a Yield's *identity* without reading what it settled to: a guard's check phase runs after the index is built, so a guard that would refuse an event has to get its chance before the stream is asked to -produce that event's result. `ReplayIndex` therefore defers the read until a -consumer asks, and reads it once. +produce that event's result. + +"Read once" spans the phases, not one accessor. The guard validates a recorded +selection and the replay path then projects it, and those observe one settled +value: the check phase is handed the retained Yields rather than the stream's +own events, and the stream is consulted at most once. Otherwise a source that +answered differently between validation and consumption would have the guard +approve one recorded target while the run executed another — exact-target +identity would say the section that ran is the one the record names, and nothing +would make that true. A malformed record fails before the recorded terminal result can be reused, with one fixed, cause-free diagnostic and nothing else: no journal text, parser @@ -7521,6 +7529,7 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a catalog or target the recorded document does not derive, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | | TX33 | Not vacuous | An uncorrupted record still replays its recorded failure | | TX34–TX37 | Totality, inside the value | Recorded markdown whose frontmatter no parser accepts, an unreadable member, a record refusing key enumeration, and a value that is not a record at all each become the one fixed diagnostic — cause-free, carrying no planted value, expanding nothing and appending nothing | +| TX43 | One read across phases | Two valid recorded selections behind one accessor — Alpha then Beta — resume as Alpha: Alpha's section executes, Beta's never does, the source is read once, and the appended Close describes the Alpha execution | | TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | | TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | From 619177192193741a839a9d83ea37c23eab01224e Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 03:47:41 -0400 Subject: [PATCH 08/14] =?UTF-8?q?=F0=9F=94=92=20Detach=20retained=20result?= =?UTF-8?q?s,=20anchor=20the=20preamble,=20and=20require=20a=20root=20impo?= =?UTF-8?q?rt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three architecture blockers. A retained result was stable only at the top: `{ ...result }` froze the outer object while `value` stayed the journal's own. A nested `target` accessor answering Alpha to the check phase and Beta to replay had an Alpha resume execute Beta. The whole result tree is now rebuilt from members read once each, so nothing the stream still owns remains reachable. The copy is not frozen through — detaching is the claim against the stream, and freezing would be one against the consumer: an eval binding restored from a journal is pushed to by the iteration that resumes on it, which LOOP49 pins. The preamble was anchored to the first outermost heading rather than the first root-flow heading, so a document opening at a deeper level than its title put an earlier addressable section inside the preamble — retained and executed by every other target while still being a target itself. A completed journal whose root import was missing reused its Close: the per-event check phase has nothing to object to in a journal that omits the event. Guards now receive the retained history once through a new `admit` phase, after every event has been offered and before terminal reuse, and an XMD replay refuses a terminal history that does not establish exactly one recognizable root import. WR13's hand-seeded journal gains the root import a real completed run always records; it still tests that a completed journal restores with no live run. --- architecture.md | 5 + packages/core/src/document-targets.ts | 9 +- packages/core/src/execute.ts | 33 ++++ .../tests/document-target-execution.test.ts | 175 ++++++++++++++++++ packages/core/tests/document-targets.test.ts | 49 +++++ packages/durable-streams/mod.ts | 2 +- packages/durable-streams/replay-guard.ts | 36 +++- packages/durable-streams/replay-index.ts | 103 ++++++++++- packages/durable-streams/run.ts | 13 ++ .../specs/protocol-specification.md | 19 +- .../tests/replay-guard.test.ts | 137 ++++++++++++++ packages/workflow/tests/workflow-run.test.ts | 13 ++ specs/executable-mdx-spec.md | 27 ++- 13 files changed, 605 insertions(+), 16 deletions(-) diff --git a/architecture.md b/architecture.md index f4ebde6d..1fb8954a 100644 --- a/architecture.md +++ b/architecture.md @@ -615,6 +615,11 @@ parses a copy in which the scanner's top-level component spans are blanked, because a Markdown parser reading raw XMD cannot tell a component's children from the root flow. +A document's preamble is what precedes its outline — the source before the first +root-flow heading, whatever its depth. A document may open at a deeper level than +the one supplying its title, and an earlier section is a target in its own right +rather than preamble every other target carries. + Selection resolves exactly once, before the document expands and before any authored effect runs. A selector may glob, but it must name exactly one catalog entry: naming none and naming several are both failures, and two sections that diff --git a/packages/core/src/document-targets.ts b/packages/core/src/document-targets.ts index 333f4861..4083525a 100644 --- a/packages/core/src/document-targets.ts +++ b/packages/core/src/document-targets.ts @@ -60,7 +60,7 @@ export interface DocumentOutline { readonly entries: readonly DocumentTarget[]; /** Canonical encoded fragments in source order, duplicates retained. */ readonly targets: readonly string[]; - /** Where the preamble ends: the first outermost heading, or the body end. */ + /** Where the preamble ends: the first root-flow heading, or the body end. */ readonly preambleEnd: number; readonly bodyLength: number; } @@ -912,7 +912,12 @@ export function outlineDocument(body: string, spans: readonly ComponentSpan[]): headings, entries, targets: entries.map((entry) => entry.target), - preambleEnd: outermost[0]!.start, + // The first heading in the root flow, whatever its depth. The preamble is + // what precedes the outline, and a document may open at a deeper level than + // the one that supplies its title — anchoring here to the shallowest + // heading would put an earlier addressable section inside the preamble, + // where every other target would retain and run it. + preambleEnd: raw[0]!.start, bodyLength: body.length, }; } diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 6b6af8ce..8edac22c 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -625,9 +625,42 @@ function holdRootSelection(root: RootDocumentSource): Operation { } return yield* next(event); }, + *admit([history], next) { + // A journal with no retained terminal result replays what it has and + // then continues live, so a root import it does not contain is one this + // run performs. A journal that *does* carry a terminal result is + // different: reusing it means standing behind a selection, and a history + // that never recorded one — or recorded two — establishes nothing to + // stand behind. + if (history.terminal && countRootImports(history.yields) !== 1) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + return yield* next(history); + }, }); } +/** + * How many retained events are recognizably the root import. + * + * Recognized by description alone, so an import that failed for reasons + * selection knows nothing about still counts as the one that happened. An + * event that will not say what it is is not counted, which leaves a history + * that offers no recognizable root import — refused above. + */ +function countRootImports(yields: readonly Yield[]): number { + let found = 0; + for (const event of yields) { + const root = attempt( + () => event.description.type === "import_component" && event.description.name === "__root__", + ); + if (root === true) { + found += 1; + } + } + return found; +} + const execFactory: ModifierFactory = (_params) => (_args, _next) => (function* () { const context = yield* useCodeBlock(); diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 21126437..1bd56ede 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -1041,6 +1041,51 @@ describe("Tier TX — one read across check and replay", () => { .filter((event) => event.type === "yield" && event.description.name === "__root__"); } + /** + * The same substitution one level down. + * + * The record itself is stable; only the `target` member beneath it answers + * afresh. Freezing the outer result stops nothing here — the guard reads a + * record naming Alpha and replay reads the same record naming Beta. + */ + it("TX46: a nested target accessor cannot substitute another section", function* () { + const alpha = yield* recordedSelection("Alpha"); + const events = yield* partialJournal("Alpha"); + const reads = { count: 0 }; + const targets = ["Alpha", "Beta"]; + + const stream = new PlantedStream(events, (event) => { + const record: Record = { ...asRecord(alpha) }; + Object.defineProperty(record, "target", { + enumerable: true, + get() { + const answer = targets[Math.min(reads.count, targets.length - 1)]; + reads.count++; + return answer; + }, + }); + return { + ...event, + result: { status: "ok", value: record as unknown as Json }, + }; + }); + + const seen: Probes = { names: [], ids: [] }; + const text = yield* scoped(function* () { + yield* useProbes(seen); + return asText( + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream })), + ); + }); + + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + expect(text).not.toContain("## Beta"); + expect(reads.count).toBe(1); + const close = stream.appended.find((event) => event.type === "close"); + expect(JSON.stringify(close)).toContain("alpha content"); + expect(JSON.stringify(close)).not.toContain("beta content"); + }); + it("TX43: a source that answers twice decides nothing twice", function* () { const alpha = yield* recordedSelection("Alpha"); const beta = yield* recordedSelection("Beta"); @@ -1083,3 +1128,133 @@ describe("Tier TX — one read across check and replay", () => { expect(JSON.stringify(close)).not.toContain("beta content"); }); }); + +/** + * A section before the title must not run as preamble. + * + * The projection rows prove the text is absent; only a component that records + * its own invocation proves the section did not execute. + */ +describe("Tier TX — preamble boundary", () => { + const OPENS_DEEP = [ + "## Before", + "", + 'before body ', + "", + "# Title", + "", + 'title content ', + "", + "## After", + "", + 'after body ', + "", + ].join("\n"); + + it("TX44: selecting the later section never executes the earlier one", function* () { + const seen: Probes = { names: [], ids: [] }; + const text = yield* run( + inlineSource(OPENS_DEEP, { target: "After" }), + new InMemoryStream(), + seen, + ); + expect(seen.names).toEqual(["title", "after"]); + expect(text).not.toContain("## Before"); + expect(text).not.toContain("before body"); + }); + + it("TX45: the earlier section still runs when it is the target", function* () { + const seen: Probes = { names: [], ids: [] }; + yield* run(inlineSource(OPENS_DEEP, { target: "Before" }), new InMemoryStream(), seen); + expect(seen.names).toEqual(["before"]); + }); +}); + +/** + * Tier TX — a retained terminal result needs the import that produced it. + * + * `durableRun` reuses a recorded Close before replaying anything, and the check + * phase only ever sees the Yields a journal actually contains. A journal whose + * root import is gone therefore offers the guard nothing to validate, and the + * Close answers for a selection that was never established — a resume asking + * for one section receives another section's completed result. + * + * A replay that means to reuse terminal history must first establish exactly + * one recognizable root import. + */ +describe("Tier TX — a terminal journal without its root import", () => { + /** A completed journal for `target`, minus its root-import Yield. */ + function* withoutRootImport(target: string): Operation { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), complete, { names: [], ids: [] }); + const stripped = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "yield" && event.description.name === "__root__") { + continue; + } + yield* stripped.append(event); + } + return stripped; + } + + /** Hold a resume against a stripped journal to the refusal contract. */ + function* refusesStripped(stream: InMemoryStream, target?: string): Operation { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const source = + target === undefined ? inlineSource(SECTIONS) : inlineSource(SECTIONS, { target }); + const error = yield* failure(source, stream, seen); + + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect(isDocumentTargetError(error)).toBe(false); + // Never the retained Close's own outcome. + expect((error as Error).message).not.toContain("alpha content"); + expect(seen.names).toEqual([]); + expect(stream.snapshot().length).toBe(before); + } + + it("TX47: a targeted journal missing its root import refuses", function* () { + yield* refusesStripped(yield* withoutRootImport("Alpha"), "Beta"); + yield* refusesStripped(yield* withoutRootImport("Alpha"), "Alpha"); + }); + + it("TX48: an untargeted journal missing its root import refuses", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS), complete, { names: [], ids: [] }); + const stripped = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "yield" && event.description.name === "__root__") { + continue; + } + yield* stripped.append(event); + } + yield* refusesStripped(stripped); + }); + + it("TX49: a duplicated root import refuses", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const doubled = new InMemoryStream(); + for (const event of complete.snapshot()) { + yield* doubled.append(event); + if (event.type === "yield" && event.description.name === "__root__") { + yield* doubled.append(event); + } + } + yield* refusesStripped(doubled, "Alpha"); + }); + + it("TX50: the control — an intact terminal journal still replays", function* () { + const stream = new InMemoryStream(); + const golden = yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { + names: [], + ids: [], + }); + const replayed = yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { + names: [], + ids: [], + }); + expect(replayed).toBe(golden); + }); +}); diff --git a/packages/core/tests/document-targets.test.ts b/packages/core/tests/document-targets.test.ts index 7b58cee4..0e2c7bd8 100644 --- a/packages/core/tests/document-targets.test.ts +++ b/packages/core/tests/document-targets.test.ts @@ -585,6 +585,55 @@ describe("Tier DT — projection", () => { }); }); +/** + * The preamble is what precedes the outline, not what precedes the title. + * + * A document may open at a deeper level than the one supplying its title. The + * shallowest heading is then not the first, and anchoring the preamble to it + * puts an earlier addressable section inside the preamble — where it is + * retained, rendered, and executed by every other target, while still being a + * target in its own right. + */ +describe("Tier DT — preamble boundary", () => { + const OPENS_DEEP = [ + "## Before", + "", + "before body", + "", + "# Title", + "", + "## After", + "", + "after body", + "", + ].join("\n"); + + it("DT63: a section before the title is addressable and is not preamble", function* () { + expect(catalog(OPENS_DEEP)).toEqual(["Before", "After"]); + expect(outline(OPENS_DEEP).preambleEnd).toBe(0); + }); + + it("DT64: selecting the later section retains neither the earlier one nor its body", function* () { + const projected = project(OPENS_DEEP, "After"); + expect(projected).toBe(["# Title", "", "## After", "", "after body", ""].join("\n")); + expect(projected).not.toContain("## Before"); + expect(projected).not.toContain("before body"); + }); + + it("DT65: the earlier section remains independently addressable", function* () { + // Its subtree runs to the next heading, so the blank line separating them + // is retained exactly as authored. + expect(project(OPENS_DEEP, "Before")).toBe(["## Before", "", "before body", "", ""].join("\n")); + }); + + it("DT66: real preamble text before the first heading is still retained", function* () { + const withPreamble = `preamble text\n\n${OPENS_DEEP}`; + expect(project(withPreamble, "After")).toBe( + ["preamble text", "", "# Title", "", "## After", "", "after body", ""].join("\n"), + ); + }); +}); + describe("Tier DT — projected parsing", () => { function* parsed(body: string, selector?: string) { return yield* parseRootMarkdownDefinition("__root__", "doc.md", body, selector); diff --git a/packages/durable-streams/mod.ts b/packages/durable-streams/mod.ts index 2d67bc3e..ed48b9c8 100644 --- a/packages/durable-streams/mod.ts +++ b/packages/durable-streams/mod.ts @@ -58,7 +58,7 @@ export type { DivergenceDecision, DivergenceInfo, DivergenceKind } from "./diver // ReplayGuard API — pluggable validation for replay staleness detection export { ReplayGuard } from "./replay-guard.ts"; -export type { ReplayOutcome } from "./replay-guard.ts"; +export type { ReplayOutcome, RetainedHistory } from "./replay-guard.ts"; // Context export { DurableContext } from "./context.ts"; diff --git a/packages/durable-streams/replay-guard.ts b/packages/durable-streams/replay-guard.ts index cec20778..67fdf4c0 100644 --- a/packages/durable-streams/replay-guard.ts +++ b/packages/durable-streams/replay-guard.ts @@ -33,7 +33,25 @@ import type { Api, Operation } from "effection"; import { createApi } from "effection/experimental"; -import type { Yield } from "./types.ts"; +import type { CoroutineId, Yield } from "./types.ts"; + +/** + * The retained history a run is about to replay, offered once. + * + * A guard's per-event check can only speak about events a journal contains. A + * journal missing something a guard requires offers it nothing to object to, + * and the recorded terminal result is then reused on the strength of history + * that was never validated. This is where a guard says whether the history as a + * whole may be replayed at all. + */ +export interface RetainedHistory { + /** The coroutine whose recorded terminal result is about to be reused. */ + readonly coroutineId: CoroutineId; + /** Every retained Yield, each owning the one cell for what it settled to. */ + readonly yields: readonly Yield[]; + /** Whether a recorded terminal result exists for that coroutine. */ + readonly terminal: boolean; +} /** * The outcome of a replay guard's decision. @@ -61,6 +79,13 @@ export type ReplayOutcome = { outcome: "replay" } | { outcome: "error"; error?: interface ReplayGuardApi { /** Phase 1: Check — gather observations before replay (I/O allowed). */ check(event: Yield): Operation; + /** + * Phase 1b: Admit — the retained history has been offered in full, and a + * recorded terminal result has not been reused yet. A guard that requires + * something of the history as a whole — that an event it validates is + * present at all, and present once — refuses here by throwing. + */ + admit(history: RetainedHistory): Operation; /** Phase 2: Decide — return replay outcome (synchronous, pure). */ decide(event: Yield): ReplayOutcome; } @@ -72,6 +97,13 @@ function* defaultCheck(_event: Yield): Operation { // No observation — pass through to next middleware or default. } +/** + * Default admit — no-op. A history nobody objects to is replayed. + */ +function* defaultAdmit(_history: RetainedHistory): Operation { + // No requirement — pass through to next middleware or default. +} + /** * Default decide — always replay. This preserves "logs are authoritative" * as the default behavior. Guards must be explicitly installed to add @@ -116,5 +148,5 @@ function defaultDecide(_event: Yield): ReplayOutcome { */ export const ReplayGuard: Api = createApi( "DurableEffection.ReplayGuard", - { check: defaultCheck, decide: defaultDecide }, + { check: defaultCheck, admit: defaultAdmit, decide: defaultDecide }, ); diff --git a/packages/durable-streams/replay-index.ts b/packages/durable-streams/replay-index.ts index 28b1be64..61e24ab4 100644 --- a/packages/durable-streams/replay-index.ts +++ b/packages/durable-streams/replay-index.ts @@ -10,7 +10,9 @@ import type { CoroutineId, DurableEvent, EffectDescription, + Json, Result, + SerializedError, Yield, } from "./types.ts"; @@ -24,6 +26,94 @@ type Settled = | { readonly kind: "result"; readonly result: Result } | { readonly kind: "refusal"; readonly refusal: unknown }; +/** + * A detached copy of one retained JSON value. + * + * Every property is read once and rebuilt, so nothing the stream still owns + * remains reachable: a nested accessor cannot answer one thing to a guard and + * another to replay, and no later mutation of the source changes what replay + * used. Keys are defined rather than assigned, because `__proto__` reaches an + * inherited setter on some runtimes and would rewrite the copy's prototype + * instead of becoming a member of it. + * + * The copy is not frozen through. Detaching is what makes the settlement + * stable against the journal; freezing would be a claim against the *consumer*, + * and replayed values are legitimately mutable — an eval binding restored from + * a journal is pushed to by the iteration that resumes on it. + * + * A cycle is refused. `Json` has none, and a value that does is not a retained + * result this can detach — refusing is remembered like any other refusal. + */ +function detachJson(value: Json, seen: Set): Json { + if (value === null || typeof value !== "object") { + return value; + } + if (seen.has(value)) { + throw new TypeError("a retained result cannot contain a cycle"); + } + seen.add(value); + try { + if (Array.isArray(value)) { + const items: Json[] = []; + for (let index = 0; index < value.length; index++) { + items.push(detachJson(value[index]!, seen)); + } + return items; + } + const detached: { [key: string]: Json } = {}; + for (const [key, member] of Object.entries(value)) { + Object.defineProperty(detached, key, { + value: detachJson(member, seen), + enumerable: true, + writable: false, + configurable: false, + }); + } + return detached; + } finally { + seen.delete(value); + } +} + +/** A detached copy of a retained failure's description. */ +function detachError(error: SerializedError): SerializedError { + const detached: SerializedError = { message: error.message }; + const name = error.name; + const stack = error.stack; + return Object.freeze({ + ...detached, + ...(name === undefined ? {} : { name }), + ...(stack === undefined ? {} : { stack }), + }); +} + +/** + * A retained result detached from everything the stream still owns. + * + * Each member is read exactly once, here, and the tree beneath it is rebuilt. + * Freezing only the outer object would leave a `value` the journal can still + * rewrite — which is the whole substitution this exists to prevent. + */ +function detachResult(result: Result): Result { + const status = result.status; + if (status === "ok") { + if (!("value" in result)) { + return Object.freeze({ status }); + } + const value = result.value; + return Object.freeze( + value === undefined ? { status } : { status, value: detachJson(value, new Set()) }, + ); + } + if (status === "err") { + if (!("error" in result)) { + throw new TypeError("a retained failure carries the error it failed with"); + } + return Object.freeze({ status, error: detachError(result.error) }); + } + return Object.freeze({ status }); +} + /** * One retained Yield, and the one cell that owns what it settled to. * @@ -43,11 +133,12 @@ type Settled = * check phase is handed as well as the one replay reads from, and the stream's * event is consulted at most once for either. * - * The snapshot is a frozen copy of the result's own members, so the members a - * consumer reads — a settled `value` above all — are read from the stream - * exactly once rather than re-read per access. Both outcomes are kept: a - * refusal is remembered and re-raised rather than retried, so a source cannot - * refuse the guard and then answer replay. + * The settlement is detached from the stream entirely, not merely frozen at the + * top: the whole result tree is rebuilt from members read once each, so a + * nested accessor beneath `value` cannot answer one thing to a guard and + * another to replay. Both outcomes are kept — a refusal is remembered and + * re-raised rather than retried, so a source cannot refuse the guard and then + * answer replay. */ class RetainedYield implements YieldEntry { readonly type = "yield" as const; @@ -65,7 +156,7 @@ class RetainedYield implements YieldEntry { get result(): Result { if (this.settled === undefined) { try { - this.settled = { kind: "result", result: Object.freeze({ ...this.event.result }) }; + this.settled = { kind: "result", result: detachResult(this.event.result) }; } catch (refusal) { this.settled = { kind: "refusal", refusal }; } diff --git a/packages/durable-streams/run.ts b/packages/durable-streams/run.ts index 0514dda0..26e1ea4b 100644 --- a/packages/durable-streams/run.ts +++ b/packages/durable-streams/run.ts @@ -110,6 +110,19 @@ export function* durableRun( // See replay-guard-spec.md §5.5. yield* runCheckPhase(replayIndex, scope); + // ── REPLAY GUARD: Admit phase ── + // The retained history has been offered in full and nothing has been reused + // yet. A guard that requires something of the history as a whole — that an + // event it validates is present, and present once — refuses here, before the + // recorded terminal result below can answer for history nobody validated. + yield* ReplayGuard.invoke(scope, "admit", [ + { + coroutineId, + yields: replayIndex.retainedYields(), + terminal: replayIndex.hasClose(coroutineId), + }, + ]); + // If the root coroutine already has a Close event in the journal, // the workflow completed in a previous run. Return the stored result // directly without re-running the workflow. diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 8359818c..7c7c00ed 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -315,10 +315,21 @@ validation meaningful — with two reads, a source answering differently between them has a guard approve one result while execution uses another, and nothing downstream can detect the substitution. -Both outcomes of that read are kept. A result is snapshotted as a frozen copy of -its own members, so a settled `value` is read from the stream exactly once -rather than re-read per access; a read that threw is remembered and re-raised -rather than retried, so a source cannot refuse the guard and then answer replay. +Both outcomes of that read are kept. The settlement is **detached**: the whole +result tree is rebuilt from members read once each, so nothing the stream still +owns remains reachable and a nested accessor cannot answer one thing to a guard +and another to replay. The detached copy is not frozen through — detaching is +the claim against the stream, while freezing would be a claim against the +consumer, and replayed values are legitimately mutable. A read that threw is +remembered and re-raised rather than retried, so a source cannot refuse the +guard and then answer replay. + +After every retained event has been offered to `check` and before any recorded +terminal result is reused, guards receive the retained history once through +`admit`. A guard that requires something of the history as a whole — that an +event it validates is present, and present once — refuses there, because a +per-event check has nothing to object to in a journal that simply omits the +event. ```typescript diff --git a/packages/durable-streams/tests/replay-guard.test.ts b/packages/durable-streams/tests/replay-guard.test.ts index a6fba6f4..1f330131 100644 --- a/packages/durable-streams/tests/replay-guard.test.ts +++ b/packages/durable-streams/tests/replay-guard.test.ts @@ -21,6 +21,7 @@ import { InMemoryStream, type Json, ReplayGuard, + ReplayIndex, type ReplayOutcome, StaleInputError, type Workflow, @@ -712,3 +713,139 @@ describe("durableRun — one retained result across check and replay", () => { expect(reads.count).toBe(1); }); }); + +/** + * Detachment reaches the whole result, not its outermost object. + * + * Freezing `{ ...result }` stops nobody: `value` is still the journal's own + * object, and an accessor beneath it answers afresh on every read. The guard + * validates one tree and replay consumes another, with nothing in between able + * to notice. + */ +/** One member of a retained JSON object, when it really is one. */ +function member(value: Json | undefined, key: string): Json | undefined { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value[key] + : undefined; +} + +describe("durableRun — a retained result detaches completely", () => { + it("a nested accessor is read once and cannot answer twice", function* () { + const answers = ["first", "second"]; + const reads = { count: 0 }; + + const nested: Record = {}; + Object.defineProperty(nested, "target", { + enumerable: true, + get() { + const answer = answers[Math.min(reads.count, answers.length - 1)]; + reads.count++; + return answer; + }, + }); + + const event = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok", value: nested }, + }; + const events = [event as unknown as DurableEvent]; + const appended = new InMemoryStream(); + const reading: DurableStream = { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + append: (durable: DurableEvent) => appended.append(durable), + }; + + const checked: unknown[] = []; + const replayed = yield* scoped(function* () { + yield* ReplayGuard.around({ + *check([yielded], next) { + checked.push( + member(yielded.result.status === "ok" ? yielded.result.value : undefined, "target"), + ); + return yield* next(yielded); + }, + }); + return yield* durableRun( + function* () { + return (yield createDurableOperation({ type: "call", name: "work" }, function* () { + throw new Error("replay must not re-execute this effect"); + })) as Json; + }, + { stream: reading }, + ); + }); + + expect(checked).toEqual(["first"]); + expect(member(replayed, "target")).toBe("first"); + expect(reads.count).toBe(1); + }); + + it("a nested member that refuses is refused once, not retried", function* () { + const reads = { count: 0 }; + const nested: Record = {}; + Object.defineProperty(nested, "target", { + enumerable: true, + get() { + reads.count++; + throw new Error("the backend will not produce this member"); + }, + }); + + const index = new ReplayIndex([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok", value: nested }, + } as unknown as DurableEvent, + ]); + const entry = index.peekYield("root"); + + for (let attempt = 0; attempt < 3; attempt++) { + let caught: unknown; + try { + void entry?.result; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not produce this member", + ); + } + // Refused once and remembered, so a source cannot refuse the guard and then + // answer replay. + expect(reads.count).toBe(1); + }); + + it("a detached result shares nothing with the source it was read from", function* () { + const list = ["a"]; + const nested = { list }; + const source = { nested }; + const index = new ReplayIndex([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok", value: source }, + } as unknown as DurableEvent, + ]); + const result = index.peekYield("root")?.result; + const value = result?.status === "ok" ? result.value : undefined; + + // The settlement itself is stable, and nothing beneath it is the source's. + expect(Object.isFrozen(result)).toBe(true); + expect(value).not.toBe(source); + expect(member(value, "nested")).not.toBe(nested); + expect(member(member(value, "nested"), "list")).not.toBe(list); + + // So rewriting the source afterwards cannot change what replay will use. + list[0] = "rewritten"; + list.push("injected"); + expect(member(member(value, "nested"), "list")).toEqual(["a"]); + }); +}); diff --git a/packages/workflow/tests/workflow-run.test.ts b/packages/workflow/tests/workflow-run.test.ts index 5eda7f7c..b3c90bc6 100644 --- a/packages/workflow/tests/workflow-run.test.ts +++ b/packages/workflow/tests/workflow-run.test.ts @@ -400,6 +400,19 @@ describe("Tier WR — workflow runs", () => { description: { type: "workflow_run", name: "workflow_run", base: "main" }, result: { status: "ok", value: { runId: "seeded-run", base: "main", pinnedCommit: COMMIT } }, }); + // The root import a real completed run always records. A retained terminal + // result is reused on the strength of the selection its root import + // established, so a journal that carries one without the other describes a + // run that never happened and is refused. + yield* stream.append({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "", content: "# Hello\n" }, + }, + }); yield* stream.append({ type: "close", coroutineId: "root", diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index bc02ec16..092de20d 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2725,7 +2725,10 @@ duplicate canonical path is reported as an ambiguity rather than resolved. Source ranges are defined against the original, unprojected body: - the **preamble** runs from the body start to immediately before the first - outermost heading; + root-flow heading, whatever its depth — a document may open at a deeper level + than the one supplying its title, and anchoring here to the outermost heading + would put an earlier addressable section inside the preamble, where every + other target would retain and run it; - an **ancestor's direct content** runs from its heading start to its first child heading's start, or to its subtree end when it has no child heading; and @@ -2952,6 +2955,13 @@ it settled to: a guard's check phase runs after the index is built, so a guard that would refuse an event has to get its chance before the stream is asked to produce that event's result. +The settlement is detached from the journal, not merely frozen at the top: the +whole result tree is rebuilt from members read once each, so a nested accessor +beneath a recorded value cannot answer one thing to a guard and another to +replay. Detaching is the stability claim; the copy is not frozen through, +because replayed values are legitimately mutable — an eval binding restored from +a journal is pushed to by the iteration that resumes on it. + "Read once" spans the phases, not one accessor. The guard validates a recorded selection and the replay path then projects it, and those observe one settled value: the check phase is handed the retained Yields rather than the stream's @@ -2971,6 +2981,17 @@ A root import whose recorded result is not `ok` is left alone: a root can fail for reasons that are not about selection, and those failures are not this protocol's to interpret. +A replay that reuses **retained terminal history** must first establish exactly +one recognizable root import. Reusing a recorded terminal result means standing +behind the selection its root import established, and a history that recorded +none — or recorded two — establishes nothing to stand behind; the per-event +check has nothing to object to, so the refusal belongs to the history as a +whole. A missing, duplicated, or unreadable required root import fails with the +same fixed cause-free diagnostic, executes no authored work, appends nothing, +and never returns the retained terminal result. A journal with no retained +terminal result is unaffected: it replays what it has and continues live, so a +root import it does not contain is one this run performs. + ### 5.5 The Component Api Expansion's context-dependent operations are exposed through one public @@ -7529,6 +7550,10 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a catalog or target the recorded document does not derive, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | | TX33 | Not vacuous | An uncorrupted record still replays its recorded failure | | TX34–TX37 | Totality, inside the value | Recorded markdown whose frontmatter no parser accepts, an unreadable member, a record refusing key enumeration, and a value that is not a record at all each become the one fixed diagnostic — cause-free, carrying no planted value, expanding nothing and appending nothing | +| DT63–DT66 | Preamble boundary | A section before the title is addressable and is not preamble; selecting a later section neither renders nor executes it; it stays independently addressable; real preamble text before the first heading is still retained | +| TX44/TX45 | Preamble execution | Selecting the later section runs no component of the earlier one, which still runs when it is the target | +| TX46 | Nested substitution | A nested `target` accessor answering Alpha then Beta cannot substitute a section: Alpha alone executes, the member is read once, and the appended Close describes Alpha | +| TX47–TX50 | Terminal history | Targeted and untargeted completed journals with the root import removed, and one with it duplicated, are refused before terminal reuse; an intact journal still replays | | TX43 | One read across phases | Two valid recorded selections behind one accessor — Alpha then Beta — resume as Alpha: Alpha's section executes, Beta's never does, the source is read once, and the appended Close describes the Alpha execution | | TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | | TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | From ce5b00e821f1bc3181187b30739fea773f953bbc Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 04:21:19 -0400 Subject: [PATCH 09/14] =?UTF-8?q?=F0=9F=94=92=20Own=20definition=20identit?= =?UTF-8?q?y=20in=20the=20journal,=20not=20in=20replaceable=20policy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six contracts from architecture review. Exact-target validation was installed through `ReplayGuard.around()`. Guards compose, and an enclosing handler may decline to call `next` — which is what composition is for, and why identity cannot live there. The execution now owns the journal it replays through: a private gate inside `readAll` reads the retained history once, validates the recorded selection and the required root-import structure against it, and hands the same snapshot onward. Public guard policy stays composable and can suppress other guards; it cannot suppress this. Partial histories are held to the same selection rule. A retained Yield now has one identity — event type, coroutine, and complete effect description — read together, detached, and kept, so the gate, the index, public policy, and replay cannot be shown different events. Settlement stays lazy and separate. Reusing a terminal result requires exactly one recognizable root import belonging to the terminal coroutine: reusing that result means standing behind the selection its import established. Detached results are ordinary mutable JSON. Detaching is a claim against the stream, not against the consumer; members stay writable and configurable, and `__proto__` remains an own data member. Target resolution now precedes schema compilation on every public path, so a caller who named a section the document does not offer hears that rather than a complaint about a schema they never reached. --- architecture.md | 14 + packages/core/src/definition.ts | 8 +- packages/core/src/execute.ts | 199 +++++---- .../tests/document-target-execution.test.ts | 377 +++++++++++++++++- packages/durable-streams/README.md | 28 +- packages/durable-streams/mod.ts | 1 + packages/durable-streams/replay-guard.ts | 16 +- packages/durable-streams/replay-index.ts | 169 +------- packages/durable-streams/retained.ts | 260 ++++++++++++ .../specs/protocol-specification.md | 21 +- .../tests/replay-guard.test.ts | 5 +- .../tests/replay-index.test.ts | 152 ++++++- specs/executable-mdx-spec.md | 47 ++- 13 files changed, 1055 insertions(+), 242 deletions(-) create mode 100644 packages/durable-streams/retained.ts diff --git a/architecture.md b/architecture.md index 1fb8954a..543e7d43 100644 --- a/architecture.md +++ b/architecture.md @@ -647,6 +647,20 @@ selection is an outcome too, and is recorded and compared as one — otherwise a journal left by a selector that matched nothing would answer a later request for a section that does exist. +Validating that identity is **execution-owned**. Replay guards are composable +policy — a handler installed further out may decline to delegate — so identity, +which must not be negotiable, is not decided there. The execution owns the +journal it replays through and validates the recorded selection inside the read +itself, ahead of public guard policy, of a retained terminal result being +reused, of authored work, and of any append. Reusing a terminal result also +requires exactly one recognizable root import belonging to the coroutine whose +result is being reused. + +Every public path validates in one order: parse the source, resolve the target, +compile schemas, then build the projected definition. A caller who named a +section the document does not offer hears that, rather than a complaint about a +schema they never reached. + A recorded selection is a closed protocol, and a record that does not satisfy it is refused rather than delegated. "This event is not the root import" and "the root import, malformed" are different answers: one continues, the other fails diff --git a/packages/core/src/definition.ts b/packages/core/src/definition.ts index 30f94aed..9671f3cc 100644 --- a/packages/core/src/definition.ts +++ b/packages/core/src/definition.ts @@ -162,11 +162,16 @@ export function* parseRootMarkdownDefinition( content: string, selector?: string, ): Operation { + // The order is the contract, and it is the same on every public path. + // Syntax first, because the outline comes from it; then the target, because a + // caller who named nothing the document offers asked the wrong question and + // should hear that rather than a complaint about a schema they did not reach; + // then the schemas; then the projected definition. const body = parseSource(path, content); - const frontmatter = yield* compileFrontmatter(body.data); const outline = outlineDocument(body.content, scanComponentSpans(body.content)); if (selector === undefined) { + const frontmatter = yield* compileFrontmatter(body.data); const bodySegments = scanSegments(body.content, { path, baseOffset: body.baseOffset, @@ -179,6 +184,7 @@ export function* parseRootMarkdownDefinition( } const entry = selectTarget(outline, selector); + const frontmatter = yield* compileFrontmatter(body.data); const newlines = newlineCounts(body.content); const bodySegments: Segment[] = []; for (const range of retainedRanges(outline, entry)) { diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 8edac22c..9bc1dc12 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -17,8 +17,10 @@ import { durableRun, createDurableOperation, ephemeral, - ReplayGuard, + retainEvents, StaleInputError, + type CoroutineId, + type DurableEvent, type DurableStream, type Yield, } from "@executablemd/durable-streams"; @@ -356,6 +358,9 @@ type SelectionOutcome = */ const UNREADABLE_ROOT_RECORD = "The recorded root document import cannot be read by this version."; +/** The coroutine a document execution's own terminal result belongs to. */ +const ROOT_COROUTINE = "root"; + /** * What a recorded event turned out to be. * @@ -573,6 +578,98 @@ function describeSelection(selection: SelectionOutcome): string { } } +/** + * The journal a document execution reads and appends through, with the + * definition-identity check that a resumed run must pass built into the read. + * + * **This authority is not middleware.** Exact canonical target is + * workflow-definition identity, and identity may not be decided by anything a + * document, a component, or an enclosing scope can replace. A public + * `ReplayGuard` handler installed further out can decline to call `next`, which + * is exactly what composable policy is allowed to do — and exactly why the + * comparison cannot live there. Here it is a step inside `readAll`, owned by + * the execution, reachable through no context and replaceable by nothing. + * + * It runs where a journal first becomes readable, so it is ahead of everything + * a wrong answer could reach: public guard policy, any retained Yield reaching + * execution, a retained Close being reused, authored work, and any append. + * + * It also owns the retained snapshot. The events it validates are the events it + * returns, so the identity and settlement it decided on are what every later + * phase observes rather than a second reading of the backend's own objects. + */ +function guardedJournal( + stream: DurableStream, + root: RootDocumentSource, + coroutineId: CoroutineId, +): DurableStream { + return { + *readAll(): Operation { + const retained = retainEvents(yield* stream.readAll()); + admitRootHistory(retained, root, coroutineId); + return retained; + }, + append: (event: DurableEvent) => stream.append(event), + }; +} + +/** + * Decide whether this run may replay the history it was handed. + * + * Synchronous and total over journal-provided values: every way the history can + * refuse to be read is the one fixed diagnostic, and the retained events it + * reads are the ones the caller keeps. + */ +function admitRootHistory( + retained: readonly DurableEvent[], + root: RootDocumentSource, + coroutineId: CoroutineId, +): void { + const imports: Yield[] = []; + let terminal = false; + for (const event of retained) { + const kind = attempt(() => event.type); + if (kind === "close") { + terminal ||= attempt(() => event.coroutineId) === coroutineId; + continue; + } + if (kind !== "yield" || event.type !== "yield") { + continue; + } + if (isRootImport(event)) { + imports.push(event); + } + } + + // A journal with no retained terminal result replays what it has and then + // continues live, so a root import it does not contain is one this run + // performs. A journal that carries one is standing behind a selection: a + // history that recorded none, recorded two, or recorded one belonging to some + // other coroutine establishes nothing for this coroutine to stand behind. + if (terminal) { + const owned = imports.filter((event) => attempt(() => event.coroutineId) === coroutineId); + if (imports.length !== 1 || owned.length !== 1) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + } + + for (const event of imports) { + admitRootSelection(event, root); + } +} + +/** Whether a retained event is recognizably the root import. */ +function isRootImport(event: DurableEvent): boolean { + return ( + attempt( + () => + event.type === "yield" && + event.description.type === "import_component" && + event.description.name === "__root__", + ) === true + ); +} + /** * Hold a resumed run to the selection its journal recorded. * @@ -587,78 +684,35 @@ function describeSelection(selection: SelectionOutcome): string { * glob naming the same section replays, and so does the same failing selector, * while any difference in outcome is stale input. * - * A recorded failed selection is reproduced here, not delegated. Nothing later - * would reproduce it with its fields intact — `durableRun` reuses a recorded - * root Close before any effect is replayed, and that path restores a + * A recorded failed selection is reproduced here, not left for later. Nothing + * later would reproduce it with its fields intact — `durableRun` reuses a + * recorded root Close before any effect is replayed, and that path restores a * deserialized error — so a recorded failure is rebuilt from its structural - * record and raised before that reuse. Either way no authored effect runs. + * record and raised ahead of that reuse. Either way no authored effect runs. * - * This is also why validation is in the check phase rather than the decide - * phase: a decision made during replay never runs for a completed journal, - * which is exactly the run whose recorded selection must still be the one being - * asked for. + * Partial histories are held to the same rule: a recorded root import that + * names another section is refused before replay continues into it. */ -function holdRootSelection(root: RootDocumentSource): Operation { - return ReplayGuard.around({ - *check([event], next) { - const recorded = recordedRootImport(event); - if (recorded.kind === "unrelated") { - return yield* next(event); - } - // Refused here, so a corrupted record can never reach the recorded - // terminal result. Nothing is delegated, nothing is executed, and no - // history is appended. - if (recorded.kind === "malformed") { - throw new Error(UNREADABLE_ROOT_RECORD); - } - const requested = requestedSelection(root, recorded.outline); - if (!sameSelection(recorded.selection, requested)) { - throw new StaleInputError( - `the recorded root document import ran ${describeSelection(recorded.selection)}, and ` + - `this run asks for ${describeSelection(requested)}. Re-run the document from the ` + - "start rather than resuming from a journal that recorded another selection.", - { coroutineId: event.coroutineId, description: event.description }, - ); - } - if (recorded.selection.kind === "failed") { - throw documentTargetError(recorded.selection.failure); - } - return yield* next(event); - }, - *admit([history], next) { - // A journal with no retained terminal result replays what it has and - // then continues live, so a root import it does not contain is one this - // run performs. A journal that *does* carry a terminal result is - // different: reusing it means standing behind a selection, and a history - // that never recorded one — or recorded two — establishes nothing to - // stand behind. - if (history.terminal && countRootImports(history.yields) !== 1) { - throw new Error(UNREADABLE_ROOT_RECORD); - } - return yield* next(history); - }, - }); -} - -/** - * How many retained events are recognizably the root import. - * - * Recognized by description alone, so an import that failed for reasons - * selection knows nothing about still counts as the one that happened. An - * event that will not say what it is is not counted, which leaves a history - * that offers no recognizable root import — refused above. - */ -function countRootImports(yields: readonly Yield[]): number { - let found = 0; - for (const event of yields) { - const root = attempt( - () => event.description.type === "import_component" && event.description.name === "__root__", +function admitRootSelection(event: Yield, root: RootDocumentSource): void { + const recorded = recordedRootImport(event); + if (recorded.kind === "unrelated") { + return; + } + if (recorded.kind === "malformed") { + throw new Error(UNREADABLE_ROOT_RECORD); + } + const requested = requestedSelection(root, recorded.outline); + if (!sameSelection(recorded.selection, requested)) { + throw new StaleInputError( + `the recorded root document import ran ${describeSelection(recorded.selection)}, and ` + + `this run asks for ${describeSelection(requested)}. Re-run the document from the ` + + "start rather than resuming from a journal that recorded another selection.", + { coroutineId: event.coroutineId, description: event.description }, ); - if (root === true) { - found += 1; - } } - return found; + if (recorded.selection.kind === "failed") { + throw documentTargetError(recorded.selection.failure); + } } const execFactory: ModifierFactory = (_params) => (_args, _next) => @@ -1279,10 +1333,6 @@ function* executeDocument(options: ExecuteOptions): Operation { at: "min" }, ); - // Installed before the durable run, so the check phase sees the recorded - // root import before `durableRun` can reuse a recorded Close. - yield* holdRootSelection(root); - // The policy is selected here — before the durable run and before any // document, frontmatter, prop, component, or eval code exists — so the // root component import is already behind the gate. What comes back is @@ -1290,8 +1340,11 @@ function* executeDocument(options: ExecuteOptions): Operation // execution that owns it. const journal = yield* useSecretDetection(secretDetection, stream); + // The journal is wrapped before it reaches `durableRun`, so the identity + // check happens inside the read that every phase downstream depends on + // rather than in middleware anything could replace. const returned = yield* durableRun(() => Execution.operations.document(props), { - stream: journal, + stream: guardedJournal(journal, root, ROOT_COROUTINE), }); // Taken rather than read, so the handoff belongs to the run that made it. const live = yield* takeLiveFailure(liveFailure); diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 1bd56ede..8563cf07 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -22,10 +22,13 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { InMemoryStream } from "@executablemd/durable-streams"; import { StaleInputError } from "@executablemd/durable-streams"; -import type { DurableEvent, DurableStream } from "@executablemd/durable-streams"; +import { detachJson, ReplayGuard } from "@executablemd/durable-streams"; +import type { DurableEvent, DurableStream, Yield } from "@executablemd/durable-streams"; +import { createApi } from "@effectionx/context-api"; import { API, useHostFiles } from "@executablemd/runtime"; import { collect } from "../src/collect.ts"; +import { useTempFileCompiler } from "../src/temp-file-compiler.ts"; import { execute } from "../src/execute.ts"; import { inspectDocument } from "../src/inspect.ts"; import { getExpansion } from "../src/expansion.ts"; @@ -1258,3 +1261,375 @@ describe("Tier TX — a terminal journal without its root import", () => { expect(replayed).toBe(golden); }); }); + +/** + * Tier TX — definition identity is not middleware. + * + * Exact canonical target is workflow-definition identity, and identity may not + * be decided by anything a document, a component, or an enclosing scope can + * replace. A public `ReplayGuard` handler installed further out may decline to + * call `next` — that is what composable policy is *for* — so a comparison + * living there is a comparison an outer handler can switch off. + * + * These install exactly such a handler and assert the answer does not change. + */ +describe("Tier TX — identity authority is execution-owned", () => { + /** A guard that swallows every stage it is given, calling `next` for none. */ + function* useSuppressingGuard(stage: "check" | "admit"): Operation { + if (stage === "check") { + yield* ReplayGuard.around({ + // deno-lint-ignore require-yield + *check() {}, + }); + return; + } + yield* ReplayGuard.around({ + // deno-lint-ignore require-yield + *admit() {}, + }); + } + + /** Resume `stream` as `target` with `install` in scope, and report what happened. */ + function* resumeWith( + stream: InMemoryStream, + target: string, + install: () => Operation, + ): Operation<{ error: unknown; seen: Probes; appended: number }> { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* install(); + yield* useProbes(seen); + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target }), stream })); + } catch (caught) { + return caught; + } + return undefined; + }); + return { error, seen, appended: stream.snapshot().length - before }; + } + + /** A completed journal for `target`. */ + function* completed(target: string): Operation { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), stream, { names: [], ids: [] }); + return stream; + } + + /** A journal for `target` with its terminal result removed. */ + function* partial(target: string): Operation { + const complete = yield* completed(target); + const stripped = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "close") { + continue; + } + yield* stripped.append(event); + } + return stripped; + } + + it("TX51: an enclosing check handler that never delegates cannot admit Beta", function* () { + const outcome = yield* resumeWith(yield* completed("Alpha"), "Beta", () => + useSuppressingGuard("check"), + ); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX52: an enclosing admit handler that never delegates cannot admit Beta", function* () { + const outcome = yield* resumeWith(yield* completed("Alpha"), "Beta", () => + useSuppressingGuard("admit"), + ); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + /** + * A guard registered under the same name by a separately loaded copy composes + * with this run's guards, because an Effection context is keyed by its name. + * That is the portability mechanism, and it is exactly why identity is not + * kept there. + */ + it("TX53: a same-name guard from another loaded copy cannot admit Beta", function* () { + const foreign = createApi<{ check(event: Yield): Operation }>( + "DurableEffection.ReplayGuard", + { + // deno-lint-ignore require-yield + *check() {}, + }, + ); + const outcome = yield* resumeWith(yield* completed("Alpha"), "Beta", function* () { + yield* foreign.around({ + // deno-lint-ignore require-yield + *check() {}, + }); + }); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX54: a partial journal is held to its recorded selection too", function* () { + const outcome = yield* resumeWith(yield* partial("Alpha"), "Beta", () => + useSuppressingGuard("check"), + ); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX55: the controls — the same target replays, complete and partial", function* () { + const whole = yield* resumeWith(yield* completed("Alpha"), "Alpha", () => + useSuppressingGuard("check"), + ); + expect(whole.error).toBe(undefined); + + const half = yield* resumeWith(yield* partial("Alpha"), "Alpha", () => + useSuppressingGuard("check"), + ); + expect(half.error).toBe(undefined); + expect(half.seen.names).toEqual(["pre", "title", "alpha", "inner"]); + }); + + it("TX56: ordinary guard composition still works", function* () { + const stream = yield* completed("Alpha"); + const observed: string[] = []; + const outcome = yield* resumeWith(stream, "Alpha", function* () { + yield* ReplayGuard.around({ + *check([event], next) { + observed.push(event.description.name); + return yield* next(event); + }, + }); + }); + expect(outcome.error).toBe(undefined); + expect(observed).toContain("__root__"); + }); +}); + +/** + * Tier TX — the root import that authorizes a terminal result is the terminal + * coroutine's own. + * + * Reusing a recorded terminal result means standing behind the selection its + * root import established. A root import belonging to some other coroutine + * established that coroutine's selection, not this one's, and two of them + * establish nothing at all. + */ +describe("Tier TX — a terminal result needs its own root import", () => { + /** A completed Alpha journal, rewritten event by event. */ + function* rewritten( + change: (events: DurableEvent[]) => DurableEvent[], + ): Operation { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const rebuilt = new InMemoryStream(); + for (const event of change(complete.snapshot())) { + yield* rebuilt.append(event); + } + return rebuilt; + } + + function isRootImport(event: DurableEvent): boolean { + return event.type === "yield" && event.description.name === "__root__"; + } + + /** Hold a resume to the refusal contract. */ + function* refuses(stream: InMemoryStream): Operation { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target: "Alpha" }), stream, seen); + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect(seen.names).toEqual([]); + expect(stream.snapshot().length).toBe(before); + } + + it("TX57: only a child coroutine carries the root import", function* () { + yield* refuses( + yield* rewritten((events) => + events.map((event) => (isRootImport(event) ? { ...event, coroutineId: "root.7" } : event)), + ), + ); + }); + + it("TX58: a valid root import plus a root-named child event", function* () { + yield* refuses( + yield* rewritten((events) => + events.flatMap((event) => + isRootImport(event) ? [event, { ...event, coroutineId: "root.7" }] : [event], + ), + ), + ); + }); + + it("TX59: no root import at all", function* () { + yield* refuses(yield* rewritten((events) => events.filter((event) => !isRootImport(event)))); + }); + + it("TX60: two root imports on the terminal coroutine", function* () { + yield* refuses( + yield* rewritten((events) => + events.flatMap((event) => (isRootImport(event) ? [event, event] : [event])), + ), + ); + }); +}); + +/** + * Tier TX — a detached retained value is still ordinary JSON. + * + * Detaching is a claim against the journal, not against the document. A + * replayed value a document goes on to update must behave as it did when the + * run first produced it. + */ +describe("Tier TX — detached values stay mutable", () => { + const MUTATES = [ + "```js eval", + "const state = { count: 0, tags: ['a'] };", + "```", + "", + "```js eval", + "state.count += 1;", + "state.tags.push('b');", + "output(`${state.count}:${state.tags.join(',')}`);", + "```", + "", + ].join("\n"); + + function* runMutating(stream: InMemoryStream): Operation { + return yield* scoped(function* () { + // The portable compiler: a data: module is not importable under every runtime. + yield* useTempFileCompiler(); + return asText(yield* collect(yield* execute({ ...inlineSource(MUTATES), stream }))); + }); + } + + it("TX61: a replayed run updates a restored object exactly as a live one does", function* () { + const live = new InMemoryStream(); + const golden = yield* runMutating(live); + expect(golden).toContain("1:a,b"); + + // Same journal, replayed: the restored binding is written to again. + expect(yield* runMutating(live)).toBe(golden); + }); + + it("TX62: `__proto__` is retained as an own data member", function* () { + const source: Record = {}; + Object.defineProperty(source, "__proto__", { + value: { polluted: true }, + enumerable: true, + writable: true, + configurable: true, + }); + const detached = detachJson(source as Json); + expect(Object.getPrototypeOf(detached)).toBe(Object.prototype); + expect(Object.getOwnPropertyNames(detached)).toEqual(["__proto__"]); + expect(({} as Record)["polluted"]).toBe(undefined); + }); + + it("TX63: nested objects and arrays detach from the journal's own", function* () { + const list = ["a"]; + const nested = { list }; + const source = { nested }; + const detached = detachJson(source as unknown as Json); + const heldNested = (detached as Record)["nested"]; + expect(heldNested).not.toBe(nested); + expect((heldNested as Record)["list"]).not.toBe(list); + + list.push("injected"); + nested.list = ["replaced"]; + expect((heldNested as Record)["list"]).toEqual(["a"]); + }); +}); + +/** + * Tier TX — one validation order on every public path. + * + * A caller who named a section the document does not offer asked the wrong + * question, and should hear that — not a complaint about a schema they never + * reached. Resolving the target before compiling schemas makes the answer the + * same whether a host inspects, runs, or resumes. + */ +describe("Tier TX — target resolution precedes schema compilation", () => { + const BROKEN_SCHEMA = [ + "---", + "props:", + " type: object", + " properties:", + " who:", + " type: not-a-json-schema-type", + "---", + "", + "# Title", + "", + "## Kept", + "", + "kept body", + "", + ].join("\n"); + + const GOOD_SCHEMA = BROKEN_SCHEMA.replace("not-a-json-schema-type", "string"); + + /** The failure each public path reports for one source and selector. */ + function* onEveryPath(source: string, target: string): Operation { + const inspected = yield* scoped(function* () { + try { + yield* inspectDocument(inlineSource(source, { target })); + } catch (error) { + return error; + } + return undefined; + }); + + const stream = new InMemoryStream(); + const live = yield* failure(inlineSource(source, { target }), stream); + // The same journal again: a recorded failed selection keeps its precedence + // and is not replaced by the recorded terminal error. + const replayed = yield* failure(inlineSource(source, { target }), stream); + return [inspected, live, replayed]; + } + + it("TX64: an unresolvable target outranks an invalid schema on all three paths", function* () { + for (const failed of yield* onEveryPath(BROKEN_SCHEMA, "Missing")) { + expect(isDocumentTargetError(failed)).toBe(true); + expect(asDocumentTargetError(failed)?.data).toMatchObject({ + kind: "no-match", + selector: "Missing", + available: ["Kept"], + }); + } + }); + + it("TX65: a resolvable target lets the invalid schema be reported", function* () { + const [inspected, live] = yield* onEveryPath(BROKEN_SCHEMA, "Kept"); + for (const failed of [inspected, live]) { + expect(isDocumentTargetError(failed)).toBe(false); + expect((failed as Error).name).toBe("PropsSchemaError"); + } + }); + + it("TX66: an unresolvable target with a valid schema still reports the target", function* () { + for (const failed of yield* onEveryPath(GOOD_SCHEMA, "Missing")) { + expect(isDocumentTargetError(failed)).toBe(true); + } + }); + + it("TX67: the control — a resolvable target and a valid schema run", function* () { + const info = yield* inspectDocument(inlineSource(GOOD_SCHEMA, { target: "Kept" })); + expect(info.target).toBe("Kept"); + const text = asText( + yield* collect( + yield* execute({ + ...inlineSource(GOOD_SCHEMA, { target: "Kept" }), + stream: new InMemoryStream(), + }), + ), + ); + expect(text).toContain("kept body"); + }); +}); diff --git a/packages/durable-streams/README.md b/packages/durable-streams/README.md index 986f45a4..1e88442a 100644 --- a/packages/durable-streams/README.md +++ b/packages/durable-streams/README.md @@ -377,15 +377,33 @@ Divergence detection catches _structural_ mismatches — the effect sequence cha The canonical example is a file-backed effect. If the workflow previously read `./component.mdx` and that file has since been edited, replaying the stored result would silently use stale content. A replay guard detects this and can halt replay with an error. -### The two-phase model +### The three stages -Every replay guard has two phases, separated by a strict I/O boundary: +A replay guard has three stages, separated by a strict I/O boundary: -**Phase 1 — `check`**: runs in generator context before replay begins. I/O is allowed. Use it to gather current state (compute file hashes, check timestamps) and cache results in the middleware closure. +**Stage 1 — `check`**: runs in generator context before replay begins, once per retained `Yield`. I/O is allowed. Use it to gather current state (compute file hashes, check timestamps) and cache results in the middleware closure. -**Phase 2 — `decide`**: runs synchronously inside the replay loop, after identity matching succeeds. Must be pure — no I/O, no side effects. Reads from the cache populated during `check` and returns a `ReplayOutcome`. +**Stage 2 — `admit`**: runs once after every retained event has been offered to `check`, and before a recorded terminal result is reused. It receives the retained history as a whole — the coroutine about to be resumed, its retained `Yield`s, and whether a terminal result exists for it. A guard that requires something of the history *as a whole* — that an event it validates is present at all, and present once — refuses here, because a per-event `check` has nothing to object to in a journal that simply omits the event. The default is a no-op. -This separation is necessary because the replay loop is synchronous. All observation-gathering must happen upfront. +**Stage 3 — `decide`**: runs synchronously inside the replay loop, after identity matching succeeds. Must be pure — no I/O, no side effects. Reads from the cache populated during `check` and returns a `ReplayOutcome`. + +The separation between generator and synchronous stages is necessary because the replay loop is synchronous. All observation-gathering must happen upfront. + +### Guards are policy, not authority + +A replay guard is **composable policy**. Guards compose through `Api.around`, and a handler installed further out may decline to call `next` — declining is what composition is for, and it means any single guard's opinion can be suppressed by another. + +That makes a guard the wrong place for anything that decides *identity*. A consumer whose durable identity must hold regardless of what a document, a component, or an enclosing scope installs owns that check itself — for example by wrapping the `DurableStream` it hands to `durableRun`, so the validation happens inside the read every later phase depends on, reachable through no context and replaceable by nothing. `@executablemd/core` does exactly that for a document's selected target. + +Use guards for staleness policy. Do not use them to enforce an invariant that must not be negotiable. + +### Retained events are read once + +Every phase of a replay reads the same events, and a journal is data a backend supplies. Events are therefore **retained**: read once and detached from whatever the backend still owns. + +A retained `Yield`'s **identity** — its event type, its coroutine, and its complete effect description — is read together and kept, so no phase can be shown a different event than the phase before it. What the event **settled to** stays lazy and separate, because the index is built before guards run and a guard that would refuse an event must get that chance before the stream is asked to produce a result. Both reads keep both outcomes: a refusal is remembered and re-raised rather than retried. + +A detached result shares no object or array with the journal, and remains ordinary mutable JSON — detaching is a claim against the *stream*, not against the consumer, and replayed values are legitimately written to. ### Writing a replay guard diff --git a/packages/durable-streams/mod.ts b/packages/durable-streams/mod.ts index ed48b9c8..db43ec59 100644 --- a/packages/durable-streams/mod.ts +++ b/packages/durable-streams/mod.ts @@ -25,6 +25,7 @@ export type { // ReplayIndex export { ReplayIndex } from "./replay-index.ts"; +export { detachJson, retainEvents, RetainedYield } from "./retained.ts"; export type { YieldEntry } from "./replay-index.ts"; // Stream interface diff --git a/packages/durable-streams/replay-guard.ts b/packages/durable-streams/replay-guard.ts index 67fdf4c0..9fa01392 100644 --- a/packages/durable-streams/replay-guard.ts +++ b/packages/durable-streams/replay-guard.ts @@ -12,14 +12,26 @@ * (e.g., content hash, status code). There is no separate metadata field — * inputs belong in the effect description, outputs belong in the result. * - * The API has two phases: + * A guard is **composable policy, not authority**. Guards compose through + * `Api.around`, and a handler installed further out may decline to call `next`. + * That is what composition is for, and it is why an invariant that must not be + * negotiable — durable identity above all — belongs somewhere a caller cannot + * replace, such as inside the `DurableStream` a consumer hands to `durableRun`. + * + * The API has three stages: * * 1. **check** (before replay begins): Runs in generator context inside * `durableRun`, after the journal is loaded but before the workflow starts. * I/O is allowed — this is where file hashing, network checks, and other * observation-gathering happens. Results are cached in middleware closures. * - * 2. **decide** (during replay): Runs synchronously inside + * 2. **admit** (after every check, before terminal reuse): Runs once with the + * retained history as a whole. A guard that requires something of the + * history rather than of one event — that an event it validates is present, + * and present once — refuses here, because a per-event check has nothing to + * object to in a journal that omits the event. Default is a no-op. + * + * 3. **decide** (during replay): Runs synchronously inside * `DurableEffect.enter()`, after identity matching succeeds but before * the stored result is fed to the generator. Must be pure and side-effect- * free. Reads from the cache populated during the check phase. diff --git a/packages/durable-streams/replay-index.ts b/packages/durable-streams/replay-index.ts index 61e24ab4..0902bf91 100644 --- a/packages/durable-streams/replay-index.ts +++ b/packages/durable-streams/replay-index.ts @@ -5,14 +5,13 @@ * to Close events. See spec §4.1. */ +import { retainEvents } from "./retained.ts"; import type { Close, CoroutineId, DurableEvent, EffectDescription, - Json, Result, - SerializedError, Yield, } from "./types.ts"; @@ -21,157 +20,10 @@ export interface YieldEntry { result: Result; } -/** What one retained Yield's single read of the stream produced. */ -type Settled = - | { readonly kind: "result"; readonly result: Result } - | { readonly kind: "refusal"; readonly refusal: unknown }; - -/** - * A detached copy of one retained JSON value. - * - * Every property is read once and rebuilt, so nothing the stream still owns - * remains reachable: a nested accessor cannot answer one thing to a guard and - * another to replay, and no later mutation of the source changes what replay - * used. Keys are defined rather than assigned, because `__proto__` reaches an - * inherited setter on some runtimes and would rewrite the copy's prototype - * instead of becoming a member of it. - * - * The copy is not frozen through. Detaching is what makes the settlement - * stable against the journal; freezing would be a claim against the *consumer*, - * and replayed values are legitimately mutable — an eval binding restored from - * a journal is pushed to by the iteration that resumes on it. - * - * A cycle is refused. `Json` has none, and a value that does is not a retained - * result this can detach — refusing is remembered like any other refusal. - */ -function detachJson(value: Json, seen: Set): Json { - if (value === null || typeof value !== "object") { - return value; - } - if (seen.has(value)) { - throw new TypeError("a retained result cannot contain a cycle"); - } - seen.add(value); - try { - if (Array.isArray(value)) { - const items: Json[] = []; - for (let index = 0; index < value.length; index++) { - items.push(detachJson(value[index]!, seen)); - } - return items; - } - const detached: { [key: string]: Json } = {}; - for (const [key, member] of Object.entries(value)) { - Object.defineProperty(detached, key, { - value: detachJson(member, seen), - enumerable: true, - writable: false, - configurable: false, - }); - } - return detached; - } finally { - seen.delete(value); - } -} - -/** A detached copy of a retained failure's description. */ -function detachError(error: SerializedError): SerializedError { - const detached: SerializedError = { message: error.message }; - const name = error.name; - const stack = error.stack; - return Object.freeze({ - ...detached, - ...(name === undefined ? {} : { name }), - ...(stack === undefined ? {} : { stack }), - }); -} - -/** - * A retained result detached from everything the stream still owns. - * - * Each member is read exactly once, here, and the tree beneath it is rebuilt. - * Freezing only the outer object would leave a `value` the journal can still - * rewrite — which is the whole substitution this exists to prevent. - */ -function detachResult(result: Result): Result { - const status = result.status; - if (status === "ok") { - if (!("value" in result)) { - return Object.freeze({ status }); - } - const value = result.value; - return Object.freeze( - value === undefined ? { status } : { status, value: detachJson(value, new Set()) }, - ); - } - if (status === "err") { - if (!("error" in result)) { - throw new TypeError("a retained failure carries the error it failed with"); - } - return Object.freeze({ status, error: detachError(result.error) }); - } - return Object.freeze({ status }); -} - -/** - * One retained Yield, and the one cell that owns what it settled to. - * - * Indexing reads a Yield's identity, because that is what indexing is for. It - * deliberately does not read the *result*: a replay guard's check phase runs - * after the index is built and before anything is replayed, and a guard that - * would have refused an event must get to refuse it before the stream is asked - * to produce what that event settled to. Reading eagerly took that chance away - * — a backend that could not produce a result failed during construction, - * carrying its own error out past every guard. - * - * The cell spans the whole replay lifecycle, not one accessor. A guard - * validates a retained result and a replay consumer then uses it, and those - * must be the same value: a source answering differently between the two would - * have the guard approve one thing and execution perform another, which no - * amount of validation downstream can detect. This is therefore the object the - * check phase is handed as well as the one replay reads from, and the stream's - * event is consulted at most once for either. - * - * The settlement is detached from the stream entirely, not merely frozen at the - * top: the whole result tree is rebuilt from members read once each, so a - * nested accessor beneath `value` cannot answer one thing to a guard and - * another to replay. Both outcomes are kept — a refusal is remembered and - * re-raised rather than retried, so a source cannot refuse the guard and then - * answer replay. - */ -class RetainedYield implements YieldEntry { - readonly type = "yield" as const; - readonly coroutineId: CoroutineId; - readonly description: EffectDescription; - private event: Yield; - private settled: Settled | undefined; - - constructor(event: Yield) { - this.event = event; - this.coroutineId = event.coroutineId; - this.description = event.description; - } - - get result(): Result { - if (this.settled === undefined) { - try { - this.settled = { kind: "result", result: detachResult(this.event.result) }; - } catch (refusal) { - this.settled = { kind: "refusal", refusal }; - } - } - if (this.settled.kind === "refusal") { - throw this.settled.refusal; - } - return this.settled.result; - } -} - export class ReplayIndex { private yields = new Map(); - /** Every retained Yield in stream order, each owning its own result cell. */ - private retained: RetainedYield[] = []; + /** Every retained Yield in stream order, each owning its own settled cells. */ + private retained: Yield[] = []; private cursors = new Map(); private closes = new Map(); /** Coroutines where replay has been disabled (run-live mode). */ @@ -179,17 +31,24 @@ export class ReplayIndex { /** Retained coroutine identities reached by the current definition. */ private claimed = new Set(); + /** + * Index a journal's events by identity, without reading what they settled to. + * + * The events are retained first — idempotently, so a caller that already + * produced the stable history hands the same objects on rather than a second + * wrapping of them, and every phase then observes one identity and one + * settlement per event. + */ constructor(events: DurableEvent[]) { - for (const event of events) { + for (const event of retainEvents(events)) { if (event.type === "yield") { let list = this.yields.get(event.coroutineId); if (!list) { list = []; this.yields.set(event.coroutineId, list); } - const entry = new RetainedYield(event); - this.retained.push(entry); - list.push(entry); + this.retained.push(event); + list.push(event); } if (event.type === "close") { this.closes.set(event.coroutineId, event); diff --git a/packages/durable-streams/retained.ts b/packages/durable-streams/retained.ts new file mode 100644 index 00000000..82f22129 --- /dev/null +++ b/packages/durable-streams/retained.ts @@ -0,0 +1,260 @@ +/** + * Retained events — what a run reads a journal as. + * + * A journal is data supplied by a backend, and every phase of a replay reads + * the same events: a private authority gate, the replay index, public guard + * policy, and the replay path itself. If those are separate reads of the + * backend's own objects, a source that answers differently between them decides + * one thing for validation and another for execution, and nothing downstream + * can detect the substitution. + * + * A retained event is therefore read once and detached. Identity — the event + * type, the coroutine it belongs to, and its complete effect description — is + * read together and kept, so no phase can be shown a different event than the + * phase before it. What the event *settled to* stays lazy and separate: the + * index is built before guards run, and a guard that would refuse an event has + * to get that chance before the stream is asked to produce its result. + */ + +import type { + CoroutineId, + DurableEvent, + EffectDescription, + Json, + Result, + SerializedError, + Yield, +} from "./types.ts"; + +/** What one retained value's single read of the stream produced. */ +type Settled = + | { readonly kind: "value"; readonly value: T } + | { readonly kind: "refusal"; readonly refusal: unknown }; + +function settle(read: () => T): Settled { + try { + return { kind: "value", value: read() }; + } catch (refusal) { + return { kind: "refusal", refusal }; + } +} + +function resolve(settled: Settled): T { + if (settled.kind === "refusal") { + throw settled.refusal; + } + return settled.value; +} + +/** + * A detached copy of one retained JSON value. + * + * Every property is read once and rebuilt, so nothing the stream still owns + * remains reachable: a nested accessor cannot answer one thing to one phase and + * another to the next, and no later mutation of the source changes what replay + * used. + * + * The copy is ordinary JSON. Detaching is the claim against the *stream*; + * making the copy immutable would be a claim against the *consumer*, and + * replayed values are legitimately mutable — an eval binding restored from a + * journal is pushed to by the iteration that resumes on it. Members are + * therefore writable and configurable like any other JSON. + * + * Keys are defined rather than assigned all the same, because `__proto__` + * reaches an inherited setter on some runtimes and would rewrite the copy's + * prototype instead of becoming a member of it. + * + * A cycle is refused. `Json` has none, and a value that does is not something + * this can detach — refusing is remembered like any other refusal. + */ +export function detachJson(value: Json, seen: Set = new Set()): Json { + if (value === null || typeof value !== "object") { + return value; + } + if (seen.has(value)) { + throw new TypeError("a retained value cannot contain a cycle"); + } + seen.add(value); + try { + if (Array.isArray(value)) { + const items: Json[] = []; + for (let index = 0; index < value.length; index++) { + items.push(detachJson(value[index]!, seen)); + } + return items; + } + const detached: { [key: string]: Json } = {}; + for (const [key, member] of Object.entries(value)) { + Object.defineProperty(detached, key, { + value: detachJson(member, seen), + enumerable: true, + writable: true, + configurable: true, + }); + } + return detached; + } finally { + seen.delete(value); + } +} + +/** A detached copy of a retained failure's description. */ +function detachError(error: SerializedError): SerializedError { + const name = error.name; + const stack = error.stack; + return { + message: error.message, + ...(name === undefined ? {} : { name }), + ...(stack === undefined ? {} : { stack }), + }; +} + +/** + * A retained result detached from everything the stream still owns. + * + * Each member is read exactly once, here, and the tree beneath it is rebuilt. + * Copying only the outer object would leave a `value` the journal can still + * rewrite, which is the substitution this exists to prevent. + */ +function detachResult(result: Result): Result { + const status = result.status; + if (status === "ok") { + if (!("value" in result)) { + return { status }; + } + const value = result.value; + return value === undefined ? { status } : { status, value: detachJson(value) }; + } + if (status === "err") { + if (!("error" in result)) { + throw new TypeError("a retained failure carries the error it failed with"); + } + return { status, error: detachError(result.error) }; + } + return { status }; +} + +/** + * A detached copy of an effect description. + * + * `type` and `name` are the identity divergence detection compares; every other + * member is extra data a guard may read. All of it is rebuilt, so a description + * cannot name one effect while one phase looks and another while the next does. + */ +function detachDescription(description: EffectDescription): EffectDescription { + // One enumeration, so every member — `type` and `name` included — is read + // exactly once. Reading them directly and then enumerating would read each of + // them twice, which is the second read this exists to remove. + const members = Object.entries(description); + let type: Json | undefined; + let name: Json | undefined; + const extra: [string, Json][] = []; + for (const [key, member] of members) { + if (key === "type") { + type = member; + continue; + } + if (key === "name") { + name = member; + continue; + } + extra.push([key, member]); + } + if (typeof type !== "string" || typeof name !== "string") { + throw new TypeError("a retained effect description carries a type and a name"); + } + const detached: EffectDescription = { type, name }; + for (const [key, member] of extra) { + Object.defineProperty(detached, key, { + value: detachJson(member), + enumerable: true, + writable: true, + configurable: true, + }); + } + return detached; +} + +/** Everything about a retained Yield except what it settled to. */ +interface RetainedIdentity { + readonly type: "yield"; + readonly coroutineId: CoroutineId; + readonly description: EffectDescription; +} + +function readIdentity(source: Yield): RetainedIdentity { + const type = source.type; + if (type !== "yield") { + throw new TypeError("a retained Yield reports its own type"); + } + const coroutineId = source.coroutineId; + if (typeof coroutineId !== "string") { + throw new TypeError("a retained Yield belongs to a coroutine"); + } + return { type, coroutineId, description: detachDescription(source.description) }; +} + +/** + * One retained Yield: one identity, and one cell for what it settled to. + * + * Identity is read together and once. Reading `type` here and `coroutineId` + * there would let a source present an unrelated event to one phase and the root + * import to the next, which is the whole reason identity is a single settled + * fact rather than three accessors. + * + * The settlement is separate and lazy on purpose: the index is built before + * guards run, so a guard that would refuse an event must get that chance before + * the stream is asked for its result. Both outcomes of both reads are kept — a + * refusal is remembered and re-raised rather than retried, so a source cannot + * refuse one phase and then answer the next. + */ +export class RetainedYield implements Yield { + private source: Yield; + private identity: Settled | undefined; + private settled: Settled | undefined; + + constructor(source: Yield) { + this.source = source; + } + + private stable(): RetainedIdentity { + this.identity ??= settle(() => readIdentity(this.source)); + return resolve(this.identity); + } + + get type(): "yield" { + return this.stable().type; + } + + get coroutineId(): CoroutineId { + return this.stable().coroutineId; + } + + get description(): EffectDescription { + return this.stable().description; + } + + get result(): Result { + this.settled ??= settle(() => detachResult(this.source.result)); + return resolve(this.settled); + } +} + +/** + * The retained form of a journal's events. + * + * Idempotent: retaining an already-retained event returns it, so a caller that + * has produced the stable history hands the same objects onward rather than a + * second wrapping of them. That is what lets one snapshot serve every phase. + * + * Only the event type is read here, which is the least a caller can read and + * still tell a Yield from a Close. Everything else is the retained event's own. + */ +export function retainEvents(events: readonly DurableEvent[]): DurableEvent[] { + return events.map((event) => { + if (event instanceof RetainedYield) { + return event; + } + return event.type === "yield" ? new RetainedYield(event) : event; + }); +} diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 7c7c00ed..d8dd57a6 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -324,12 +324,31 @@ consumer, and replayed values are legitimately mutable. A read that threw is remembered and re-raised rather than retried, so a source cannot refuse the guard and then answer replay. +A retained Yield's **identity** — its event type, its coroutine, and its +complete effect description — is read together and once, and kept. Reading those +members separately would let a source present an unrelated event to one phase +and a significant one to the next, which is the same substitution as answering +differently about a result, one level up. Identity and settlement are separate +cells; both keep both outcomes. + +The detached result is ordinary mutable JSON. Detaching is the claim against the +stream; making the copy immutable would be a claim against the consumer, and +replayed values are legitimately written to. + After every retained event has been offered to `check` and before any recorded terminal result is reused, guards receive the retained history once through `admit`. A guard that requires something of the history as a whole — that an event it validates is present, and present once — refuses there, because a per-event check has nothing to object to in a journal that simply omits the -event. +event. Its default is a no-op. + +Replay guards are **composable policy, not authority**. Guards compose through +middleware, and a handler installed further out may decline to delegate — which +is what composition is for. An invariant that must not be negotiable therefore +does not belong in a guard. A consumer whose durable identity depends on +validating retained history owns that validation itself, for example by wrapping +the `DurableStream` it passes to `durableRun` so the check happens inside the +read every later phase depends on. ```typescript diff --git a/packages/durable-streams/tests/replay-guard.test.ts b/packages/durable-streams/tests/replay-guard.test.ts index 1f330131..85616296 100644 --- a/packages/durable-streams/tests/replay-guard.test.ts +++ b/packages/durable-streams/tests/replay-guard.test.ts @@ -823,6 +823,8 @@ describe("durableRun — a retained result detaches completely", () => { }); it("a detached result shares nothing with the source it was read from", function* () { + // Detachment is the claim, not immutability: the copy is ordinary JSON and + // a consumer may write to it, which `updates an existing property` covers. const list = ["a"]; const nested = { list }; const source = { nested }; @@ -837,8 +839,7 @@ describe("durableRun — a retained result detaches completely", () => { const result = index.peekYield("root")?.result; const value = result?.status === "ok" ? result.value : undefined; - // The settlement itself is stable, and nothing beneath it is the source's. - expect(Object.isFrozen(result)).toBe(true); + // Nothing beneath the settlement is the source's. expect(value).not.toBe(source); expect(member(value, "nested")).not.toBe(nested); expect(member(member(value, "nested"), "list")).not.toBe(list); diff --git a/packages/durable-streams/tests/replay-index.test.ts b/packages/durable-streams/tests/replay-index.test.ts index b046d999..e39e711c 100644 --- a/packages/durable-streams/tests/replay-index.test.ts +++ b/packages/durable-streams/tests/replay-index.test.ts @@ -10,7 +10,8 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { ReplayIndex } from "../replay-index.ts"; -import type { DurableEvent, Json } from "../types.ts"; +import { RetainedYield } from "../retained.ts"; +import type { DurableEvent, Json, Yield } from "../types.ts"; function yieldEvent( coroutineId: string, @@ -383,3 +384,152 @@ describe("ReplayIndex — result access ordering", () => { expect(answers).toBe(1); }); }); + +/** + * A retained Yield has one identity, read once. + * + * Identity is what every phase uses to decide *which* event it is looking at: + * the private authority gate finds the root import by it, the index groups by + * it, guards read it, and replay matches on it. Read separately, a source can + * present an unrelated event to one phase and the root import to the next — + * which is the same substitution as answering differently about a result, one + * level up. + */ +describe("RetainedYield — one stable identity", () => { + /** A Yield whose identity members answer differently on each read. */ + function shifting(answers: { + type?: string[]; + coroutineId?: string[]; + name?: string[]; + extra?: Json[]; + }): { event: DurableEvent; reads: { count: number } } { + const reads = { count: 0 }; + // Each member counts its own reads, so one accessor's answers cannot be + // shifted by another's; `reads.count` totals them. + const answering = (list: T[] | undefined, fallback: T): (() => T) => { + const answers = list ?? [fallback]; + let taken = 0; + return () => { + const answer = answers[Math.min(taken, answers.length - 1)]!; + taken++; + reads.count++; + return answer; + }; + }; + const description: Record = {}; + Object.defineProperty(description, "type", { enumerable: true, value: "import_component" }); + Object.defineProperty(description, "name", { + enumerable: true, + get: answering(answers.name, "__root__"), + }); + Object.defineProperty(description, "marker", { + enumerable: true, + get: answering(answers.extra, "stable"), + }); + const event: Record = { result: { status: "ok", value: 1 } }; + Object.defineProperty(event, "type", { + enumerable: true, + get: answering(answers.type, "yield"), + }); + Object.defineProperty(event, "coroutineId", { + enumerable: true, + get: answering(answers.coroutineId, "root"), + }); + Object.defineProperty(event, "description", { enumerable: true, value: description }); + return { event: event as unknown as DurableEvent, reads }; + } + + it("a name that turns into __root__ is seen the same way by every phase", function* () { + const { event } = shifting({ name: ["unrelated", "__root__"] }); + const retained = new RetainedYield(event as Yield); + const answers = [ + retained.description.name, + retained.description.name, + retained.description.name, + ]; + expect(answers).toEqual(["unrelated", "unrelated", "unrelated"]); + }); + + it("a name that turns away from __root__ is seen the same way by every phase", function* () { + const { event } = shifting({ name: ["__root__", "unrelated"] }); + const retained = new RetainedYield(event as Yield); + expect(retained.description.name).toBe("__root__"); + expect(retained.description.name).toBe("__root__"); + }); + + it("a shifting coroutine id settles once", function* () { + const { event } = shifting({ coroutineId: ["root", "root.3"] }); + const retained = new RetainedYield(event as Yield); + expect(retained.coroutineId).toBe("root"); + expect(retained.coroutineId).toBe("root"); + }); + + it("a shifting event type settles once", function* () { + const { event } = shifting({ type: ["yield", "close"] }); + const retained = new RetainedYield(event as Yield); + expect(retained.type).toBe("yield"); + expect(retained.type).toBe("yield"); + }); + + it("a shifting nested description member settles once", function* () { + const { event } = shifting({ extra: ["stable", "swapped"] }); + const retained = new RetainedYield(event as Yield); + expect(retained.description["marker"]).toBe("stable"); + expect(retained.description["marker"]).toBe("stable"); + }); + + it("identity is read once for all three members together", function* () { + const { event, reads } = shifting({}); + const retained = new RetainedYield(event as Yield); + void retained.type; + void retained.coroutineId; + void retained.description; + void retained.description["marker"]; + // Four members across two objects, each read exactly once. + expect(reads.count).toBe(4); + }); + + it("an identity accessor that refuses is refused once, not retried", function* () { + const reads = { count: 0 }; + const event: Record = { + type: "yield", + description: { type: "call", name: "work" }, + result: { status: "ok" }, + }; + Object.defineProperty(event, "coroutineId", { + enumerable: true, + get() { + reads.count++; + throw new Error("the backend will not say which coroutine this is"); + }, + }); + const retained = new RetainedYield(event as unknown as Yield); + for (let attempt = 0; attempt < 3; attempt++) { + let caught: unknown; + try { + void retained.coroutineId; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not say which coroutine this is", + ); + } + expect(reads.count).toBe(1); + }); + + it("the description is detached from the source", function* () { + const marker = { nested: "original" }; + const source = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work", marker }, + result: { status: "ok" }, + }; + const retained = new RetainedYield(source as unknown as Yield); + const held = retained.description["marker"]; + expect(held).not.toBe(marker); + marker.nested = "rewritten"; + expect(held).toEqual({ nested: "original" }); + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 092de20d..035f7a41 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2762,7 +2762,29 @@ The live root import records the **exact canonical target**, never the caller's selector. An untargeted import records no target member at all, so journals written before targets existed stay readable by untargeted runs. -A replay guard validates the selection before the recorded run is reused. It +Selection validation is **execution-owned and non-contextual**. Exact canonical +target is workflow-definition identity, and identity may not be decided by +anything a document, a component, or an enclosing scope can replace — a public +replay guard installed further out may decline to delegate, which is what +composable policy is for. The check therefore lives inside the journal the +execution hands to `durableRun`: it reads the retained history once, owns the +retained snapshot every later phase observes, validates the recorded selection +and the required root-import structure against it, and passes that same snapshot +on. It runs ahead of public guard policy, of any retained Yield reaching +execution, of a retained terminal result being reused, of authored work, and of +any append. Public `ReplayGuard` policy remains composable and may short-circuit +other public guards; it cannot suppress this. + +Reusing a recorded terminal result additionally requires exactly one +recognizable root import in the retained history, belonging to the coroutine +whose terminal result is being reused. Reusing that result means standing behind +the selection its root import established; a history that recorded none, +recorded two, or recorded one belonging to another coroutine establishes nothing +to stand behind. A partial history is held to the same selection rule: a +recorded root import that names another section is refused before replay +continues into it. + +The validation itself proceeds as follows. It resolves the current selector against the *recorded* content and requires the same selection outcome; the recorded content is then what the projection is taken from. A different selector naming the same section replays, and so does @@ -2774,6 +2796,25 @@ could not match. The check runs before a completed run's recorded terminal result can be reused, so a finished journal cannot answer for a selection it never made. +##### One validation order + +`inspectDocument()`, a live `execute()`, and a replayed `execute()` validate a +root document in one order: + +1. parse the source far enough to obtain the body and its outline; +2. resolve the requested target; +3. compile frontmatter, props, and return schemas; +4. build and project the selected definition. + +Resolving the target before compiling schemas is what makes the answer the same +on all three paths, and it is the useful order: a caller who named a section the +document does not offer asked the wrong question, and should hear that rather +than a complaint about a schema they never reached. So an unresolvable target +with an invalid props schema raises `DocumentTargetError`; a resolvable target +with an invalid schema raises `PropsSchemaError`; an unresolvable target with a +valid schema raises `DocumentTargetError`. A recorded failed selection keeps the +same precedence on replay and is not replaced by the recorded terminal error. + ##### Naming a root document `@executablemd/core` exposes the shared shapes: @@ -7554,6 +7595,10 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX44/TX45 | Preamble execution | Selecting the later section runs no component of the earlier one, which still runs when it is the target | | TX46 | Nested substitution | A nested `target` accessor answering Alpha then Beta cannot substitute a section: Alpha alone executes, the member is read once, and the appended Close describes Alpha | | TX47–TX50 | Terminal history | Targeted and untargeted completed journals with the root import removed, and one with it duplicated, are refused before terminal reuse; an intact journal still replays | +| TX51–TX56 | Identity authority | A completed or partial Alpha journal resumed as Beta is refused with an enclosing `check` handler that never delegates, with the equivalent `admit` handler, and with a same-name guard from another loaded copy; same-target replay and ordinary guard composition are the controls | +| TX57–TX60 | Terminal binding | A root import on a child coroutine, a valid one plus a root-named child event, none at all, and two on the terminal coroutine each refuse before terminal reuse | +| TX61–TX63 | Detached but mutable | A replayed run updates a restored object exactly as a live one does; `__proto__` is an own data member; nested objects and arrays detach from the journal's own | +| TX64–TX67 | Validation order | An unresolvable target outranks an invalid schema on inspection, live execution, and replay; a resolvable target lets the schema failure be reported; the control runs | | TX43 | One read across phases | Two valid recorded selections behind one accessor — Alpha then Beta — resume as Alpha: Alpha's section executes, Beta's never does, the source is read once, and the appended Close describes the Alpha execution | | TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | | TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | From 27e1ead991decb67a4c0fe0e6db9ff50bc10bb40 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 04:45:03 -0400 Subject: [PATCH 10/14] =?UTF-8?q?=F0=9F=94=92=20Retain=20every=20event=20t?= =?UTF-8?q?he=20history=20decides=20with,=20not=20only=20its=20yields?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `retainEvents` read `event.type` to classify and `RetainedYield` read `source.type` again, and Close events were handed on as the backend's own objects. A Close could therefore belong to a child coroutine while admission asked and to the root while the index did: the gate saw no terminal result for `root` and admitted a history whose root import had been removed, and the run then returned the recorded output — Alpha's, for a request for Beta. The discriminator is now settled by the classification that chooses an event's retained kind and never read from the source again. Close events are retained too, with their coroutine settled once and their result detached and memoized. An event that refuses to say what it is is refused from every member, and a refusal is remembered rather than retried. A Yield's settlement stays lazy, so public guard check remains the first ordinary consumer. The gate refuses a history it cannot classify, and requires the root import whenever any completion was recorded: a Close means some coroutine of this execution finished, and every coroutine it has exists because the root document was imported. Retained events present their members as ordinary own properties, so they spread, serialize, and compare like the plain events they stand for. `detachJson` and the retained classes are internal again; only `retainEvents` stays public, because core owns its own journal gate across the package boundary. Their behavior is exercised from durable-streams' own tests. --- architecture.md | 4 +- packages/core/src/execute.ts | 28 +- .../tests/document-target-execution.test.ts | 151 +++++-- packages/durable-streams/README.md | 8 +- packages/durable-streams/mod.ts | 5 +- packages/durable-streams/retained.ts | 170 +++++-- .../specs/protocol-specification.md | 29 +- .../tests/replay-index.test.ts | 152 +------ .../durable-streams/tests/retained.test.ts | 416 ++++++++++++++++++ specs/executable-mdx-spec.md | 9 +- 10 files changed, 730 insertions(+), 242 deletions(-) create mode 100644 packages/durable-streams/tests/retained.test.ts diff --git a/architecture.md b/architecture.md index 543e7d43..ed58fe71 100644 --- a/architecture.md +++ b/architecture.md @@ -652,7 +652,9 @@ policy — a handler installed further out may decline to delegate — so identi which must not be negotiable, is not decided there. The execution owns the journal it replays through and validates the recorded selection inside the read itself, ahead of public guard policy, of a retained terminal result being -reused, of authored work, and of any append. Reusing a terminal result also +reused, of authored work, and of any append. The retained history it owns covers +every event that takes part in that decision, a recorded completion included, so +no event can present one identity to the validation and another to the run. Reusing a terminal result also requires exactly one recognizable root import belonging to the coroutine whose result is being reused. diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 9bc1dc12..7bfb651d 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -628,12 +628,27 @@ function admitRootHistory( const imports: Yield[] = []; let terminal = false; for (const event of retained) { + // The retained history has already settled every discriminator, so one that + // still refuses is a history this run cannot describe — not an event to + // skip past on the way to reusing a terminal result. const kind = attempt(() => event.type); + if (kind === undefined) { + throw new Error(UNREADABLE_ROOT_RECORD); + } if (kind === "close") { - terminal ||= attempt(() => event.coroutineId) === coroutineId; + // Any recorded completion at all, not only this coroutine's. A Close + // means some coroutine of this document execution finished, and every + // coroutine it has exists because the root document was imported — so a + // history holding one while the import that authorized it is absent + // describes a run that never happened, whichever coroutine the Close + // claims to belong to. + if (attempt(() => event.coroutineId) === undefined) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + terminal = true; continue; } - if (kind !== "yield" || event.type !== "yield") { + if (event.type !== "yield") { continue; } if (isRootImport(event)) { @@ -641,11 +656,12 @@ function admitRootHistory( } } - // A journal with no retained terminal result replays what it has and then + // A journal that recorded no completion replays what it has and then // continues live, so a root import it does not contain is one this run - // performs. A journal that carries one is standing behind a selection: a - // history that recorded none, recorded two, or recorded one belonging to some - // other coroutine establishes nothing for this coroutine to stand behind. + // performs. A journal that recorded one is standing behind a selection: a + // history that recorded no import, recorded two, or recorded one belonging to + // some other coroutine establishes nothing for this coroutine to stand + // behind. if (terminal) { const owned = imports.filter((event) => attempt(() => event.coroutineId) === coroutineId); if (imports.length !== 1 || owned.length !== 1) { diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 8563cf07..7695f00f 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -22,7 +22,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { InMemoryStream } from "@executablemd/durable-streams"; import { StaleInputError } from "@executablemd/durable-streams"; -import { detachJson, ReplayGuard } from "@executablemd/durable-streams"; +import { ReplayGuard } from "@executablemd/durable-streams"; import type { DurableEvent, DurableStream, Yield } from "@executablemd/durable-streams"; import { createApi } from "@effectionx/context-api"; import { API, useHostFiles } from "@executablemd/runtime"; @@ -1501,49 +1501,51 @@ describe("Tier TX — detached values stay mutable", () => { "", ].join("\n"); - function* runMutating(stream: InMemoryStream): Operation { + function* runMutating(stream: DurableStream): Operation { return yield* scoped(function* () { - // The portable compiler: a data: module is not importable under every runtime. + // The portable compiler: a data: module is not importable under every + // runtime. yield* useTempFileCompiler(); return asText(yield* collect(yield* execute({ ...inlineSource(MUTATES), stream }))); }); } - it("TX61: a replayed run updates a restored object exactly as a live one does", function* () { - const live = new InMemoryStream(); - const golden = yield* runMutating(live); + /** + * A genuine partial replay: the binding is restored from the journal and then + * written to by an iteration that runs live. + * + * The first eval block is retained, so `state` comes back detached from the + * journal. The second is dropped along with the terminal result, so the + * update runs for real. A detached member that is not writable fails here. + */ + it("TX61: a restored binding is mutated by the live continuation", function* () { + const complete = new InMemoryStream(); + const golden = yield* runMutating(complete); expect(golden).toContain("1:a,b"); - // Same journal, replayed: the restored binding is written to again. - expect(yield* runMutating(live)).toBe(golden); - }); - - it("TX62: `__proto__` is retained as an own data member", function* () { - const source: Record = {}; - Object.defineProperty(source, "__proto__", { - value: { polluted: true }, - enumerable: true, - writable: true, - configurable: true, - }); - const detached = detachJson(source as Json); - expect(Object.getPrototypeOf(detached)).toBe(Object.prototype); - expect(Object.getOwnPropertyNames(detached)).toEqual(["__proto__"]); - expect(({} as Record)["polluted"]).toBe(undefined); - }); + const evals = complete + .snapshot() + .filter((event) => event.type === "yield" && event.description.type === "eval"); + expect(evals.length).toBeGreaterThan(1); - it("TX63: nested objects and arrays detach from the journal's own", function* () { - const list = ["a"]; - const nested = { list }; - const source = { nested }; - const detached = detachJson(source as unknown as Json); - const heldNested = (detached as Record)["nested"]; - expect(heldNested).not.toBe(nested); - expect((heldNested as Record)["list"]).not.toBe(list); + const partial = new InMemoryStream(); + let kept = 0; + for (const event of complete.snapshot()) { + if (event.type === "close") { + continue; + } + if (event.type === "yield" && event.description.type === "eval") { + kept += 1; + if (kept > 1) { + continue; + } + } + yield* partial.append(event); + } - list.push("injected"); - nested.list = ["replaced"]; - expect((heldNested as Record)["list"]).toEqual(["a"]); + // The first block replays and restores `state`; the second runs live and + // writes to it. + expect(yield* runMutating(partial)).toBe(golden); }); }); @@ -1606,8 +1608,9 @@ describe("Tier TX — target resolution precedes schema compilation", () => { }); it("TX65: a resolvable target lets the invalid schema be reported", function* () { - const [inspected, live] = yield* onEveryPath(BROKEN_SCHEMA, "Kept"); - for (const failed of [inspected, live]) { + // All three paths, replay included: the recorded terminal error is the + // schema failure, and nothing replaces it with a target failure. + for (const failed of yield* onEveryPath(BROKEN_SCHEMA, "Kept")) { expect(isDocumentTargetError(failed)).toBe(false); expect((failed as Error).name).toBe("PropsSchemaError"); } @@ -1633,3 +1636,79 @@ describe("Tier TX — target resolution precedes schema compilation", () => { expect(text).toContain("kept body"); }); }); + +/** + * Tier TX — a Close cannot change coroutines between phases. + * + * The admission gate decides whether a terminal result exists; the index + * decides whose it is. Read separately, a Close can belong to a child while the + * gate asks and to the root while the index does — the gate admits a history it + * believes has no terminal result, and the run then returns one, for a section + * nobody requested. + * + * Recorded before the fix, resuming as Beta against an Alpha journal whose root + * import was removed: + * + * ```json + * {"requested":"Beta","ok":true,"value":"# Title\n\n## Alpha\n\nalpha content\n\n"} + * ``` + */ +describe("Tier TX — a shifting terminal coroutine", () => { + it("TX68: a Close that moves from a child to the root returns nothing", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + + let reads = 0; + const events: DurableEvent[] = []; + for (const event of complete.snapshot()) { + if (event.type === "yield" && event.description.name === "__root__") { + continue; + } + if (event.type === "close") { + const close: Record = { type: "close", result: event.result }; + Object.defineProperty(close, "coroutineId", { + enumerable: true, + get() { + reads += 1; + return reads === 1 ? "root.7" : "root"; + }, + }); + events.push(close as unknown as DurableEvent); + continue; + } + events.push(event); + } + + const appended: DurableEvent[] = []; + const stream: DurableStream = { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* useProbes(seen); + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream })); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + + // Never Alpha's retained output, and never a partial answer either. + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect((error as Error).message).not.toContain("alpha content"); + expect(seen.names).toEqual([]); + expect(appended).toEqual([]); + // The coroutine was asked once, so there was never a second answer. + expect(reads).toBe(1); + }); +}); diff --git a/packages/durable-streams/README.md b/packages/durable-streams/README.md index 1e88442a..13ab7ccb 100644 --- a/packages/durable-streams/README.md +++ b/packages/durable-streams/README.md @@ -401,7 +401,13 @@ Use guards for staleness policy. Do not use them to enforce an invariant that mu Every phase of a replay reads the same events, and a journal is data a backend supplies. Events are therefore **retained**: read once and detached from whatever the backend still owns. -A retained `Yield`'s **identity** — its event type, its coroutine, and its complete effect description — is read together and kept, so no phase can be shown a different event than the phase before it. What the event **settled to** stays lazy and separate, because the index is built before guards run and a guard that would refuse an event must get that chance before the stream is asked to produce a result. Both reads keep both outcomes: a refusal is remembered and re-raised rather than retried. +**Every** event that participates in admission, indexing, or terminal reuse is retained — `Close` as well as `Yield`. A `Close` decides whether a coroutine has a terminal result to reuse, so leaving it as the backend's own object lets it belong to a child coroutine while one phase asks and to the root while the next does. + +The **discriminator** is settled by the classification that chooses an event's retained kind, and never read from the source again. **Identity** — the coroutine an event belongs to, and a `Yield`'s complete effect description — is settled once too, so no phase can be shown a different event than the phase before it. An event that refuses to say what it is is refused from every member. + +A `Yield`'s **settlement** stays lazy and separate, because the index is built before guards run and a guard that would refuse an event must get that chance before the stream is asked to produce a result. A `Close` keeps its own cell, memoized the same way, so every later read receives the same detached answer. Every cell keeps both outcomes: a refusal is remembered and re-raised rather than retried. + +A retained event presents its members as ordinary own properties, so it spreads, serializes, and compares like the plain event a backend would have supplied. A detached result shares no object or array with the journal, and remains ordinary mutable JSON — detaching is a claim against the *stream*, not against the consumer, and replayed values are legitimately written to. diff --git a/packages/durable-streams/mod.ts b/packages/durable-streams/mod.ts index db43ec59..36511ba6 100644 --- a/packages/durable-streams/mod.ts +++ b/packages/durable-streams/mod.ts @@ -25,7 +25,10 @@ export type { // ReplayIndex export { ReplayIndex } from "./replay-index.ts"; -export { detachJson, retainEvents, RetainedYield } from "./retained.ts"; +// `retainEvents` is public because `@executablemd/core` owns its own journal +// gate and must produce the stable history across the package boundary. The +// retained classes and the detach helpers stay internal. +export { retainEvents } from "./retained.ts"; export type { YieldEntry } from "./replay-index.ts"; // Stream interface diff --git a/packages/durable-streams/retained.ts b/packages/durable-streams/retained.ts index 82f22129..73a7f102 100644 --- a/packages/durable-streams/retained.ts +++ b/packages/durable-streams/retained.ts @@ -8,15 +8,26 @@ * one thing for validation and another for execution, and nothing downstream * can detect the substitution. * - * A retained event is therefore read once and detached. Identity — the event - * type, the coroutine it belongs to, and its complete effect description — is - * read together and kept, so no phase can be shown a different event than the - * phase before it. What the event *settled to* stays lazy and separate: the - * index is built before guards run, and a guard that would refuse an event has - * to get that chance before the stream is asked to produce its result. + * A retained event is therefore read once and detached. **Every** event that + * participates in admission, indexing, or terminal reuse is retained, Close as + * well as Yield: a Close decides whether a coroutine has a terminal result to + * reuse, so leaving it as the backend's own object lets it belong to a child + * coroutine while one phase asks and to the root while the next does. + * + * The discriminator is settled once, by the classification that chooses a + * retained event's kind, and never read from the source again. Identity — the + * coroutine an event belongs to, and a Yield's complete effect description — is + * settled once too, so no phase can be shown a different event than the phase + * before it. + * + * A Yield's *settlement* stays lazy and separate: the index is built before + * guards run, and a guard that would refuse an event has to get that chance + * before the stream is asked to produce its result. A Close keeps its own cell, + * memoized the same way, so every later read receives the same detached answer. */ import type { + Close, CoroutineId, DurableEvent, EffectDescription, @@ -177,21 +188,23 @@ function detachDescription(description: EffectDescription): EffectDescription { /** Everything about a retained Yield except what it settled to. */ interface RetainedIdentity { - readonly type: "yield"; readonly coroutineId: CoroutineId; readonly description: EffectDescription; } -function readIdentity(source: Yield): RetainedIdentity { - const type = source.type; - if (type !== "yield") { - throw new TypeError("a retained Yield reports its own type"); - } +function readCoroutineId(source: { coroutineId: CoroutineId }): CoroutineId { const coroutineId = source.coroutineId; if (typeof coroutineId !== "string") { - throw new TypeError("a retained Yield belongs to a coroutine"); + throw new TypeError("a retained event belongs to a coroutine"); } - return { type, coroutineId, description: detachDescription(source.description) }; + return coroutineId; +} + +function readIdentity(source: Yield): RetainedIdentity { + return { + coroutineId: readCoroutineId(source), + description: detachDescription(source.description), + }; } /** @@ -208,35 +221,102 @@ function readIdentity(source: Yield): RetainedIdentity { * refusal is remembered and re-raised rather than retried, so a source cannot * refuse one phase and then answer the next. */ -export class RetainedYield implements Yield { - private source: Yield; - private identity: Settled | undefined; - private settled: Settled | undefined; +/** + * Present a retained member the way the event it stands for presents it. + * + * Own and enumerable, so a retained event spreads, serializes, and compares + * like the plain event a backend would have supplied. The settled cells stay + * genuinely private, which is what keeps them out of all of that. + */ +function present(target: object, name: string, read: () => T): void { + Object.defineProperty(target, name, { enumerable: true, get: read }); +} + +class RetainedYield implements Yield { + /** + * Settled by the classification that chose this wrapper, never re-read. A + * second read of the source's own discriminator is a second chance for it to + * answer differently, and an event that classifies as a Yield here and a + * Close there is the same substitution one level further out. + */ + declare readonly type: "yield"; + declare readonly coroutineId: CoroutineId; + declare readonly description: EffectDescription; + declare readonly result: Result; + #source: Yield; + #identity: Settled | undefined; + #settled: Settled | undefined; constructor(source: Yield) { - this.source = source; + this.#source = source; + present(this, "type", () => "yield" as const); + present(this, "coroutineId", () => this.#stable().coroutineId); + present(this, "description", () => this.#stable().description); + present(this, "result", () => { + this.#settled ??= settle(() => detachResult(this.#source.result)); + return resolve(this.#settled); + }); } - private stable(): RetainedIdentity { - this.identity ??= settle(() => readIdentity(this.source)); - return resolve(this.identity); + #stable(): RetainedIdentity { + this.#identity ??= settle(() => readIdentity(this.#source)); + return resolve(this.#identity); } +} - get type(): "yield" { - return this.stable().type; - } +/** + * One retained Close: settled identity, and one cell for its terminal result. + * + * A Close decides whether a coroutine has a terminal result to reuse, so it + * participates in admission exactly as a Yield does. Left as the backend's own + * object it could belong to a child coroutine while one phase asks and to the + * root while the next does — a history nobody could admit, reused as a result + * nobody asked for. + */ +class RetainedClose implements Close { + declare readonly type: "close"; + declare readonly coroutineId: CoroutineId; + declare readonly result: Result; + #source: Close; + #identity: Settled | undefined; + #settled: Settled | undefined; - get coroutineId(): CoroutineId { - return this.stable().coroutineId; + constructor(source: Close) { + this.#source = source; + present(this, "type", () => "close" as const); + present(this, "coroutineId", () => { + this.#identity ??= settle(() => readCoroutineId(this.#source)); + return resolve(this.#identity); + }); + present(this, "result", () => { + this.#settled ??= settle(() => detachResult(this.#source.result)); + return resolve(this.#settled); + }); } +} - get description(): EffectDescription { - return this.stable().description; - } +/** + * An event that would not say what it is. + * + * Classification is the one thing every later phase depends on, so an event + * that refuses it is not an event this history can describe. The refusal is + * remembered and re-raised from every member, rather than retried — a source + * that refuses one phase must not answer the next. + */ +class RetainedRefusal implements Yield { + declare readonly type: "yield"; + declare readonly coroutineId: CoroutineId; + declare readonly description: EffectDescription; + declare readonly result: Result; + #refusal: unknown; - get result(): Result { - this.settled ??= settle(() => detachResult(this.source.result)); - return resolve(this.settled); + constructor(refusal: unknown) { + this.#refusal = refusal; + for (const name of ["type", "coroutineId", "description", "result"]) { + present(this, name, (): never => { + throw this.#refusal; + }); + } } } @@ -252,9 +332,29 @@ export class RetainedYield implements Yield { */ export function retainEvents(events: readonly DurableEvent[]): DurableEvent[] { return events.map((event) => { - if (event instanceof RetainedYield) { + if (isRetained(event)) { return event; } - return event.type === "yield" ? new RetainedYield(event) : event; + // The one read of the source's discriminator. Whatever it says here is what + // the retained event reports from now on, to every phase. + const classified = settle(() => event.type); + if (classified.kind === "refusal") { + return new RetainedRefusal(classified.refusal); + } + if (classified.value === "yield") { + return new RetainedYield(event as Yield); + } + if (classified.value === "close") { + return new RetainedClose(event as Close); + } + return new RetainedRefusal(new TypeError("a retained event is a yield or a close")); }); } + +function isRetained(event: DurableEvent): boolean { + return ( + event instanceof RetainedYield || + event instanceof RetainedClose || + event instanceof RetainedRefusal + ); +} diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index d8dd57a6..180d40a5 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -324,12 +324,23 @@ consumer, and replayed values are legitimately mutable. A read that threw is remembered and re-raised rather than retried, so a source cannot refuse the guard and then answer replay. -A retained Yield's **identity** — its event type, its coroutine, and its -complete effect description — is read together and once, and kept. Reading those -members separately would let a source present an unrelated event to one phase -and a significant one to the next, which is the same substitution as answering -differently about a result, one level up. Identity and settlement are separate -cells; both keep both outcomes. +**Every** event that participates in admission, indexing, or terminal reuse is +retained — Close as well as Yield. A Close decides whether a coroutine has a +terminal result to reuse, so leaving it as the backend's own object lets it +belong to a child coroutine while one phase asks and to the root while the next +does: the phase that admits the history sees no terminal result, and the phase +that reuses one sees it. + +The discriminator is settled by the classification that chooses an event's +retained kind, and never read from the source again. Identity — the coroutine an +event belongs to, and a Yield's complete effect description — is settled once +too, so no phase can be shown a different event than the phase before it. An +event that refuses to say what it is is refused from every member. + +A Yield's settlement stays lazy, so a guard's check remains the first ordinary +consumer of a recorded result. A Close keeps its own cell, memoized the same +way. Every cell keeps both outcomes: a refusal is remembered and re-raised +rather than retried. The detached result is ordinary mutable JSON. Detaching is the claim against the stream; making the copy immutable would be a claim against the consumer, and @@ -361,11 +372,13 @@ class ReplayIndex { private closes = new Map(); constructor(events: DurableEvent[]) { - for (const event of events) { + // Retained first, idempotently: a caller that already produced the stable + // history hands the same objects on rather than a second wrapping of them. + for (const event of retainEvents(events)) { if (event.type === "yield") { const list = this.yields.get(event.coroutineId) ?? []; // Identity now; the result when a consumer asks. See below. - list.push(new RetainedYield(event)); + list.push(event); this.yields.set(event.coroutineId, list); } if (event.type === "close") { diff --git a/packages/durable-streams/tests/replay-index.test.ts b/packages/durable-streams/tests/replay-index.test.ts index e39e711c..b046d999 100644 --- a/packages/durable-streams/tests/replay-index.test.ts +++ b/packages/durable-streams/tests/replay-index.test.ts @@ -10,8 +10,7 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { ReplayIndex } from "../replay-index.ts"; -import { RetainedYield } from "../retained.ts"; -import type { DurableEvent, Json, Yield } from "../types.ts"; +import type { DurableEvent, Json } from "../types.ts"; function yieldEvent( coroutineId: string, @@ -384,152 +383,3 @@ describe("ReplayIndex — result access ordering", () => { expect(answers).toBe(1); }); }); - -/** - * A retained Yield has one identity, read once. - * - * Identity is what every phase uses to decide *which* event it is looking at: - * the private authority gate finds the root import by it, the index groups by - * it, guards read it, and replay matches on it. Read separately, a source can - * present an unrelated event to one phase and the root import to the next — - * which is the same substitution as answering differently about a result, one - * level up. - */ -describe("RetainedYield — one stable identity", () => { - /** A Yield whose identity members answer differently on each read. */ - function shifting(answers: { - type?: string[]; - coroutineId?: string[]; - name?: string[]; - extra?: Json[]; - }): { event: DurableEvent; reads: { count: number } } { - const reads = { count: 0 }; - // Each member counts its own reads, so one accessor's answers cannot be - // shifted by another's; `reads.count` totals them. - const answering = (list: T[] | undefined, fallback: T): (() => T) => { - const answers = list ?? [fallback]; - let taken = 0; - return () => { - const answer = answers[Math.min(taken, answers.length - 1)]!; - taken++; - reads.count++; - return answer; - }; - }; - const description: Record = {}; - Object.defineProperty(description, "type", { enumerable: true, value: "import_component" }); - Object.defineProperty(description, "name", { - enumerable: true, - get: answering(answers.name, "__root__"), - }); - Object.defineProperty(description, "marker", { - enumerable: true, - get: answering(answers.extra, "stable"), - }); - const event: Record = { result: { status: "ok", value: 1 } }; - Object.defineProperty(event, "type", { - enumerable: true, - get: answering(answers.type, "yield"), - }); - Object.defineProperty(event, "coroutineId", { - enumerable: true, - get: answering(answers.coroutineId, "root"), - }); - Object.defineProperty(event, "description", { enumerable: true, value: description }); - return { event: event as unknown as DurableEvent, reads }; - } - - it("a name that turns into __root__ is seen the same way by every phase", function* () { - const { event } = shifting({ name: ["unrelated", "__root__"] }); - const retained = new RetainedYield(event as Yield); - const answers = [ - retained.description.name, - retained.description.name, - retained.description.name, - ]; - expect(answers).toEqual(["unrelated", "unrelated", "unrelated"]); - }); - - it("a name that turns away from __root__ is seen the same way by every phase", function* () { - const { event } = shifting({ name: ["__root__", "unrelated"] }); - const retained = new RetainedYield(event as Yield); - expect(retained.description.name).toBe("__root__"); - expect(retained.description.name).toBe("__root__"); - }); - - it("a shifting coroutine id settles once", function* () { - const { event } = shifting({ coroutineId: ["root", "root.3"] }); - const retained = new RetainedYield(event as Yield); - expect(retained.coroutineId).toBe("root"); - expect(retained.coroutineId).toBe("root"); - }); - - it("a shifting event type settles once", function* () { - const { event } = shifting({ type: ["yield", "close"] }); - const retained = new RetainedYield(event as Yield); - expect(retained.type).toBe("yield"); - expect(retained.type).toBe("yield"); - }); - - it("a shifting nested description member settles once", function* () { - const { event } = shifting({ extra: ["stable", "swapped"] }); - const retained = new RetainedYield(event as Yield); - expect(retained.description["marker"]).toBe("stable"); - expect(retained.description["marker"]).toBe("stable"); - }); - - it("identity is read once for all three members together", function* () { - const { event, reads } = shifting({}); - const retained = new RetainedYield(event as Yield); - void retained.type; - void retained.coroutineId; - void retained.description; - void retained.description["marker"]; - // Four members across two objects, each read exactly once. - expect(reads.count).toBe(4); - }); - - it("an identity accessor that refuses is refused once, not retried", function* () { - const reads = { count: 0 }; - const event: Record = { - type: "yield", - description: { type: "call", name: "work" }, - result: { status: "ok" }, - }; - Object.defineProperty(event, "coroutineId", { - enumerable: true, - get() { - reads.count++; - throw new Error("the backend will not say which coroutine this is"); - }, - }); - const retained = new RetainedYield(event as unknown as Yield); - for (let attempt = 0; attempt < 3; attempt++) { - let caught: unknown; - try { - void retained.coroutineId; - } catch (error) { - caught = error; - } - expect((caught as Error | undefined)?.message).toBe( - "the backend will not say which coroutine this is", - ); - } - expect(reads.count).toBe(1); - }); - - it("the description is detached from the source", function* () { - const marker = { nested: "original" }; - const source = { - type: "yield", - coroutineId: "root", - description: { type: "call", name: "work", marker }, - result: { status: "ok" }, - }; - const retained = new RetainedYield(source as unknown as Yield); - const held = retained.description["marker"]; - expect(held).not.toBe(marker); - marker.nested = "rewritten"; - expect(held).toEqual({ nested: "original" }); - }); -}); diff --git a/packages/durable-streams/tests/retained.test.ts b/packages/durable-streams/tests/retained.test.ts new file mode 100644 index 00000000..c441c18d --- /dev/null +++ b/packages/durable-streams/tests/retained.test.ts @@ -0,0 +1,416 @@ +/** + * Retained history — one settled answer per event, for every phase. + * + * A journal is data a backend supplies, and every phase of a replay reads the + * same events: a consumer's own admission gate, the replay index, public guard + * policy, and the replay path. Where those are separate reads of the backend's + * objects, a source that answers differently between them decides one thing for + * validation and another for execution. + * + * Classification is the read everything else rests on: an event that is a Yield + * to one phase and a Close to the next cannot be reasoned about at all. These + * go through `retainEvents`, which is how a run actually obtains its history, + * rather than constructing wrappers directly. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { detachJson, retainEvents } from "../retained.ts"; +import { ReplayIndex } from "../replay-index.ts"; +import type { DurableEvent, Json } from "../types.ts"; + +/** An event whose members answer from a list, counting reads per member. */ +function shifting( + members: Record, + fixed: Record = {}, +): { event: DurableEvent; reads: Record } { + const reads: Record = {}; + const event: Record = { ...fixed }; + for (const [key, answers] of Object.entries(members)) { + reads[key] = 0; + Object.defineProperty(event, key, { + enumerable: true, + get() { + const index = Math.min(reads[key]!, answers.length - 1); + reads[key] = reads[key]! + 1; + return answers[index]; + }, + }); + } + return { event: event as unknown as DurableEvent, reads }; +} + +function retain(event: DurableEvent): DurableEvent { + return retainEvents([event])[0]!; +} + +describe("retained history — classification settles once", () => { + it("an event that turns from Yield into Close stays a Yield", function* () { + const { event, reads } = shifting( + { type: ["yield", "close"] }, + { + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok" }, + }, + ); + const retained = retain(event); + expect(retained.type).toBe("yield"); + expect(retained.type).toBe("yield"); + // Classified once; the wrapper never asks the source again. + expect(reads["type"]).toBe(1); + }); + + it("an event that turns from Close into Yield stays a Close", function* () { + const { event, reads } = shifting( + { type: ["close", "yield"] }, + { coroutineId: "root", result: { status: "ok", value: "kept" } }, + ); + const retained = retain(event); + expect(retained.type).toBe("close"); + expect(retained.type).toBe("close"); + expect(reads["type"]).toBe(1); + }); + + it("a discriminator that refuses is refused from every member, once", function* () { + let asked = 0; + const event: Record = {}; + Object.defineProperty(event, "type", { + enumerable: true, + get() { + asked++; + throw new Error("the backend will not say what this event is"); + }, + }); + const retained = retain(event as unknown as DurableEvent); + for (const read of [() => retained.type, () => retained.coroutineId, () => retained.result]) { + let caught: unknown; + try { + read(); + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not say what this event is", + ); + } + expect(asked).toBe(1); + }); + + it("an event that is neither is refused rather than passed through", function* () { + const retained = retain({ type: "something-else" } as unknown as DurableEvent); + let caught: unknown; + try { + void retained.type; + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(TypeError); + }); + + it("retaining an already-retained event returns it unchanged", function* () { + const once = retain({ + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok" }, + }); + expect(retainEvents([once])[0]).toBe(once); + }); +}); + +describe("retained history — a Close settles once", () => { + it("a coroutine that moves from a child to the root stays the child's", function* () { + const { event, reads } = shifting( + { coroutineId: ["root.7", "root"] }, + { type: "close", result: { status: "ok", value: "alpha" } }, + ); + const retained = retain(event); + expect(retained.coroutineId).toBe("root.7"); + expect(retained.coroutineId).toBe("root.7"); + expect(reads["coroutineId"]).toBe(1); + + // And the index — the phase that decides whether a terminal result exists — + // sees exactly what the first reader saw. + const index = new ReplayIndex([retained]); + expect(index.hasClose("root")).toBe(false); + expect(index.hasClose("root.7")).toBe(true); + }); + + it("a coroutine that moves from the root to a child stays the root's", function* () { + const { event } = shifting( + { coroutineId: ["root", "root.7"] }, + { type: "close", result: { status: "ok", value: "alpha" } }, + ); + const index = new ReplayIndex([retain(event)]); + expect(index.hasClose("root")).toBe(true); + expect(index.hasClose("root.7")).toBe(false); + }); + + it("a successful terminal result that changes is answered once", function* () { + const { event, reads } = shifting( + { + result: [ + { status: "ok", value: "first" }, + { status: "ok", value: "second" }, + ], + }, + { type: "close", coroutineId: "root" }, + ); + const retained = retain(event); + expect(retained.result).toEqual({ status: "ok", value: "first" }); + expect(retained.result).toEqual({ status: "ok", value: "first" }); + expect(reads["result"]).toBe(1); + }); + + it("a failed terminal result that changes is answered once", function* () { + const { event } = shifting( + { + result: [ + { status: "err", error: { message: "first" } }, + { status: "err", error: { message: "second" } }, + ], + }, + { type: "close", coroutineId: "root" }, + ); + const retained = retain(event); + expect(retained.result).toEqual({ status: "err", error: { message: "first" } }); + expect(retained.result).toEqual({ status: "err", error: { message: "first" } }); + }); + + it("a terminal result that refuses and then answers stays refused", function* () { + let asked = 0; + const event: Record = { type: "close", coroutineId: "root" }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + asked++; + if (asked === 1) { + throw new Error("the backend will not produce this result"); + } + return { status: "ok", value: "answered later" }; + }, + }); + const retained = retain(event as unknown as DurableEvent); + for (let attempt = 0; attempt < 3; attempt++) { + let caught: unknown; + try { + void retained.result; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not produce this result", + ); + } + expect(asked).toBe(1); + }); + + it("a terminal result is detached from the source", function* () { + const value = { list: ["a"] }; + const retained = retain({ + type: "close", + coroutineId: "root", + result: { status: "ok", value }, + } as unknown as DurableEvent); + const held = retained.result; + value.list.push("injected"); + expect(held).toEqual({ status: "ok", value: { list: ["a"] } }); + }); + + it("the control — an intact history classifies and settles normally", function* () { + const events: DurableEvent[] = [ + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository" } }, + }, + { type: "close", coroutineId: "root", result: { status: "ok", value: "done" } }, + ]; + const index = new ReplayIndex(retainEvents(events)); + expect(index.peekYield("root")?.description).toEqual({ + type: "import_component", + name: "__root__", + }); + expect(index.hasClose("root")).toBe(true); + expect(index.getClose("root")?.result).toEqual({ status: "ok", value: "done" }); + }); +}); + +describe("retained history — a Yield's identity settles once", () => { + const BASE = { result: { status: "ok", value: 1 } }; + + it("a name that turns into __root__ is seen the same way by every phase", function* () { + const { event } = shifting( + { name: ["unrelated", "__root__"] }, + { type: "yield", coroutineId: "root" }, + ); + const description = { type: "import_component" }; + Object.defineProperty(description, "name", { + enumerable: true, + get: Object.getOwnPropertyDescriptor(event, "name")!.get!, + }); + const retained = retain({ + type: "yield", + coroutineId: "root", + description, + ...BASE, + } as unknown as DurableEvent); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("unrelated"); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("unrelated"); + }); + + it("a name that turns away from __root__ keeps naming the root import", function* () { + const { event } = shifting( + { name: ["__root__", "unrelated"] }, + { type: "yield", coroutineId: "root" }, + ); + const description = { type: "import_component" }; + Object.defineProperty(description, "name", { + enumerable: true, + get: Object.getOwnPropertyDescriptor(event, "name")!.get!, + }); + const retained = retain({ + type: "yield", + coroutineId: "root", + description, + ...BASE, + } as unknown as DurableEvent); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("__root__"); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("__root__"); + }); + + it("a shifting coroutine id settles once", function* () { + const { event, reads } = shifting( + { coroutineId: ["root", "root.3"] }, + { type: "yield", description: { type: "call", name: "work" }, ...BASE }, + ); + const retained = retain(event); + expect(retained.coroutineId).toBe("root"); + expect(retained.coroutineId).toBe("root"); + expect(reads["coroutineId"]).toBe(1); + }); + + it("a shifting nested description member settles once", function* () { + const description: Record = { type: "call", name: "work" }; + let asked = 0; + Object.defineProperty(description, "marker", { + enumerable: true, + get() { + asked++; + return asked === 1 ? "stable" : "swapped"; + }, + }); + const retained = retain({ + type: "yield", + coroutineId: "root", + description, + ...BASE, + } as unknown as DurableEvent); + const read = () => (retained.type === "yield" ? retained.description["marker"] : undefined); + expect(read()).toBe("stable"); + expect(read()).toBe("stable"); + expect(asked).toBe(1); + }); + + it("an identity accessor that refuses is refused once, not retried", function* () { + let asked = 0; + const event: Record = { + type: "yield", + description: { type: "call", name: "work" }, + ...BASE, + }; + Object.defineProperty(event, "coroutineId", { + enumerable: true, + get() { + asked++; + throw new Error("the backend will not say which coroutine this is"); + }, + }); + const retained = retain(event as unknown as DurableEvent); + for (let attempt = 0; attempt < 3; attempt++) { + let caught: unknown; + try { + void retained.coroutineId; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not say which coroutine this is", + ); + } + expect(asked).toBe(1); + }); + + it("a Yield's settlement stays lazy, so a guard can refuse before it is read", function* () { + let asked = 0; + const event: Record = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + asked++; + return { status: "ok", value: 1 }; + }, + }); + const index = new ReplayIndex(retainEvents([event as unknown as DurableEvent])); + expect(index.peekYield("root")?.description.name).toBe("work"); + expect(asked).toBe(0); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: 1 }); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: 1 }); + expect(asked).toBe(1); + }); +}); + +describe("retained history — detached values stay ordinary JSON", () => { + it("`__proto__` is retained as an own data member", function* () { + const source: Record = {}; + Object.defineProperty(source, "__proto__", { + value: { polluted: true }, + enumerable: true, + writable: true, + configurable: true, + }); + const detached = detachJson(source as Json); + expect(Object.getPrototypeOf(detached)).toBe(Object.prototype); + expect(Object.getOwnPropertyNames(detached)).toEqual(["__proto__"]); + expect(({} as Record)["polluted"]).toBe(undefined); + }); + + it("nested objects and arrays detach from the journal's own", function* () { + const list = ["a"]; + const nested = { list }; + const detached = detachJson({ nested } as unknown as Json); + const held = (detached as Record)["nested"]; + expect(held).not.toBe(nested); + expect((held as Record)["list"]).not.toBe(list); + + list.push("injected"); + nested.list = ["replaced"]; + expect((held as Record)["list"]).toEqual(["a"]); + }); + + it("a detached member is writable and configurable", function* () { + const detached = detachJson({ count: 0, tags: ["a"] } as unknown as Json); + const held = detached as Record; + held["count"] = 1; + (held["tags"] as Json[]).push("b"); + expect(held).toEqual({ count: 1, tags: ["a", "b"] }); + }); + + it("a cycle is refused rather than followed", function* () { + const looped: Record = {}; + looped["self"] = looped; + let caught: unknown; + try { + detachJson(looped as Json); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(TypeError); + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 035f7a41..38e54ef3 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2770,9 +2770,12 @@ composable policy is for. The check therefore lives inside the journal the execution hands to `durableRun`: it reads the retained history once, owns the retained snapshot every later phase observes, validates the recorded selection and the required root-import structure against it, and passes that same snapshot -on. It runs ahead of public guard policy, of any retained Yield reaching -execution, of a retained terminal result being reused, of authored work, and of -any append. Public `ReplayGuard` policy remains composable and may short-circuit +on. It runs ahead of public guard policy, of any retained event reaching execution, +of a retained terminal result being reused, of authored work, and of any append. +The snapshot it owns covers **every** event that participates in admission, +indexing, or terminal reuse — a recorded completion as much as a recorded effect +— so a completion cannot belong to one coroutine while admission asks and to +another while the run reuses it. Public `ReplayGuard` policy remains composable and may short-circuit other public guards; it cannot suppress this. Reusing a recorded terminal result additionally requires exactly one From 0a1223afbdd94d43ba5a9bb0de6bce730b4a810b Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 08:47:38 -0400 Subject: [PATCH 11/14] =?UTF-8?q?=F0=9F=94=92=20Settle=20a=20retained=20te?= =?UTF-8?q?rminal=20result=20while=20the=20history=20is=20retained?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A retained Close detached its result only when the getter was first read. The execution-owned gate completed without touching it, so public ReplayGuard policy — code any enclosing scope may install — ran in the interval between admission and terminal reuse with the backend still owning the answer. Resuming an intact Alpha journal under a `check` handler that rewrote the raw Close result returned `"planted after admission"`, with nothing appended: a run admitted on one history and completed on another. A Close now settles both its coroutine and its result as the history is retained, before admission can complete, and admission recognizes the retained terminal result so a refusal becomes this run's fixed cause-free diagnostic rather than a failure surfacing later. Memoizing on first access closes repeated reads and leaves the window open, so it is not what this does. A Yield's settlement stays lazy, keeping public guard check its first ordinary consumer. --- packages/core/src/execute.ts | 8 + .../tests/document-target-execution.test.ts | 146 ++++++++++++++++++ packages/durable-streams/README.md | 6 +- packages/durable-streams/retained.ts | 28 ++-- .../specs/protocol-specification.md | 15 +- .../durable-streams/tests/retained.test.ts | 29 ++++ specs/executable-mdx-spec.md | 9 +- 7 files changed, 223 insertions(+), 18 deletions(-) diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 7bfb651d..24576bca 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -642,9 +642,17 @@ function admitRootHistory( // history holding one while the import that authorized it is absent // describes a run that never happened, whichever coroutine the Close // claims to belong to. + // + // Its result is recognized here as well as its coroutine. The retained + // history has already settled both, and forcing them now is what makes a + // refusal this run's fixed diagnostic rather than a failure surfacing + // later, out of some other phase's hands. if (attempt(() => event.coroutineId) === undefined) { throw new Error(UNREADABLE_ROOT_RECORD); } + if (attempt(() => event.result) === undefined) { + throw new Error(UNREADABLE_ROOT_RECORD); + } terminal = true; continue; } diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 7695f00f..586fe9b4 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -1712,3 +1712,149 @@ describe("Tier TX — a shifting terminal coroutine", () => { expect(reads).toBe(1); }); }); + +/** + * Tier TX — nothing the backend still owns survives admission. + * + * The private gate accepts a history; public `ReplayGuard` policy runs next; + * terminal reuse happens after that. If a Close still points at the backend's + * own result through all of it, those two later phases are a window in which + * the answer can be replaced — and public policy is code any enclosing scope + * may install. + * + * Recorded before the fix, resuming Alpha under a `check` handler that rewrites + * the raw Close result: + * + * ```json + * {"ok":true,"value":"planted after admission","appended":0} + * ``` + */ +describe("Tier TX — a terminal result cannot be replaced after admission", () => { + /** An intact Alpha journal whose Close carries a caller-owned result. */ + function* plantableJournal(): Operation<{ + events: DurableEvent[]; + planted: Record; + }> { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + + const events: DurableEvent[] = []; + let planted: Record = {}; + for (const event of complete.snapshot()) { + if (event.type === "close") { + const value = parseJson(event.result.status === "ok" ? (event.result.value ?? null) : null); + if (isJsonObject(value)) { + planted = value; + } + events.push({ + type: "close", + coroutineId: "root", + result: { status: "ok", value }, + } as DurableEvent); + continue; + } + events.push(event); + } + return { events, planted }; + } + + function reading(events: DurableEvent[], appended: DurableEvent[]): DurableStream { + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + } + + it("TX69: public policy cannot rewrite the terminal result it was shown", function* () { + const { events, planted } = yield* plantableJournal(); + const appended: DurableEvent[] = []; + const stream = reading(events, appended); + + const text = yield* scoped(function* () { + yield* ReplayGuard.around({ + *check([event], next) { + // Public policy, running between private admission and terminal + // reuse, rewriting what the backend still owns. + planted["output"] = "planted after admission"; + planted["value"] = "planted after admission"; + return yield* next(event); + }, + }); + return asText( + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream })), + ); + }); + + expect(text).toContain("alpha content"); + expect(text).not.toContain("planted after admission"); + expect(appended).toEqual([]); + }); + + it("TX70: a terminal result that refuses is the fixed diagnostic", function* () { + const { events } = yield* plantableJournal(); + let asked = 0; + const refusing = events.map((event) => { + if (event.type !== "close") { + return event; + } + const close: Record = { type: "close", coroutineId: "root" }; + Object.defineProperty(close, "result", { + enumerable: true, + get() { + asked += 1; + if (asked === 1) { + throw new Error("the backend will not produce this result"); + } + return { status: "ok", value: "answered later" }; + }, + }); + return close as unknown as DurableEvent; + }); + + const appended: DurableEvent[] = []; + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* useProbes(seen); + try { + yield* collect( + yield* execute({ + ...inlineSource(SECTIONS, { target: "Alpha" }), + stream: reading(refusing, appended), + }), + ); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect((error as Error).message).not.toContain("answered later"); + // Read once during retention, and never retried afterwards. + expect(asked).toBe(1); + expect(seen.names).toEqual([]); + expect(appended).toEqual([]); + }); + + it("TX71: the control — an intact terminal journal still replays", function* () { + const { events } = yield* plantableJournal(); + const appended: DurableEvent[] = []; + const text = asText( + yield* collect( + yield* execute({ + ...inlineSource(SECTIONS, { target: "Alpha" }), + stream: reading(events, appended), + }), + ), + ); + expect(text).toContain("alpha content"); + expect(appended).toEqual([]); + }); +}); diff --git a/packages/durable-streams/README.md b/packages/durable-streams/README.md index 13ab7ccb..5ba0b3f1 100644 --- a/packages/durable-streams/README.md +++ b/packages/durable-streams/README.md @@ -405,7 +405,11 @@ Every phase of a replay reads the same events, and a journal is data a backend s The **discriminator** is settled by the classification that chooses an event's retained kind, and never read from the source again. **Identity** — the coroutine an event belongs to, and a `Yield`'s complete effect description — is settled once too, so no phase can be shown a different event than the phase before it. An event that refuses to say what it is is refused from every member. -A `Yield`'s **settlement** stays lazy and separate, because the index is built before guards run and a guard that would refuse an event must get that chance before the stream is asked to produce a result. A `Close` keeps its own cell, memoized the same way, so every later read receives the same detached answer. Every cell keeps both outcomes: a refusal is remembered and re-raised rather than retried. +A `Yield`'s **settlement** stays lazy and separate, because the index is built before guards run and a guard that would refuse an event must get that chance before the stream is asked to produce a result. + +A `Close`'s result is settled **while the history is retained**, not at a first later read. A `Close` carries what a completed run hands back, and deferring that read leaves an interval — between the moment a consumer's own admission accepts the history and the moment terminal reuse consumes it — in which the backend still owns the answer and can replace it. Reading once at a later getter closes repeated reads and leaves that window open. + +Every cell keeps both outcomes: a refusal is remembered and re-raised rather than retried, so retaining a history never fails and a refusal reaches whichever phase asks. A retained event presents its members as ordinary own properties, so it spreads, serializes, and compares like the plain event a backend would have supplied. diff --git a/packages/durable-streams/retained.ts b/packages/durable-streams/retained.ts index 73a7f102..58dca7f8 100644 --- a/packages/durable-streams/retained.ts +++ b/packages/durable-streams/retained.ts @@ -277,21 +277,25 @@ class RetainedClose implements Close { declare readonly type: "close"; declare readonly coroutineId: CoroutineId; declare readonly result: Result; - #source: Close; - #identity: Settled | undefined; - #settled: Settled | undefined; + #identity: Settled; + #settled: Settled; constructor(source: Close) { - this.#source = source; + // Settled here, while the history is being retained, rather than at a first + // later read. A Close carries the result a completed run hands back, and + // deferring that read leaves an interval — between the moment a consumer's + // private admission accepts the history and the moment terminal reuse + // consumes it — in which the backend still owns the answer and can replace + // it. Reading once at a later getter closes repeated reads and leaves that + // window open. + // + // Settling cannot throw: a refusal is captured and re-raised from the + // getter, so retaining a history is never the thing that fails. + this.#identity = settle(() => readCoroutineId(source)); + this.#settled = settle(() => detachResult(source.result)); present(this, "type", () => "close" as const); - present(this, "coroutineId", () => { - this.#identity ??= settle(() => readCoroutineId(this.#source)); - return resolve(this.#identity); - }); - present(this, "result", () => { - this.#settled ??= settle(() => detachResult(this.#source.result)); - return resolve(this.#settled); - }); + present(this, "coroutineId", () => resolve(this.#identity)); + present(this, "result", () => resolve(this.#settled)); } } diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 180d40a5..38fb6f0d 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -338,9 +338,18 @@ too, so no phase can be shown a different event than the phase before it. An event that refuses to say what it is is refused from every member. A Yield's settlement stays lazy, so a guard's check remains the first ordinary -consumer of a recorded result. A Close keeps its own cell, memoized the same -way. Every cell keeps both outcomes: a refusal is remembered and re-raised -rather than retried. +consumer of a recorded result. + +A Close's result is settled while the history is retained. A Close carries what +a completed run hands back, and deferring that read leaves an interval — between +the moment a consumer's own admission accepts the history and the moment +terminal reuse consumes it — in which the backend still owns the answer and can +replace it. Memoizing on first access closes repeated reads and leaves that +window open. + +Every cell keeps both outcomes: a refusal is remembered and re-raised rather +than retried, so retaining a history never fails and a refusal reaches whichever +phase asks. The detached result is ordinary mutable JSON. Detaching is the claim against the stream; making the copy immutable would be a claim against the consumer, and diff --git a/packages/durable-streams/tests/retained.test.ts b/packages/durable-streams/tests/retained.test.ts index c441c18d..72f6251c 100644 --- a/packages/durable-streams/tests/retained.test.ts +++ b/packages/durable-streams/tests/retained.test.ts @@ -206,6 +206,26 @@ describe("retained history — a Close settles once", () => { expect(asked).toBe(1); }); + /** + * Detached while the history is retained, not at a first later read. + * + * Memoizing on first access closes repeated reads and leaves the interval + * between a consumer's admission and terminal reuse open: nobody has touched + * the getter yet, so the backend still owns the answer. + */ + it("a terminal result is detached before anyone reads it", function* () { + const value: Record = { output: "original" }; + const retained = retain({ + type: "close", + coroutineId: "root", + result: { status: "ok", value }, + } as unknown as DurableEvent); + + // Mutated before the retained result has ever been read. + value["output"] = "planted after retention"; + expect(retained.result).toEqual({ status: "ok", value: { output: "original" } }); + }); + it("a terminal result is detached from the source", function* () { const value = { list: ["a"] }; const retained = retain({ @@ -218,6 +238,15 @@ describe("retained history — a Close settles once", () => { expect(held).toEqual({ status: "ok", value: { list: ["a"] } }); }); + it("a cancelled terminal result is retained as itself", function* () { + const retained = retain({ + type: "close", + coroutineId: "root", + result: { status: "cancelled" }, + } as unknown as DurableEvent); + expect(retained.result).toEqual({ status: "cancelled" }); + }); + it("the control — an intact history classifies and settles normally", function* () { const events: DurableEvent[] = [ { diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 38e54ef3..36e0af8f 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2775,7 +2775,10 @@ of a retained terminal result being reused, of authored work, and of any append. The snapshot it owns covers **every** event that participates in admission, indexing, or terminal reuse — a recorded completion as much as a recorded effect — so a completion cannot belong to one coroutine while admission asks and to -another while the run reuses it. Public `ReplayGuard` policy remains composable and may short-circuit +another while the run reuses it, and cannot carry one result while admission +accepts it and another while the run consumes it. A recorded completion's result +is detached as the history is retained, and recognized during admission, so +nothing the backend still owns reaches a later phase. Public `ReplayGuard` policy remains composable and may short-circuit other public guards; it cannot suppress this. Reusing a recorded terminal result additionally requires exactly one @@ -7600,7 +7603,9 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX47–TX50 | Terminal history | Targeted and untargeted completed journals with the root import removed, and one with it duplicated, are refused before terminal reuse; an intact journal still replays | | TX51–TX56 | Identity authority | A completed or partial Alpha journal resumed as Beta is refused with an enclosing `check` handler that never delegates, with the equivalent `admit` handler, and with a same-name guard from another loaded copy; same-target replay and ordinary guard composition are the controls | | TX57–TX60 | Terminal binding | A root import on a child coroutine, a valid one plus a root-named child event, none at all, and two on the terminal coroutine each refuse before terminal reuse | -| TX61–TX63 | Detached but mutable | A replayed run updates a restored object exactly as a live one does; `__proto__` is an own data member; nested objects and arrays detach from the journal's own | +| TX61 | Detached but mutable | A partial replay restores a binding from the journal and the live continuation writes to it, matching the complete run exactly | +| TX68 | Shifting terminal coroutine | A Close that moves from a child to the root returns nothing: the fixed diagnostic, no retained output, nothing expanded, nothing appended, the coroutine asked once | +| TX69–TX71 | Post-admission replacement | Public guard policy rewriting the backend's own terminal result does not change what replay returns; a terminal result that refuses is the fixed diagnostic, read once, with nothing expanded and nothing appended; the intact control replays | | TX64–TX67 | Validation order | An unresolvable target outranks an invalid schema on inspection, live execution, and replay; a resolvable target lets the schema failure be reported; the control runs | | TX43 | One read across phases | Two valid recorded selections behind one accessor — Alpha then Beta — resume as Alpha: Alpha's section executes, Beta's never does, the source is read once, and the appended Close describes the Alpha execution | | TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | From d7ad191f8169968e9410bcd342d030c942d580fe Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 09:08:35 -0400 Subject: [PATCH 12/14] =?UTF-8?q?=F0=9F=94=92=20Let=20policy=20observe=20t?= =?UTF-8?q?he=20retained=20history=20without=20holding=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Public ReplayGuard check, admit and decide received the authoritative retained events, so policy — code any enclosing scope may install — could rewrite what the private gate had already validated and what replay was about to consume. A partial Alpha journal resumed as Alpha, with check rewriting the retained root-import target to Beta, executed the Beta projection. A retained effect named A, renamed to B by check, let a workflow asking for B consume A's result without B ever running. One history now has three views. The authoritative retained graph is what admission validated and replay consumes, detached from the backend and frozen through so nothing that receives it can rewrite a later decision. Guards receive an isolated deep mutable copy per invocation, so middleware composes as freely as before and nothing it writes flows back. What workflow code receives is a fresh mutable copy taken from the authority at consumption, so a resumed binding stays ordinary JSON its own continuation writes to. --- architecture.md | 9 +- .../tests/document-target-execution.test.ts | 186 ++++++++++++++++++ packages/durable-streams/README.md | 10 + packages/durable-streams/effect.ts | 24 ++- packages/durable-streams/retained.ts | 102 +++++++++- packages/durable-streams/run.ts | 14 +- .../specs/protocol-specification.md | 16 ++ .../tests/replay-guard.test.ts | 143 ++++++++++++++ specs/executable-mdx-spec.md | 10 +- 9 files changed, 497 insertions(+), 17 deletions(-) diff --git a/architecture.md b/architecture.md index ed58fe71..4d261be4 100644 --- a/architecture.md +++ b/architecture.md @@ -654,7 +654,14 @@ journal it replays through and validates the recorded selection inside the read itself, ahead of public guard policy, of a retained terminal result being reused, of authored work, and of any append. The retained history it owns covers every event that takes part in that decision, a recorded completion included, so -no event can present one identity to the validation and another to the run. Reusing a terminal result also +no event can present one identity to the validation and another to the run. + +Three views of that history stay distinct. The **authoritative** one is what +admission validated and replay consumes, and it is immutable to policy. +Replaceable public policy reads an **isolated observation** of it, so a handler +composes freely without acquiring the authority the gate exists to withhold. +What a document receives is a **fresh mutable copy**, because a resumed binding +is ordinary data its own continuation writes to. Reusing a terminal result also requires exactly one recognizable root import belonging to the coroutine whose result is being reused. diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 586fe9b4..9b218d7d 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -1858,3 +1858,189 @@ describe("Tier TX — a terminal result cannot be replaced after admission", () expect(appended).toEqual([]); }); }); + +/** + * Tier TX — public policy observes; it does not hold authority. + * + * The execution-owned gate validates a history and replay consumes it. Between + * those, public `ReplayGuard` policy runs — `check`, then `admit`, then `decide` + * during replay — and it is code any enclosing scope may install. Handing it the + * retained events themselves would let it rewrite the root selection, the + * recorded content, an effect description, or a result *after* admission had + * accepted them, which is exactly the authority the private gate exists to keep + * out of public hands. + * + * Recorded before the fix, resuming a partial Alpha journal as Alpha with + * `check` rewriting the retained root-import target to Beta: the run executed + * the Beta projection. + */ +describe("Tier TX — guards observe a copy, not the retained history", () => { + /** A partial Alpha journal: the recorded import, no terminal result. */ + function* partialAlpha(): Operation { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const partial = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "close") { + continue; + } + yield* partial.append(event); + } + return partial; + } + + /** Rewrite a root-import observation the way hostile policy would. */ + function rewriteRoot(event: Yield, change: (record: Record) => void): void { + if (event.description.name !== "__root__" || event.result.status !== "ok") { + return; + } + const record = event.result.value; + if (isJsonObject(record)) { + change(record as unknown as Record); + } + } + + /** Resume Alpha with `install` in scope; report what ran. */ + function* resume( + stream: InMemoryStream, + install: () => Operation, + ): Operation<{ seen: Probes; text: string | undefined; error: unknown; appended: number }> { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const outcome = yield* scoped(function* () { + yield* install(); + yield* useProbes(seen); + try { + return { + text: asText( + yield* collect( + yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream }), + ), + ), + error: undefined, + }; + } catch (error) { + return { text: undefined, error }; + } + }); + return { seen, ...outcome, appended: stream.snapshot().length - before }; + } + + /** Every root mismatch has the same shape of answer. */ + function expectAlphaOnly(outcome: { + seen: Probes; + text: string | undefined; + error: unknown; + }): void { + expect(outcome.seen.names).not.toContain("beta"); + expect(outcome.text ?? "").not.toContain("beta content"); + if (outcome.error === undefined) { + expect(outcome.text).toContain("alpha content"); + } + } + + it("TX72: check cannot rewrite the retained root target", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + rewriteRoot(event, (record) => { + record["target"] = "Beta"; + }); + return yield* next(event); + }, + }); + }); + expectAlphaOnly(outcome); + }); + + it("TX73: admit cannot rewrite the retained root target", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *admit([history], next) { + for (const event of history.yields) { + rewriteRoot(event, (record) => { + record["target"] = "Beta"; + }); + } + return yield* next(history); + }, + }); + }); + expectAlphaOnly(outcome); + }); + + it("TX74: decide cannot rewrite the retained root target", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + decide([event], next) { + rewriteRoot(event, (record) => { + record["target"] = "Beta"; + }); + return next(event); + }, + }); + }); + expectAlphaOnly(outcome); + }); + + it("TX75: the recorded content and failure record are equally out of reach", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + rewriteRoot(event, (record) => { + record["content"] = "# Rewritten\\n\\n## Beta\\n\\nrewritten beta\\n"; + record["kind"] = "target-failure"; + record["failure"] = { + kind: "no-match", + selector: "Alpha", + matches: [], + available: [], + }; + }); + return yield* next(event); + }, + }); + }); + expectAlphaOnly(outcome); + expect(outcome.text ?? "").not.toContain("rewritten beta"); + }); + + it("TX76: a completed journal refused for a mismatch appends nothing", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const outcome = yield* scoped(function* () { + const seen: Probes = { names: [], ids: [] }; + const before = complete.snapshot().length; + yield* useProbes(seen); + let error: unknown; + try { + yield* collect( + yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream: complete }), + ); + } catch (caught) { + error = caught; + } + return { seen, error, appended: complete.snapshot().length - before }; + }); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX77: downstream guard composition still reads its observation", function* () { + const observed: string[] = []; + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + // An annotation on the observation: composition still works, and + // nothing it writes reaches replay. + Object.assign(event.description, { annotated: true }); + observed.push(event.description.name); + return yield* next(event); + }, + }); + }); + expect(observed).toContain("__root__"); + expectAlphaOnly(outcome); + }); +}); diff --git a/packages/durable-streams/README.md b/packages/durable-streams/README.md index 5ba0b3f1..f4f6c21c 100644 --- a/packages/durable-streams/README.md +++ b/packages/durable-streams/README.md @@ -389,6 +389,16 @@ A replay guard has three stages, separated by a strict I/O boundary: The separation between generator and synchronous stages is necessary because the replay loop is synchronous. All observation-gathering must happen upfront. +### Three views of one history + +A replay distinguishes three things, and conflating any two of them hands authority to whoever holds the wrong one: + +1. **The authoritative retained history.** What a consumer's own admission validated and what replay consumes. Detached from the backend and immutable to policy — its descriptions and results are frozen through, so nothing that receives it can rewrite what replay will decide. +2. **Isolated guard observations.** What `check`, `admit`, and `decide` receive: a deep, mutable copy made per invocation. Middleware may read, annotate, and compose over it freely; nothing it writes reaches replay. +3. **Values delivered to workflow code.** A fresh mutable copy taken from the authority at the moment of consumption. A document that resumes on a restored binding writes to it, so replayed values stay ordinary JSON. + +Handing policy the authoritative events would let a guard rename effect A to B — so a workflow asking for B consumes A's result without B ever running — or rewrite a recorded root selection after admission accepted it. + ### Guards are policy, not authority A replay guard is **composable policy**. Guards compose through `Api.around`, and a handler installed further out may decline to call `next` — declining is what composition is for, and it means any single guard's opinion can be suppressed by another. diff --git a/packages/durable-streams/effect.ts b/packages/durable-streams/effect.ts index 6eb28571..d3feb8bb 100644 --- a/packages/durable-streams/effect.ts +++ b/packages/durable-streams/effect.ts @@ -36,6 +36,7 @@ import { type LiveDurableOperationCoordinator, } from "./live-coordinator.ts"; import { ReplayGuard } from "./replay-guard.ts"; +import { consumable, observeEvent } from "./retained.ts"; import { protocolToEffection, serializeError } from "./serialize.ts"; import type { CoroutineView, @@ -133,12 +134,24 @@ function checkReplay( // Description matches — now check replay guards before replaying. // ── REPLAY GUARD: Decide phase ── - const yieldEvent: Yield = { + // An isolated observation, like the check and admit phases: a decision is + // policy, and policy reads. Handing the retained description or result + // here would let a guard rewrite what replay is about to consume. + const observed = observeEvent({ type: "yield", coroutineId: ctx.coroutineId, description: entry.description, result: entry.result, - }; + }); + const yieldEvent: Yield = + observed.type === "yield" + ? observed + : { + type: "yield", + coroutineId: ctx.coroutineId, + description: desc, + result: entry.result, + }; const outcome = ReplayGuard.invoke(routine.scope, "decide", [yieldEvent]); if (outcome.outcome === "error") { @@ -157,8 +170,11 @@ function checkReplay( // All guards approved — consume the entry and advance cursor ctx.replayIndex.consumeYield(ctx.coroutineId); - // Feed stored result synchronously - resolve(protocolToEffection(entry.result)); + // Feed stored result synchronously, as a fresh mutable copy: the + // authoritative result stays frozen so policy cannot rewrite it, while a + // document that resumes on a restored binding still writes to what it + // receives. + resolve(protocolToEffection(consumable(entry.result))); return { path: "replayed", teardown: (exit) => exit(VOID_OK) }; } diff --git a/packages/durable-streams/retained.ts b/packages/durable-streams/retained.ts index 58dca7f8..959af6c7 100644 --- a/packages/durable-streams/retained.ts +++ b/packages/durable-streams/retained.ts @@ -109,15 +109,39 @@ export function detachJson(value: Json, seen: Set = new Set()): Json { } } +/** + * A detached copy, frozen through. + * + * The authoritative retained graph is what admission validated and what replay + * consumes, so nothing that reaches a caller may write to it. Freezing is the + * claim against *policy*, not against a workflow: what a document finally + * receives is a fresh mutable copy taken from this, never this. + */ +function sealJson(value: Json): Json { + const detached = detachJson(value); + freezeDeep(detached); + return detached; +} + +function freezeDeep(value: Json): void { + if (value === null || typeof value !== "object") { + return; + } + Object.freeze(value); + for (const member of Array.isArray(value) ? value : Object.values(value)) { + freezeDeep(member); + } +} + /** A detached copy of a retained failure's description. */ function detachError(error: SerializedError): SerializedError { const name = error.name; const stack = error.stack; - return { + return Object.freeze({ message: error.message, ...(name === undefined ? {} : { name }), ...(stack === undefined ? {} : { stack }), - }; + }); } /** @@ -134,15 +158,15 @@ function detachResult(result: Result): Result { return { status }; } const value = result.value; - return value === undefined ? { status } : { status, value: detachJson(value) }; + return Object.freeze(value === undefined ? { status } : { status, value: sealJson(value) }); } if (status === "err") { if (!("error" in result)) { throw new TypeError("a retained failure carries the error it failed with"); } - return { status, error: detachError(result.error) }; + return Object.freeze({ status, error: detachError(result.error) }); } - return { status }; + return Object.freeze({ status }); } /** @@ -177,13 +201,13 @@ function detachDescription(description: EffectDescription): EffectDescription { const detached: EffectDescription = { type, name }; for (const [key, member] of extra) { Object.defineProperty(detached, key, { - value: detachJson(member), + value: sealJson(member), enumerable: true, - writable: true, - configurable: true, + writable: false, + configurable: false, }); } - return detached; + return Object.freeze(detached); } /** Everything about a retained Yield except what it settled to. */ @@ -362,3 +386,63 @@ function isRetained(event: DurableEvent): boolean { event instanceof RetainedRefusal ); } + +/** + * An isolated observation of a retained event, for public policy to read. + * + * A replay guard is composable policy, and composition means handlers read, + * annotate, and pass along. What it must never mean is that a handler edits the + * history the execution already validated: the authoritative graph is what + * admission accepted and what replay consumes, and a guard that could rewrite a + * root selection or an effect description after admission would hold exactly + * the authority the private gate exists to keep out of public hands. + * + * So policy reads a copy. It is deep and mutable, so middleware may compose over + * it as freely as it likes, and nothing it does reaches replay. + */ +export function observeEvent(event: DurableEvent): DurableEvent { + if (event.type === "close") { + return { type: "close", coroutineId: event.coroutineId, result: consumable(event.result) }; + } + return { + type: "yield", + coroutineId: event.coroutineId, + description: observeDescription(event.description), + result: consumable(event.result), + }; +} + +function observeDescription(description: EffectDescription): EffectDescription { + const copy: EffectDescription = { type: description.type, name: description.name }; + for (const [key, member] of Object.entries(description)) { + if (key === "type" || key === "name") { + continue; + } + Object.defineProperty(copy, key, { + value: detachJson(member), + enumerable: true, + writable: true, + configurable: true, + }); + } + return copy; +} + +/** + * A retained result as a consumer may hold it: ordinary mutable JSON. + * + * The authoritative copy is frozen so policy cannot rewrite it. A document + * that resumes on a restored binding writes to it, so what a workflow receives + * is a fresh copy taken from that authority rather than the authority itself. + */ +export function consumable(result: Result): Result { + if (result.status === "ok") { + return "value" in result && result.value !== undefined + ? { status: "ok", value: detachJson(result.value) } + : { status: "ok" }; + } + if (result.status === "err") { + return { status: "err", error: { ...result.error } }; + } + return { status: "cancelled" }; +} diff --git a/packages/durable-streams/run.ts b/packages/durable-streams/run.ts index 26e1ea4b..976bee3a 100644 --- a/packages/durable-streams/run.ts +++ b/packages/durable-streams/run.ts @@ -19,6 +19,7 @@ import { DurableContext } from "./context.ts"; import { activeDurabilityFailure, appendDurableEvent } from "./durability.ts"; import { EarlyReturnDivergenceError, TerminalDivergenceError } from "./errors.ts"; import { ReplayGuard } from "./replay-guard.ts"; +import { observeEvent } from "./retained.ts"; import { ReplayIndex } from "./replay-index.ts"; import { deserializeError, serializeError } from "./serialize.ts"; import type { DurableStream } from "./stream.ts"; @@ -46,7 +47,13 @@ function unalignedReplay(replayIndex: ReplayIndex, coroutineId: string) { */ function* runCheckPhase(replayIndex: ReplayIndex, scope: Scope): Operation { for (const event of replayIndex.retainedYields()) { - yield* ReplayGuard.invoke(scope, "check", [event]); + // An isolated observation, not the retained event. Guards compose by + // reading and passing along; what composition must not become is the power + // to edit a history the execution already validated. + const observed = observeEvent(event); + if (observed.type === "yield") { + yield* ReplayGuard.invoke(scope, "check", [observed]); + } } } @@ -118,7 +125,10 @@ export function* durableRun( yield* ReplayGuard.invoke(scope, "admit", [ { coroutineId, - yields: replayIndex.retainedYields(), + yields: replayIndex.retainedYields().flatMap((event) => { + const observed = observeEvent(event); + return observed.type === "yield" ? [observed] : []; + }), terminal: replayIndex.hasClose(coroutineId), }, ]); diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 38fb6f0d..401bb5e7 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -362,6 +362,22 @@ event it validates is present, and present once — refuses there, because a per-event check has nothing to object to in a journal that simply omits the event. Its default is a no-op. +A replay distinguishes three views of one history, and conflating any two hands +authority to whoever holds the wrong one: + +1. the **authoritative retained history**, which admission validated and replay + consumes — detached from the backend and immutable to policy, its + descriptions and results frozen through; +2. **isolated guard observations**, a deep mutable copy made per guard + invocation, over which middleware composes freely and from which nothing + flows back; and +3. **values delivered to workflow code**, a fresh mutable copy taken from the + authority at consumption, so replayed values stay ordinary JSON. + +Handing policy the authoritative events would let a guard rename effect A to B — +so a workflow asking for B consumes A's result without B ever running — or +rewrite a recorded root selection after admission accepted it. + Replay guards are **composable policy, not authority**. Guards compose through middleware, and a handler installed further out may decline to delegate — which is what composition is for. An invariant that must not be negotiable therefore diff --git a/packages/durable-streams/tests/replay-guard.test.ts b/packages/durable-streams/tests/replay-guard.test.ts index 85616296..0ae603d8 100644 --- a/packages/durable-streams/tests/replay-guard.test.ts +++ b/packages/durable-streams/tests/replay-guard.test.ts @@ -22,6 +22,7 @@ import { type Json, ReplayGuard, ReplayIndex, + retainEvents, type ReplayOutcome, StaleInputError, type Workflow, @@ -850,3 +851,145 @@ describe("durableRun — a retained result detaches completely", () => { expect(member(member(value, "nested"), "list")).toEqual(["a"]); }); }); + +/** + * Guard policy cannot rewrite the identity or result replay consumes. + * + * A replay guard is composable policy. Composition means reading, annotating, + * and passing along; it must not mean editing the history the execution already + * validated. Given the retained events themselves, a guard could rename effect + * A to B and have a workflow asking for B consume A's result, without B ever + * running. + */ +describe("replay guard — observation is isolated from replay authority", () => { + function journal(): DurableEvent[] { + return [ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "A" }, + result: { status: "ok", value: "A-result" }, + }, + ]; + } + + function reading(events: DurableEvent[], appended: DurableEvent[]): DurableStream { + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + } + + /** Run a workflow asking for `name`, with `install` in scope. */ + function* asking( + name: string, + events: DurableEvent[], + install: () => Operation, + ): Operation<{ value: unknown; error: unknown; ran: boolean; appended: DurableEvent[] }> { + const appended: DurableEvent[] = []; + let ran = false; + const outcome = yield* scoped(function* () { + yield* install(); + try { + const value = yield* durableRun( + function* () { + return (yield createDurableOperation({ type: "call", name }, function* () { + ran = true; + return `${name}-live`; + })) as Json; + }, + { stream: reading(events, appended) }, + ); + return { value, error: undefined }; + } catch (error) { + return { value: undefined, error }; + } + }); + return { ...outcome, ran, appended }; + } + + it("check cannot rename A to B and feed B the A result", function* () { + const outcome = yield* asking("B", journal(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + Object.assign(event.description, { name: "B" }); + return yield* next(event); + }, + }); + }); + // B never received A's result: either B ran live, or the run failed. + expect(outcome.value).not.toBe("A-result"); + if (outcome.error === undefined) { + expect(outcome.ran).toBe(true); + expect(outcome.value).toBe("B-live"); + } + }); + + it("admit cannot rename A to B and feed B the A result", function* () { + const outcome = yield* asking("B", journal(), function* () { + yield* ReplayGuard.around({ + *admit([history], next) { + for (const event of history.yields) { + Object.assign(event.description, { name: "B" }); + } + return yield* next(history); + }, + }); + }); + expect(outcome.value).not.toBe("A-result"); + }); + + it("decide cannot rewrite the result replay is about to feed", function* () { + const outcome = yield* asking("A", journal(), function* () { + yield* ReplayGuard.around({ + decide([event], next) { + if (event.result.status === "ok") { + Object.assign(event.result, { value: "rewritten-by-policy" }); + } + return next(event); + }, + }); + }); + expect(outcome.value).toBe("A-result"); + expect(outcome.ran).toBe(false); + }); + + it("a public ReplayIndex observation cannot change what replay decides", function* () { + const events = journal(); + const index = new ReplayIndex(retainEvents(events)); + const entry = index.peekYield("root"); + expect(entry).toBeDefined(); + + // Whatever a caller does with what the index hands back, the retained + // answer is unchanged. + try { + Object.assign(entry!.description, { name: "B" }); + } catch { + // A frozen description refuses the write outright, which is the same + // conclusion reached more directly. + } + expect(index.peekYield("root")?.description.name).toBe("A"); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: "A-result" }); + }); + + it("the control — an unmodified guard still replays the recorded result", function* () { + const seen: string[] = []; + const outcome = yield* asking("A", journal(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + seen.push(event.description.name); + return yield* next(event); + }, + }); + }); + expect(seen).toEqual(["A"]); + expect(outcome.value).toBe("A-result"); + expect(outcome.ran).toBe(false); + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 36e0af8f..b0173333 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2778,7 +2778,15 @@ indexing, or terminal reuse — a recorded completion as much as a recorded effe another while the run reuses it, and cannot carry one result while admission accepts it and another while the run consumes it. A recorded completion's result is detached as the history is retained, and recognized during admission, so -nothing the backend still owns reaches a later phase. Public `ReplayGuard` policy remains composable and may short-circuit +nothing the backend still owns reaches a later phase. + +That snapshot is also immutable to public policy. Guard `check`, `admit`, and +`decide` receive an isolated copy rather than the retained events, so a handler +cannot rewrite a recorded target, the recorded content, a selection failure, an +effect description, or a result after admission accepted it. What a document +finally receives is a third thing again: a fresh mutable copy taken from the +authority at consumption, so a resumed binding is still ordinary JSON its own +continuation writes to. Public `ReplayGuard` policy remains composable and may short-circuit other public guards; it cannot suppress this. Reusing a recorded terminal result additionally requires exactly one From 0eab238615c2cb3a96c637c1792776414ec8f7b0 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 09:24:41 -0400 Subject: [PATCH 13/14] =?UTF-8?q?=F0=9F=94=92=20Apply=20the=20authority=20?= =?UTF-8?q?boundary=20to=20every=20Result=20path?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two paths were missed. `durableRun` returned a completed Close's value straight from the frozen authority, so a caller of a finished run received a frozen object where a live run had given ordinary JSON and mutation threw. And a successful settlement carrying no value was left writable, so a caller of a public `ReplayIndex` observation could add one before replay read it. Every retained settlement is now frozen through — a success with a value, a `Result` with none, a failure, and a cancellation, on Yield and Close alike — and a completed run hands back a fresh consumer copy exactly as a replayed effect does. Mutating what one completed replay returned cannot reach the authority or the next replay, and completed replay still runs nothing and appends nothing. The specification and README now say plainly that "mutable replayed value" means the fresh consumer copy, and that the retained Result is frozen through. --- packages/durable-streams/README.md | 4 +- packages/durable-streams/retained.ts | 6 +- packages/durable-streams/run.ts | 9 +- .../specs/protocol-specification.md | 10 +- .../tests/replay-guard.test.ts | 133 ++++++++++++++++++ 5 files changed, 157 insertions(+), 5 deletions(-) diff --git a/packages/durable-streams/README.md b/packages/durable-streams/README.md index f4f6c21c..9260205a 100644 --- a/packages/durable-streams/README.md +++ b/packages/durable-streams/README.md @@ -395,7 +395,9 @@ A replay distinguishes three things, and conflating any two of them hands author 1. **The authoritative retained history.** What a consumer's own admission validated and what replay consumes. Detached from the backend and immutable to policy — its descriptions and results are frozen through, so nothing that receives it can rewrite what replay will decide. 2. **Isolated guard observations.** What `check`, `admit`, and `decide` receive: a deep, mutable copy made per invocation. Middleware may read, annotate, and compose over it freely; nothing it writes reaches replay. -3. **Values delivered to workflow code.** A fresh mutable copy taken from the authority at the moment of consumption. A document that resumes on a restored binding writes to it, so replayed values stay ordinary JSON. +3. **Values delivered to workflow code.** A fresh mutable copy taken from the authority at the moment of consumption — for a replayed effect and for a completed run's own return value alike. This is the only thing "mutable replayed value" ever means: a document that resumes on a restored binding writes to its copy, and writing to it cannot reach the authority or the next replay. + +The authority is frozen through in **every** settlement shape — a success with a value, a `Result` with none, a failure, and a cancellation, on `Yield` and `Close` alike. An envelope left writable is one a public observation could add a value to before replay reads it. Handing policy the authoritative events would let a guard rename effect A to B — so a workflow asking for B consumes A's result without B ever running — or rewrite a recorded root selection after admission accepted it. diff --git a/packages/durable-streams/retained.ts b/packages/durable-streams/retained.ts index 959af6c7..38022446 100644 --- a/packages/durable-streams/retained.ts +++ b/packages/durable-streams/retained.ts @@ -154,8 +154,12 @@ function detachError(error: SerializedError): SerializedError { function detachResult(result: Result): Result { const status = result.status; if (status === "ok") { + // Every successful shape, including the one that settled to nothing. A + // `Result` carries no value to detach, but the envelope is still + // authority — left writable, a caller of a public observation could add one + // before replay reads it. if (!("value" in result)) { - return { status }; + return Object.freeze({ status }); } const value = result.value; return Object.freeze(value === undefined ? { status } : { status, value: sealJson(value) }); diff --git a/packages/durable-streams/run.ts b/packages/durable-streams/run.ts index 976bee3a..31a55b88 100644 --- a/packages/durable-streams/run.ts +++ b/packages/durable-streams/run.ts @@ -19,7 +19,7 @@ import { DurableContext } from "./context.ts"; import { activeDurabilityFailure, appendDurableEvent } from "./durability.ts"; import { EarlyReturnDivergenceError, TerminalDivergenceError } from "./errors.ts"; import { ReplayGuard } from "./replay-guard.ts"; -import { observeEvent } from "./retained.ts"; +import { consumable, observeEvent } from "./retained.ts"; import { ReplayIndex } from "./replay-index.ts"; import { deserializeError, serializeError } from "./serialize.ts"; import type { DurableStream } from "./stream.ts"; @@ -139,7 +139,12 @@ export function* durableRun( if (replayIndex.hasClose(coroutineId)) { const closeEvent = replayIndex.getClose(coroutineId)!; if (closeEvent.result.status === "ok") { - return closeEvent.result.value as T; + // A fresh consumer copy, exactly as a replayed Yield's result is. The + // retained settlement is frozen so policy cannot rewrite it; what a + // caller receives from a completed run is ordinary data it may hold and + // change, and changing it cannot reach the next replay. + const settled = consumable(closeEvent.result); + return (settled.status === "ok" ? settled.value : undefined) as T; } else if (closeEvent.result.status === "err") { throw deserializeError(closeEvent.result.error); } else { diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 401bb5e7..d397864b 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -372,7 +372,15 @@ authority to whoever holds the wrong one: invocation, over which middleware composes freely and from which nothing flows back; and 3. **values delivered to workflow code**, a fresh mutable copy taken from the - authority at consumption, so replayed values stay ordinary JSON. + authority at consumption — for a replayed effect and for a completed run's + own return value alike. This is the only thing "mutable replayed value" + means: writing to that copy reaches neither the authority nor the next + replay. + +The authority is frozen through in every settlement shape: a success with a +value, a `Result` with none, a failure, and a cancellation, on Yield and +Close alike. An envelope left writable is one a public observation could add a +value to before replay reads it. Handing policy the authoritative events would let a guard rename effect A to B — so a workflow asking for B consumes A's result without B ever running — or diff --git a/packages/durable-streams/tests/replay-guard.test.ts b/packages/durable-streams/tests/replay-guard.test.ts index 0ae603d8..9a18fed6 100644 --- a/packages/durable-streams/tests/replay-guard.test.ts +++ b/packages/durable-streams/tests/replay-guard.test.ts @@ -22,6 +22,7 @@ import { type Json, ReplayGuard, ReplayIndex, + type Result, retainEvents, type ReplayOutcome, StaleInputError, @@ -993,3 +994,135 @@ describe("replay guard — observation is isolated from replay authority", () => expect(outcome.ran).toBe(false); }); }); + +/** + * Every Result shape is authority; every consumer value is a copy. + * + * The distinction has to hold for all four settlements, not only the one that + * carries data. A `Result` left writable is an envelope a public + * observation could add a value to before replay reads it, and a completed run + * that hands back the retained value itself gives a caller a frozen object + * where the live run gave ordinary JSON. + */ +describe("replay authority — every settlement, and its consumer copy", () => { + function completed(value: Json): DurableEvent[] { + return [{ type: "close", coroutineId: "root", result: { status: "ok", value } }]; + } + + function reading(events: DurableEvent[], appended: DurableEvent[]): DurableStream { + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + } + + function* replayCompleted( + events: DurableEvent[], + appended: DurableEvent[], + ran: { live: boolean }, + ): Operation { + return yield* durableRun( + function* () { + ran.live = true; + return "live" as Json; + }, + { stream: reading(events, appended) }, + ); + } + + it("a completed run hands back ordinary mutable JSON", function* () { + const appended: DurableEvent[] = []; + const ran = { live: false }; + const events = completed({ count: 1, tags: ["a"], nested: { deep: true } }); + const value = yield* replayCompleted(events, appended, ran); + + const held = value as Record; + held["count"] = 2; + (held["tags"] as Json[]).push("b"); + (held["nested"] as Record)["deep"] = false; + expect(held).toEqual({ count: 2, tags: ["a", "b"], nested: { deep: false } }); + + // Completed replay ran nothing and wrote nothing. + expect(ran.live).toBe(false); + expect(appended).toEqual([]); + }); + + it("mutating a completed value cannot change the next replay", function* () { + const events = completed({ count: 1 }); + const first = yield* replayCompleted(events, [], { live: false }); + (first as Record)["count"] = 99; + + const second = yield* replayCompleted(events, [], { live: false }); + expect(second).toEqual({ count: 1 }); + }); + + it("a retained ok-without-value settlement is authority", function* () { + const index = new ReplayIndex( + retainEvents([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok" }, + }, + ]), + ); + const settled = index.peekYield("root")?.result; + expect(settled).toEqual({ status: "ok" }); + try { + Object.assign(settled as object, { value: "planted" }); + } catch { + // A frozen settlement refuses outright, which is the same conclusion. + } + expect(index.peekYield("root")?.result).toEqual({ status: "ok" }); + }); + + it("a retained void Close settlement is authority", function* () { + const index = new ReplayIndex( + retainEvents([{ type: "close", coroutineId: "root", result: { status: "ok" } }]), + ); + const settled = index.getClose("root")?.result; + try { + Object.assign(settled as object, { value: "planted" }); + } catch { + // As above. + } + expect(index.getClose("root")?.result).toEqual({ status: "ok" }); + }); + + it("ok(value), ok(void), err and cancelled are all frozen through", function* () { + const results: Result[] = [ + { status: "ok", value: { nested: ["a"] } }, + { status: "ok" }, + { status: "err", error: { message: "failed" } }, + { status: "cancelled" }, + ]; + for (const result of results) { + const index = new ReplayIndex( + retainEvents([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result, + }, + ]), + ); + const settled = index.peekYield("root")!.result; + expect(Object.isFrozen(settled)).toBe(true); + if (settled.status === "ok" && "value" in settled && settled.value !== undefined) { + expect(Object.isFrozen(settled.value)).toBe(true); + expect(Object.isFrozen(member(settled.value, "nested"))).toBe(true); + } + if (settled.status === "err") { + expect(Object.isFrozen(settled.error)).toBe(true); + } + } + }); +}); From d8cd0492b58650bad3525df56203c7eb7fd748a0 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 09:52:12 -0400 Subject: [PATCH 14/14] =?UTF-8?q?=F0=9F=94=92=20Carry=20the=20journal=20wi?= =?UTF-8?q?tness=20across=20the=20admission=20wrapper?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Journal provenance is not transitive: a wrapper is unproven unless its wrapping site carries the source's witness onto it. Core puts two wrappers on the journal a host supplies — the secret filter, which preserves explicitly, and the execution-owned target-admission gate, which did not. A live coordinator therefore received no provenance, and a Workspace provider refused the operation before any transaction. The gate is a trusted wrapping site: core installs it before any document code exists and it delegates every append to the exact stream it was handed. It now transfers the witness that exact source already has, and nothing more — it establishes none, so an unproven journal stays unproven, a generic guard or a custom wrapper gains nothing, and a separately loaded copy cannot transfer the canonical witness. The workflow specification, architecture.md, the durable-stream README and the executable-MDX prose named the secret filter as the one trusted site; the admission wrapper is now named as the second. --- architecture.md | 7 +- packages/core/src/execute.ts | 13 +- .../tests/document-target-execution.test.ts | 167 +++++++++++++++++- packages/durable-streams/README.md | 7 + specs/executable-mdx-spec.md | 10 +- specs/workflow-spec.md | 6 +- 6 files changed, 203 insertions(+), 7 deletions(-) diff --git a/architecture.md b/architecture.md index 4d261be4..0aa7174b 100644 --- a/architecture.md +++ b/architecture.md @@ -1019,7 +1019,12 @@ It qualifies only while every one of these holds: it; - establishment and transfer occur only through the canonical module's `establishJournalProvenance()` and `preserveJournalProvenance()`, one fresh - witness per stream, with duplicate establishment refused; and + witness per stream, with duplicate establishment refused; +- transfer happens only at a trusted wrapping site — one installed before any + code the journal's content could influence, delegating to the exact stream it + was handed. A document execution's journal passes through two: the secret + filter and the execution-owned target-admission wrapper. Each transfers only + what its source already had, so an unproven journal stays unproven; and - it retains no execution, lifecycle, journal content or provider state. The exception exists because the exact-object, anti-forgery and loaded-copy diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 24576bca..c8fd4a83 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -17,6 +17,7 @@ import { durableRun, createDurableOperation, ephemeral, + preserveJournalProvenance, retainEvents, StaleInputError, type CoroutineId, @@ -597,13 +598,22 @@ function describeSelection(selection: SelectionOutcome): string { * It also owns the retained snapshot. The events it validates are the events it * returns, so the identity and settlement it decided on are what every later * phase observes rather than a second reading of the backend's own objects. + * + * It is a trusted wrapping site, and says so explicitly. Journal provenance is + * not transitive: a wrapper is unproven unless a wrapping site carries its + * source's witness onto it, and a run whose journal is unproven is refused by a + * Workspace provider before any transaction. This wrapper qualifies because + * core installs it before any document code exists and it delegates every + * append to the exact stream it was handed. What it transfers is only the + * witness that exact source already has — it establishes none, so an unproven + * source stays unproven and a wrapper somebody else built gains nothing. */ function guardedJournal( stream: DurableStream, root: RootDocumentSource, coroutineId: CoroutineId, ): DurableStream { - return { + const admitting: DurableStream = { *readAll(): Operation { const retained = retainEvents(yield* stream.readAll()); admitRootHistory(retained, root, coroutineId); @@ -611,6 +621,7 @@ function guardedJournal( }, append: (event: DurableEvent) => stream.append(event), }; + return preserveJournalProvenance(stream, admitting); } /** diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts index 9b218d7d..32d7d31a 100644 --- a/packages/core/tests/document-target-execution.test.ts +++ b/packages/core/tests/document-target-execution.test.ts @@ -22,14 +22,24 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { InMemoryStream } from "@executablemd/durable-streams"; import { StaleInputError } from "@executablemd/durable-streams"; -import { ReplayGuard } from "@executablemd/durable-streams"; +import { + createDurableOperation, + defaultLiveDurableOperationCoordinator, + establishJournalProvenance, + guardDurableStream, + ReplayGuard, +} from "@executablemd/durable-streams"; +import type { + JournalProvenance, + LiveDurableOperationCoordinator, +} from "@executablemd/durable-streams"; import type { DurableEvent, DurableStream, Yield } from "@executablemd/durable-streams"; import { createApi } from "@effectionx/context-api"; import { API, useHostFiles } from "@executablemd/runtime"; import { collect } from "../src/collect.ts"; import { useTempFileCompiler } from "../src/temp-file-compiler.ts"; -import { execute } from "../src/execute.ts"; +import { execute, Execution } from "../src/execute.ts"; import { inspectDocument } from "../src/inspect.ts"; import { getExpansion } from "../src/expansion.ts"; import { registerComponents } from "../src/components/registration.ts"; @@ -2044,3 +2054,156 @@ describe("Tier TX — guards observe a copy, not the retained history", () => { expectAlphaOnly(outcome); }); }); + +/** + * Tier TX — the witness survives every wrapper a run puts on its journal. + * + * Journal provenance is deliberately non-transitive (#425): a wrapper is + * unproven unless a trusted wrapping site carries its source's witness onto it, + * and a run whose journal is unproven is refused by a Workspace provider before + * any transaction. Core puts two wrappers on the journal a host supplies — the + * secret filter, and the execution-owned target-admission gate — so both have + * to be explicit about it or a live coordinator receives nothing. + * + * Recorded before the fix, at the exact head: + * + * ```json + * {"filtered":true,"identityGate":false,"identityGateMissing":true} + * ``` + * + * These exercise `execute()` rather than the wrapper in isolation, because what + * is being measured is what reaches live coordination after every wrapper. + */ +describe("Tier TX — journal provenance across the admission gate", () => { + /** What a live durable operation's coordinator was handed. */ + interface Coordinated { + witnesses: (JournalProvenance | undefined)[]; + } + + /** + * A coordinator that records the provenance it is handed. + * + * Installed on a durable operation raised from inside a real document + * execution, so what it sees is what core's journal — after secret filtering + * and target admission — actually delivers to live coordination. + */ + function witnessing(seen: Coordinated): LiveDurableOperationCoordinator { + return { + *run(execute, publish, activateFailure, journalProvenance) { + seen.witnesses.push(journalProvenance); + return yield* defaultLiveDurableOperationCoordinator.run( + execute, + publish, + activateFailure, + journalProvenance, + ); + }, + }; + } + + /** Raise one coordinated durable operation inside the document's own run. */ + function* useWitnessProbe(seen: Coordinated): Operation { + yield* Execution.around({ + *document([props], next) { + yield createDurableOperation( + { type: "probe", name: "provenance" }, + // deno-lint-ignore require-yield + function* () { + return "probed"; + }, + { coordinator: witnessing(seen) }, + ); + return yield* next(props); + }, + }); + } + + const LIVE = [ + "# Title", + "", + "## Alpha", + "", + "alpha content", + "", + "## Beta", + "", + "beta content", + "", + ].join("\n"); + + /** Run `source` against `stream`, reporting what coordination witnessed. */ + function* observing( + stream: DurableStream, + source: RootDocumentSource, + settings: { secretDetection?: boolean } = {}, + ): Operation { + const seen: Coordinated = { witnesses: [] }; + yield* scoped(function* () { + yield* useWitnessProbe(seen); + yield* collect(yield* execute({ ...source, stream, ...settings })); + }); + return seen; + } + + it("TX78: an untargeted run delivers the selected journal's exact witness", function* () { + const stream = new InMemoryStream(); + const witness = establishJournalProvenance(stream); + const seen = yield* observing(stream, inlineSource(LIVE)); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(witness); + } + }); + + it("TX79: a targeted run delivers the same exact witness", function* () { + const stream = new InMemoryStream(); + const witness = establishJournalProvenance(stream); + const seen = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(witness); + } + }); + + it("TX80: secret detection disabled delivers the same exact witness", function* () { + const stream = new InMemoryStream(); + const witness = establishJournalProvenance(stream); + const seen = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" }), { + secretDetection: false, + }); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(witness); + } + }); + + it("TX81: an unproven journal stays unproven through both wrappers", function* () { + const stream = new InMemoryStream(); + const seen = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(undefined); + } + }); + + it("TX82: an ordinary wrapper of a proven journal is not promoted", function* () { + const backend = new InMemoryStream(); + establishJournalProvenance(backend); + // A wrapper nobody trusted: generic guarding, not a wrapping site. + const ordinary = guardDurableStream(backend, function* () {}); + const seen = yield* observing(ordinary, inlineSource(LIVE, { target: "Alpha" })); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(undefined); + } + }); + + it("TX83: replay reaches no live coordination at all", function* () { + const stream = new InMemoryStream(); + establishJournalProvenance(stream); + yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + + const replayed = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + expect(replayed.witnesses).toEqual([]); + }); +}); diff --git a/packages/durable-streams/README.md b/packages/durable-streams/README.md index 9260205a..28364d85 100644 --- a/packages/durable-streams/README.md +++ b/packages/durable-streams/README.md @@ -746,6 +746,13 @@ transfers only the witness already associated with the exact source: an unproven source leaves the target unproven, and nesting trusted wrappers carries the same witness through each one. +A wrapping site is trusted when it is installed before any code the journal's +own content could influence, and delegates to the exact stream it was handed. +`@executablemd/core` has two, and a document execution's journal passes through +both: the secret filter, and the execution-owned wrapper that admits a run's +recorded target before replay. Each transfers only what its source already had, +so an unproven journal stays unproven through both. + --- ## Long-running workflows diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index b0173333..be902cee 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2770,7 +2770,15 @@ composable policy is for. The check therefore lives inside the journal the execution hands to `durableRun`: it reads the retained history once, owns the retained snapshot every later phase observes, validates the recorded selection and the required root-import structure against it, and passes that same snapshot -on. It runs ahead of public guard policy, of any retained event reaching execution, +on. It is also a trusted journal-provenance wrapping site. Provenance is not +transitive, so a wrapper is unproven unless its wrapping site carries the source +witness onto it, and a run whose journal is unproven is refused by a Workspace +provider before any transaction. This wrapper qualifies — core installs it +before any document code exists and it delegates every append to the exact +stream it was handed — and it transfers only the witness that source already +has, establishing none. + +It runs ahead of public guard policy, of any retained event reaching execution, of a retained terminal result being reused, of authored work, and of any append. The snapshot it owns covers **every** event that participates in admission, indexing, or terminal reuse — a recorded completion as much as a recorded effect diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index f6793b92..39b2e61c 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -482,8 +482,10 @@ capturing the root. The Deno provider binds an adapter-private proof operation to one exact WorkflowRun handle through module-private executor identity. It establishes the canonical durable-stream module's journal provenance for that run's journal and retains that exact witness. The generic pre-persistence guard -preserves nothing; the trusted secret-filter wrapping site preserves the -witness explicitly, including through nested trusted wrappers. The provider +preserves nothing. Two trusted wrapping sites preserve the witness explicitly, +and a run's journal passes through both: the secret filter, and the +execution-owned target-admission wrapper core installs before any document code +exists. Nesting them carries the same witness through each. The provider receives the invocation's exact executor identity and journal provenance from its execution-owned capability and validates them before it opens the caller-owned transaction. It refuses a foreign executor, foreign or absent