diff --git a/architecture.md b/architecture.md index 294bb3ea..0aa7174b 100644 --- a/architecture.md +++ b/architecture.md @@ -24,13 +24,14 @@ Existing documents and code get aligned to this section retroactively. | middleware | applied by the lexical structure, used by runtime execution | | workflow run | a workflow being carried out with its progress and outcome recorded durably; document executions perform its work, while ongoing effects remain scoped to the document execution in which they run | | document execution | one evaluation of a root document initiated through `execute()`, producing one output stream and one completion result while reading and appending a durable journal; its ongoing effects belong to the Effection scope in which the evaluation runs | -| workflow definition | what a workflow run is a run of: a versioned descriptor naming an immutable object — its format and object ID — together with the repository-relative path of the root document inside it. A repository locator is not part of it, and it is distinct from every Repository created inside the run's Workspace | +| workflow definition | what a workflow run is a run of: a versioned descriptor naming an immutable object — its format and object ID — together with the repository-relative path of the root document inside it, and the exact canonical document target when one is selected. A repository locator is not part of it, and it is distinct from every Repository created inside the run's Workspace | | retrieval metadata | replaceable, credential-free information about where a workflow definition can be fetched from now; it takes no part in run identity and is reauthorized by the host before use | | stop reason | why a workflow run or a document execution stopped: a categorical host code, or a reference to an already-filtered journal event | | run ID | an opaque stable public identifier generated by the host or selected by an authorized caller; it associates the run's durable records and effects, remains unchanged for the life of the run, and has no semantics beyond equality and lifecycle addressing | | definition base | the Git revision supplied to choose a workflow definition's pinned commit | | Repository base | the optional Git revision from which one named Workspace Repository initializes its primary checkout | | pinned commit | the commit obtained by resolving a base once; it remains the workflow run's starting repository state even as the run creates descendant commits | +| document target | an addressable static heading in a root document's own Markdown flow, named by the canonical path of heading labels that reaches it; selecting one executes the preamble, each ancestor's own content, and that heading's complete subtree | | expansion | one logical evaluation of an authored executable element within a document execution | | expansion ID | a deterministic identifier for one logical expansion; restoring or retrying that expansion preserves the ID, while a distinct evaluation requested by the document receives another | | Git capability | the contextual interface through which workflow infrastructure queries the Git repository associated with the current working directory | @@ -603,6 +604,81 @@ interpreter; graceful Worker shutdown alone is insufficient. Worker Shell exposes no native executable or host PATH and is not described as POSIX or native Bash. +## Document targets + +A root document addresses its own sections. The outline is discovered from the +document's static Markdown alone: only headings in the root flow are targets, +and a heading is addressable only when its text is statically rendered — a +heading generated inside a component, or one carrying an interpolation, has no +stable address, and neither does anything beneath it. Discovery therefore +parses a copy in which the scanner's top-level component spans are blanked, +because a Markdown parser reading raw XMD cannot tell a component's children +from the root flow. + +A document's preamble is what precedes its outline — the source before the first +root-flow heading, whatever its depth. A document may open at a deeper level than +the one supplying its title, and an earlier section is a target in its own right +rather than preamble every other target carries. + +Selection resolves exactly once, before the document expands and before any +authored effect runs. A selector may glob, but it must name exactly one catalog +entry: naming none and naming several are both failures, and two sections that +canonicalize to the same path stay two entries so the ambiguity is reported +rather than resolved arbitrarily. + +The selector and the target it resolves to are different things, and only one of +them is identity. + +**The exact canonical target is definition identity.** It is what ran, so it is +what a document execution records durably, what a targeted workflow definition +carries, and what a resumed run is checked against. + +**A caller's glob is non-authoritative invocation metadata.** It describes what +a caller asked for — two callers may spell one request differently — and it +never substitutes for the exact target: it does not occupy the recorded +exact-target field, it never enters a workflow definition, and it is never +re-resolved against a newer checkout to decide what a resumed run means. A glob +is retained in exactly one place, a failed selection's structural record, and +only so that an ordinary failed execution can be reproduced. + +A resumed run re-resolves the current selector against the *recorded* content +and refuses to continue unless the outcome is the one recorded. A failed +selection is an outcome too, and is recorded and compared as one — otherwise a +journal left by a selector that matched nothing would answer a later request for +a section that does exist. + +Validating that identity is **execution-owned**. Replay guards are composable +policy — a handler installed further out may decline to delegate — so identity, +which must not be negotiable, is not decided there. The execution owns the +journal it replays through and validates the recorded selection inside the read +itself, ahead of public guard policy, of a retained terminal result being +reused, of authored work, and of any append. The retained history it owns covers +every event that takes part in that decision, a recorded completion included, so +no event can present one identity to the validation and another to the run. + +Three views of that history stay distinct. The **authoritative** one is what +admission validated and replay consumes, and it is immutable to policy. +Replaceable public policy reads an **isolated observation** of it, so a handler +composes freely without acquiring the authority the gate exists to withhold. +What a document receives is a **fresh mutable copy**, because a resumed binding +is ordinary data its own continuation writes to. Reusing a terminal result also +requires exactly one recognizable root import belonging to the coroutine whose +result is being reused. + +Every public path validates in one order: parse the source, resolve the target, +compile schemas, then build the projected definition. A caller who named a +section the document does not offer hears that, rather than a complaint about a +schema they never reached. + +A recorded selection is a closed protocol, and a record that does not satisfy it +is refused rather than delegated. "This event is not the root import" and "the +root import, malformed" are different answers: one continues, the other fails +before the recorded terminal result can be reused, without executing authored +work or appending history. The record carries the content it was taken from, so +the selection is verified against that content rather than merely parsed, and +reading it is total — every way journal-controlled data can refuse to be read is +the same refusal, carrying nothing the record supplied. + ## Expansion identity Core describes the executable element currently being expanded: @@ -639,6 +715,14 @@ JavaScript object identity. document execution receives expansion identity without installing workflow middleware. +Selecting a document target does not disturb any of this. Projection retains +the original source ranges and scans each one under its own origin, so a +retained element keeps the offset and line it was authored at, and with them +its expansion ID. Two runs of the same document under different targets +therefore agree on the ID of every element they both retain, and may share IDs +without sharing effects — run identity and workflow-definition identity are +what tell those runs apart. + ## Two layers Error handling has two layers: @@ -935,7 +1019,12 @@ It qualifies only while every one of these holds: it; - establishment and transfer occur only through the canonical module's `establishJournalProvenance()` and `preserveJournalProvenance()`, one fresh - witness per stream, with duplicate establishment refused; and + witness per stream, with duplicate establishment refused; +- transfer happens only at a trusted wrapping site — one installed before any + code the journal's content could influence, delegating to the exact stream it + was handed. A document execution's journal passes through two: the secret + filter and the execution-owned target-admission wrapper. Each transfers only + what its source already had, so an unproven journal stays unproven; and - it retains no execution, lifecycle, journal content or provider state. The exception exists because the exact-object, anti-forgery and loaded-copy @@ -992,6 +1081,7 @@ Status is measured against main. | `` / `printErrors(fn)` | prints failures | built on main | | `` region `output` mode | an undecided error fails the document execution | built on main | | `Expansion` / `getExpansion()` | describes the current logical element expansion | built on main | +| document targets | catalogs a root document's addressable static headings, resolves one selector to one exact target, and projects the document to it before expansion | built on the #412 stack; `xmd targets`, targeted `xmd run`, and the targeted workflow definition are unbuilt | | `useWorkflow()` / `getWorkflowRun()` | associates one document execution with a workflow run | built on main | | `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main | | workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; public workflow execution is unbuilt | diff --git a/packages/core/mod.ts b/packages/core/mod.ts index d1c66473..45af4b8a 100644 --- a/packages/core/mod.ts +++ b/packages/core/mod.ts @@ -130,8 +130,25 @@ export type { ExecutionApi, DocumentExecution, } from "./src/execute.ts"; -export { INLINE_SOURCE_PATH, inlineSource, rootSourcePath } from "./src/root-source.ts"; -export type { InlineRootDocument, RootDocumentSource } from "./src/root-source.ts"; +export { + fileSource, + formatDocumentReference, + INLINE_SOURCE_PATH, + inlineSource, + rootSourcePath, +} from "./src/root-source.ts"; +export type { + FileRootDocument, + InlineRootDocument, + RootDocumentSource, +} from "./src/root-source.ts"; +export { + asDocumentTargetError, + DocumentTargetError, + isDocumentTargetError, + parseDocumentTargetFailure, +} from "./src/document-targets.ts"; +export type { DocumentTargetErrorKind, DocumentTargetFailure } from "./src/document-targets.ts"; export { inspectComponent, inspectDocument } from "./src/inspect.ts"; export type { ComponentInfo, diff --git a/packages/core/src/definition.ts b/packages/core/src/definition.ts index f3796d96..9671f3cc 100644 --- a/packages/core/src/definition.ts +++ b/packages/core/src/definition.ts @@ -1,8 +1,11 @@ -import type { Operation } from "effection"; -import type { ComponentDefinition } from "./types.ts"; +import { Ok } from "effection"; +import type { Operation, Result } from "effection"; +import type { ComponentDefinition, Segment } from "./types.ts"; import { parseFrontmatter } from "./frontmatter.ts"; import { compilePropsSchema, compileReturnsSchema } from "./validate.ts"; -import { scanSegments } from "./scanner.ts"; +import { scanComponentSpans, scanSegments } from "./scanner.ts"; +import { findTarget, outlineDocument, retainedRanges, selectTarget } from "./document-targets.ts"; +import type { DocumentOutline } from "./document-targets.ts"; import matter from "gray-matter"; @@ -15,56 +18,201 @@ export function isFunctionComponentPath(path: string): boolean { return path.endsWith(".ts"); } +/** A document's frontmatter data, its markdown body, and where the body sits. */ +interface ParsedSource { + data: Record; + content: string; + baseOffset: number; + baseLine: number; +} + /** - * Parse markdown source into a component definition. Execution and - * inspection share this so their frontmatter and schema behavior cannot - * drift: both compile the props and return schemas, so a malformed schema - * fails the same way whether the document runs or is only described. + * Split frontmatter from the markdown body without reading either. + * + * The markdown body is a verbatim suffix of the raw file, so the body start + * is computed by length — never by content search, which could false-match + * body text repeated inside frontmatter. The invariant check turns any + * gray-matter normalization surprise into a loud error instead of silently + * wrong source positions. */ -export function* parseMarkdownDefinition( - name: string, - path: string, - content: string, -): Operation { +function parseSource(path: string, content: string): ParsedSource { const parsed = matter(content); - const { meta, props, returns } = parseFrontmatter(parsed.data); - yield* compilePropsSchema(props); - if (returns !== undefined) { - yield* compileReturnsSchema(returns); - } - // The markdown body is a verbatim suffix of the raw file, so the body start - // is computed by length — never by content search, which could false-match - // body text repeated inside frontmatter. The invariant check turns any - // gray-matter normalization surprise into a loud error instead of silently - // wrong source positions. - const bodyStart = content.length - parsed.content.length; - if (content.slice(bodyStart) !== parsed.content) { + const baseOffset = content.length - parsed.content.length; + if (content.slice(baseOffset) !== parsed.content) { throw new Error(`frontmatter parse did not preserve the markdown body verbatim: ${path}`); } let baseLine = 1; - for (let i = 0; i < bodyStart; i++) { + for (let i = 0; i < baseOffset; i++) { if (content[i] === "\n") { baseLine++; } } - const bodySegments = scanSegments(parsed.content, { - path, - baseOffset: bodyStart, - baseLine, - }); + return { data: parsed.data, content: parsed.content, baseOffset, baseLine }; +} +/** The static heading structure a document's body offers as targets. */ +export function documentOutline(path: string, content: string): DocumentOutline { + const body = parseSource(path, content).content; + return outlineDocument(body, scanComponentSpans(body)); +} + +/** + * The exact canonical target a selector names in this document's content, or + * the failure describing why it names none. + * + * Synchronous and free of effects, so the resolution that decides *what* runs + * happens before anything runs — including inside the durable operation that + * records the root, and inside a replay guard reading recorded content. The + * outcome comes back rather than being thrown because both of those callers + * record it as data before anyone reports it. + */ +export function resolveDocumentTarget( + path: string, + content: string, + selector: string, +): Result { + const found = findTarget(documentOutline(path, content), selector); + return found.ok ? Ok(found.value.target) : found; +} + +interface CompiledFrontmatter { + meta: Record; + props: ComponentDefinition["props"]; + returns: ComponentDefinition["returns"]; +} + +function* compileFrontmatter(data: Record): Operation { + const { meta, props, returns } = parseFrontmatter(data); + yield* compilePropsSchema(props); + if (returns !== undefined) { + yield* compileReturnsSchema(returns); + } + return { meta, props, returns }; +} + +function buildDefinition( + name: string, + path: string, + frontmatter: CompiledFrontmatter, + bodySegments: Segment[], +): ComponentDefinition { // `returns` stays absent in text mode: absence is what distinguishes a text // component from one that explicitly declares a string return. const definition: ComponentDefinition = { kind: "markdown", name, path, - meta, - props, + meta: frontmatter.meta, + props: frontmatter.props, bodySegments, }; - if (returns !== undefined) { - definition.returns = returns; + if (frontmatter.returns !== undefined) { + definition.returns = frontmatter.returns; } return definition; } + +/** + * Parse markdown source into a component definition. Execution and + * inspection share this so their frontmatter and schema behavior cannot + * drift: both compile the props and return schemas, so a malformed schema + * fails the same way whether the document runs or is only described. + */ +export function* parseMarkdownDefinition( + name: string, + path: string, + content: string, +): Operation { + const body = parseSource(path, content); + const frontmatter = yield* compileFrontmatter(body.data); + return buildDefinition( + name, + path, + frontmatter, + scanSegments(body.content, { path, baseOffset: body.baseOffset, baseLine: body.baseLine }), + ); +} + +/** A root document as parsed: what it declares, and what it addresses. */ +export interface ParsedRootDocument { + definition: ComponentDefinition; + /** Canonical encoded target fragments in document order, duplicates kept. */ + targets: readonly string[]; + /** The exact canonical target selected, when one was requested. */ + target?: string; +} + +/** + * Parse a root document, projecting it to one target when a selector asks for + * one. + * + * Selection happens here, before any segment exists, so a selector that names + * nothing or names several sections fails with nothing expanded. Without a + * selector the whole body is scanned exactly as an ordinary markdown component + * is. + * + * A projection scans each retained range on its own, with the origin that range + * has in the original file, rather than scanning a concatenated string. Skipped + * source therefore cannot renumber what follows it: a retained element keeps + * the offset and line it was authored at, and with them its expansion ID. + */ +export function* parseRootMarkdownDefinition( + name: string, + path: string, + content: string, + selector?: string, +): Operation { + // The order is the contract, and it is the same on every public path. + // Syntax first, because the outline comes from it; then the target, because a + // caller who named nothing the document offers asked the wrong question and + // should hear that rather than a complaint about a schema they did not reach; + // then the schemas; then the projected definition. + const body = parseSource(path, content); + const outline = outlineDocument(body.content, scanComponentSpans(body.content)); + + if (selector === undefined) { + const frontmatter = yield* compileFrontmatter(body.data); + const bodySegments = scanSegments(body.content, { + path, + baseOffset: body.baseOffset, + baseLine: body.baseLine, + }); + return { + definition: buildDefinition(name, path, frontmatter, bodySegments), + targets: outline.targets, + }; + } + + const entry = selectTarget(outline, selector); + const frontmatter = yield* compileFrontmatter(body.data); + const newlines = newlineCounts(body.content); + const bodySegments: Segment[] = []; + for (const range of retainedRanges(outline, entry)) { + bodySegments.push( + ...scanSegments(body.content.slice(range.start, range.end), { + path, + baseOffset: body.baseOffset + range.start, + baseLine: body.baseLine + newlines[range.start]!, + }), + ); + } + return { + definition: buildDefinition(name, path, frontmatter, bodySegments), + targets: outline.targets, + target: entry.target, + }; +} + +/** How many newlines precede each offset, so a retained range knows its line. */ +function newlineCounts(body: string): number[] { + const counts = new Array(body.length + 1); + let seen = 0; + for (let i = 0; i < body.length; i++) { + counts[i] = seen; + if (body[i] === "\n") { + seen++; + } + } + counts[body.length] = seen; + return counts; +} diff --git a/packages/core/src/document-targets.ts b/packages/core/src/document-targets.ts new file mode 100644 index 00000000..4083525a --- /dev/null +++ b/packages/core/src/document-targets.ts @@ -0,0 +1,1019 @@ +/** + * Document targets (spec §5.4). + * + * A target is an addressable static heading in a root document's Markdown flow. + * Selecting one projects the document down to the preamble, the direct content + * of every ancestor needed to reach it, and its complete subtree — so a section + * of a document runs on its own without its siblings. + * + * Two properties shape everything here. + * + * The outline is discovered from *static* Markdown only. Component children can + * hold text that looks like a heading, and Remark cannot tell the difference: a + * blank line inside component children ends its HTML block and the child heading + * surfaces as a root heading. Discovery therefore parses a masked copy of the + * body, where every top-level component span is replaced by spaces of the same + * length. Offsets, lines, and everything outside those spans are untouched, so + * the mask changes what is *seen*, never where anything *is*. + * + * Projection retains original source ranges rather than a rebuilt document. Each + * retained range is scanned with its own origin, so every retained element keeps + * the offset and line it was authored at — which is what keeps expansion + * identifiers equal between a full run and a targeted one. + */ + +import { Err, Ok } from "effection"; +import type { Result } from "effection"; +import { remark } from "remark"; +import { toString as mdastToString } from "mdast-util-to-string"; + +import type { ComponentSpan } from "./scanner.ts"; + +/** A half-open slice of the original document body. */ +export interface SourceRange { + readonly start: number; + readonly end: number; +} + +/** One catalog entry: an addressable heading and the path that reaches it. */ +export interface DocumentTarget { + /** The canonical encoded target fragment, without a leading `#`. */ + readonly target: string; + /** The decoded, normalized labels the fragment encodes. */ + readonly labels: readonly string[]; + /** Which heading in the outline this entry addresses. */ + readonly heading: number; +} + +interface OutlineHeading { + readonly depth: number; + readonly start: number; + readonly end: number; + readonly parent: number | undefined; + readonly addressable: boolean; + readonly label: string; +} + +/** The static heading structure of one document body, and what it addresses. */ +export interface DocumentOutline { + readonly headings: readonly OutlineHeading[]; + readonly entries: readonly DocumentTarget[]; + /** Canonical encoded fragments in source order, duplicates retained. */ + readonly targets: readonly string[]; + /** Where the preamble ends: the first root-flow heading, or the body end. */ + readonly preambleEnd: number; + readonly bodyLength: number; +} + +/** Why a requested target did not resolve to exactly one catalog entry. */ +export type DocumentTargetErrorKind = "invalid-selector" | "no-match" | "multiple-matches"; + +const KINDS: readonly DocumentTargetErrorKind[] = [ + "invalid-selector", + "no-match", + "multiple-matches", +]; + +const KIND_WORDING: ReadonlyMap = new Map([ + ["invalid-selector", "is not a valid document target selector"], + ["no-match", "matches no document target"], + ["multiple-matches", "matches more than one document target"], +]); + +/** + * The structural tag a document-target failure carries. + * + * Namespaced and stable, because it is the whole recognition mechanism. Two + * loaded copies of this package are two classes, so `instanceof` answers false + * between them; a failure built by one copy has to be recognized by the other + * on exactly the same terms as one built here (AGENTS.md rule 15). + */ +const DOCUMENT_TARGET_FAILURE = "executablemd.document-target-failure"; + +/** + * Why one requested selector did not name exactly one section, as data. + * + * Frozen and rebuilt from validated parts wherever it crosses a boundary. The + * selector is retained because reproducing an ordinary failed execution needs + * to say what was asked for — it is sanitized invocation metadata, never + * identity, and it never stands in for an exact target. + */ +export interface DocumentTargetFailure { + readonly type: typeof DOCUMENT_TARGET_FAILURE; + readonly kind: DocumentTargetErrorKind; + /** The selector fragment as it was requested, still encoded. */ + readonly selector: string; + /** Canonical encoded targets the selector matched; empty unless ambiguous. */ + readonly matches: readonly string[]; + /** Every canonical encoded target the document offers. */ + readonly available: readonly string[]; +} + +/** The fields a failure carries, without the tag that authenticates them. */ +interface TargetFailureFields { + kind: DocumentTargetErrorKind; + selector: string; + matches: string[]; + available: string[]; +} + +function attempt(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** Whether this is an Error, without trusting its prototype chain. */ +function isError(value: unknown): value is Error { + return attempt(() => value instanceof Error) === true; +} + +/** One property, read through a trap that may refuse or fail. */ +function property(target: object, name: string): unknown { + return attempt(() => Reflect.get(target, name)); +} + +function stringList(value: unknown): string[] | undefined { + return attempt(() => { + if (!Array.isArray(value)) { + return undefined; + } + const items: string[] = []; + for (let index = 0; index < value.length; index++) { + const item = index in value ? value[index] : undefined; + if (typeof item !== "string") { + return undefined; + } + items.push(item); + } + return items; + }); +} + +/** + * The canonical labels a canonical target encodes. + * + * Only called on an entry already proven canonical, so decoding cannot fail; + * the guard is here because this reads data a candidate supplied. + */ +function targetLabels(target: string): string[] | undefined { + const labels: string[] = []; + for (const level of target.split("/")) { + const decoded = decodePercentEncoded(level); + if (decoded === undefined) { + return undefined; + } + labels.push(decoded); + } + return labels; +} + +/** A dense list of canonical encoded targets, copied out of the candidate. */ +function targetList(value: unknown): string[] | undefined { + const items = stringList(value); + if (items === undefined) { + return undefined; + } + return items.every((item) => isCanonicalTarget(item)) ? items : undefined; +} + +/** + * Whether these fields describe an outcome selection could actually have + * reached. + * + * The check re-derives the outcome rather than trusting the three fields to + * agree: the selector is parsed, matched against the catalog the candidate + * supplied, and the result compared with the matches it claims. A record whose + * kind, selector, matches, and catalog cannot all be true at once is refused, + * so an inconsistent journal or a hand-built candidate cannot describe a + * selection that never happened. + */ +function consistentOutcome(fields: TargetFailureFields): boolean { + const levels = parseSelector(fields.selector); + if (fields.kind === "invalid-selector") { + return levels === undefined && fields.matches.length === 0; + } + if (levels === undefined) { + return false; + } + const derived = fields.available.filter((target) => { + const labels = targetLabels(target); + return labels !== undefined && matchPath(levels, labels); + }); + if (fields.kind === "no-match") { + return derived.length === 0 && fields.matches.length === 0; + } + return derived.length > 1 && sameList(derived, fields.matches); +} + +/** + * Read a candidate's failure fields, rebuilding every one of them. + * + * Total: an unreadable property, a missing one, a kind outside the closed set, + * a sparse list, an entry that is not a canonical encoded target, and a + * combination of kind, selector, matches and catalog that no selection could + * have produced are all "not this shape" rather than a throw. + * + * Nothing the candidate owns is retained. Every entry is read once and copied + * into a fresh array, so a list that is mutated afterwards — or reached through + * a Proxy that is later revoked — cannot change what comes back. + */ +function targetFailureFields(value: unknown): TargetFailureFields | undefined { + return attempt(() => { + if (!isRecord(value)) { + return undefined; + } + const kind = KINDS.find((candidate) => candidate === property(value, "kind")); + const selector = property(value, "selector"); + const matches = targetList(property(value, "matches")); + const available = targetList(property(value, "available")); + if (kind === undefined || typeof selector !== "string") { + return undefined; + } + if (matches === undefined || available === undefined) { + return undefined; + } + const fields = { kind, selector, matches, available }; + return consistentOutcome(fields) ? fields : undefined; + }); +} + +/** Freeze validated fields into the failure data an error carries. */ +function sealFailure(fields: TargetFailureFields): DocumentTargetFailure { + return Object.freeze({ + type: DOCUMENT_TARGET_FAILURE, + kind: fields.kind, + selector: fields.selector, + matches: Object.freeze([...fields.matches]), + available: Object.freeze([...fields.available]), + }); +} + +/** Exactly the members failure data describes, and nothing else. */ +const FAILURE_DATA_MEMBERS: readonly string[] = [ + "available", + "kind", + "matches", + "selector", + "type", +]; + +/** + * Whether this object's own members are exactly the contract's. + * + * Own *property names* rather than enumerable keys, and own symbols with them: + * a non-enumerable extra is still payload the contract does not describe, and a + * symbol-keyed one survives spreading and `Object.assign`, which are exactly the + * mechanisms a consumer uses to pass data on. + */ +function hasOnlyDataMembers(value: object): boolean { + const names = attempt(() => [...Object.getOwnPropertyNames(value)].sort()); + if (names === undefined || names.length !== FAILURE_DATA_MEMBERS.length) { + return false; + } + if (!names.every((name, index) => name === FAILURE_DATA_MEMBERS[index])) { + return false; + } + return attempt(() => Object.getOwnPropertySymbols(value).length) === 0; +} + +/** + * The failure data this value carries, if it carries valid, tagged data. + * + * Every field is checked, the exact member set with them, and the outcome the + * fields describe is re-derived rather than trusted. What comes back is built + * here from validated parts: the candidate's own arrays are never adopted, so a + * nested list that is mutable, mutated later, or reached through a revocable + * Proxy cannot reach a caller. + */ +export function parseDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined { + return attempt(() => { + if (!isRecord(value) || property(value, "type") !== DOCUMENT_TARGET_FAILURE) { + return undefined; + } + if (!hasOnlyDataMembers(value)) { + return undefined; + } + const fields = targetFailureFields(value); + return fields === undefined ? undefined : sealFailure(fields); + }); +} + +/** Exactly the members a journal's failure record holds: the data, untagged. */ +const RECORD_MEMBERS: readonly string[] = ["available", "kind", "matches", "selector"]; + +/** + * The failure a journal record describes, rebuilt and sealed. + * + * The record is untagged — its place inside a recorded root-import selection is + * what identifies it — so this validates the fields and supplies the tag, + * rather than requiring a tag the journal never held. The member set is closed + * all the same: an extra key is data this contract does not describe, and a + * journal is not a place to carry undescribed data forward from. + */ +export function recordedDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined { + return attempt(() => { + if (!isRecord(value)) { + return undefined; + } + const names = attempt(() => [...Object.getOwnPropertyNames(value)].sort()); + if (names === undefined || names.length !== RECORD_MEMBERS.length) { + return undefined; + } + if (!names.every((name, index) => name === RECORD_MEMBERS[index])) { + return undefined; + } + const fields = targetFailureFields(value); + return fields === undefined ? undefined : sealFailure(fields); + }); +} + +/** + * The one diagnostic a failure carries, derived from its data alone. + * + * Recognition compares against this, so the message cannot disagree with the + * fields. Every reference is canonically encoded and the selector is JSON + * quoted, so a heading holding a control character cannot reach a diagnostic + * literally. + */ +function documentTargetMessage(failure: DocumentTargetFailure): string { + const ambiguous = failure.kind === "multiple-matches"; + const listed = ambiguous ? failure.matches : failure.available; + const heading = ambiguous ? "Matched targets:" : "Available targets:"; + return ( + `${JSON.stringify(failure.selector)} ${KIND_WORDING.get(failure.kind)}.\n` + + (listed.length === 0 + ? "The document has no targets." + : `${heading}\n${listed.map((target) => ` ${target}`).join("\n")}`) + ); +} + +/** + * Everything a constructor here puts on the Error itself, and nothing else. + * + * `message` and `stack` are non-enumerable own properties of every Error, so + * what remains enumerable is exactly what this constructor assigned. + */ +const FAILURE_MEMBERS: readonly string[] = ["data", "name"]; + +function hasOnlyContractMembers(error: Error): boolean { + const keys = attempt(() => [...Object.keys(error)].sort()); + if (keys === undefined || keys.length !== FAILURE_MEMBERS.length) { + return false; + } + if (!keys.every((key, index) => key === FAILURE_MEMBERS[index])) { + return false; + } + const payload = attempt(() => + Object.getOwnPropertySymbols(error).filter( + (symbol) => Object.getOwnPropertyDescriptor(error, symbol)?.enumerable === true, + ), + ); + return payload !== undefined && payload.length === 0; +} + +/** + * A requested document target that does not name exactly one section. + * + * An ordinary invocation failure: the caller asked for something the document + * does not offer, and nothing durable or contained is involved. It is raised + * before the document expands, so a run that cannot decide what to execute + * executes nothing. + * + * Its data is the contract; the message is derived from it. Construct one from + * validated data — `documentTargetError()` — rather than from parts, so a + * failure rebuilt at a journal boundary is indistinguishable from the one the + * live run raised. + */ +export class DocumentTargetError extends Error { + readonly data: DocumentTargetFailure; + + constructor(data: DocumentTargetFailure) { + super(documentTargetMessage(data)); + this.name = "DocumentTargetError"; + this.data = data; + } +} + +/** Build the failure this selector produced, from parts this module owns. */ +export function documentTargetFailure( + kind: DocumentTargetErrorKind, + selector: string, + matches: readonly string[], + available: readonly string[], +): DocumentTargetFailure { + return sealFailure({ + kind, + selector, + matches: [...matches], + available: [...available], + }); +} + +/** + * Rebuild the error a failure describes. + * + * The one constructor call outside this module's own selection path, so a + * replayed failure and a live one are the same object shape carrying the same + * fields — a caller cannot tell which run raised it, and does not have to. + */ +export function documentTargetError(data: DocumentTargetFailure): DocumentTargetError { + return new DocumentTargetError(data); +} + +/** + * The validated failure data this error carries, if it satisfies the whole + * contract rather than merely the tag. + * + * Structural throughout, so a failure constructed by a separately loaded copy + * of this package is read on exactly the same terms as one constructed here. + * The name is checked rather than the class for the same reason: a second + * copy's constructor is a different function producing the same name. + * + * The shell is checked as strictly as the data. The message has to be the one + * the data derives, so a diagnostic cannot disagree with the fields it claims; + * there can be no cause and no enumerable member beyond the contract, so a + * candidate cannot carry a path, a foreign object, or a second message under a + * recognized tag. + */ +function readDocumentTargetError(error: unknown): DocumentTargetFailure | undefined { + return attempt(() => { + if (!isError(error)) { + return undefined; + } + const data = parseDocumentTargetFailure(property(error, "data")); + if (data === undefined) { + return undefined; + } + if (property(error, "name") !== "DocumentTargetError") { + return undefined; + } + if (property(error, "message") !== documentTargetMessage(data)) { + return undefined; + } + if (property(error, "cause") !== undefined) { + return undefined; + } + return hasOnlyContractMembers(error) ? data : undefined; + }); +} + +/** Whether this failure satisfies the whole document-target contract. */ +export function isDocumentTargetError(error: unknown): boolean { + return readDocumentTargetError(error) !== undefined; +} + +/** + * The document-target failure this error describes, as a local error. + * + * A fresh error built from reconstructed data, never the candidate itself. + * Nothing here has a fail-stop's reason to preserve object identity — this is + * an ordinary invocation failure, and what a caller needs is the outcome, not + * the instance that reported it. Returning the candidate would hand on whatever + * it owns: nested arrays somebody else can still mutate, a revocable Proxy, a + * prototype with accessors. Rebuilding costs one allocation and removes all of + * it, so the result stays readable however the original is treated afterwards. + */ +export function asDocumentTargetError(error: unknown): DocumentTargetError | undefined { + const data = readDocumentTargetError(error); + return data === undefined ? undefined : new DocumentTargetError(data); +} + +/** Whether two failures describe the same selection outcome, field by field. */ +export function sameDocumentTargetFailure( + left: DocumentTargetFailure, + right: DocumentTargetFailure, +): boolean { + return ( + left.kind === right.kind && + left.selector === right.selector && + sameList(left.matches, right.matches) && + sameList(left.available, right.available) + ); +} + +function sameList(left: readonly string[], right: readonly string[]): boolean { + return left.length === right.length && left.every((item, index) => item === right[index]); +} + +const UNRESERVED = /^[A-Za-z0-9\-._~]$/; +const HEX = /^[0-9A-Fa-f]$/; + +const ENCODER = new TextEncoder(); + +function encodeCharacter(character: string): string { + let encoded = ""; + for (const byte of ENCODER.encode(character)) { + encoded += `%${byte.toString(16).toUpperCase().padStart(2, "0")}`; + } + return encoded; +} + +/** + * Percent-encode one canonical label. Everything outside RFC 3986's unreserved + * set is escaped, so `/`, `*`, `#`, and `%` inside a heading cannot be read as + * hierarchy or operator syntax. + */ +export function encodeTargetLabel(label: string): string { + let encoded = ""; + for (const character of label) { + encoded += UNRESERVED.test(character) ? character : encodeCharacter(character); + } + return encoded; +} + +/** + * Percent-encode a decoded filesystem path. Separators survive as raw `/`; a + * `/` that is part of a filename cannot be told apart from one afterwards, so + * this is a formatter for paths the caller already holds, not a round trip. + */ +export function encodeDocumentPath(path: string): string { + let encoded = ""; + for (const character of path) { + encoded += + character === "/" || UNRESERVED.test(character) ? character : encodeCharacter(character); + } + return encoded; +} + +/** + * Decode one percent-encoded chunk, or `undefined` when it is not decodable. + * + * Malformed escapes, byte sequences that are not UTF-8, and NUL are all + * refused rather than repaired: a selector that cannot be read exactly is not a + * selector this can match against. `+` is an ordinary character — this is URI + * path syntax, not a form encoding. + */ +export function decodePercentEncoded(text: string): string | undefined { + const characters = Array.from(text); + const bytes: number[] = []; + for (let index = 0; index < characters.length; index++) { + const character = characters[index]!; + if (character !== "%") { + for (const byte of ENCODER.encode(character)) { + bytes.push(byte); + } + continue; + } + const high = characters[index + 1]; + const low = characters[index + 2]; + if (high === undefined || low === undefined || !HEX.test(high) || !HEX.test(low)) { + return undefined; + } + bytes.push(Number.parseInt(`${high}${low}`, 16)); + index += 2; + } + try { + const decoded = new TextDecoder("utf-8", { fatal: true }).decode(new Uint8Array(bytes)); + return decoded.includes("\u0000") ? undefined : decoded; + } catch { + return undefined; + } +} + +/** + * The canonical form of rendered heading text: NFC, every run of Unicode + * whitespace collapsed to one ASCII space, trimmed, case preserved. + */ +export function normalizeLabel(text: string): string { + return text.normalize("NFC").replace(/\s+/gu, " ").trim(); +} + +/** + * Whether a fragment is already an exact canonical target. + * + * A level is canonical only when decoding it, normalizing the label, and + * re-encoding that label reproduce the level byte for byte. Requiring the whole + * round trip is what makes this total: it rejects a wildcard operator, an empty + * level, a lowercase escape, a raw `#`, an NFD spelling, a tab, and leading, + * trailing, or uncollapsed whitespace without naming any of them, because none + * of them is what this module would have written. + */ +export function isCanonicalTarget(target: string): boolean { + if (target.length === 0) { + return false; + } + return target.split("/").every((level) => { + const decoded = decodePercentEncoded(level); + if (decoded === undefined || decoded.length === 0) { + return false; + } + const label = normalizeLabel(decoded); + return label === decoded && encodeTargetLabel(label) === level; + }); +} + +type LevelPart = + | { readonly kind: "literal"; readonly text: string } + | { readonly kind: "wildcard" }; + +type SelectorLevel = + | { readonly kind: "recursive" } + | { readonly kind: "label"; readonly parts: readonly LevelPart[] }; + +/** + * Parse a target selector into levels, or `undefined` when the syntax is not a + * selector at all. + * + * Raw `/` separates levels and raw `*` is an operator, so the split happens + * before decoding: `%2F` stays a slash inside one label and `%2A` stays a + * literal asterisk. Only the literal chunks between operators are decoded. + */ +function parseSelector(selector: string): readonly SelectorLevel[] | undefined { + if (selector.length === 0 || selector.startsWith("/") || selector.endsWith("/")) { + return undefined; + } + // A raw `#` is the reference's own delimiter, so it never reaches a selector + // by the supported route and cannot be written back into one. `%23` addresses + // a heading that really contains it. + if (selector.includes("#")) { + return undefined; + } + const levels: SelectorLevel[] = []; + for (const raw of selector.split("/")) { + if (raw.length === 0) { + return undefined; + } + if (raw === "**") { + levels.push({ kind: "recursive" }); + continue; + } + const chunks: string[] = []; + for (const chunk of raw.split("*")) { + const decoded = decodePercentEncoded(chunk); + if (decoded === undefined) { + return undefined; + } + chunks.push(decoded.normalize("NFC").replace(/\s+/gu, " ")); + } + // Only the outer edges are trimmed: whitespace beside a wildcard is part of + // what the author asked to match, while the whole level is compared against + // an already-trimmed label. + chunks[0] = chunks[0]!.trimStart(); + chunks[chunks.length - 1] = chunks[chunks.length - 1]!.trimEnd(); + + const parts: LevelPart[] = []; + for (const [index, chunk] of chunks.entries()) { + if (index > 0 && parts[parts.length - 1]?.kind !== "wildcard") { + parts.push({ kind: "wildcard" }); + } + if (chunk.length > 0) { + parts.push({ kind: "literal", text: chunk }); + } + } + levels.push({ kind: "label", parts }); + } + return levels; +} + +/** + * Whether one level's parts match one label, by code point. + * + * A reachability sweep rather than backtracking: each part advances a set of + * positions the label could have been consumed to, so a selector holding many + * wildcards costs the product of its size and the label's, never an exponential + * search. + */ +function matchLabel(parts: readonly LevelPart[], label: readonly string[]): boolean { + let reachable = new Array(label.length + 1).fill(false); + reachable[0] = true; + for (const part of parts) { + const next = new Array(label.length + 1).fill(false); + if (part.kind === "wildcard") { + let open = false; + for (let index = 0; index <= label.length; index++) { + open ||= reachable[index]!; + next[index] = open; + } + } else { + const literal = Array.from(part.text); + for (let index = 0; index + literal.length <= label.length; index++) { + if (!reachable[index]) { + continue; + } + if (literal.every((character, offset) => label[index + offset] === character)) { + next[index + literal.length] = true; + } + } + } + reachable = next; + } + return reachable[label.length]!; +} + +/** Whether a parsed selector matches a canonical label path. */ +function matchPath(levels: readonly SelectorLevel[], path: readonly string[]): boolean { + const characters = path.map((label) => Array.from(label)); + let reachable = new Array(path.length + 1).fill(false); + reachable[0] = true; + for (const level of levels) { + const next = new Array(path.length + 1).fill(false); + if (level.kind === "recursive") { + let open = false; + for (let index = 0; index <= path.length; index++) { + open ||= reachable[index]!; + next[index] = open; + } + } else { + for (let index = 0; index < path.length; index++) { + if (reachable[index] && matchLabel(level.parts, characters[index]!)) { + next[index + 1] = true; + } + } + } + reachable = next; + } + return reachable[path.length]!; +} + +/** + * The one catalog entry a selector names. + * + * Zero matches and several matches are both failures, and both are decided + * here — before the document expands — so an ambiguous request never runs half + * a document to discover it was ambiguous. Duplicate canonical paths stay + * duplicate entries, which is what makes that ambiguity observable at all. + */ +export function selectTarget(outline: DocumentOutline, selector: string): DocumentTarget { + const found = findTarget(outline, selector); + if (found.ok) { + return found.value; + } + throw found.error; +} + +/** The entry a selector names, or the failure describing why it names none. */ +export function findTarget(outline: DocumentOutline, selector: string): Result { + const fail = ( + kind: DocumentTargetErrorKind, + matches: readonly string[], + ): Result => + Err(documentTargetError(documentTargetFailure(kind, selector, matches, outline.targets))); + + const levels = parseSelector(selector); + if (levels === undefined) { + return fail("invalid-selector", []); + } + const matched = outline.entries.filter((entry) => matchPath(levels, entry.labels)); + const first = matched[0]; + if (first === undefined) { + return fail("no-match", []); + } + if (matched.length > 1) { + return fail( + "multiple-matches", + matched.map((entry) => entry.target), + ); + } + return Ok(first); +} + +/** + * The interpolation forms a heading's own source may not contain. + * + * A heading whose text is computed is not a stable address, so it is not one. + * `\{` escapes an interpolation back into literal text, which stays static and + * stays addressable. + */ +const INTERPOLATION = + /(\\?)\{(?:(?:meta|props)\.[^}]+|[A-Za-z_$][A-Za-z0-9_$]*(?:\.[A-Za-z_$][A-Za-z0-9_$]*)*)\}/g; + +function hasUnescapedInterpolation(source: string): boolean { + for (const match of source.matchAll(INTERPOLATION)) { + if (match[1] !== "\\") { + return true; + } + } + return false; +} + +/** + * A copy of the body with every top-level component span blanked. + * + * Length, newline positions, and every offset are preserved, so a heading found + * in the mask sits at the same place in the original. + */ +function maskComponents(body: string, spans: readonly ComponentSpan[]): string { + if (spans.length === 0) { + return body; + } + let masked = ""; + let cursor = 0; + for (const span of spans) { + masked += body.slice(cursor, span.start); + masked += body.slice(span.start, span.end).replace(/[^\n]/g, " "); + cursor = span.end; + } + return masked + body.slice(cursor); +} + +interface RawHeading { + readonly depth: number; + readonly start: number; + readonly end: number; + readonly text: string; +} + +function rootHeadings(masked: string): RawHeading[] { + const headings: RawHeading[] = []; + for (const child of remark().parse(masked).children) { + if (child.type !== "heading") { + continue; + } + const start = child.position?.start.offset; + const end = child.position?.end.offset; + if (start === undefined || end === undefined) { + continue; + } + headings.push({ + depth: child.depth, + start, + end, + // Read from the masked tree, which is the original text for every heading + // that does not overlap a component span — and one that does is refused + // below, so no label is ever built from blanked source. + text: mdastToString(child, { includeHtml: false, includeImageAlt: true }), + }); + } + return headings; +} + +function overlapsComponent(heading: RawHeading, spans: readonly ComponentSpan[]): boolean { + return spans.some((span) => span.start < heading.end && heading.start < span.end); +} + +/** + * Discover the outline of one document body and the targets it addresses. + * + * The hierarchy is the standard outline stack — a heading's parent is the + * nearest preceding heading with a smaller depth — so skipped depths are + * ordinary. "Outermost" is the smallest depth present, not `h1`. + * + * A single outermost heading is the document's title: it is retained in every + * projection and takes no level in any path, which is why a document that opens + * with one title still addresses its sections by their own names. + */ +export function outlineDocument(body: string, spans: readonly ComponentSpan[]): DocumentOutline { + const raw = rootHeadings(maskComponents(body, spans)); + if (raw.length === 0) { + return { + headings: [], + entries: [], + targets: [], + preambleEnd: body.length, + bodyLength: body.length, + }; + } + + const outermostDepth = Math.min(...raw.map((heading) => heading.depth)); + const outermost = raw.filter((heading) => heading.depth === outermostDepth); + const titleIndex = outermost.length === 1 ? raw.indexOf(outermost[0]!) : undefined; + + const headings: OutlineHeading[] = []; + const stack: number[] = []; + for (const [index, heading] of raw.entries()) { + while (stack.length > 0 && raw[stack[stack.length - 1]!]!.depth >= heading.depth) { + stack.pop(); + } + const label = normalizeLabel(heading.text); + headings.push({ + depth: heading.depth, + start: heading.start, + end: heading.end, + parent: stack[stack.length - 1], + addressable: + label.length > 0 && + !overlapsComponent(heading, spans) && + !hasUnescapedInterpolation(body.slice(heading.start, heading.end)), + label, + }); + stack.push(index); + } + + const entries: DocumentTarget[] = []; + for (let index = 0; index < headings.length; index++) { + const labels = pathLabels(headings, index, titleIndex); + if (labels !== undefined) { + entries.push({ + target: labels.map(encodeTargetLabel).join("/"), + labels, + heading: index, + }); + } + } + + return { + headings, + entries, + targets: entries.map((entry) => entry.target), + // The first heading in the root flow, whatever its depth. The preamble is + // what precedes the outline, and a document may open at a deeper level than + // the one that supplies its title — anchoring here to the shallowest + // heading would put an earlier addressable section inside the preamble, + // where every other target would retain and run it. + preambleEnd: raw[0]!.start, + bodyLength: body.length, + }; +} + +/** + * The canonical path a heading is addressed by, or `undefined` when it has + * none. + * + * Every level has to be addressable: a heading under one whose text is not + * static cannot be named, so its subtree is unreachable. The sole title is not + * a level, which is what lets a static section under a computed title stay + * addressable. + */ +function pathLabels( + headings: readonly OutlineHeading[], + index: number, + titleIndex: number | undefined, +): readonly string[] | undefined { + if (index === titleIndex) { + return undefined; + } + const labels: string[] = []; + for (let current: number | undefined = index; current !== undefined; ) { + if (current !== titleIndex) { + if (!headings[current]!.addressable) { + return undefined; + } + labels.unshift(headings[current]!.label); + } + current = headings[current]!.parent; + } + return labels.length === 0 ? undefined : labels; +} + +/** Where a heading's own subtree ends: the next heading at its depth or above. */ +function subtreeEnd(outline: DocumentOutline, index: number): number { + const depth = outline.headings[index]!.depth; + for (let next = index + 1; next < outline.headings.length; next++) { + if (outline.headings[next]!.depth <= depth) { + return outline.headings[next]!.start; + } + } + return outline.bodyLength; +} + +/** An ancestor's own content: its heading through its first child heading. */ +function directPrefixEnd(outline: DocumentOutline, index: number): number { + const child = outline.headings[index + 1]; + if (child !== undefined && child.depth > outline.headings[index]!.depth) { + return child.start; + } + return subtreeEnd(outline, index); +} + +/** + * The original source ranges a target retains: the preamble, each ancestor's + * own content, and the selected subtree. + * + * The ranges are returned in source order and never overlap, so scanning them + * in turn reproduces authored positions exactly. Sibling subtrees fall in the + * gaps between them and are never scanned, which is what keeps their + * components, resources, and code blocks from running at all. + */ +export function retainedRanges( + outline: DocumentOutline, + entry: DocumentTarget, +): readonly SourceRange[] { + const ancestors: number[] = []; + for ( + let current = outline.headings[entry.heading]!.parent; + current !== undefined; + current = outline.headings[current]!.parent + ) { + ancestors.unshift(current); + } + + const ranges: SourceRange[] = [{ start: 0, end: outline.preambleEnd }]; + for (const ancestor of ancestors) { + ranges.push({ + start: outline.headings[ancestor]!.start, + end: directPrefixEnd(outline, ancestor), + }); + } + ranges.push({ + start: outline.headings[entry.heading]!.start, + end: subtreeEnd(outline, entry.heading), + }); + + const retained: SourceRange[] = []; + let consumed = 0; + for (const range of ranges) { + const start = Math.max(range.start, consumed); + if (start < range.end) { + retained.push({ start, end: range.end }); + consumed = range.end; + } + } + return retained; +} diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 8426f37f..c8fd4a83 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -17,7 +17,13 @@ import { durableRun, createDurableOperation, ephemeral, + preserveJournalProvenance, + retainEvents, + StaleInputError, + type CoroutineId, + type DurableEvent, type DurableStream, + type Yield, } from "@executablemd/durable-streams"; import { exec, readTextFile, cwd } from "@executablemd/runtime"; import { cwd as processCwd } from "@effectionx/fs"; @@ -35,7 +41,7 @@ import type { ReturnsSchema, Segment, } from "./types.ts"; -import { parseJson, parseJsonObject } from "./json.ts"; +import { isJsonObject, parseJson, parseJsonObject } from "./json.ts"; import { compilePropsSchema, compileReturnsSchema, @@ -43,7 +49,23 @@ import { validateProps, } from "./validate.ts"; import { useParseCompiler } from "./components/parse-schema.ts"; -import { isFunctionComponentPath, parseMarkdownDefinition } from "./definition.ts"; +import { + documentOutline, + isFunctionComponentPath, + parseMarkdownDefinition, + parseRootMarkdownDefinition, + resolveDocumentTarget, +} from "./definition.ts"; +import { + asDocumentTargetError, + documentTargetError, + documentTargetFailure, + findTarget, + isCanonicalTarget, + recordedDocumentTargetFailure, + sameDocumentTargetFailure, +} from "./document-targets.ts"; +import type { DocumentOutline, DocumentTargetFailure } from "./document-targets.ts"; import { parseReturnsDeclaration } from "./frontmatter.ts"; import { expandSegments, @@ -129,9 +151,40 @@ export type ExecuteOptions = RootDocumentSource & ExecuteSettings; * implementation up again in the scope that is running now. */ type DurableSelection = - | { kind: "repository"; path: string; content: string } + | { kind: "repository"; path: string; content: string; target?: string } + | { kind: "target-failure"; path: string; content: string; failure: TargetFailureRecord } | { kind: "registered"; origin: string; reserved: boolean }; +/** + * A selection that named no single section, as the journal holds it. + * + * A failed selection is an observation of the document, not an accident: the + * text was read, and it does not offer what was asked for. Recording it as data + * — rather than letting the effect fail and keeping only a serialized message — + * is what lets a resumed run tell "the same request, failing the same way" from + * "a different request the recorded run never made", and what lets the failure + * be rebuilt with its fields intact instead of reduced to prose. + * + * `selector` is sanitized invocation metadata. It is never identity: it does + * not occupy the exact-target field, and it never reaches a workflow + * definition. + */ +type TargetFailureRecord = { + kind: string; + selector: string; + matches: string[]; + available: string[]; +}; + +function targetFailureRecord(failure: DocumentTargetFailure): TargetFailureRecord { + return { + kind: failure.kind, + selector: failure.selector, + matches: [...failure.matches], + available: [...failure.available], + }; +} + function* durableImportComponent( name: string, root: RootDocumentSource | undefined, @@ -145,10 +198,30 @@ function* durableImportComponent( // Inside the durable operation, so the journal holds the root's identity // and its text: a replay restores both without reading anything, whether // the source was a file or supplied. + // + // The selector resolves here too, against the text this operation is + // about to record, so the exact target the run executed is part of the + // record rather than something a later read has to rediscover. Only the + // exact target is recorded — a glob describes what the caller asked + // for, not what ran. + const path = rootSourcePath(root); + const content = yield* readRootSource(root); + if (root.target === undefined) { + return { kind: "repository", path, content }; + } + const resolved = resolveDocumentTarget(path, content, root.target); + if (resolved.ok) { + return { kind: "repository", path, content, target: resolved.value }; + } + const failure = asDocumentTargetError(resolved.error); + if (failure === undefined) { + throw resolved.error; + } return { - kind: "repository", - path: rootSourcePath(root), - content: yield* readRootSource(root), + kind: "target-failure", + path, + content, + failure: targetFailureRecord(failure.data), }; } @@ -177,6 +250,17 @@ function* durableImportComponent( }, )) as DurableSelection; + // Rebuilt here rather than carried out of the durable operation, so a replayed + // failed selection and a live one raise the same error with the same fields. + // Parsed rather than trusted: the record is journal data. + if (selection.kind === "target-failure") { + const failure = recordedDocumentTargetFailure(selection.failure); + if (failure === undefined) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + throw documentTargetError(failure); + } + if (selection.kind === "registered") { // The function was never journaled. Find the implementation the recorded // origin names in the registry this run has; refusing when it is gone is @@ -193,7 +277,7 @@ function* durableImportComponent( return found.definition; } - const { path, content } = selection; + const { path, content, target } = selection; // Function component: .ts file — import() the module if (isFunctionComponentPath(path)) { @@ -238,7 +322,13 @@ function* durableImportComponent( return definition; } - // Markdown component: parse at runtime — deterministic from content + // Markdown component: parse at runtime — deterministic from content. + // A recorded target projects the recorded content, so a resumed run executes + // the same section from the same text the first run recorded, whatever the + // file on disk says now. + if (target !== undefined) { + return (yield* ephemeral(parseRootMarkdownDefinition(name, path, content, target))).definition; + } return yield* ephemeral(parseMarkdownDefinition(name, path, content)); } @@ -246,6 +336,420 @@ function isFunctionComponent(value: unknown): value is FunctionComponent { return typeof value === "function"; } +/** + * What one run's selector decided: the whole document, one exact section, or a + * failure that named none. + * + * Selection is compared as an outcome rather than as a target string, because a + * failed selection is an outcome too. Without the third case a journal written + * by one selector that matched nothing would answer a later request for a + * section that does exist. + */ +type SelectionOutcome = + | { kind: "whole" } + | { kind: "exact"; target: string } + | { kind: "failed"; failure: DocumentTargetFailure }; + +/** + * What the fixed diagnostic says when a recorded root import cannot be read, + * and all it says. + * + * Cause-free: the record is journal data, and quoting it back would put + * whatever it holds into a diagnostic. + */ +const UNREADABLE_ROOT_RECORD = "The recorded root document import cannot be read by this version."; + +/** The coroutine a document execution's own terminal result belongs to. */ +const ROOT_COROUTINE = "root"; + +/** + * What a recorded event turned out to be. + * + * "Not the root import" and "the root import, malformed" are deliberately + * different answers. Collapsing them into one absent value is what would let a + * corrupted record fall through to the recorded terminal result, which is the + * failure this distinction exists to prevent. + */ +type RootImportRecord = + | { kind: "unrelated" } + | { kind: "malformed" } + | { kind: "read"; outline: DocumentOutline; selection: SelectionOutcome }; + +const UNRELATED: RootImportRecord = { kind: "unrelated" }; +const MALFORMED: RootImportRecord = { kind: "malformed" }; + +/** + * Read a value that may refuse to be read. + * + * Every value this boundary touches comes from the journal, and a journal is + * data: a property may be an accessor that throws, a key list may come from a + * Proxy that refuses, and content may be markdown whose frontmatter no parser + * accepts. None of those is a failure of this run — they are ways of saying the + * record cannot be read — so none of them may travel as an error of its own. + * + * Synchronous throughout, so nothing an Effection scope owns passes through + * here: this cannot swallow a cancellation or a durability failure, because + * neither can arise inside a synchronous parse. + */ +function attempt(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +/** + * Parse a recorded root import as a closed protocol. + * + * Two selection shapes are supported and nothing else: a repository selection + * with an optional canonical target, and a failed selection with an exact + * failure record. An unknown kind, a missing or mistyped member, an extra + * member, a noncanonical target, and failure data that no selection could have + * produced are each malformed rather than absent. + * + * A result that is not `ok` is left alone. A root import can fail for reasons + * that have nothing to do with selection — an unreadable file — and those + * recorded failures are not this protocol's to interpret. + */ +/** + * A value the journal refused to produce. + * + * Distinct from `undefined`, which is an ordinary absent value. Reading a + * member and finding nothing there, and reading a member that will not say what + * is there, are different facts about a record, and one of them is a refusal: + * conflating them is how "the root import will not say what it settled to" + * became "this is not the root import" and fell through to terminal-result + * reuse. + */ +const UNREADABLE: unique symbol = Symbol("unreadable"); + +/** One read of journal-controlled data: its value, or a refusal. */ +function read(get: () => T): T | typeof UNREADABLE { + try { + return get(); + } catch { + return UNREADABLE; + } +} + +/** The settlements the protocol recognizes as an ordinary failed root import. */ +const SETTLED_FAILURES: readonly string[] = ["err", "cancelled"]; + +function recordedRootImport(event: Yield): RootImportRecord { + // Identification first. An event that will not say what it is cannot be + // claimed as the root import, so it stays unrelated. + const description = read(() => event.description); + if (description === UNREADABLE) { + return UNRELATED; + } + const type = read(() => description.type); + const name = read(() => description.name); + if (type !== "import_component" || name !== "__root__") { + return UNRELATED; + } + + // Identified. From here the event owes this protocol an answer, and every way + // of not giving one is malformed — except the ordinary failed settlement, + // which is a root import that failed for reasons selection knows nothing + // about. + const result = read(() => event.result); + if (result === UNREADABLE || typeof result !== "object" || result === null) { + return MALFORMED; + } + const status = read(() => result.status); + if (status !== "ok") { + return typeof status === "string" && SETTLED_FAILURES.includes(status) ? UNRELATED : MALFORMED; + } + const value = read(() => ("value" in result ? result.value : undefined)); + if (value === UNREADABLE || value === undefined) { + return MALFORMED; + } + return attempt(() => readRootSelection(value)) ?? MALFORMED; +} + +function readRootSelection(value: unknown): RootImportRecord { + // Parsed rather than read in place. `parseJson` walks every property once and + // rebuilds the record, so a trap that throws or a value that is not JSON is + // discovered here — and every read below is of this run's own copy rather + // than of an object the journal still controls. + const record = parseJson(value); + if (!isJsonObject(record)) { + return MALFORMED; + } + const content = record["content"]; + const path = record["path"]; + if (typeof content !== "string" || typeof path !== "string") { + return MALFORMED; + } + const kind = record["kind"]; + const members = Object.keys(record).length; + // Parsing the recorded content is part of reading the record, for every + // shape. It is what the verification below compares against, and doing it + // here means a later read of the same content cannot be the first to + // discover that it does not parse. + const outline = documentOutline(path, content); + + if (kind === "repository") { + const target = record["target"]; + if (target === undefined) { + return members === 3 ? { kind: "read", outline, selection: { kind: "whole" } } : MALFORMED; + } + if (members !== 4 || typeof target !== "string" || !isCanonicalTarget(target)) { + return MALFORMED; + } + // The recorded content is here, so the target is verified against it rather + // than merely parsed: a well-formed target the recorded document does not + // offer describes a selection that never happened. + const resolved = findTarget(outline, target); + if (!resolved.ok || resolved.value.target !== target) { + return MALFORMED; + } + return { kind: "read", outline, selection: { kind: "exact", target } }; + } + + if (kind === "target-failure") { + const failure = recordedDocumentTargetFailure(record["failure"]); + if (members !== 4 || failure === undefined) { + return MALFORMED; + } + // Same standard for a failure: the recorded selector must fail against the + // recorded content in exactly the way the record claims. That verifies the + // catalog and the matches too, which no amount of shape checking could. + const rederived = findTarget(outline, failure.selector); + if (rederived.ok) { + return MALFORMED; + } + const actual = asDocumentTargetError(rederived.error); + if (actual === undefined || !sameDocumentTargetFailure(actual.data, failure)) { + return MALFORMED; + } + return { kind: "read", outline, selection: { kind: "failed", failure } }; + } + + return MALFORMED; +} + +/** + * What this run's selector decides against the outline the journal recorded. + * + * Takes the outline the record already produced rather than the content, so + * this cannot be the call that discovers unparseable recorded markdown — that + * discovery belongs to reading the record, where it is malformed rather than an + * error of this run's own. + */ +function requestedSelection(root: RootDocumentSource, outline: DocumentOutline): SelectionOutcome { + if (root.target === undefined) { + return { kind: "whole" }; + } + const found = findTarget(outline, root.target); + if (found.ok) { + return { kind: "exact", target: found.value.target }; + } + const failure = asDocumentTargetError(found.error); + // A failure this module did not build is not a selection outcome that can be + // compared, so it cannot be shown compatible with anything. + return failure === undefined + ? { kind: "failed", failure: documentTargetFailure("invalid-selector", root.target, [], []) } + : { kind: "failed", failure: failure.data }; +} + +function sameSelection(recorded: SelectionOutcome, requested: SelectionOutcome): boolean { + if (recorded.kind === "whole" || requested.kind === "whole") { + return recorded.kind === requested.kind; + } + if (recorded.kind === "exact" || requested.kind === "exact") { + return ( + recorded.kind === "exact" && + requested.kind === "exact" && + recorded.target === requested.target + ); + } + return sameDocumentTargetFailure(recorded.failure, requested.failure); +} + +function describeSelection(selection: SelectionOutcome): string { + switch (selection.kind) { + case "whole": + return "the whole document"; + case "exact": + return `the target ${JSON.stringify(selection.target)}`; + case "failed": + return `a selector that names no single target (${selection.failure.kind})`; + } +} + +/** + * The journal a document execution reads and appends through, with the + * definition-identity check that a resumed run must pass built into the read. + * + * **This authority is not middleware.** Exact canonical target is + * workflow-definition identity, and identity may not be decided by anything a + * document, a component, or an enclosing scope can replace. A public + * `ReplayGuard` handler installed further out can decline to call `next`, which + * is exactly what composable policy is allowed to do — and exactly why the + * comparison cannot live there. Here it is a step inside `readAll`, owned by + * the execution, reachable through no context and replaceable by nothing. + * + * It runs where a journal first becomes readable, so it is ahead of everything + * a wrong answer could reach: public guard policy, any retained Yield reaching + * execution, a retained Close being reused, authored work, and any append. + * + * It also owns the retained snapshot. The events it validates are the events it + * returns, so the identity and settlement it decided on are what every later + * phase observes rather than a second reading of the backend's own objects. + * + * It is a trusted wrapping site, and says so explicitly. Journal provenance is + * not transitive: a wrapper is unproven unless a wrapping site carries its + * source's witness onto it, and a run whose journal is unproven is refused by a + * Workspace provider before any transaction. This wrapper qualifies because + * core installs it before any document code exists and it delegates every + * append to the exact stream it was handed. What it transfers is only the + * witness that exact source already has — it establishes none, so an unproven + * source stays unproven and a wrapper somebody else built gains nothing. + */ +function guardedJournal( + stream: DurableStream, + root: RootDocumentSource, + coroutineId: CoroutineId, +): DurableStream { + const admitting: DurableStream = { + *readAll(): Operation { + const retained = retainEvents(yield* stream.readAll()); + admitRootHistory(retained, root, coroutineId); + return retained; + }, + append: (event: DurableEvent) => stream.append(event), + }; + return preserveJournalProvenance(stream, admitting); +} + +/** + * Decide whether this run may replay the history it was handed. + * + * Synchronous and total over journal-provided values: every way the history can + * refuse to be read is the one fixed diagnostic, and the retained events it + * reads are the ones the caller keeps. + */ +function admitRootHistory( + retained: readonly DurableEvent[], + root: RootDocumentSource, + coroutineId: CoroutineId, +): void { + const imports: Yield[] = []; + let terminal = false; + for (const event of retained) { + // The retained history has already settled every discriminator, so one that + // still refuses is a history this run cannot describe — not an event to + // skip past on the way to reusing a terminal result. + const kind = attempt(() => event.type); + if (kind === undefined) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + if (kind === "close") { + // Any recorded completion at all, not only this coroutine's. A Close + // means some coroutine of this document execution finished, and every + // coroutine it has exists because the root document was imported — so a + // history holding one while the import that authorized it is absent + // describes a run that never happened, whichever coroutine the Close + // claims to belong to. + // + // Its result is recognized here as well as its coroutine. The retained + // history has already settled both, and forcing them now is what makes a + // refusal this run's fixed diagnostic rather than a failure surfacing + // later, out of some other phase's hands. + if (attempt(() => event.coroutineId) === undefined) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + if (attempt(() => event.result) === undefined) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + terminal = true; + continue; + } + if (event.type !== "yield") { + continue; + } + if (isRootImport(event)) { + imports.push(event); + } + } + + // A journal that recorded no completion replays what it has and then + // continues live, so a root import it does not contain is one this run + // performs. A journal that recorded one is standing behind a selection: a + // history that recorded no import, recorded two, or recorded one belonging to + // some other coroutine establishes nothing for this coroutine to stand + // behind. + if (terminal) { + const owned = imports.filter((event) => attempt(() => event.coroutineId) === coroutineId); + if (imports.length !== 1 || owned.length !== 1) { + throw new Error(UNREADABLE_ROOT_RECORD); + } + } + + for (const event of imports) { + admitRootSelection(event, root); + } +} + +/** Whether a retained event is recognizably the root import. */ +function isRootImport(event: DurableEvent): boolean { + return ( + attempt( + () => + event.type === "yield" && + event.description.type === "import_component" && + event.description.name === "__root__", + ) === true + ); +} + +/** + * Hold a resumed run to the selection its journal recorded. + * + * Only `type` and `name` decide whether a journal entry matches, and the root + * import's name is the same for every selector — so without this, resuming with + * a different one would restore the recorded content and then project a section + * the recorded run never executed, or restore a recorded selection failure as + * the answer to a request that would have succeeded. + * + * The current selector is resolved against the *recorded* content, so what is + * compared is what each run decided, not what each caller typed: a different + * glob naming the same section replays, and so does the same failing selector, + * while any difference in outcome is stale input. + * + * A recorded failed selection is reproduced here, not left for later. Nothing + * later would reproduce it with its fields intact — `durableRun` reuses a + * recorded root Close before any effect is replayed, and that path restores a + * deserialized error — so a recorded failure is rebuilt from its structural + * record and raised ahead of that reuse. Either way no authored effect runs. + * + * Partial histories are held to the same rule: a recorded root import that + * names another section is refused before replay continues into it. + */ +function admitRootSelection(event: Yield, root: RootDocumentSource): void { + const recorded = recordedRootImport(event); + if (recorded.kind === "unrelated") { + return; + } + if (recorded.kind === "malformed") { + throw new Error(UNREADABLE_ROOT_RECORD); + } + const requested = requestedSelection(root, recorded.outline); + if (!sameSelection(recorded.selection, requested)) { + throw new StaleInputError( + `the recorded root document import ran ${describeSelection(recorded.selection)}, and ` + + `this run asks for ${describeSelection(requested)}. Re-run the document from the ` + + "start rather than resuming from a journal that recorded another selection.", + { coroutineId: event.coroutineId, description: event.description }, + ); + } + if (recorded.selection.kind === "failed") { + throw documentTargetError(recorded.selection.failure); + } +} + const execFactory: ModifierFactory = (_params) => (_args, _next) => (function* () { const context = yield* useCodeBlock(); @@ -871,8 +1375,11 @@ function* executeDocument(options: ExecuteOptions): Operation // execution that owns it. const journal = yield* useSecretDetection(secretDetection, stream); + // The journal is wrapped before it reaches `durableRun`, so the identity + // check happens inside the read that every phase downstream depends on + // rather than in middleware anything could replace. const returned = yield* durableRun(() => Execution.operations.document(props), { - stream: journal, + stream: guardedJournal(journal, root, ROOT_COROUTINE), }); // Taken rather than read, so the handoff belongs to the run that made it. const live = yield* takeLiveFailure(liveFailure); diff --git a/packages/core/src/inspect.ts b/packages/core/src/inspect.ts index bf3617f9..941669c4 100644 --- a/packages/core/src/inspect.ts +++ b/packages/core/src/inspect.ts @@ -2,7 +2,11 @@ import type { Operation } from "effection"; import { readTextFile } from "@executablemd/runtime"; import type { ComponentOrigin, PropsSchema, ReturnsSchema } from "./types.ts"; -import { isFunctionComponentPath, parseMarkdownDefinition } from "./definition.ts"; +import { + isFunctionComponentPath, + parseMarkdownDefinition, + parseRootMarkdownDefinition, +} from "./definition.ts"; import { Component } from "./component-api.ts"; import { selectComponent } from "./components/select.ts"; import { readRootSource, rootSourcePath } from "./root-source.ts"; @@ -38,6 +42,22 @@ export interface DocumentInfo { * the default, so the mode — not the schema — tells the two apart. */ returnMode: "text" | "value"; + + /** + * Every target the document addresses, as canonical encoded fragments without + * the document path or a leading `#`, in document order. + * + * Duplicates are retained: two sections that canonicalize to the same path + * are an ambiguity a caller can see rather than one a selector resolves + * arbitrarily. + */ + readonly targets: readonly string[]; + + /** + * The exact canonical target the requested selector resolved to. Present only + * when a target was requested and resolved; it is never the caller's glob. + */ + readonly target?: string; } /** @@ -47,6 +67,10 @@ export interface DocumentInfo { * validation as execution, but never expands the document, evaluates a * code block, imports a body component, starts an agent, or creates a * journal — so describing a document is always free of its effects. + * + * Target discovery and selection happen here too. A requested selector that + * names no section, or several, fails as a `DocumentTargetError` — before + * anything is expanded, and without a journal ever existing. */ export function* inspectDocument(options: InspectOptions): Operation { const path = rootSourcePath(options); @@ -58,7 +82,8 @@ export function* inspectDocument(options: InspectOptions): Operation` identity. */ -export function inlineSource(source: string): InlineRootDocument { - return { path: INLINE_SOURCE_PATH, source }; +export function inlineSource( + source: string, + options?: { readonly target?: string }, +): InlineRootDocument { + const target = options?.target; + return { + path: INLINE_SOURCE_PATH, + source, + ...(target === undefined ? {} : { target }), + }; +} + +/** + * A file root document from a URI-style document reference. + * + * The reference is `#`, split at the + * first raw `#`. The path is percent-decoded; the fragment is not, because + * `%2F` has to stay distinguishable from the raw `/` that separates target + * levels — the selector parser splits the hierarchy and operator syntax first + * and decodes only the literal chunks between them. + * + * A reference that cannot be read fails with a `TypeError` carrying nothing but + * fixed wording: the input is a command-line argument, and echoing it back + * would put arbitrary bytes into a diagnostic. + * + * A filename containing `#` is written `%23`, and one containing a literal + * `%HH` sequence is written `%25HH`. + */ +export function fileSource(reference: string): FileRootDocument { + const fragment = reference.indexOf("#"); + const encodedPath = fragment === -1 ? reference : reference.slice(0, fragment); + const path = decodePercentEncoded(encodedPath); + if (path === undefined || path.length === 0) { + throw new TypeError(INVALID_REFERENCE); + } + return fragment === -1 ? { path } : { path, target: reference.slice(fragment + 1) }; +} + +/** + * The canonical reference for a document, and optionally one exact target + * inside it. + * + * The path arrives decoded and is encoded here; the target arrives already + * canonical — `DocumentInfo.target`, or a stored workflow definition's — and is + * validated rather than encoded again, so a canonical `%2F` is never turned + * into `%252F`. Making an authored glob canonical is the selector parser's job, + * not this one's. + * + * This is the one formatter diagnostics, command output, and workflow handoff + * use, so a reference printed by one of them is a reference the others accept. + */ +export function formatDocumentReference(path: string, target?: string): string { + if (path.length === 0) { + throw new TypeError(INVALID_REFERENCE); + } + // The round trip is the rule, not a sample of it: a path only formats when + // decoding what this would write reproduces it exactly. NUL, which the + // decoder refuses, and an unpaired surrogate, which encodes lossily to the + // replacement character, both fail here rather than producing a reference + // that names a different file than the one asked about. + const encoded = encodeDocumentPath(path); + if (decodePercentEncoded(encoded) !== path) { + throw new TypeError(INVALID_REFERENCE); + } + if (target === undefined) { + return encoded; + } + if (!isCanonicalTarget(target)) { + throw new TypeError(INVALID_REFERENCE); + } + return `${encoded}#${target}`; } /** The identity printed errors and source positions report for this root. */ diff --git a/packages/core/src/scanner.ts b/packages/core/src/scanner.ts index 71e83bef..6f07ff48 100644 --- a/packages/core/src/scanner.ts +++ b/packages/core/src/scanner.ts @@ -112,6 +112,19 @@ function positionAt(index: PositionIndex, offset: number): SourcePosition { }; } +/** + * Where one top-level component invocation begins and ends in the scanned text. + * + * Half-open, relative to the text handed to the scanner. Internal: this is the + * scanner's own boundary decision, recorded so heading discovery can blank the + * regions this scanner owns before a Markdown parser looks at them. It is not a + * public authority over what a component is. + */ +export interface ComponentSpan { + readonly start: number; + readonly end: number; +} + /** * Scan raw markdown text into segments. * @@ -120,8 +133,16 @@ function positionAt(index: PositionIndex, offset: number): SourcePosition { * * When `origin` is provided, component invocations carry `position` values * expressed in the original file's coordinates. + * + * When `spans` is provided, each top-level component invocation records its + * source span into it, in source order. Collecting them changes nothing about + * the segments produced. */ -export function scanSegments(text: string, origin?: SourceOrigin): Segment[] { +export function scanSegments( + text: string, + origin?: SourceOrigin, + spans?: ComponentSpan[], +): Segment[] { const index: PositionIndex = { origin, lineStarts: computeLineStarts(text) }; const segments: Segment[] = []; let pos = 0; @@ -184,6 +205,7 @@ export function scanSegments(text: string, origin?: SourceOrigin): Segment[] { pushText(segments, text.slice(textStart, pos)); } segments.push(component.segment); + spans?.push({ start: pos, end: component.end }); pos = component.end; textStart = pos; continue; @@ -201,6 +223,13 @@ export function scanSegments(text: string, origin?: SourceOrigin): Segment[] { return segments; } +/** The source spans of the top-level component invocations in `text`. */ +export function scanComponentSpans(text: string): ComponentSpan[] { + const spans: ComponentSpan[] = []; + scanSegments(text, undefined, spans); + return spans; +} + interface FenceOpen { fenceChar: string; fenceLen: number; diff --git a/packages/core/tests/document-target-execution.test.ts b/packages/core/tests/document-target-execution.test.ts new file mode 100644 index 00000000..32d7d31a --- /dev/null +++ b/packages/core/tests/document-target-execution.test.ts @@ -0,0 +1,2209 @@ +/** + * Tier TX — targeted document execution and replay (spec §5.4, §6.11). + * + * A projected document is not a rendering exercise: what it must prove is that + * a skipped sibling *did not run*, that a retained element kept the identity it + * has in a full run, and that a journal recorded against one section cannot be + * resumed as another. + * + * Every "did not run" assertion is made from a component that records its own + * invocation, not from absent text — text can be absent because it rendered + * empty. Every identity assertion reads the expansion ID the engine derived, + * not a position that merely looks unchanged. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped, until } from "effection"; +import type { Operation } from "effection"; +import { rm, writeTextFile } from "@effectionx/fs"; +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import { StaleInputError } from "@executablemd/durable-streams"; +import { + createDurableOperation, + defaultLiveDurableOperationCoordinator, + establishJournalProvenance, + guardDurableStream, + ReplayGuard, +} from "@executablemd/durable-streams"; +import type { + JournalProvenance, + LiveDurableOperationCoordinator, +} from "@executablemd/durable-streams"; +import type { DurableEvent, DurableStream, Yield } from "@executablemd/durable-streams"; +import { createApi } from "@effectionx/context-api"; +import { API, useHostFiles } from "@executablemd/runtime"; + +import { collect } from "../src/collect.ts"; +import { useTempFileCompiler } from "../src/temp-file-compiler.ts"; +import { execute, Execution } from "../src/execute.ts"; +import { inspectDocument } from "../src/inspect.ts"; +import { getExpansion } from "../src/expansion.ts"; +import { registerComponents } from "../src/components/registration.ts"; +import { + asDocumentTargetError, + DocumentTargetError, + isDocumentTargetError, +} from "../src/document-targets.ts"; +import { isJsonObject, parseJson } from "../src/json.ts"; +import { fileSource, formatDocumentReference, inlineSource } from "../src/root-source.ts"; +import type { RootDocumentSource } from "../src/root-source.ts"; +import type { Json } from "../src/types.ts"; +import { asText } from "./helpers.ts"; + +/** What every `` in a run reported, in the order it expanded. */ +interface Probes { + names: string[]; + ids: string[]; +} + +/** + * `` — proof of expansion. + * + * A component that records its own invocation and its expansion ID. Absent + * output would not distinguish "skipped" from "rendered nothing"; an absent + * entry here can only mean the element never expanded. + */ +function* useProbes(seen: Probes): Operation { + yield* registerComponents([ + { + name: "Probe", + origin: "tier-tx", + props: { type: "object", properties: { name: { type: "string" } } }, + *fn(props) { + const name = props["name"]; + seen.names.push(typeof name === "string" ? name : "?"); + seen.ids.push((yield* getExpansion()).id); + return `[${typeof name === "string" ? name : "?"}]`; + }, + }, + ]); +} + +/** A directory the contextual cwd points at, removed when the test ends. */ +function* useWorkspace(files: Record): Operation { + const root = yield* until(mkdtemp(join(tmpdir(), "xmd-targets-"))); + yield* ensure(() => rm(root, { recursive: true, force: true })); + for (const [name, content] of Object.entries(files)) { + yield* writeTextFile(join(root, name), content); + } + yield* API.Env.around( + { + *cwd() { + return root; + }, + }, + { at: "min" }, + ); + yield* useHostFiles(); + return root; +} + +function rootImports(stream: InMemoryStream) { + return stream + .snapshot() + .flatMap((event) => + event.type === "yield" && + event.description.type === "import_component" && + event.description.name === "__root__" + ? [event] + : [], + ); +} + +function closes(stream: InMemoryStream) { + return stream.snapshot().filter((event) => event.type === "close"); +} + +/** Run a document and report both its text and what expanded. */ +function run(root: RootDocumentSource, stream: InMemoryStream, seen: Probes): Operation { + return scoped(function* () { + yield* useProbes(seen); + return asText(yield* collect(yield* execute({ ...root, stream }))); + }); +} + +/** The failure a run produced, refusing to pass a success off as one. */ +function* failure( + root: RootDocumentSource, + stream: InMemoryStream, + seen: Probes = { names: [], ids: [] }, +): Operation { + try { + yield* run(root, stream, seen); + } catch (error) { + return error; + } + throw new Error("the run completed instead of failing"); +} + +const SECTIONS = [ + 'preamble ', + "", + "# Title", + "", + 'title content ', + "", + "## Alpha", + "", + 'alpha content ', + "", + "### Inner", + "", + 'inner content ', + "", + "## Beta", + "", + 'beta content ', + "", + "```sh exec", + "echo beta-ran", + "```", + "", +].join("\n"); + +describe("Tier TX — targeted execution", () => { + it("TX1: only the preamble, the ancestors, and the subtree expand", function* () { + const seen: Probes = { names: [], ids: [] }; + const text = yield* run( + inlineSource(SECTIONS, { target: "Alpha/Inner" }), + new InMemoryStream(), + seen, + ); + + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + expect(text).toContain("# Title"); + expect(text).toContain("## Alpha"); + expect(text).toContain("### Inner"); + expect(text).not.toContain("## Beta"); + }); + + it("TX2: a skipped sibling's components and code blocks never run", function* () { + const seen: Probes = { names: [], ids: [] }; + yield* scoped(function* () { + yield* API.Process.around({ + *exec([options], _next) { + throw new Error(`a skipped code block ran: ${JSON.stringify(options.command)}`); + }, + }); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), new InMemoryStream(), seen); + }); + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + }); + + it("TX3: selecting a non-leaf expands every descendant", function* () { + const seen: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), new InMemoryStream(), seen); + expect(seen.names).toContain("inner"); + }); + + it("TX4: a retained element keeps the expansion ID it has in a full run", function* () { + const whole: Probes = { names: [], ids: [] }; + const targeted: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS), new InMemoryStream(), whole); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), new InMemoryStream(), targeted); + + const idOf = (probes: Probes, name: string) => probes.ids[probes.names.indexOf(name)]; + expect(targeted.names).toEqual(["pre", "title", "beta"]); + expect(idOf(targeted, "beta")).toBe(idOf(whole, "beta")); + expect(idOf(targeted, "pre")).toBe(idOf(whole, "pre")); + }); + + /** + * The identifier is derived from position, not from what ran. Two targets + * that retain the same element therefore agree with each other and with the + * full run — and seeding identity with the target string would break all + * three at once. + */ + it("TX5: two different targets agree on a shared retained element", function* () { + const alpha: Probes = { names: [], ids: [] }; + const beta: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), new InMemoryStream(), alpha); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), new InMemoryStream(), beta); + expect(beta.ids[beta.names.indexOf("title")]).toBe(alpha.ids[alpha.names.indexOf("title")]); + }); + + it("TX6: a file root and an inline root behave identically", function* () { + const workspace = yield* useWorkspace({ "doc.md": SECTIONS }); + const fromFile: Probes = { names: [], ids: [] }; + const fromText: Probes = { names: [], ids: [] }; + const fileText = yield* run( + fileSource(formatDocumentReference(join(workspace, "doc.md"), "Beta")), + new InMemoryStream(), + fromFile, + ); + const inlineText = yield* run( + inlineSource(SECTIONS, { target: "Beta" }), + new InMemoryStream(), + fromText, + ); + expect(fromFile.names).toEqual(fromText.names); + expect(fileText).toBe(inlineText); + }); + + it("TX7: root props and frontmatter apply to the projected body", function* () { + const body = [ + "---", + "title: Doc", + "props:", + " who:", + " type: string", + "---", + "", + "# {meta.title}", + "", + "## Greeting", + "", + "hello {props.who}", + "", + "## Skipped", + "", + "skipped {props.who}", + "", + ].join("\n"); + const text = yield* scoped(function* () { + const execution = yield* execute({ + ...inlineSource(body, { target: "Greeting" }), + stream: new InMemoryStream(), + props: { who: "world" }, + }); + return asText(yield* collect(execution)); + }); + expect(text).toContain("# Doc"); + expect(text).toContain("hello world"); + expect(text).not.toContain("skipped"); + }); + + it("TX8: a value root returns from the projected body", function* () { + const body = [ + "---", + "returns:", + " type: object", + " properties:", + " picked:", + " type: string", + "---", + "", + "# Title", + "", + "## Kept", + "", + '', + "", + ].join("\n"); + const value = yield* collect( + yield* execute({ + ...inlineSource(body, { target: "Kept" }), + stream: new InMemoryStream(), + }), + ); + expect(value).toEqual({ picked: "kept" }); + }); + + it("TX9: `` in the projected body selects what is emitted", function* () { + const body = [ + "# Title", + "", + "title text", + "", + "## Kept", + "", + "", + "chosen", + "", + "", + "not chosen", + "", + ].join("\n"); + const text = asText( + yield* collect( + yield* execute({ + ...inlineSource(body, { target: "Kept" }), + stream: new InMemoryStream(), + }), + ), + ); + expect(text.trim()).toBe("chosen"); + }); + + /** + * Structural preflight applies to the projected body, so a violation the + * caller did not select is not a violation of what runs. + */ + it("TX10: an invalid structure in a skipped sibling is irrelevant", function* () { + const body = [ + "# Title", + "", + "## Kept", + "", + 'kept body ', + "", + "## Broken", + "", + '', + "", + ].join("\n"); + const seen: Probes = { names: [], ids: [] }; + const text = yield* run(inlineSource(body, { target: "Kept" }), new InMemoryStream(), seen); + expect(seen.names).toEqual(["kept"]); + expect(text).toContain("kept body"); + expect(text).not.toContain(" requires"); + }); + + it("TX11: an invalid structure in the retained range fails before any effect", function* () { + const body = [ + "# Title", + "", + '', + "", + "## Kept", + "", + '', + "", + ].join("\n"); + const seen: Probes = { names: [], ids: [] }; + const text = yield* run(inlineSource(body, { target: "Kept" }), new InMemoryStream(), seen); + expect(seen.names).toEqual([]); + expect(text).toContain(" requires"); + }); + + /** + * Resolution sits inside the durable root import, so a target failure travels + * out of `execute()` the way every failure crossing that boundary does: by + * name and message, its class left behind with the journal round trip. The + * typed `DocumentTargetError` is what `inspectDocument()` reports, and + * inspection is where a host resolves a selector before running anything. + */ + it("TX12: a target that resolves to nothing runs no authored effect", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream, seen); + expect(isDocumentTargetError(error)).toBe(true); + expect(asDocumentTargetError(error)?.data).toMatchObject({ + kind: "no-match", + selector: "Missing", + matches: [], + }); + expect(seen.names).toEqual([]); + // The selection was recorded as an observation, and the effect succeeded: + // what failed is the document, deterministically, from that record. + expect(rootImports(stream).map((event) => event.result.status)).toEqual(["ok"]); + expect(stream.snapshot().filter((event) => event.type === "yield").length).toBe(1); + }); + + it("TX13: an ambiguous target runs no authored effect", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream, seen); + expect(asDocumentTargetError(error)?.data.kind).toBe("multiple-matches"); + expect(seen.names).toEqual([]); + expect(stream.snapshot().filter((event) => event.type === "yield").length).toBe(1); + }); + + it("TX14: inspection resolves a target without expanding a component", function* () { + const seen: Probes = { names: [], ids: [] }; + const info = yield* scoped(function* () { + yield* useProbes(seen); + return yield* inspectDocument(inlineSource(SECTIONS, { target: "**/Inner" })); + }); + expect(info.target).toBe("Alpha/Inner"); + expect(seen.names).toEqual([]); + }); +}); + +describe("Tier TX — targeted replay", () => { + it("TX15: the journal records the exact target, never the glob", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + yield* run(inlineSource(SECTIONS, { target: "**/I*" }), stream, seen); + + const imports = rootImports(stream); + expect(imports.length).toBe(1); + expect(imports[0]).toMatchObject({ + result: { status: "ok", value: { target: "Alpha/Inner" } }, + }); + }); + + it("TX16: an untargeted run records no target member at all", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + const recorded = rootImports(stream)[0]; + expect(recorded?.result.status).toBe("ok"); + const value = recorded?.result.status === "ok" ? recorded.result.value : undefined; + expect(value !== null && typeof value === "object" && "target" in value).toBe(false); + }); + + it("TX17: a different selector naming the same section replays", function* () { + const stream = new InMemoryStream(); + const first: Probes = { names: [], ids: [] }; + const golden = yield* run(inlineSource(SECTIONS, { target: "Alpha/Inner" }), stream, first); + + const second: Probes = { names: [], ids: [] }; + const replayed = yield* run(inlineSource(SECTIONS, { target: "**/I*" }), stream, second); + + expect(replayed).toBe(golden); + expect(rootImports(stream).length).toBe(1); + }); + + /** + * The reuse this has to beat is the root Close, which `durableRun` honours + * before any effect is replayed. Validating in the decide phase alone would + * leave a completed journal answering for a section it never ran. + */ + it("TX18: a different exact target refuses to reuse a completed journal", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { names: [], ids: [] }); + expect(closes(stream).length).toBeGreaterThan(0); + + const error = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect(error).toBeInstanceOf(StaleInputError); + expect((error as Error).message).toContain("Alpha"); + expect((error as Error).message).toContain("Beta"); + }); + + it("TX19: an untargeted request refuses a targeted journal", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), stream, { names: [], ids: [] }); + + const error = yield* failure(inlineSource(SECTIONS), stream); + expect(error).toBeInstanceOf(StaleInputError); + expect((error as Error).message).toContain("the whole document"); + }); + + it("TX20: a targeted request refuses an untargeted journal", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + + const error = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect(error).toBeInstanceOf(StaleInputError); + }); + + it("TX21: a selector the recorded content no longer resolves fails stale", function* () { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { names: [], ids: [] }); + + // "**" is ambiguous against the recorded content, so this run cannot show + // that it means the recorded section. + const error = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream); + expect(error).toBeInstanceOf(StaleInputError); + // Stale input is what this is, and it carries nothing else: the guard + // retains no failure object from the selection it could not match. + expect((error as Error).cause).toBe(undefined); + expect(isDocumentTargetError(error)).toBe(false); + }); + + /** + * The defect this stack shipped first, and the reason a failed selection is + * recorded structurally rather than left to the effect's own failure. + * + * `Missing` matched nothing, so the run failed and `durableRun` closed the + * root. A later request for `Good` — which the document really offers — was + * then answered with the recorded `Missing` error, because the guard + * delegated past every `err` result and the completed Close short-circuited + * everything after it. + */ + it("TX24: a journal from a failed selector never answers a valid one", function* () { + const stream = new InMemoryStream(); + const first = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + expect(asDocumentTargetError(first)?.data.selector).toBe("Missing"); + + const second = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect(second).toBeInstanceOf(StaleInputError); + expect((second as Error).message).not.toContain("Missing"); + }); + + it("TX25: the same failing selector replays its own recorded failure", function* () { + const stream = new InMemoryStream(); + const seen: Probes = { names: [], ids: [] }; + const first = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + const replayed = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream, seen); + + expect(isDocumentTargetError(replayed)).toBe(true); + expect(asDocumentTargetError(replayed)?.data).toEqual(asDocumentTargetError(first)?.data); + expect(seen.names).toEqual([]); + // One recorded import, and it is the first run's. + expect(rootImports(stream).length).toBe(1); + }); + + it("TX26: one failed selection never answers for another kind of failure", function* () { + const stream = new InMemoryStream(); + yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + + // Ambiguous rather than unmatched: a different outcome, not a different + // spelling of the same one. + const ambiguous = yield* failure(inlineSource(SECTIONS, { target: "**" }), stream); + expect(ambiguous).toBeInstanceOf(StaleInputError); + + // Invalid syntax rather than unmatched. + const invalid = yield* failure(inlineSource(SECTIONS, { target: "/bad" }), stream); + expect(invalid).toBeInstanceOf(StaleInputError); + + // A different selector that also matches nothing is still a different + // request, and the recorded failure describes the one that was made. + const other = yield* failure(inlineSource(SECTIONS, { target: "AlsoMissing" }), stream); + expect(other).toBeInstanceOf(StaleInputError); + }); + + it("TX27: live and replayed selection failures are the same structural error", function* () { + const stream = new InMemoryStream(); + const live = yield* failure(inlineSource(SECTIONS, { target: "**/N*/Deep" }), stream); + const replayed = yield* failure(inlineSource(SECTIONS, { target: "**/N*/Deep" }), stream); + + for (const error of [live, replayed]) { + expect(isDocumentTargetError(error)).toBe(true); + expect(asDocumentTargetError(error)?.data.selector).toBe("**/N*/Deep"); + expect(asDocumentTargetError(error)?.data.available).toEqual([ + "Alpha", + "Alpha/Inner", + "Beta", + ]); + } + }); + + it("TX22: an untargeted journal still replays for an untargeted run", function* () { + const stream = new InMemoryStream(); + const golden = yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + const replayed = yield* run(inlineSource(SECTIONS), stream, { names: [], ids: [] }); + expect(replayed).toBe(golden); + expect(rootImports(stream).length).toBe(1); + }); + + /** + * A replay projects the text the journal holds. Rewriting the file between + * runs would change which section a re-resolved selector names if the current + * copy were consulted; it does not, so the replayed output is the first run's. + */ + it("TX23: replay projects the recorded content, not the file on disk", function* () { + const workspace = yield* useWorkspace({ "doc.md": SECTIONS }); + const reference = formatDocumentReference(join(workspace, "doc.md"), "Beta"); + const stream = new InMemoryStream(); + const golden = yield* run(fileSource(reference), stream, { names: [], ids: [] }); + + yield* writeTextFile( + join(workspace, "doc.md"), + ["# Title", "", "## Beta", "", "rewritten beta", ""].join("\n"), + ); + + const replayed = yield* run(fileSource(reference), stream, { names: [], ids: [] }); + expect(replayed).toBe(golden); + expect(replayed).not.toContain("rewritten beta"); + }); +}); + +/** + * Tier TX — a corrupted root-import record fails closed. + * + * "Not the root import" and "the root import, malformed" have to be different + * answers. A boundary that returns one absent value for both delegates a + * corrupted record onward, and `durableRun` then reuses the recorded terminal + * result — replaying a failure or a success the record no longer describes. + * + * Every case here starts from a *valid* completed journal and corrupts only the + * recorded selection, so what is being measured is the parse and nothing else. + * Each is resumed twice: once with the selector that produced the journal, once + * with a selector that would succeed against a healthy record. Both must be + * refused with the fixed diagnostic, and neither may expand anything or append + * history. + */ +/** The one thing an unreadable recorded root import says. */ +const UNREADABLE_RECORD = "The recorded root document import cannot be read by this version."; + +describe("Tier TX — malformed recorded selections", () => { + const UNREADABLE = UNREADABLE_RECORD; + + /** A completed journal whose root import recorded a failed selection. */ + function* failedJournal(): Operation { + const stream = new InMemoryStream(); + yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream); + return stream; + } + + /** Rewrite the recorded root-import selection, keeping everything else. */ + function* corrupt( + stream: InMemoryStream, + change: (record: Record) => Record, + ): Operation { + const corrupted = new InMemoryStream(); + for (const event of stream.snapshot()) { + const record = + event.type === "yield" && + event.description.name === "__root__" && + event.result.status === "ok" + ? event.result.value + : undefined; + if (event.type === "yield" && isJsonObject(record)) { + yield* corrupted.append({ + ...event, + result: { status: "ok", value: parseJson(change({ ...record })) }, + }); + continue; + } + yield* corrupted.append(event); + } + return corrupted; + } + + /** Every way a corrupted record must be refused, resumed both ways. */ + function* refuses( + change: (record: Record) => Record, + ): Operation { + const healthy = yield* failedJournal(); + const before = (yield* corrupt(healthy, change)).snapshot().length; + + for (const target of ["Missing", "Beta"]) { + const stream = yield* corrupt(healthy, change); + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target }), stream, seen); + + expect((error as Error).message).toBe(UNREADABLE); + expect((error as Error).cause).toBe(undefined); + // Not the recorded failure, and not a document-target failure at all. + expect(isDocumentTargetError(error)).toBe(false); + expect((error as Error).message).not.toContain("Missing"); + // Nothing expanded, and no history was appended on top of the corruption. + expect(seen.names).toEqual([]); + expect(stream.snapshot().length).toBe(before); + } + } + + it("TX28: a missing or non-array `available` is refused", function* () { + yield* refuses((record) => ({ + ...record, + failure: omit(record["failure"], "available"), + })); + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), available: "Beta" }, + })); + }); + + it("TX29: an unknown selection kind is refused", function* () { + yield* refuses((record) => ({ ...record, kind: "something-else" })); + yield* refuses((record) => omit(record, "kind")); + }); + + it("TX30: extra data in the record or the failure is refused", function* () { + yield* refuses((record) => ({ ...record, extra: "payload" })); + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), extra: "payload" }, + })); + }); + + /** + * Two different facts, and only the second belongs to this protocol. + * + * `Beta ` and `a%2fb` are not canonical encodings, so the data parser refuses + * them (DT58). `../../etc/passwd` *is* a canonical four-level heading path + * (DT59) — what refuses it here is that the recorded document's derived + * catalog does not contain it. Shape checking alone would accept it. + */ + it("TX31: a catalog the recorded document does not derive is refused", function* () { + for (const available of [["../../etc/passwd"], ["Beta "], ["a%2fb"]]) { + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), available }, + })); + } + // The same rule on a successful repository selection's own target: `beta` + // is a perfectly canonical target that this document simply does not have. + const healthy = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Beta" }), healthy, { names: [], ids: [] }); + const stream = yield* corrupt(healthy, (record) => ({ ...record, target: "beta" })); + const error = yield* failure(inlineSource(SECTIONS, { target: "Beta" }), stream); + expect((error as Error).message).toBe(UNREADABLE); + }); + + it("TX32: semantically inconsistent kind and matches are refused", function* () { + // `no-match` carrying matches, and a selector that really does match. + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), matches: ["Beta"] }, + })); + yield* refuses((record) => ({ + ...record, + failure: { ...asRecord(record["failure"]), selector: "Beta" }, + })); + // `multiple-matches` with a single match. + yield* refuses((record) => ({ + ...record, + failure: { + ...asRecord(record["failure"]), + kind: "multiple-matches", + selector: "Beta", + matches: ["Beta"], + }, + })); + }); + + it("TX33: a valid record still replays, so the refusals are not vacuous", function* () { + const stream = yield* failedJournal(); + const seen: Probes = { names: [], ids: [] }; + const replayed = yield* failure(inlineSource(SECTIONS, { target: "Missing" }), stream, seen); + expect(isDocumentTargetError(replayed)).toBe(true); + expect((replayed as Error).message).not.toBe(UNREADABLE); + expect(seen.names).toEqual([]); + }); +}); + +function asRecord(value: unknown): Record { + return typeof value === "object" && value !== null ? { ...(value as object) } : {}; +} + +function omit(value: unknown, key: string): Record { + const record = asRecord(value); + delete record[key]; + return record; +} + +/** + * Tier TX — reading a recorded root selection is total. + * + * The record is journal data, and journal data has ways of refusing to be read + * that are not defects of this run: a property may be an accessor that throws, + * a key list may come from a Proxy that refuses, and recorded markdown may have + * frontmatter no parser accepts. Each must become the one fixed answer, not an + * error of its own — an exception escaping here would carry a parser message, a + * path, or whatever a hostile record planted, and would do it from a boundary + * whose whole job is to refuse. + * + * The seam is the stream, not the record. `InMemoryStream` structured-clones on + * append, so nothing hostile can be *stored* in one; what a run actually + * consumes is whatever `readAll()` hands back. `PlantedStream` is therefore a + * stream that answers reads with a substituted root-import result, which is the + * shape a damaged or hostile backend really has. + * + * Each row plants a distinctive value and asserts it reaches nothing. + */ +const PLANTED = "pl4nted-s3cret"; + +/** + * A stream that answers reads with a substituted recorded root import. + * + * The substitution is a function of the healthy event, so a row can plant at + * any depth: inside a normally readable `result.value`, or on the envelope + * itself — `result`, `result.status`, `result.value` — which is where a damaged + * backend's unreadability actually lives. + */ +class PlantedStream implements DurableStream { + readonly appended: DurableEvent[] = []; + + constructor( + private readonly events: readonly DurableEvent[], + private readonly plant: (event: DurableEvent) => DurableEvent, + ) {} + + // deno-lint-ignore require-yield + *readAll(): Operation { + return this.events.map((event) => + event.type === "yield" && event.description.name === "__root__" ? this.plant(event) : event, + ); + } + + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + this.appended.push(event); + } +} + +/** + * Substitute the recorded root-import result value. + * + * `Result.value` is typed `Json`, and the point of these rows is to put + * something there that is not — which is what a damaged backend does and what + * the boundary has to survive. The casts in this file exist for that reason and + * no other. + */ +function plantValue(value: unknown): (event: DurableEvent) => DurableEvent { + return (event) => ({ ...event, result: { status: "ok", value: value as Json } }); +} + +/** Replace part of the recorded root-import envelope with a refusing accessor. */ +function plantEnvelope( + member: "result" | "status" | "value", +): (event: DurableEvent) => DurableEvent { + const refuse = () => { + throw new Error(`envelope refused: ${PLANTED}`); + }; + return (event) => { + if (member === "result") { + const planted: Record = { ...event }; + Object.defineProperty(planted, "result", { enumerable: true, get: refuse }); + return planted as unknown as DurableEvent; + } + const result: Record = + member === "status" + ? { value: { kind: "repository", path: "doc.md", content: SECTIONS } } + : { status: "ok" }; + Object.defineProperty(result, member, { enumerable: true, get: refuse }); + return { ...event, result: result as unknown as DurableEvent["result"] }; + }; +} + +describe("Tier TX — unreadable recorded selections", () => { + /** A completed journal, then a stream that answers reads with `plant`. */ + function* plantedStream(plant: (event: DurableEvent) => DurableEvent): Operation { + const healthy = new InMemoryStream(); + yield* failure(inlineSource(SECTIONS, { target: "Missing" }), healthy); + return new PlantedStream(healthy.snapshot(), plant); + } + + /** + * Resume a planted journal both ways and hold every refusal to the contract. + * + * Both directions matter and they fail differently when the boundary is + * wrong: the original `Missing` selector would replay the recorded failure, + * and the different, genuinely valid `Beta` selector would replay that same + * `Missing` failure — an outcome for a request nobody made. + */ + function* refusesTotally(plant: (event: DurableEvent) => DurableEvent): Operation { + for (const target of ["Missing", "Beta"]) { + const stream = yield* plantedStream(plant); + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* useProbes(seen); + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target }), stream })); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + // Never the recorded failure, for either request. + expect(isDocumentTargetError(error)).toBe(false); + // Nothing the record planted escapes, by any route a consumer would use. + const rendered = `${String(error)} ${(error as Error).stack ?? ""} ${JSON.stringify({ + ...(error as object), + })}`; + expect(rendered).not.toContain(PLANTED); + expect(rendered).not.toContain("Missing"); + // Nothing expanded, and nothing was appended on top of the record. + expect(seen.names).toEqual([]); + expect(stream.appended).toEqual([]); + } + } + + it("TX34: recorded content whose frontmatter no parser accepts is malformed", function* () { + // Valid JSON, valid string, unparseable document: without a total boundary + // the YAML parser's own failure escapes, carrying the planted text with it. + yield* refusesTotally( + plantValue({ + kind: "target-failure", + path: `${PLANTED}.md`, + content: `---\n: [unbalanced\n ${PLANTED}\n---\n\n# T\n`, + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, + }), + ); + }); + + it("TX35: an unreadable member is malformed, and says nothing about itself", function* () { + const record: Record = { + kind: "target-failure", + path: "doc.md", + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, + }; + Object.defineProperty(record, "content", { + enumerable: true, + get() { + throw new Error(`accessor refused: ${PLANTED}`); + }, + }); + yield* refusesTotally(plantValue(record)); + }); + + it("TX36: a record that refuses key enumeration is malformed", function* () { + yield* refusesTotally( + plantValue( + new Proxy( + { + kind: "target-failure", + path: "doc.md", + content: "# T\n\n## Beta\n", + failure: { kind: "no-match", selector: "Missing", matches: [], available: [] }, + }, + { + ownKeys() { + throw new Error(`enumeration refused: ${PLANTED}`); + }, + }, + ), + ), + ); + }); + + it("TX37: a value that is not a record at all is malformed", function* () { + for (const planted of [`a string ${PLANTED}`, 7, null, [PLANTED]]) { + yield* refusesTotally(plantValue(planted)); + } + }); + + /** + * The envelope, not its contents. + * + * Recognizing the event and reading its settled value are different questions, + * and answering both with one absent value conflates "this is not the root + * import" with "the root import will not say what it settled to". The second + * then delegates, and `durableRun` reuses the recorded terminal result — so a + * request for a section that really exists is answered with the failure of a + * request nobody made. + */ + it("TX38: a successful root result whose value refuses to be read is malformed", function* () { + yield* refusesTotally(plantEnvelope("value")); + }); + + it("TX39: a root result whose settlement refuses to be read is malformed", function* () { + yield* refusesTotally(plantEnvelope("status")); + }); + + /** + * The envelope itself, which used to be an exception. + * + * `ReplayIndex` read every Yield's result while building itself — before any + * guard's check phase — so a stream that would not produce the envelope + * failed inside indexing and carried its own error out past every refusal. + * Indexing now reads identity and leaves the result alone until a consumer + * asks, which puts this case back inside the same sanitized refusal as the + * rest. The ordering property itself is pinned in the durable-streams suite. + */ + it("TX40: a root event whose result refuses to be read is malformed", function* () { + yield* refusesTotally(plantEnvelope("result")); + }); + + it("TX41: a successful root result with no value at all is malformed", function* () { + yield* refusesTotally((event) => ({ ...event, result: { status: "ok" } })); + }); + + /** + * The other side of the classification: an ordinary failed settlement on the + * root import is recognized and left alone, because a root can fail for + * reasons that are not about selection. + */ + it("TX42: an ordinary failed root settlement is not this protocol's to refuse", function* () { + const stream = yield* plantedStream((event) => ({ + ...event, + result: { status: "err", error: { message: "the file went away" } }, + })); + const error = yield* scoped(function* () { + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream })); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + // Not refused by this protocol. Because the guard leaves the event alone, + // the completed journal's terminal result is reused as it always is — and + // that path deserializes, so what arrives is the recorded outcome's message + // rather than a reconstructed typed failure. That is the ordinary durable + // behavior an unclaimed event should get. + expect((error as Error).message).not.toBe(UNREADABLE_RECORD); + expect((error as Error).message).toContain("matches no document target"); + }); +}); + +/** + * Tier TX — the guard and replay read one settled value, not two. + * + * A recorded result is validated in the check phase and consumed during replay. + * If those are two reads of the stream's own event, a source that answers + * differently between them decides one thing for the guard and another for + * execution — the guard approves the section it was shown, and the run executes + * the section it was handed. Exact-target identity says the section that ran is + * the section the record names, and two reads make that unenforceable. + * + * Both answers here are *valid*: two real recorded selections from two real + * runs. Nothing is malformed, nothing is refused, and the only thing separating + * a correct run from a wrong one is that the value is read once. + */ +describe("Tier TX — one read across check and replay", () => { + /** The recorded root-import value of a real run against `target`. */ + function* recordedSelection(target: string): Operation { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), stream, { names: [], ids: [] }); + const recorded = stream + .snapshot() + .find((event) => event.type === "yield" && event.description.name === "__root__"); + if (recorded === undefined || recorded.result.status !== "ok") { + throw new Error("no recorded root import"); + } + const value = recorded.result.value; + if (value === undefined) { + throw new Error("recorded root import carried no value"); + } + return value; + } + + /** + * A partial journal: the recorded root import and nothing after it. + * + * Without a Close there is no completed-run shortcut, so the run replays the + * import and then continues live — which is the shape that lets the guard and + * the replay consumer both reach the same retained event. + */ + function* partialJournal(target: string): Operation { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), stream, { names: [], ids: [] }); + return stream + .snapshot() + .filter((event) => event.type === "yield" && event.description.name === "__root__"); + } + + /** + * The same substitution one level down. + * + * The record itself is stable; only the `target` member beneath it answers + * afresh. Freezing the outer result stops nothing here — the guard reads a + * record naming Alpha and replay reads the same record naming Beta. + */ + it("TX46: a nested target accessor cannot substitute another section", function* () { + const alpha = yield* recordedSelection("Alpha"); + const events = yield* partialJournal("Alpha"); + const reads = { count: 0 }; + const targets = ["Alpha", "Beta"]; + + const stream = new PlantedStream(events, (event) => { + const record: Record = { ...asRecord(alpha) }; + Object.defineProperty(record, "target", { + enumerable: true, + get() { + const answer = targets[Math.min(reads.count, targets.length - 1)]; + reads.count++; + return answer; + }, + }); + return { + ...event, + result: { status: "ok", value: record as unknown as Json }, + }; + }); + + const seen: Probes = { names: [], ids: [] }; + const text = yield* scoped(function* () { + yield* useProbes(seen); + return asText( + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream })), + ); + }); + + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + expect(text).not.toContain("## Beta"); + expect(reads.count).toBe(1); + const close = stream.appended.find((event) => event.type === "close"); + expect(JSON.stringify(close)).toContain("alpha content"); + expect(JSON.stringify(close)).not.toContain("beta content"); + }); + + it("TX43: a source that answers twice decides nothing twice", function* () { + const alpha = yield* recordedSelection("Alpha"); + const beta = yield* recordedSelection("Beta"); + const events = yield* partialJournal("Alpha"); + const answers: Json[] = [alpha, beta]; + const reads = { count: 0 }; + + const stream = new PlantedStream(events, (event) => { + const result: Record = { status: "ok" }; + Object.defineProperty(result, "value", { + enumerable: true, + get() { + const answer = answers[Math.min(reads.count, answers.length - 1)]!; + reads.count++; + return answer; + }, + }); + return { ...event, result: result as unknown as DurableEvent["result"] }; + }); + + const seen: Probes = { names: [], ids: [] }; + const text = yield* scoped(function* () { + yield* useProbes(seen); + return asText( + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream })), + ); + }); + + // The guard approved Alpha, so Alpha is what ran. + expect(seen.names).toEqual(["pre", "title", "alpha", "inner"]); + expect(text).toContain("### Inner"); + expect(text).not.toContain("## Beta"); + // And the record was consulted once, so there was never a second answer to + // disagree with the first. + expect(reads.count).toBe(1); + // What the run published describes the section the guard approved. + const close = stream.appended.find((event) => event.type === "close"); + expect(close).toBeDefined(); + expect(JSON.stringify(close)).toContain("alpha content"); + expect(JSON.stringify(close)).not.toContain("beta content"); + }); +}); + +/** + * A section before the title must not run as preamble. + * + * The projection rows prove the text is absent; only a component that records + * its own invocation proves the section did not execute. + */ +describe("Tier TX — preamble boundary", () => { + const OPENS_DEEP = [ + "## Before", + "", + 'before body ', + "", + "# Title", + "", + 'title content ', + "", + "## After", + "", + 'after body ', + "", + ].join("\n"); + + it("TX44: selecting the later section never executes the earlier one", function* () { + const seen: Probes = { names: [], ids: [] }; + const text = yield* run( + inlineSource(OPENS_DEEP, { target: "After" }), + new InMemoryStream(), + seen, + ); + expect(seen.names).toEqual(["title", "after"]); + expect(text).not.toContain("## Before"); + expect(text).not.toContain("before body"); + }); + + it("TX45: the earlier section still runs when it is the target", function* () { + const seen: Probes = { names: [], ids: [] }; + yield* run(inlineSource(OPENS_DEEP, { target: "Before" }), new InMemoryStream(), seen); + expect(seen.names).toEqual(["before"]); + }); +}); + +/** + * Tier TX — a retained terminal result needs the import that produced it. + * + * `durableRun` reuses a recorded Close before replaying anything, and the check + * phase only ever sees the Yields a journal actually contains. A journal whose + * root import is gone therefore offers the guard nothing to validate, and the + * Close answers for a selection that was never established — a resume asking + * for one section receives another section's completed result. + * + * A replay that means to reuse terminal history must first establish exactly + * one recognizable root import. + */ +describe("Tier TX — a terminal journal without its root import", () => { + /** A completed journal for `target`, minus its root-import Yield. */ + function* withoutRootImport(target: string): Operation { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), complete, { names: [], ids: [] }); + const stripped = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "yield" && event.description.name === "__root__") { + continue; + } + yield* stripped.append(event); + } + return stripped; + } + + /** Hold a resume against a stripped journal to the refusal contract. */ + function* refusesStripped(stream: InMemoryStream, target?: string): Operation { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const source = + target === undefined ? inlineSource(SECTIONS) : inlineSource(SECTIONS, { target }); + const error = yield* failure(source, stream, seen); + + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect(isDocumentTargetError(error)).toBe(false); + // Never the retained Close's own outcome. + expect((error as Error).message).not.toContain("alpha content"); + expect(seen.names).toEqual([]); + expect(stream.snapshot().length).toBe(before); + } + + it("TX47: a targeted journal missing its root import refuses", function* () { + yield* refusesStripped(yield* withoutRootImport("Alpha"), "Beta"); + yield* refusesStripped(yield* withoutRootImport("Alpha"), "Alpha"); + }); + + it("TX48: an untargeted journal missing its root import refuses", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS), complete, { names: [], ids: [] }); + const stripped = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "yield" && event.description.name === "__root__") { + continue; + } + yield* stripped.append(event); + } + yield* refusesStripped(stripped); + }); + + it("TX49: a duplicated root import refuses", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const doubled = new InMemoryStream(); + for (const event of complete.snapshot()) { + yield* doubled.append(event); + if (event.type === "yield" && event.description.name === "__root__") { + yield* doubled.append(event); + } + } + yield* refusesStripped(doubled, "Alpha"); + }); + + it("TX50: the control — an intact terminal journal still replays", function* () { + const stream = new InMemoryStream(); + const golden = yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { + names: [], + ids: [], + }); + const replayed = yield* run(inlineSource(SECTIONS, { target: "Alpha" }), stream, { + names: [], + ids: [], + }); + expect(replayed).toBe(golden); + }); +}); + +/** + * Tier TX — definition identity is not middleware. + * + * Exact canonical target is workflow-definition identity, and identity may not + * be decided by anything a document, a component, or an enclosing scope can + * replace. A public `ReplayGuard` handler installed further out may decline to + * call `next` — that is what composable policy is *for* — so a comparison + * living there is a comparison an outer handler can switch off. + * + * These install exactly such a handler and assert the answer does not change. + */ +describe("Tier TX — identity authority is execution-owned", () => { + /** A guard that swallows every stage it is given, calling `next` for none. */ + function* useSuppressingGuard(stage: "check" | "admit"): Operation { + if (stage === "check") { + yield* ReplayGuard.around({ + // deno-lint-ignore require-yield + *check() {}, + }); + return; + } + yield* ReplayGuard.around({ + // deno-lint-ignore require-yield + *admit() {}, + }); + } + + /** Resume `stream` as `target` with `install` in scope, and report what happened. */ + function* resumeWith( + stream: InMemoryStream, + target: string, + install: () => Operation, + ): Operation<{ error: unknown; seen: Probes; appended: number }> { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* install(); + yield* useProbes(seen); + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target }), stream })); + } catch (caught) { + return caught; + } + return undefined; + }); + return { error, seen, appended: stream.snapshot().length - before }; + } + + /** A completed journal for `target`. */ + function* completed(target: string): Operation { + const stream = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target }), stream, { names: [], ids: [] }); + return stream; + } + + /** A journal for `target` with its terminal result removed. */ + function* partial(target: string): Operation { + const complete = yield* completed(target); + const stripped = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "close") { + continue; + } + yield* stripped.append(event); + } + return stripped; + } + + it("TX51: an enclosing check handler that never delegates cannot admit Beta", function* () { + const outcome = yield* resumeWith(yield* completed("Alpha"), "Beta", () => + useSuppressingGuard("check"), + ); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX52: an enclosing admit handler that never delegates cannot admit Beta", function* () { + const outcome = yield* resumeWith(yield* completed("Alpha"), "Beta", () => + useSuppressingGuard("admit"), + ); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + /** + * A guard registered under the same name by a separately loaded copy composes + * with this run's guards, because an Effection context is keyed by its name. + * That is the portability mechanism, and it is exactly why identity is not + * kept there. + */ + it("TX53: a same-name guard from another loaded copy cannot admit Beta", function* () { + const foreign = createApi<{ check(event: Yield): Operation }>( + "DurableEffection.ReplayGuard", + { + // deno-lint-ignore require-yield + *check() {}, + }, + ); + const outcome = yield* resumeWith(yield* completed("Alpha"), "Beta", function* () { + yield* foreign.around({ + // deno-lint-ignore require-yield + *check() {}, + }); + }); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX54: a partial journal is held to its recorded selection too", function* () { + const outcome = yield* resumeWith(yield* partial("Alpha"), "Beta", () => + useSuppressingGuard("check"), + ); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX55: the controls — the same target replays, complete and partial", function* () { + const whole = yield* resumeWith(yield* completed("Alpha"), "Alpha", () => + useSuppressingGuard("check"), + ); + expect(whole.error).toBe(undefined); + + const half = yield* resumeWith(yield* partial("Alpha"), "Alpha", () => + useSuppressingGuard("check"), + ); + expect(half.error).toBe(undefined); + expect(half.seen.names).toEqual(["pre", "title", "alpha", "inner"]); + }); + + it("TX56: ordinary guard composition still works", function* () { + const stream = yield* completed("Alpha"); + const observed: string[] = []; + const outcome = yield* resumeWith(stream, "Alpha", function* () { + yield* ReplayGuard.around({ + *check([event], next) { + observed.push(event.description.name); + return yield* next(event); + }, + }); + }); + expect(outcome.error).toBe(undefined); + expect(observed).toContain("__root__"); + }); +}); + +/** + * Tier TX — the root import that authorizes a terminal result is the terminal + * coroutine's own. + * + * Reusing a recorded terminal result means standing behind the selection its + * root import established. A root import belonging to some other coroutine + * established that coroutine's selection, not this one's, and two of them + * establish nothing at all. + */ +describe("Tier TX — a terminal result needs its own root import", () => { + /** A completed Alpha journal, rewritten event by event. */ + function* rewritten( + change: (events: DurableEvent[]) => DurableEvent[], + ): Operation { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const rebuilt = new InMemoryStream(); + for (const event of change(complete.snapshot())) { + yield* rebuilt.append(event); + } + return rebuilt; + } + + function isRootImport(event: DurableEvent): boolean { + return event.type === "yield" && event.description.name === "__root__"; + } + + /** Hold a resume to the refusal contract. */ + function* refuses(stream: InMemoryStream): Operation { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const error = yield* failure(inlineSource(SECTIONS, { target: "Alpha" }), stream, seen); + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect(seen.names).toEqual([]); + expect(stream.snapshot().length).toBe(before); + } + + it("TX57: only a child coroutine carries the root import", function* () { + yield* refuses( + yield* rewritten((events) => + events.map((event) => (isRootImport(event) ? { ...event, coroutineId: "root.7" } : event)), + ), + ); + }); + + it("TX58: a valid root import plus a root-named child event", function* () { + yield* refuses( + yield* rewritten((events) => + events.flatMap((event) => + isRootImport(event) ? [event, { ...event, coroutineId: "root.7" }] : [event], + ), + ), + ); + }); + + it("TX59: no root import at all", function* () { + yield* refuses(yield* rewritten((events) => events.filter((event) => !isRootImport(event)))); + }); + + it("TX60: two root imports on the terminal coroutine", function* () { + yield* refuses( + yield* rewritten((events) => + events.flatMap((event) => (isRootImport(event) ? [event, event] : [event])), + ), + ); + }); +}); + +/** + * Tier TX — a detached retained value is still ordinary JSON. + * + * Detaching is a claim against the journal, not against the document. A + * replayed value a document goes on to update must behave as it did when the + * run first produced it. + */ +describe("Tier TX — detached values stay mutable", () => { + const MUTATES = [ + "```js eval", + "const state = { count: 0, tags: ['a'] };", + "```", + "", + "```js eval", + "state.count += 1;", + "state.tags.push('b');", + "output(`${state.count}:${state.tags.join(',')}`);", + "```", + "", + ].join("\n"); + + function* runMutating(stream: DurableStream): Operation { + return yield* scoped(function* () { + // The portable compiler: a data: module is not importable under every + // runtime. + yield* useTempFileCompiler(); + return asText(yield* collect(yield* execute({ ...inlineSource(MUTATES), stream }))); + }); + } + + /** + * A genuine partial replay: the binding is restored from the journal and then + * written to by an iteration that runs live. + * + * The first eval block is retained, so `state` comes back detached from the + * journal. The second is dropped along with the terminal result, so the + * update runs for real. A detached member that is not writable fails here. + */ + it("TX61: a restored binding is mutated by the live continuation", function* () { + const complete = new InMemoryStream(); + const golden = yield* runMutating(complete); + expect(golden).toContain("1:a,b"); + + const evals = complete + .snapshot() + .filter((event) => event.type === "yield" && event.description.type === "eval"); + expect(evals.length).toBeGreaterThan(1); + + const partial = new InMemoryStream(); + let kept = 0; + for (const event of complete.snapshot()) { + if (event.type === "close") { + continue; + } + if (event.type === "yield" && event.description.type === "eval") { + kept += 1; + if (kept > 1) { + continue; + } + } + yield* partial.append(event); + } + + // The first block replays and restores `state`; the second runs live and + // writes to it. + expect(yield* runMutating(partial)).toBe(golden); + }); +}); + +/** + * Tier TX — one validation order on every public path. + * + * A caller who named a section the document does not offer asked the wrong + * question, and should hear that — not a complaint about a schema they never + * reached. Resolving the target before compiling schemas makes the answer the + * same whether a host inspects, runs, or resumes. + */ +describe("Tier TX — target resolution precedes schema compilation", () => { + const BROKEN_SCHEMA = [ + "---", + "props:", + " type: object", + " properties:", + " who:", + " type: not-a-json-schema-type", + "---", + "", + "# Title", + "", + "## Kept", + "", + "kept body", + "", + ].join("\n"); + + const GOOD_SCHEMA = BROKEN_SCHEMA.replace("not-a-json-schema-type", "string"); + + /** The failure each public path reports for one source and selector. */ + function* onEveryPath(source: string, target: string): Operation { + const inspected = yield* scoped(function* () { + try { + yield* inspectDocument(inlineSource(source, { target })); + } catch (error) { + return error; + } + return undefined; + }); + + const stream = new InMemoryStream(); + const live = yield* failure(inlineSource(source, { target }), stream); + // The same journal again: a recorded failed selection keeps its precedence + // and is not replaced by the recorded terminal error. + const replayed = yield* failure(inlineSource(source, { target }), stream); + return [inspected, live, replayed]; + } + + it("TX64: an unresolvable target outranks an invalid schema on all three paths", function* () { + for (const failed of yield* onEveryPath(BROKEN_SCHEMA, "Missing")) { + expect(isDocumentTargetError(failed)).toBe(true); + expect(asDocumentTargetError(failed)?.data).toMatchObject({ + kind: "no-match", + selector: "Missing", + available: ["Kept"], + }); + } + }); + + it("TX65: a resolvable target lets the invalid schema be reported", function* () { + // All three paths, replay included: the recorded terminal error is the + // schema failure, and nothing replaces it with a target failure. + for (const failed of yield* onEveryPath(BROKEN_SCHEMA, "Kept")) { + expect(isDocumentTargetError(failed)).toBe(false); + expect((failed as Error).name).toBe("PropsSchemaError"); + } + }); + + it("TX66: an unresolvable target with a valid schema still reports the target", function* () { + for (const failed of yield* onEveryPath(GOOD_SCHEMA, "Missing")) { + expect(isDocumentTargetError(failed)).toBe(true); + } + }); + + it("TX67: the control — a resolvable target and a valid schema run", function* () { + const info = yield* inspectDocument(inlineSource(GOOD_SCHEMA, { target: "Kept" })); + expect(info.target).toBe("Kept"); + const text = asText( + yield* collect( + yield* execute({ + ...inlineSource(GOOD_SCHEMA, { target: "Kept" }), + stream: new InMemoryStream(), + }), + ), + ); + expect(text).toContain("kept body"); + }); +}); + +/** + * Tier TX — a Close cannot change coroutines between phases. + * + * The admission gate decides whether a terminal result exists; the index + * decides whose it is. Read separately, a Close can belong to a child while the + * gate asks and to the root while the index does — the gate admits a history it + * believes has no terminal result, and the run then returns one, for a section + * nobody requested. + * + * Recorded before the fix, resuming as Beta against an Alpha journal whose root + * import was removed: + * + * ```json + * {"requested":"Beta","ok":true,"value":"# Title\n\n## Alpha\n\nalpha content\n\n"} + * ``` + */ +describe("Tier TX — a shifting terminal coroutine", () => { + it("TX68: a Close that moves from a child to the root returns nothing", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + + let reads = 0; + const events: DurableEvent[] = []; + for (const event of complete.snapshot()) { + if (event.type === "yield" && event.description.name === "__root__") { + continue; + } + if (event.type === "close") { + const close: Record = { type: "close", result: event.result }; + Object.defineProperty(close, "coroutineId", { + enumerable: true, + get() { + reads += 1; + return reads === 1 ? "root.7" : "root"; + }, + }); + events.push(close as unknown as DurableEvent); + continue; + } + events.push(event); + } + + const appended: DurableEvent[] = []; + const stream: DurableStream = { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* useProbes(seen); + try { + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream })); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + + // Never Alpha's retained output, and never a partial answer either. + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect((error as Error).message).not.toContain("alpha content"); + expect(seen.names).toEqual([]); + expect(appended).toEqual([]); + // The coroutine was asked once, so there was never a second answer. + expect(reads).toBe(1); + }); +}); + +/** + * Tier TX — nothing the backend still owns survives admission. + * + * The private gate accepts a history; public `ReplayGuard` policy runs next; + * terminal reuse happens after that. If a Close still points at the backend's + * own result through all of it, those two later phases are a window in which + * the answer can be replaced — and public policy is code any enclosing scope + * may install. + * + * Recorded before the fix, resuming Alpha under a `check` handler that rewrites + * the raw Close result: + * + * ```json + * {"ok":true,"value":"planted after admission","appended":0} + * ``` + */ +describe("Tier TX — a terminal result cannot be replaced after admission", () => { + /** An intact Alpha journal whose Close carries a caller-owned result. */ + function* plantableJournal(): Operation<{ + events: DurableEvent[]; + planted: Record; + }> { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + + const events: DurableEvent[] = []; + let planted: Record = {}; + for (const event of complete.snapshot()) { + if (event.type === "close") { + const value = parseJson(event.result.status === "ok" ? (event.result.value ?? null) : null); + if (isJsonObject(value)) { + planted = value; + } + events.push({ + type: "close", + coroutineId: "root", + result: { status: "ok", value }, + } as DurableEvent); + continue; + } + events.push(event); + } + return { events, planted }; + } + + function reading(events: DurableEvent[], appended: DurableEvent[]): DurableStream { + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + } + + it("TX69: public policy cannot rewrite the terminal result it was shown", function* () { + const { events, planted } = yield* plantableJournal(); + const appended: DurableEvent[] = []; + const stream = reading(events, appended); + + const text = yield* scoped(function* () { + yield* ReplayGuard.around({ + *check([event], next) { + // Public policy, running between private admission and terminal + // reuse, rewriting what the backend still owns. + planted["output"] = "planted after admission"; + planted["value"] = "planted after admission"; + return yield* next(event); + }, + }); + return asText( + yield* collect(yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream })), + ); + }); + + expect(text).toContain("alpha content"); + expect(text).not.toContain("planted after admission"); + expect(appended).toEqual([]); + }); + + it("TX70: a terminal result that refuses is the fixed diagnostic", function* () { + const { events } = yield* plantableJournal(); + let asked = 0; + const refusing = events.map((event) => { + if (event.type !== "close") { + return event; + } + const close: Record = { type: "close", coroutineId: "root" }; + Object.defineProperty(close, "result", { + enumerable: true, + get() { + asked += 1; + if (asked === 1) { + throw new Error("the backend will not produce this result"); + } + return { status: "ok", value: "answered later" }; + }, + }); + return close as unknown as DurableEvent; + }); + + const appended: DurableEvent[] = []; + const seen: Probes = { names: [], ids: [] }; + const error = yield* scoped(function* () { + yield* useProbes(seen); + try { + yield* collect( + yield* execute({ + ...inlineSource(SECTIONS, { target: "Alpha" }), + stream: reading(refusing, appended), + }), + ); + } catch (caught) { + return caught; + } + throw new Error("the run completed instead of failing"); + }); + + expect((error as Error).message).toBe(UNREADABLE_RECORD); + expect((error as Error).cause).toBe(undefined); + expect((error as Error).message).not.toContain("answered later"); + // Read once during retention, and never retried afterwards. + expect(asked).toBe(1); + expect(seen.names).toEqual([]); + expect(appended).toEqual([]); + }); + + it("TX71: the control — an intact terminal journal still replays", function* () { + const { events } = yield* plantableJournal(); + const appended: DurableEvent[] = []; + const text = asText( + yield* collect( + yield* execute({ + ...inlineSource(SECTIONS, { target: "Alpha" }), + stream: reading(events, appended), + }), + ), + ); + expect(text).toContain("alpha content"); + expect(appended).toEqual([]); + }); +}); + +/** + * Tier TX — public policy observes; it does not hold authority. + * + * The execution-owned gate validates a history and replay consumes it. Between + * those, public `ReplayGuard` policy runs — `check`, then `admit`, then `decide` + * during replay — and it is code any enclosing scope may install. Handing it the + * retained events themselves would let it rewrite the root selection, the + * recorded content, an effect description, or a result *after* admission had + * accepted them, which is exactly the authority the private gate exists to keep + * out of public hands. + * + * Recorded before the fix, resuming a partial Alpha journal as Alpha with + * `check` rewriting the retained root-import target to Beta: the run executed + * the Beta projection. + */ +describe("Tier TX — guards observe a copy, not the retained history", () => { + /** A partial Alpha journal: the recorded import, no terminal result. */ + function* partialAlpha(): Operation { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const partial = new InMemoryStream(); + for (const event of complete.snapshot()) { + if (event.type === "close") { + continue; + } + yield* partial.append(event); + } + return partial; + } + + /** Rewrite a root-import observation the way hostile policy would. */ + function rewriteRoot(event: Yield, change: (record: Record) => void): void { + if (event.description.name !== "__root__" || event.result.status !== "ok") { + return; + } + const record = event.result.value; + if (isJsonObject(record)) { + change(record as unknown as Record); + } + } + + /** Resume Alpha with `install` in scope; report what ran. */ + function* resume( + stream: InMemoryStream, + install: () => Operation, + ): Operation<{ seen: Probes; text: string | undefined; error: unknown; appended: number }> { + const before = stream.snapshot().length; + const seen: Probes = { names: [], ids: [] }; + const outcome = yield* scoped(function* () { + yield* install(); + yield* useProbes(seen); + try { + return { + text: asText( + yield* collect( + yield* execute({ ...inlineSource(SECTIONS, { target: "Alpha" }), stream }), + ), + ), + error: undefined, + }; + } catch (error) { + return { text: undefined, error }; + } + }); + return { seen, ...outcome, appended: stream.snapshot().length - before }; + } + + /** Every root mismatch has the same shape of answer. */ + function expectAlphaOnly(outcome: { + seen: Probes; + text: string | undefined; + error: unknown; + }): void { + expect(outcome.seen.names).not.toContain("beta"); + expect(outcome.text ?? "").not.toContain("beta content"); + if (outcome.error === undefined) { + expect(outcome.text).toContain("alpha content"); + } + } + + it("TX72: check cannot rewrite the retained root target", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + rewriteRoot(event, (record) => { + record["target"] = "Beta"; + }); + return yield* next(event); + }, + }); + }); + expectAlphaOnly(outcome); + }); + + it("TX73: admit cannot rewrite the retained root target", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *admit([history], next) { + for (const event of history.yields) { + rewriteRoot(event, (record) => { + record["target"] = "Beta"; + }); + } + return yield* next(history); + }, + }); + }); + expectAlphaOnly(outcome); + }); + + it("TX74: decide cannot rewrite the retained root target", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + decide([event], next) { + rewriteRoot(event, (record) => { + record["target"] = "Beta"; + }); + return next(event); + }, + }); + }); + expectAlphaOnly(outcome); + }); + + it("TX75: the recorded content and failure record are equally out of reach", function* () { + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + rewriteRoot(event, (record) => { + record["content"] = "# Rewritten\\n\\n## Beta\\n\\nrewritten beta\\n"; + record["kind"] = "target-failure"; + record["failure"] = { + kind: "no-match", + selector: "Alpha", + matches: [], + available: [], + }; + }); + return yield* next(event); + }, + }); + }); + expectAlphaOnly(outcome); + expect(outcome.text ?? "").not.toContain("rewritten beta"); + }); + + it("TX76: a completed journal refused for a mismatch appends nothing", function* () { + const complete = new InMemoryStream(); + yield* run(inlineSource(SECTIONS, { target: "Alpha" }), complete, { names: [], ids: [] }); + const outcome = yield* scoped(function* () { + const seen: Probes = { names: [], ids: [] }; + const before = complete.snapshot().length; + yield* useProbes(seen); + let error: unknown; + try { + yield* collect( + yield* execute({ ...inlineSource(SECTIONS, { target: "Beta" }), stream: complete }), + ); + } catch (caught) { + error = caught; + } + return { seen, error, appended: complete.snapshot().length - before }; + }); + expect(outcome.error).toBeInstanceOf(StaleInputError); + expect(outcome.seen.names).toEqual([]); + expect(outcome.appended).toBe(0); + }); + + it("TX77: downstream guard composition still reads its observation", function* () { + const observed: string[] = []; + const outcome = yield* resume(yield* partialAlpha(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + // An annotation on the observation: composition still works, and + // nothing it writes reaches replay. + Object.assign(event.description, { annotated: true }); + observed.push(event.description.name); + return yield* next(event); + }, + }); + }); + expect(observed).toContain("__root__"); + expectAlphaOnly(outcome); + }); +}); + +/** + * Tier TX — the witness survives every wrapper a run puts on its journal. + * + * Journal provenance is deliberately non-transitive (#425): a wrapper is + * unproven unless a trusted wrapping site carries its source's witness onto it, + * and a run whose journal is unproven is refused by a Workspace provider before + * any transaction. Core puts two wrappers on the journal a host supplies — the + * secret filter, and the execution-owned target-admission gate — so both have + * to be explicit about it or a live coordinator receives nothing. + * + * Recorded before the fix, at the exact head: + * + * ```json + * {"filtered":true,"identityGate":false,"identityGateMissing":true} + * ``` + * + * These exercise `execute()` rather than the wrapper in isolation, because what + * is being measured is what reaches live coordination after every wrapper. + */ +describe("Tier TX — journal provenance across the admission gate", () => { + /** What a live durable operation's coordinator was handed. */ + interface Coordinated { + witnesses: (JournalProvenance | undefined)[]; + } + + /** + * A coordinator that records the provenance it is handed. + * + * Installed on a durable operation raised from inside a real document + * execution, so what it sees is what core's journal — after secret filtering + * and target admission — actually delivers to live coordination. + */ + function witnessing(seen: Coordinated): LiveDurableOperationCoordinator { + return { + *run(execute, publish, activateFailure, journalProvenance) { + seen.witnesses.push(journalProvenance); + return yield* defaultLiveDurableOperationCoordinator.run( + execute, + publish, + activateFailure, + journalProvenance, + ); + }, + }; + } + + /** Raise one coordinated durable operation inside the document's own run. */ + function* useWitnessProbe(seen: Coordinated): Operation { + yield* Execution.around({ + *document([props], next) { + yield createDurableOperation( + { type: "probe", name: "provenance" }, + // deno-lint-ignore require-yield + function* () { + return "probed"; + }, + { coordinator: witnessing(seen) }, + ); + return yield* next(props); + }, + }); + } + + const LIVE = [ + "# Title", + "", + "## Alpha", + "", + "alpha content", + "", + "## Beta", + "", + "beta content", + "", + ].join("\n"); + + /** Run `source` against `stream`, reporting what coordination witnessed. */ + function* observing( + stream: DurableStream, + source: RootDocumentSource, + settings: { secretDetection?: boolean } = {}, + ): Operation { + const seen: Coordinated = { witnesses: [] }; + yield* scoped(function* () { + yield* useWitnessProbe(seen); + yield* collect(yield* execute({ ...source, stream, ...settings })); + }); + return seen; + } + + it("TX78: an untargeted run delivers the selected journal's exact witness", function* () { + const stream = new InMemoryStream(); + const witness = establishJournalProvenance(stream); + const seen = yield* observing(stream, inlineSource(LIVE)); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(witness); + } + }); + + it("TX79: a targeted run delivers the same exact witness", function* () { + const stream = new InMemoryStream(); + const witness = establishJournalProvenance(stream); + const seen = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(witness); + } + }); + + it("TX80: secret detection disabled delivers the same exact witness", function* () { + const stream = new InMemoryStream(); + const witness = establishJournalProvenance(stream); + const seen = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" }), { + secretDetection: false, + }); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(witness); + } + }); + + it("TX81: an unproven journal stays unproven through both wrappers", function* () { + const stream = new InMemoryStream(); + const seen = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(undefined); + } + }); + + it("TX82: an ordinary wrapper of a proven journal is not promoted", function* () { + const backend = new InMemoryStream(); + establishJournalProvenance(backend); + // A wrapper nobody trusted: generic guarding, not a wrapping site. + const ordinary = guardDurableStream(backend, function* () {}); + const seen = yield* observing(ordinary, inlineSource(LIVE, { target: "Alpha" })); + expect(seen.witnesses.length).toBeGreaterThan(0); + for (const observed of seen.witnesses) { + expect(observed).toBe(undefined); + } + }); + + it("TX83: replay reaches no live coordination at all", function* () { + const stream = new InMemoryStream(); + establishJournalProvenance(stream); + yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + + const replayed = yield* observing(stream, inlineSource(LIVE, { target: "Alpha" })); + expect(replayed.witnesses).toEqual([]); + }); +}); diff --git a/packages/core/tests/document-targets.test.ts b/packages/core/tests/document-targets.test.ts new file mode 100644 index 00000000..0e2c7bd8 --- /dev/null +++ b/packages/core/tests/document-targets.test.ts @@ -0,0 +1,1041 @@ +/** + * Tier DT — document targets (spec §5.4). + * + * A target is an addressable static heading, and selecting one runs the + * preamble, each ancestor's own content, and that heading's subtree. These + * assert the three properties the feature stands on. + * + * **Discovery cannot see inside a component.** The masked parse is not a + * refinement of a Remark parse — a component child holding a blank line and a + * `#` line surfaces as a root heading without it, so DT13 fails outright + * against raw Remark discovery. + * + * **A selector resolves exactly once, before anything runs.** Zero matches and + * several matches are both failures, and duplicate canonical paths stay + * duplicates so the ambiguity is visible rather than silently resolved. + * + * **Projection retains original ranges.** The assertions read the projected + * source and the scanned positions rather than a rendering, because the + * position is what an expansion identifier is derived from. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Operation } from "effection"; + +import { + asDocumentTargetError, + DocumentTargetError, + encodeTargetLabel, + isCanonicalTarget, + isDocumentTargetError, + normalizeLabel, + parseDocumentTargetFailure, + outlineDocument, + retainedRanges, + selectTarget, +} from "../src/document-targets.ts"; +import { scanComponentSpans } from "../src/scanner.ts"; +import { parseRootMarkdownDefinition } from "../src/definition.ts"; +import { fileSource, formatDocumentReference, inlineSource } from "../src/root-source.ts"; +import { inspectDocument } from "../src/inspect.ts"; + +function outline(body: string) { + return outlineDocument(body, scanComponentSpans(body)); +} + +function catalog(body: string): readonly string[] { + return outline(body).targets; +} + +function project(body: string, selector: string): string { + const found = outline(body); + const entry = selectTarget(found, selector); + return retainedRanges(found, entry) + .map((range) => body.slice(range.start, range.end)) + .join(""); +} + +/** The failure a selector produced, refusing to pass a success off as one. */ +function refusal(body: string, selector: string): DocumentTargetError { + try { + selectTarget(outline(body), selector); + } catch (error) { + if (error instanceof DocumentTargetError) { + return error; + } + throw error; + } + throw new Error(`${selector} resolved instead of failing`); +} + +const SECTIONS = [ + "preamble", + "", + "# Title", + "", + "intro", + "", + "## Test", + "", + "test intro", + "", + "### Node", + "", + "node body", + "", + "### Bun", + "", + "bun body", + "", + "## Other", + "", + "other body", + "", +].join("\n"); + +describe("Tier DT — document target catalog", () => { + it("DT1: catalogs ATX headings in source order under a sole title", function* () { + expect(catalog(SECTIONS)).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + }); + + it("DT2: a Setext heading is an ordinary outline heading", function* () { + const body = ["Title", "=====", "", "Section", "-------", "", "body", ""].join("\n"); + expect(catalog(body)).toEqual(["Section"]); + }); + + it("DT3: a skipped depth still nests, and the depth itself is not the path", function* () { + const body = ["# Title", "", "#### Deep", "", "body", ""].join("\n"); + expect(catalog(body)).toEqual(["Deep"]); + }); + + it("DT4: the outermost depth is the smallest present, not h1", function* () { + const body = ["## A", "", "### A1", "", "## B", ""].join("\n"); + expect(catalog(body)).toEqual(["A", "A/A1", "B"]); + }); + + it("DT5: several outermost headings all take a path level", function* () { + const body = ["# A", "", "## A1", "", "# B", "", "## B1", ""].join("\n"); + expect(catalog(body)).toEqual(["A", "A/A1", "B", "B/B1"]); + }); + + it("DT6: matching is case sensitive", function* () { + const body = ["# Title", "", "## Test", ""].join("\n"); + expect(refusal(body, "test").data.kind).toBe("no-match"); + expect(selectTarget(outline(body), "Test").target).toBe("Test"); + }); + + it("DT7: a label is the statically rendered text, formatting removed", function* () { + const body = [ + "# Title", + "", + "## **Bold** and _italic_", + "", + "## A [link](https://example.test/x) here", + "", + "## Inline `code` text", + "", + "## Alt ![a picture](img.png) text", + "", + "## Tagged text", + "", + ].join("\n"); + expect(catalog(body)).toEqual([ + "Bold%20and%20italic", + "A%20link%20here", + "Inline%20code%20text", + "Alt%20a%20picture%20text", + "Tagged%20text", + ]); + }); + + it("DT8: NFC-equivalent spellings are one label, and Unicode space collapses", function* () { + const decomposed = ["# Title", "", "## Café   name", ""].join("\n"); + expect(catalog(decomposed)).toEqual([encodeTargetLabel("Café name")]); + expect(normalizeLabel("Café   name")).toBe("Café name"); + // The precomposed spelling addresses the decomposed heading. + expect(selectTarget(outline(decomposed), encodeTargetLabel("Café name")).labels).toEqual([ + "Café name", + ]); + }); + + it("DT9: a heading that renders no text is not addressable", function* () { + const body = ["# Title", "", "##", "", "body", "", "## Real", ""].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT10: reserved characters are percent-encoded, never left as syntax", function* () { + const body = [ + "# Title", + "", + "## a/b", + "", + "## 100% done", + "", + "## C\\# sharp", + "", + "## star \\* here", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["a%2Fb", "100%25%20done", "C%23%20sharp", "star%20%2A%20here"]); + // `%2F` addresses one label containing a slash; a raw `/` would be hierarchy. + expect(selectTarget(outline(body), "a%2Fb").labels).toEqual(["a/b"]); + expect(refusal(body, "a/b").data.kind).toBe("no-match"); + // `%2A` is a literal asterisk; a raw `*` is the operator. + expect(selectTarget(outline(body), "star%20%2A%20here").labels).toEqual(["star * here"]); + }); + + it("DT11: duplicate canonical paths stay duplicate entries", function* () { + const body = ["# Title", "", "## Same", "", "one", "", "## Same", "", "two", ""].join("\n"); + expect(catalog(body)).toEqual(["Same", "Same"]); + const ambiguous = refusal(body, "Same"); + expect(ambiguous.data.kind).toBe("multiple-matches"); + expect(ambiguous.data.matches).toEqual(["Same", "Same"]); + }); + + it("DT12: only root-flow headings count", function* () { + const body = [ + "# Title", + "", + "> # Quoted", + "", + "- # Listed", + "", + "```md", + "# Fenced", + "```", + "", + "```sh exec", + "# Executed", + "```", + "", + "
", + "# Raw html child", + "
", + "", + "## Real", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + /** + * The regression that decides the parser boundary. + * + * Remark ends an HTML block at a blank line, so a component child holding one + * puts every following `#` line at the root of the tree. Discovery therefore + * parses a masked copy in which the component's whole span is blanked. Remove + * the mask and `Inner` appears here. + */ + it("DT13: a component child's apparent headings are never targets", function* () { + const body = [ + "# Title", + "", + "", + "", + "# Inner", + "", + "some text", + "", + "## Inner two", + "", + "", + "", + "## Real", + "", + "real body", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT14: a heading overlapping component syntax is not addressable", function* () { + const body = ["# Title", "", "## Head tail", "", "## Real", ""].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT15: an interpolated heading is not addressable, and blocks its subtree", function* () { + const body = [ + "# Title", + "", + "## {meta.name}", + "", + "### Under computed", + "", + "## {binding}", + "", + "## {props.a.b}", + "", + "## Real", + "", + ].join("\n"); + expect(catalog(body)).toEqual(["Real"]); + }); + + it("DT16: escaped interpolation is static text and stays addressable", function* () { + const body = ["# Title", "", "## \\{meta.name\\}", "", "body", ""].join("\n"); + expect(catalog(body)).toEqual(["%7Bmeta.name%7D"]); + expect(selectTarget(outline(body), "%7Bmeta.name%7D").labels).toEqual(["{meta.name}"]); + }); + + /** + * The title is not a path level, so it is not a level that has to be + * addressable either — which is the whole reason the exception exists. + */ + it("DT17: a computed sole title still leaves its sections addressable", function* () { + const body = ["# {meta.title}", "", "## Real", "", "### Deeper", ""].join("\n"); + expect(catalog(body)).toEqual(["Real", "Real/Deeper"]); + }); + + it("DT18: a document with no heading has an empty catalog", function* () { + expect(catalog("just prose\n")).toEqual([]); + expect(refusal("just prose\n", "Anything").data.available).toEqual([]); + }); + + it("DT19: a sole title is itself no target", function* () { + expect(catalog("# Only\n\nbody\n")).toEqual([]); + }); +}); + +describe("Tier DT — target selectors", () => { + it("DT20: a literal selector matches one whole label", function* () { + expect(selectTarget(outline(SECTIONS), "Test/Node").labels).toEqual(["Test", "Node"]); + expect(refusal(SECTIONS, "Nod").data.kind).toBe("no-match"); + }); + + it("DT21: `*` matches within one level, in any position, more than once", function* () { + expect(selectTarget(outline(SECTIONS), "Test/N*").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "Test/*ode").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "Test/N*d*").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "*ther").target).toBe("Other"); + // One `*` never crosses a level boundary. + expect(refusal(SECTIONS, "*Node").data.kind).toBe("no-match"); + }); + + it("DT22: `**` matches zero or more complete levels", function* () { + expect(selectTarget(outline(SECTIONS), "**/Node").target).toBe("Test/Node"); + expect(selectTarget(outline(SECTIONS), "**/Other").target).toBe("Other"); + expect(selectTarget(outline(SECTIONS), "Other/**").target).toBe("Other"); + expect(selectTarget(outline(SECTIONS), "**/Bun/**").target).toBe("Test/Bun"); + }); + + it("DT23: a selector must name exactly one entry", function* () { + expect(refusal(SECTIONS, "**").data.kind).toBe("multiple-matches"); + expect(refusal(SECTIONS, "**").data.matches).toEqual([ + "Test", + "Test/Node", + "Test/Bun", + "Other", + ]); + expect(refusal(SECTIONS, "Missing").data.kind).toBe("no-match"); + expect(refusal(SECTIONS, "Missing").data.matches).toEqual([]); + expect(refusal(SECTIONS, "Missing").data.available).toEqual([ + "Test", + "Test/Node", + "Test/Bun", + "Other", + ]); + }); + + it("DT24: malformed selector syntax is refused as syntax", function* () { + for (const selector of ["", "/Test", "Test/", "Test//Node", "%zz", "Test/%2"]) { + expect(refusal(SECTIONS, selector).data.kind).toBe("invalid-selector"); + } + }); + + it("DT25: percent decoding is URI path decoding — `+` is a plus", function* () { + const body = ["# Title", "", "## a+b", "", "## a b", ""].join("\n"); + expect(catalog(body)).toEqual(["a%2Bb", "a%20b"]); + expect(selectTarget(outline(body), "a+b").labels).toEqual(["a+b"]); + expect(selectTarget(outline(body), "a%2Bb").labels).toEqual(["a+b"]); + expect(selectTarget(outline(body), "a%20b").labels).toEqual(["a b"]); + }); + + it("DT26: a malformed or NUL-bearing escape never decodes", function* () { + expect(refusal(SECTIONS, "%00").data.kind).toBe("invalid-selector"); + // A lone continuation byte is not UTF-8. + expect(refusal(SECTIONS, "%80").data.kind).toBe("invalid-selector"); + }); + + /** + * A backtracking matcher answers this in exponential time; the reachability + * sweep answers it in the product of the two lengths. A regression to + * backtracking does not fail this assertion — it never reaches it. + */ + it("DT27: a wildcard-dense selector against a long label terminates", function* () { + const label = "a".repeat(120); + const body = ["# Title", "", `## ${label}`, ""].join("\n"); + const selector = `${"*a".repeat(30)}*b`; + expect(refusal(body, selector).data.kind).toBe("no-match"); + expect(selectTarget(outline(body), `${"*a".repeat(30)}*`).labels).toEqual([label]); + }); + + it("DT28: whitespace beside a wildcard is matched; only the outer edges trim", function* () { + const spaced = ["# Title", "", "## alpha beta gamma", ""].join("\n"); + const joined = ["# Title", "", "## alphabetagamma", ""].join("\n"); + expect(selectTarget(outline(spaced), "alpha%20*%20gamma").labels).toEqual(["alpha beta gamma"]); + // The spaces around the wildcard are part of what was asked for. + expect(refusal(joined, "alpha%20*%20gamma").data.kind).toBe("no-match"); + // The level's own outer whitespace is not, so a padded selector still lands. + expect(selectTarget(outline(spaced), "%20alpha*gamma%20").labels).toEqual(["alpha beta gamma"]); + }); +}); + +describe("Tier DT — canonical references", () => { + it("DT29: a reference splits at the first raw `#`", function* () { + expect(fileSource("README.md")).toEqual({ path: "README.md" }); + expect(fileSource("README.md#Test/Node")).toEqual({ + path: "README.md", + target: "Test/Node", + }); + // A `#` inside the filename is written `%23`; the fragment keeps its own. + expect(fileSource("odd%23name.md#A%23B")).toEqual({ + path: "odd#name.md", + target: "A%23B", + }); + }); + + it("DT30: a path keeps its separators and decodes its escapes", function* () { + expect(fileSource("docs/sub%20dir/a.md").path).toBe("docs/sub dir/a.md"); + // A literal `%HH` in a filename is spelled `%25HH`. + expect(fileSource("lit%2520.md").path).toBe("lit%20.md"); + }); + + it("DT31: an unreadable reference says only that", function* () { + for (const reference of ["", "#Test", "a%zz.md", "a%00b.md"]) { + let caught: unknown; + try { + fileSource(reference); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(TypeError); + expect((caught as Error).message).toBe("Invalid document reference"); + expect(Object.hasOwn(caught as Error, "cause")).toBe(false); + } + }); + + it("DT32: formatting encodes the path and validates an exact target", function* () { + expect(formatDocumentReference("README.md")).toBe("README.md"); + expect(formatDocumentReference("docs/a b.md")).toBe("docs/a%20b.md"); + expect(formatDocumentReference("odd#name.md", "A%23B")).toBe("odd%23name.md#A%23B"); + // Already canonical: encoded once, never twice. + expect(formatDocumentReference("a.md", "a%2Fb")).toBe("a.md#a%2Fb"); + }); + + it("DT33: formatting refuses anything that is not an exact canonical target", function* () { + for (const target of ["", "Test/", "/Test", "Test/*", "**", "a/b*c", "a b", "a%2fb"]) { + let caught: unknown; + try { + formatDocumentReference("a.md", target); + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe("Invalid document reference"); + } + expect(isCanonicalTarget("Test/Node")).toBe(true); + expect(isCanonicalTarget("Test/%2A")).toBe(true); + expect(isCanonicalTarget("Test/*")).toBe(false); + }); + + /** + * Canonical means "exactly what the encoder would have written". Anything + * that decodes to a label needing normalization is a spelling of a target, + * not the target — accepting one would let two spellings of one section + * become two workflow-definition identities. + */ + it("DT48: a level is canonical only through the whole round trip", function* () { + for (const canonical of ["Caf%C3%A9", "a%20b", "A%2FB", "%2A", "%23", "a%2Bb", "Test/Node"]) { + expect(isCanonicalTarget(canonical)).toBe(true); + expect(formatDocumentReference("a.md", canonical)).toBe(`a.md#${canonical}`); + } + const rejected = [ + "Cafe%CC%81", // NFD — normalization would change it + "a%09b", // a tab is not an ASCII space + "a%20%20b", // uncollapsed whitespace + "%20a", // leading whitespace + "a%20", // trailing whitespace + "a%2fb", // lowercase escape + "A#B", // a raw `#` is the reference delimiter + "a//b", // an empty level + "a*b", // a raw wildcard operator + "%00", // NUL + ]; + for (const target of rejected) { + expect(isCanonicalTarget(target)).toBe(false); + } + }); + + it("DT49: a raw `#` is never a literal selector character, but `%23` is", function* () { + const body = ["# Title", "", "## A#B", "", "## Real", ""].join("\n"); + expect(catalog(body)).toEqual(["A%23B", "Real"]); + expect(selectTarget(outline(body), "A%23B").labels).toEqual(["A#B"]); + expect(refusal(body, "A#B").data.kind).toBe("invalid-selector"); + }); + + /** + * The formatter may only produce references the parser reads back. Sampling + * the rule would miss the two ways encoding loses information, so the + * implementation checks the round trip itself and these pin both losses. + */ + it("DT50: formatting refuses a path it could not encode losslessly", function* () { + // NUL, which the decoder refuses outright, and both halves of a broken + // surrogate pair, which encode lossily to the replacement character. + for (const path of ["a\u0000b.md", "lone\uD800.md", "trail\uDC00.md"]) { + let caught: unknown; + try { + formatDocumentReference(path); + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe("Invalid document reference"); + } + }); + + it("DT51: every formatted reference parses back to what it named", function* () { + const paths = [ + "README.md", + "docs/sub dir/a.md", + "odd#name.md", + "lit%20.md", + "café/ü.md", + "star*.md", + "a+b.md", + ]; + for (const path of paths) { + expect(fileSource(formatDocumentReference(path))).toEqual({ path }); + expect(fileSource(formatDocumentReference(path, "A%2FB"))).toEqual({ + path, + target: "A%2FB", + }); + } + }); +}); + +describe("Tier DT — projection", () => { + it("DT34: preamble, ancestor content, and the whole selected subtree", function* () { + expect(project(SECTIONS, "Test/Node")).toBe( + [ + "preamble", + "", + "# Title", + "", + "intro", + "", + "## Test", + "", + "test intro", + "", + "### Node", + "", + "node body", + "", + "", + ].join("\n"), + ); + }); + + it("DT35: selecting a non-leaf keeps every descendant", function* () { + const projected = project(SECTIONS, "Test"); + expect(projected).toContain("### Node"); + expect(projected).toContain("### Bun"); + expect(projected).not.toContain("## Other"); + expect(projected).not.toContain("other body"); + }); + + it("DT36: sibling subtrees are absent, earlier and later alike", function* () { + const projected = project(SECTIONS, "Test/Bun"); + expect(projected).toContain("bun body"); + expect(projected).not.toContain("node body"); + expect(projected).not.toContain("### Node"); + expect(projected).not.toContain("other body"); + }); + + it("DT37: retained headings and the sole title stay in the projection", function* () { + const projected = project(SECTIONS, "Test/Node"); + expect(projected).toContain("# Title"); + expect(projected).toContain("intro"); + expect(projected).toContain("## Test"); + expect(projected).toContain("test intro"); + }); + + it("DT38: with several outermost headings none is retained by default", function* () { + const body = ["pre", "", "# A", "", "a body", "", "# B", "", "b body", ""].join("\n"); + expect(project(body, "B")).toBe(["pre", "", "# B", "", "b body", ""].join("\n")); + }); + + it("DT39: an ancestor keeps only its own content, not an earlier sibling's", function* () { + const body = [ + "# Title", + "", + "title content", + "", + "## First", + "", + "first content", + "", + "## Second", + "", + "second content", + "", + ].join("\n"); + expect(project(body, "Second")).toBe( + ["# Title", "", "title content", "", "## Second", "", "second content", ""].join("\n"), + ); + }); +}); + +/** + * The preamble is what precedes the outline, not what precedes the title. + * + * A document may open at a deeper level than the one supplying its title. The + * shallowest heading is then not the first, and anchoring the preamble to it + * puts an earlier addressable section inside the preamble — where it is + * retained, rendered, and executed by every other target, while still being a + * target in its own right. + */ +describe("Tier DT — preamble boundary", () => { + const OPENS_DEEP = [ + "## Before", + "", + "before body", + "", + "# Title", + "", + "## After", + "", + "after body", + "", + ].join("\n"); + + it("DT63: a section before the title is addressable and is not preamble", function* () { + expect(catalog(OPENS_DEEP)).toEqual(["Before", "After"]); + expect(outline(OPENS_DEEP).preambleEnd).toBe(0); + }); + + it("DT64: selecting the later section retains neither the earlier one nor its body", function* () { + const projected = project(OPENS_DEEP, "After"); + expect(projected).toBe(["# Title", "", "## After", "", "after body", ""].join("\n")); + expect(projected).not.toContain("## Before"); + expect(projected).not.toContain("before body"); + }); + + it("DT65: the earlier section remains independently addressable", function* () { + // Its subtree runs to the next heading, so the blank line separating them + // is retained exactly as authored. + expect(project(OPENS_DEEP, "Before")).toBe(["## Before", "", "before body", "", ""].join("\n")); + }); + + it("DT66: real preamble text before the first heading is still retained", function* () { + const withPreamble = `preamble text\n\n${OPENS_DEEP}`; + expect(project(withPreamble, "After")).toBe( + ["preamble text", "", "# Title", "", "## After", "", "after body", ""].join("\n"), + ); + }); +}); + +describe("Tier DT — projected parsing", () => { + function* parsed(body: string, selector?: string) { + return yield* parseRootMarkdownDefinition("__root__", "doc.md", body, selector); + } + + it("DT40: a retained element keeps the offset and line it was authored at", function* () { + const body = [ + "# Title", + "", + "## Skipped", + "", + "x".repeat(400), + "", + "## Kept", + "", + "", + "", + ].join("\n"); + + const whole = yield* parsed(body); + const targeted = yield* parsed(body, "Kept"); + const positionOf = (definition: { bodySegments: readonly unknown[] }) => + definition.bodySegments + .flatMap((segment) => + typeof segment === "object" && + segment !== null && + "type" in segment && + segment.type === "component" + ? [segment] + : [], + ) + .map((segment) => (segment as { position?: unknown }).position); + + expect(positionOf(whole.definition)).toEqual(positionOf(targeted.definition)); + expect(targeted.target).toBe("Kept"); + }); + + it("DT41: CRLF source keeps its original offsets and lines too", function* () { + const body = [ + "# Title", + "", + "## Skipped", + "", + "skipped body", + "", + "## Kept", + "", + "", + "", + ].join("\r\n"); + const whole = yield* parsed(body); + const targeted = yield* parsed(body, "Kept"); + const componentsOf = (segments: readonly unknown[]) => + segments.flatMap((segment) => + typeof segment === "object" && + segment !== null && + "type" in segment && + segment.type === "component" + ? [segment as { position?: { offset: number; line: number } }] + : [], + ); + expect(componentsOf(targeted.definition.bodySegments)[0]?.position).toEqual( + componentsOf(whole.definition.bodySegments)[0]?.position, + ); + }); + + it("DT42: frontmatter, props, and the return mode survive projection", function* () { + const body = [ + "---", + "title: Doc", + "props:", + " name:", + " type: string", + "returns:", + " type: object", + "---", + "", + "# Title", + "", + "## Kept", + "", + "kept", + "", + ].join("\n"); + const targeted = yield* parsed(body, "Kept"); + expect(targeted.definition.meta).toEqual({ title: "Doc" }); + expect(targeted.definition.props).toMatchObject({ properties: { name: { type: "string" } } }); + expect(targeted.definition.returns).toMatchObject({ type: "object" }); + expect(targeted.targets).toEqual(["Kept"]); + }); + + it("DT43: the untargeted parse still scans the whole body", function* () { + const whole = yield* parsed(SECTIONS); + expect(whole.target).toBe(undefined); + expect(whole.targets).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + const text = whole.definition.bodySegments + .map((segment) => (segment.type === "text" ? segment.content : "")) + .join(""); + expect(text).toBe(SECTIONS); + }); +}); + +describe("Tier DT — inspection", () => { + it("DT44: inspection reports the catalog without selecting anything", function* (): Operation { + const info = yield* inspectDocument(inlineSource(SECTIONS)); + expect(info.targets).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + expect(info.target).toBe(undefined); + }); + + it("DT45: inspection resolves a glob to the exact canonical target", function* (): Operation { + const info = yield* inspectDocument(inlineSource(SECTIONS, { target: "**/N*" })); + expect(info.target).toBe("Test/Node"); + expect(info.targets).toEqual(["Test", "Test/Node", "Test/Bun", "Other"]); + }); + + it("DT46: an unresolvable target fails inspection", function* (): Operation { + let caught: unknown; + try { + yield* inspectDocument(inlineSource(SECTIONS, { target: "Nope" })); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(DocumentTargetError); + expect((caught as DocumentTargetError).data.kind).toBe("no-match"); + expect((caught as DocumentTargetError).data.selector).toBe("Nope"); + }); + + it("DT47: the error's data is frozen and rebuilt, not the parser's arrays", function* () { + const error = refusal(SECTIONS, "**"); + expect(Object.isFrozen(error.data.matches)).toBe(true); + expect(Object.isFrozen(error.data.available)).toBe(true); + expect(error.data.matches).not.toBe(outline(SECTIONS).targets); + // Encoded throughout, so a control character in a heading cannot reach a + // diagnostic literally. + expect(error.message).toContain('"**"'); + for (const line of error.message.split("\n").slice(1)) { + expect(line).not.toMatch(/[\u0000-\u001F]/); + } + }); +}); + +/** + * Recognition is the whole contract, so it is tested as one. + * + * The boundary is closed and it reconstructs: a candidate is validated field by + * field and a *fresh local* error is built from the result. Nothing the + * candidate owns is handed on, which is why these mutate and revoke the + * originals afterwards and assert the answer is unchanged. + * + * A separately loaded copy of this package is a different class producing the + * same name and the same tagged data, and must be read on exactly the same + * terms. Everything else — payload, a list that is not a catalog, fields that no + * selection could have produced — must be refused. + */ +describe("Tier DT — structural recognition", () => { + const CATALOG = ["# T", "", "## Alpha", "", "## Beta", ""].join("\n"); + + /** Genuine failure data, so the fixtures cannot drift from the real thing. */ + const GENUINE = refusal(CATALOG, "Missing").data; + const MESSAGE = refusal(CATALOG, "Missing").message; + + function data(change: Record = {}): Record { + return { + type: "executablemd.document-target-failure", + kind: GENUINE.kind, + selector: GENUINE.selector, + matches: [...GENUINE.matches], + available: [...GENUINE.available], + ...change, + }; + } + + function shell(payload: unknown, message = MESSAGE): Error { + const error = new Error(message); + error.name = "DocumentTargetError"; + Object.assign(error, { data: payload }); + return error; + } + + /** What a separately loaded copy produces: same shape, its own class. */ + function foreignError(): Error { + class DocumentTargetError extends Error { + readonly data = Object.freeze(data()); + constructor() { + super(MESSAGE); + this.name = "DocumentTargetError"; + } + } + return new DocumentTargetError(); + } + + it("DT52: a failure from another loaded copy is read on the same terms", function* () { + const foreign = foreignError(); + expect(foreign instanceof DocumentTargetError).toBe(false); + expect(isDocumentTargetError(foreign)).toBe(true); + expect(asDocumentTargetError(foreign)?.data.kind).toBe("no-match"); + }); + + it("DT53: this module's own failure is recognized", function* () { + const own = refusal(CATALOG, "Missing"); + expect(isDocumentTargetError(own)).toBe(true); + expect(asDocumentTargetError(own)?.data).toEqual(own.data); + }); + + /** + * The reconstruction claim, made where it can fail. A boundary that returned + * the candidate would pass every field assertion above and still hand a + * caller arrays somebody else can rewrite. + */ + it("DT54: recognition returns a fresh local error, never the candidate", function* () { + const foreign = foreignError(); + const safe = asDocumentTargetError(foreign); + + expect(safe).not.toBe(foreign); + expect(safe).toBeInstanceOf(DocumentTargetError); + const original = foreign as unknown as { data: { available: unknown } }; + expect(safe?.data).not.toBe(original.data); + expect(safe?.data.available).not.toBe(original.data.available); + expect(Object.isFrozen(safe?.data)).toBe(true); + expect(Object.isFrozen(safe?.data.available)).toBe(true); + }); + + it("DT55: a mutable nested list is copied, and later mutation changes nothing", function* () { + const available = ["Alpha", "Beta"]; + // Frozen outer data around a list its owner can still rewrite. + const candidate = shell(Object.freeze(data({ available }))); + const safe = asDocumentTargetError(candidate); + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + + available.push("Injected"); + available[0] = "Rewritten"; + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + expect(safe?.message).toBe(MESSAGE); + }); + + it("DT56: a revoked Proxy cannot reach through a result already built", function* () { + const revocable = Proxy.revocable(["Alpha", "Beta"], {}); + const candidate = shell(Object.freeze(data({ available: revocable.proxy }))); + const safe = asDocumentTargetError(candidate); + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + + revocable.revoke(); + // Reading the result must not touch the revoked original. + expect(safe?.data.available).toEqual(["Alpha", "Beta"]); + expect(safe?.message).toBe(MESSAGE); + expect(String(safe)).toContain("Alpha"); + // A candidate whose Proxy is already revoked is simply refused. + expect(isDocumentTargetError(candidate)).toBe(false); + expect(asDocumentTargetError(candidate)).toBe(undefined); + }); + + it("DT57: data-level extras are refused, enumerable, hidden, or symbol-keyed", function* () { + const enumerable = shell(Object.freeze(data({ extra: "payload" }))); + expect(isDocumentTargetError(enumerable)).toBe(false); + + const hidden = Object.freeze( + Object.defineProperty(data(), "extra", { value: "/etc/passwd", enumerable: false }), + ); + expect(isDocumentTargetError(shell(hidden))).toBe(false); + + const symbolic = Object.freeze( + Object.defineProperty(data(), Symbol.for("payload"), { + value: "/etc/passwd", + enumerable: true, + }), + ); + expect(isDocumentTargetError(shell(symbolic))).toBe(false); + }); + + /** + * Tested through the data parser, not through an Error shell. + * + * A shell carries a message derived from its data, so changing the data + * without rebuilding the message makes recognition fail on the message — + * which would make every case here green whether or not list validation + * exists. Going straight at `parseDocumentTargetFailure` keeps each case + * load-bearing. + */ + it("DT58: a list entry that is not an encoded canonical target is refused", function* () { + const rejected: unknown[][] = [ + // A raw space, tab or no-break space is not how a label is encoded. + ["Alpha Beta"], + ["Alpha\u0009Beta"], + ["Alpha\u00A0Beta"], + // Edge whitespace survives no round trip. + ["Alpha "], + // A lowercase escape is not what the encoder writes. + ["a%2fb"], + // NUL never decodes. + ["%00"], + // Not strings at all. + [1], + [null], + // A sparse list is not a dense one. + Object.assign(Array.from({ length: 2 }) as unknown[], { 0: "Alpha" }), + ]; + for (const available of rejected) { + expect(parseDocumentTargetFailure(Object.freeze(data({ available })))).toBe(undefined); + } + // The control: the same shape with a valid catalog parses. + expect(parseDocumentTargetFailure(Object.freeze(data()))).toBeDefined(); + }); + + /** + * `.` and `..` are ordinary heading labels — a document may really have a + * section called `..` — so `../../etc/passwd` is a well-formed four-level + * canonical *heading path*, never filesystem authority, and nothing here + * resolves it against a filesystem. + * + * Structural parsing therefore accepts it when the rest of the failure is + * consistent. What refuses it is the journal protocol, which compares the + * record against the catalog the recorded document derives — see TX31. + */ + it("DT59: a dotted heading path is canonical, and parses when consistent", function* () { + for (const path of ["../../etc/passwd", "Alpha/../Beta", "..", "."]) { + expect(isCanonicalTarget(path)).toBe(true); + } + const parsed = parseDocumentTargetFailure( + Object.freeze(data({ available: ["../../etc/passwd", "Alpha/../Beta"] })), + ); + expect(parsed?.available).toEqual(["../../etc/passwd", "Alpha/../Beta"]); + }); + + it("DT60: fields no selection could have produced are refused", function* () { + const inconsistent: Record[] = [ + // `no-match` whose selector really does match the catalog. + data({ kind: "no-match", selector: "Alpha", matches: [] }), + // `no-match` carrying matches. + data({ kind: "no-match", matches: ["Alpha"] }), + // `multiple-matches` with one match. + data({ kind: "multiple-matches", selector: "Alpha", matches: ["Alpha"] }), + // `multiple-matches` claiming a match outside the catalog. + data({ kind: "multiple-matches", selector: "**", matches: ["Alpha", "Gamma"] }), + // `invalid-selector` whose selector parses perfectly well. + data({ kind: "invalid-selector", selector: "Alpha" }), + // A kind outside the closed set. + data({ kind: "made-up" }), + // A missing member. + (() => { + const partial = data(); + delete partial["available"]; + return partial; + })(), + // A member of the wrong type. + data({ available: "Alpha" }), + data({ selector: 7 }), + ]; + for (const candidate of inconsistent) { + expect(parseDocumentTargetFailure(Object.freeze(candidate))).toBe(undefined); + } + }); + + it("DT61: the Error shell is closed too", function* () { + const withCause = shell(Object.freeze(data())); + Object.assign(withCause, { cause: new Error("foreign") }); + expect(isDocumentTargetError(withCause)).toBe(false); + + const withPayload = shell(Object.freeze(data())); + Object.assign(withPayload, { path: "/etc/passwd" }); + expect(isDocumentTargetError(withPayload)).toBe(false); + + // A diagnostic that does not derive from the data it claims. + expect(isDocumentTargetError(shell(Object.freeze(data()), "something else"))).toBe(false); + + for (const candidate of [undefined, null, "a string", new Error(MESSAGE), {}]) { + expect(isDocumentTargetError(candidate)).toBe(false); + expect(asDocumentTargetError(candidate)).toBe(undefined); + } + }); + + /** + * The payload question asked the way a consumer would ask it: after the + * boundary, is any of it still reachable by the ordinary means of passing an + * error on? + */ + it("DT62: no planted payload survives the boundary", function* () { + const planted = Object.freeze( + Object.defineProperty(data(), Symbol.for("secret"), { + value: "s3cret", + enumerable: true, + }), + ); + // Refused outright, so nothing to survive. + expect(asDocumentTargetError(shell(planted))).toBe(undefined); + + // And for a candidate that is accepted, the result carries only the + // contract: no extra own member, string or symbol, on the data or the error. + const safe = asDocumentTargetError(foreignError()); + expect(safe).toBeDefined(); + expect(Object.getOwnPropertyNames(safe?.data ?? {}).sort()).toEqual([ + "available", + "kind", + "matches", + "selector", + "type", + ]); + expect(Object.getOwnPropertySymbols(safe?.data ?? {})).toEqual([]); + expect(Object.keys(safe ?? {}).sort()).toEqual(["data", "name"]); + expect(Object.getOwnPropertySymbols(safe ?? {})).toEqual([]); + expect(JSON.stringify({ ...safe })).not.toContain("s3cret"); + expect(String(safe)).toBe(`DocumentTargetError: ${MESSAGE}`); + // A journal round trip rebuilds the same data from the same fields. + expect(parseDocumentTargetFailure(JSON.parse(JSON.stringify(safe?.data)))).toEqual(safe?.data); + }); +}); diff --git a/packages/durable-streams/README.md b/packages/durable-streams/README.md index 986f45a4..28364d85 100644 --- a/packages/durable-streams/README.md +++ b/packages/durable-streams/README.md @@ -377,15 +377,55 @@ Divergence detection catches _structural_ mismatches — the effect sequence cha The canonical example is a file-backed effect. If the workflow previously read `./component.mdx` and that file has since been edited, replaying the stored result would silently use stale content. A replay guard detects this and can halt replay with an error. -### The two-phase model +### The three stages -Every replay guard has two phases, separated by a strict I/O boundary: +A replay guard has three stages, separated by a strict I/O boundary: -**Phase 1 — `check`**: runs in generator context before replay begins. I/O is allowed. Use it to gather current state (compute file hashes, check timestamps) and cache results in the middleware closure. +**Stage 1 — `check`**: runs in generator context before replay begins, once per retained `Yield`. I/O is allowed. Use it to gather current state (compute file hashes, check timestamps) and cache results in the middleware closure. -**Phase 2 — `decide`**: runs synchronously inside the replay loop, after identity matching succeeds. Must be pure — no I/O, no side effects. Reads from the cache populated during `check` and returns a `ReplayOutcome`. +**Stage 2 — `admit`**: runs once after every retained event has been offered to `check`, and before a recorded terminal result is reused. It receives the retained history as a whole — the coroutine about to be resumed, its retained `Yield`s, and whether a terminal result exists for it. A guard that requires something of the history *as a whole* — that an event it validates is present at all, and present once — refuses here, because a per-event `check` has nothing to object to in a journal that simply omits the event. The default is a no-op. -This separation is necessary because the replay loop is synchronous. All observation-gathering must happen upfront. +**Stage 3 — `decide`**: runs synchronously inside the replay loop, after identity matching succeeds. Must be pure — no I/O, no side effects. Reads from the cache populated during `check` and returns a `ReplayOutcome`. + +The separation between generator and synchronous stages is necessary because the replay loop is synchronous. All observation-gathering must happen upfront. + +### Three views of one history + +A replay distinguishes three things, and conflating any two of them hands authority to whoever holds the wrong one: + +1. **The authoritative retained history.** What a consumer's own admission validated and what replay consumes. Detached from the backend and immutable to policy — its descriptions and results are frozen through, so nothing that receives it can rewrite what replay will decide. +2. **Isolated guard observations.** What `check`, `admit`, and `decide` receive: a deep, mutable copy made per invocation. Middleware may read, annotate, and compose over it freely; nothing it writes reaches replay. +3. **Values delivered to workflow code.** A fresh mutable copy taken from the authority at the moment of consumption — for a replayed effect and for a completed run's own return value alike. This is the only thing "mutable replayed value" ever means: a document that resumes on a restored binding writes to its copy, and writing to it cannot reach the authority or the next replay. + +The authority is frozen through in **every** settlement shape — a success with a value, a `Result` with none, a failure, and a cancellation, on `Yield` and `Close` alike. An envelope left writable is one a public observation could add a value to before replay reads it. + +Handing policy the authoritative events would let a guard rename effect A to B — so a workflow asking for B consumes A's result without B ever running — or rewrite a recorded root selection after admission accepted it. + +### Guards are policy, not authority + +A replay guard is **composable policy**. Guards compose through `Api.around`, and a handler installed further out may decline to call `next` — declining is what composition is for, and it means any single guard's opinion can be suppressed by another. + +That makes a guard the wrong place for anything that decides *identity*. A consumer whose durable identity must hold regardless of what a document, a component, or an enclosing scope installs owns that check itself — for example by wrapping the `DurableStream` it hands to `durableRun`, so the validation happens inside the read every later phase depends on, reachable through no context and replaceable by nothing. `@executablemd/core` does exactly that for a document's selected target. + +Use guards for staleness policy. Do not use them to enforce an invariant that must not be negotiable. + +### Retained events are read once + +Every phase of a replay reads the same events, and a journal is data a backend supplies. Events are therefore **retained**: read once and detached from whatever the backend still owns. + +**Every** event that participates in admission, indexing, or terminal reuse is retained — `Close` as well as `Yield`. A `Close` decides whether a coroutine has a terminal result to reuse, so leaving it as the backend's own object lets it belong to a child coroutine while one phase asks and to the root while the next does. + +The **discriminator** is settled by the classification that chooses an event's retained kind, and never read from the source again. **Identity** — the coroutine an event belongs to, and a `Yield`'s complete effect description — is settled once too, so no phase can be shown a different event than the phase before it. An event that refuses to say what it is is refused from every member. + +A `Yield`'s **settlement** stays lazy and separate, because the index is built before guards run and a guard that would refuse an event must get that chance before the stream is asked to produce a result. + +A `Close`'s result is settled **while the history is retained**, not at a first later read. A `Close` carries what a completed run hands back, and deferring that read leaves an interval — between the moment a consumer's own admission accepts the history and the moment terminal reuse consumes it — in which the backend still owns the answer and can replace it. Reading once at a later getter closes repeated reads and leaves that window open. + +Every cell keeps both outcomes: a refusal is remembered and re-raised rather than retried, so retaining a history never fails and a refusal reaches whichever phase asks. + +A retained event presents its members as ordinary own properties, so it spreads, serializes, and compares like the plain event a backend would have supplied. + +A detached result shares no object or array with the journal, and remains ordinary mutable JSON — detaching is a claim against the *stream*, not against the consumer, and replayed values are legitimately written to. ### Writing a replay guard @@ -706,6 +746,13 @@ transfers only the witness already associated with the exact source: an unproven source leaves the target unproven, and nesting trusted wrappers carries the same witness through each one. +A wrapping site is trusted when it is installed before any code the journal's +own content could influence, and delegates to the exact stream it was handed. +`@executablemd/core` has two, and a document execution's journal passes through +both: the secret filter, and the execution-owned wrapper that admits a run's +recorded target before replay. Each transfers only what its source already had, +so an unproven journal stays unproven through both. + --- ## Long-running workflows diff --git a/packages/durable-streams/effect.ts b/packages/durable-streams/effect.ts index 6eb28571..d3feb8bb 100644 --- a/packages/durable-streams/effect.ts +++ b/packages/durable-streams/effect.ts @@ -36,6 +36,7 @@ import { type LiveDurableOperationCoordinator, } from "./live-coordinator.ts"; import { ReplayGuard } from "./replay-guard.ts"; +import { consumable, observeEvent } from "./retained.ts"; import { protocolToEffection, serializeError } from "./serialize.ts"; import type { CoroutineView, @@ -133,12 +134,24 @@ function checkReplay( // Description matches — now check replay guards before replaying. // ── REPLAY GUARD: Decide phase ── - const yieldEvent: Yield = { + // An isolated observation, like the check and admit phases: a decision is + // policy, and policy reads. Handing the retained description or result + // here would let a guard rewrite what replay is about to consume. + const observed = observeEvent({ type: "yield", coroutineId: ctx.coroutineId, description: entry.description, result: entry.result, - }; + }); + const yieldEvent: Yield = + observed.type === "yield" + ? observed + : { + type: "yield", + coroutineId: ctx.coroutineId, + description: desc, + result: entry.result, + }; const outcome = ReplayGuard.invoke(routine.scope, "decide", [yieldEvent]); if (outcome.outcome === "error") { @@ -157,8 +170,11 @@ function checkReplay( // All guards approved — consume the entry and advance cursor ctx.replayIndex.consumeYield(ctx.coroutineId); - // Feed stored result synchronously - resolve(protocolToEffection(entry.result)); + // Feed stored result synchronously, as a fresh mutable copy: the + // authoritative result stays frozen so policy cannot rewrite it, while a + // document that resumes on a restored binding still writes to what it + // receives. + resolve(protocolToEffection(consumable(entry.result))); return { path: "replayed", teardown: (exit) => exit(VOID_OK) }; } diff --git a/packages/durable-streams/mod.ts b/packages/durable-streams/mod.ts index 2d67bc3e..36511ba6 100644 --- a/packages/durable-streams/mod.ts +++ b/packages/durable-streams/mod.ts @@ -25,6 +25,10 @@ export type { // ReplayIndex export { ReplayIndex } from "./replay-index.ts"; +// `retainEvents` is public because `@executablemd/core` owns its own journal +// gate and must produce the stable history across the package boundary. The +// retained classes and the detach helpers stay internal. +export { retainEvents } from "./retained.ts"; export type { YieldEntry } from "./replay-index.ts"; // Stream interface @@ -58,7 +62,7 @@ export type { DivergenceDecision, DivergenceInfo, DivergenceKind } from "./diver // ReplayGuard API — pluggable validation for replay staleness detection export { ReplayGuard } from "./replay-guard.ts"; -export type { ReplayOutcome } from "./replay-guard.ts"; +export type { ReplayOutcome, RetainedHistory } from "./replay-guard.ts"; // Context export { DurableContext } from "./context.ts"; diff --git a/packages/durable-streams/replay-guard.ts b/packages/durable-streams/replay-guard.ts index cec20778..9fa01392 100644 --- a/packages/durable-streams/replay-guard.ts +++ b/packages/durable-streams/replay-guard.ts @@ -12,14 +12,26 @@ * (e.g., content hash, status code). There is no separate metadata field — * inputs belong in the effect description, outputs belong in the result. * - * The API has two phases: + * A guard is **composable policy, not authority**. Guards compose through + * `Api.around`, and a handler installed further out may decline to call `next`. + * That is what composition is for, and it is why an invariant that must not be + * negotiable — durable identity above all — belongs somewhere a caller cannot + * replace, such as inside the `DurableStream` a consumer hands to `durableRun`. + * + * The API has three stages: * * 1. **check** (before replay begins): Runs in generator context inside * `durableRun`, after the journal is loaded but before the workflow starts. * I/O is allowed — this is where file hashing, network checks, and other * observation-gathering happens. Results are cached in middleware closures. * - * 2. **decide** (during replay): Runs synchronously inside + * 2. **admit** (after every check, before terminal reuse): Runs once with the + * retained history as a whole. A guard that requires something of the + * history rather than of one event — that an event it validates is present, + * and present once — refuses here, because a per-event check has nothing to + * object to in a journal that omits the event. Default is a no-op. + * + * 3. **decide** (during replay): Runs synchronously inside * `DurableEffect.enter()`, after identity matching succeeds but before * the stored result is fed to the generator. Must be pure and side-effect- * free. Reads from the cache populated during the check phase. @@ -33,7 +45,25 @@ import type { Api, Operation } from "effection"; import { createApi } from "effection/experimental"; -import type { Yield } from "./types.ts"; +import type { CoroutineId, Yield } from "./types.ts"; + +/** + * The retained history a run is about to replay, offered once. + * + * A guard's per-event check can only speak about events a journal contains. A + * journal missing something a guard requires offers it nothing to object to, + * and the recorded terminal result is then reused on the strength of history + * that was never validated. This is where a guard says whether the history as a + * whole may be replayed at all. + */ +export interface RetainedHistory { + /** The coroutine whose recorded terminal result is about to be reused. */ + readonly coroutineId: CoroutineId; + /** Every retained Yield, each owning the one cell for what it settled to. */ + readonly yields: readonly Yield[]; + /** Whether a recorded terminal result exists for that coroutine. */ + readonly terminal: boolean; +} /** * The outcome of a replay guard's decision. @@ -61,6 +91,13 @@ export type ReplayOutcome = { outcome: "replay" } | { outcome: "error"; error?: interface ReplayGuardApi { /** Phase 1: Check — gather observations before replay (I/O allowed). */ check(event: Yield): Operation; + /** + * Phase 1b: Admit — the retained history has been offered in full, and a + * recorded terminal result has not been reused yet. A guard that requires + * something of the history as a whole — that an event it validates is + * present at all, and present once — refuses here by throwing. + */ + admit(history: RetainedHistory): Operation; /** Phase 2: Decide — return replay outcome (synchronous, pure). */ decide(event: Yield): ReplayOutcome; } @@ -72,6 +109,13 @@ function* defaultCheck(_event: Yield): Operation { // No observation — pass through to next middleware or default. } +/** + * Default admit — no-op. A history nobody objects to is replayed. + */ +function* defaultAdmit(_history: RetainedHistory): Operation { + // No requirement — pass through to next middleware or default. +} + /** * Default decide — always replay. This preserves "logs are authoritative" * as the default behavior. Guards must be explicitly installed to add @@ -116,5 +160,5 @@ function defaultDecide(_event: Yield): ReplayOutcome { */ export const ReplayGuard: Api = createApi( "DurableEffection.ReplayGuard", - { check: defaultCheck, decide: defaultDecide }, + { check: defaultCheck, admit: defaultAdmit, decide: defaultDecide }, ); diff --git a/packages/durable-streams/replay-index.ts b/packages/durable-streams/replay-index.ts index 0d8e0e95..0902bf91 100644 --- a/packages/durable-streams/replay-index.ts +++ b/packages/durable-streams/replay-index.ts @@ -5,7 +5,15 @@ * to Close events. See spec §4.1. */ -import type { Close, CoroutineId, DurableEvent, EffectDescription, Result } from "./types.ts"; +import { retainEvents } from "./retained.ts"; +import type { + Close, + CoroutineId, + DurableEvent, + EffectDescription, + Result, + Yield, +} from "./types.ts"; export interface YieldEntry { description: EffectDescription; @@ -14,6 +22,8 @@ export interface YieldEntry { export class ReplayIndex { private yields = new Map(); + /** Every retained Yield in stream order, each owning its own settled cells. */ + private retained: Yield[] = []; private cursors = new Map(); private closes = new Map(); /** Coroutines where replay has been disabled (run-live mode). */ @@ -21,18 +31,24 @@ export class ReplayIndex { /** Retained coroutine identities reached by the current definition. */ private claimed = new Set(); + /** + * Index a journal's events by identity, without reading what they settled to. + * + * The events are retained first — idempotently, so a caller that already + * produced the stable history hands the same objects on rather than a second + * wrapping of them, and every phase then observes one identity and one + * settlement per event. + */ constructor(events: DurableEvent[]) { - for (const event of events) { + for (const event of retainEvents(events)) { if (event.type === "yield") { let list = this.yields.get(event.coroutineId); if (!list) { list = []; this.yields.set(event.coroutineId, list); } - list.push({ - description: event.description, - result: event.result, - }); + this.retained.push(event); + list.push(event); } if (event.type === "close") { this.closes.set(event.coroutineId, event); @@ -40,6 +56,16 @@ export class ReplayIndex { } } + /** + * Every retained Yield in stream order, as the events a check phase sees. + * + * The same objects the replay path consumes, so a guard and a later consumer + * observe one settled result rather than two reads of the stream. + */ + retainedYields(): Yield[] { + return [...this.retained]; + } + /** * Disable replay for a coroutine (run-live mode). * diff --git a/packages/durable-streams/retained.ts b/packages/durable-streams/retained.ts new file mode 100644 index 00000000..38022446 --- /dev/null +++ b/packages/durable-streams/retained.ts @@ -0,0 +1,452 @@ +/** + * Retained events — what a run reads a journal as. + * + * A journal is data supplied by a backend, and every phase of a replay reads + * the same events: a private authority gate, the replay index, public guard + * policy, and the replay path itself. If those are separate reads of the + * backend's own objects, a source that answers differently between them decides + * one thing for validation and another for execution, and nothing downstream + * can detect the substitution. + * + * A retained event is therefore read once and detached. **Every** event that + * participates in admission, indexing, or terminal reuse is retained, Close as + * well as Yield: a Close decides whether a coroutine has a terminal result to + * reuse, so leaving it as the backend's own object lets it belong to a child + * coroutine while one phase asks and to the root while the next does. + * + * The discriminator is settled once, by the classification that chooses a + * retained event's kind, and never read from the source again. Identity — the + * coroutine an event belongs to, and a Yield's complete effect description — is + * settled once too, so no phase can be shown a different event than the phase + * before it. + * + * A Yield's *settlement* stays lazy and separate: the index is built before + * guards run, and a guard that would refuse an event has to get that chance + * before the stream is asked to produce its result. A Close keeps its own cell, + * memoized the same way, so every later read receives the same detached answer. + */ + +import type { + Close, + CoroutineId, + DurableEvent, + EffectDescription, + Json, + Result, + SerializedError, + Yield, +} from "./types.ts"; + +/** What one retained value's single read of the stream produced. */ +type Settled = + | { readonly kind: "value"; readonly value: T } + | { readonly kind: "refusal"; readonly refusal: unknown }; + +function settle(read: () => T): Settled { + try { + return { kind: "value", value: read() }; + } catch (refusal) { + return { kind: "refusal", refusal }; + } +} + +function resolve(settled: Settled): T { + if (settled.kind === "refusal") { + throw settled.refusal; + } + return settled.value; +} + +/** + * A detached copy of one retained JSON value. + * + * Every property is read once and rebuilt, so nothing the stream still owns + * remains reachable: a nested accessor cannot answer one thing to one phase and + * another to the next, and no later mutation of the source changes what replay + * used. + * + * The copy is ordinary JSON. Detaching is the claim against the *stream*; + * making the copy immutable would be a claim against the *consumer*, and + * replayed values are legitimately mutable — an eval binding restored from a + * journal is pushed to by the iteration that resumes on it. Members are + * therefore writable and configurable like any other JSON. + * + * Keys are defined rather than assigned all the same, because `__proto__` + * reaches an inherited setter on some runtimes and would rewrite the copy's + * prototype instead of becoming a member of it. + * + * A cycle is refused. `Json` has none, and a value that does is not something + * this can detach — refusing is remembered like any other refusal. + */ +export function detachJson(value: Json, seen: Set = new Set()): Json { + if (value === null || typeof value !== "object") { + return value; + } + if (seen.has(value)) { + throw new TypeError("a retained value cannot contain a cycle"); + } + seen.add(value); + try { + if (Array.isArray(value)) { + const items: Json[] = []; + for (let index = 0; index < value.length; index++) { + items.push(detachJson(value[index]!, seen)); + } + return items; + } + const detached: { [key: string]: Json } = {}; + for (const [key, member] of Object.entries(value)) { + Object.defineProperty(detached, key, { + value: detachJson(member, seen), + enumerable: true, + writable: true, + configurable: true, + }); + } + return detached; + } finally { + seen.delete(value); + } +} + +/** + * A detached copy, frozen through. + * + * The authoritative retained graph is what admission validated and what replay + * consumes, so nothing that reaches a caller may write to it. Freezing is the + * claim against *policy*, not against a workflow: what a document finally + * receives is a fresh mutable copy taken from this, never this. + */ +function sealJson(value: Json): Json { + const detached = detachJson(value); + freezeDeep(detached); + return detached; +} + +function freezeDeep(value: Json): void { + if (value === null || typeof value !== "object") { + return; + } + Object.freeze(value); + for (const member of Array.isArray(value) ? value : Object.values(value)) { + freezeDeep(member); + } +} + +/** A detached copy of a retained failure's description. */ +function detachError(error: SerializedError): SerializedError { + const name = error.name; + const stack = error.stack; + return Object.freeze({ + message: error.message, + ...(name === undefined ? {} : { name }), + ...(stack === undefined ? {} : { stack }), + }); +} + +/** + * A retained result detached from everything the stream still owns. + * + * Each member is read exactly once, here, and the tree beneath it is rebuilt. + * Copying only the outer object would leave a `value` the journal can still + * rewrite, which is the substitution this exists to prevent. + */ +function detachResult(result: Result): Result { + const status = result.status; + if (status === "ok") { + // Every successful shape, including the one that settled to nothing. A + // `Result` carries no value to detach, but the envelope is still + // authority — left writable, a caller of a public observation could add one + // before replay reads it. + if (!("value" in result)) { + return Object.freeze({ status }); + } + const value = result.value; + return Object.freeze(value === undefined ? { status } : { status, value: sealJson(value) }); + } + if (status === "err") { + if (!("error" in result)) { + throw new TypeError("a retained failure carries the error it failed with"); + } + return Object.freeze({ status, error: detachError(result.error) }); + } + return Object.freeze({ status }); +} + +/** + * A detached copy of an effect description. + * + * `type` and `name` are the identity divergence detection compares; every other + * member is extra data a guard may read. All of it is rebuilt, so a description + * cannot name one effect while one phase looks and another while the next does. + */ +function detachDescription(description: EffectDescription): EffectDescription { + // One enumeration, so every member — `type` and `name` included — is read + // exactly once. Reading them directly and then enumerating would read each of + // them twice, which is the second read this exists to remove. + const members = Object.entries(description); + let type: Json | undefined; + let name: Json | undefined; + const extra: [string, Json][] = []; + for (const [key, member] of members) { + if (key === "type") { + type = member; + continue; + } + if (key === "name") { + name = member; + continue; + } + extra.push([key, member]); + } + if (typeof type !== "string" || typeof name !== "string") { + throw new TypeError("a retained effect description carries a type and a name"); + } + const detached: EffectDescription = { type, name }; + for (const [key, member] of extra) { + Object.defineProperty(detached, key, { + value: sealJson(member), + enumerable: true, + writable: false, + configurable: false, + }); + } + return Object.freeze(detached); +} + +/** Everything about a retained Yield except what it settled to. */ +interface RetainedIdentity { + readonly coroutineId: CoroutineId; + readonly description: EffectDescription; +} + +function readCoroutineId(source: { coroutineId: CoroutineId }): CoroutineId { + const coroutineId = source.coroutineId; + if (typeof coroutineId !== "string") { + throw new TypeError("a retained event belongs to a coroutine"); + } + return coroutineId; +} + +function readIdentity(source: Yield): RetainedIdentity { + return { + coroutineId: readCoroutineId(source), + description: detachDescription(source.description), + }; +} + +/** + * One retained Yield: one identity, and one cell for what it settled to. + * + * Identity is read together and once. Reading `type` here and `coroutineId` + * there would let a source present an unrelated event to one phase and the root + * import to the next, which is the whole reason identity is a single settled + * fact rather than three accessors. + * + * The settlement is separate and lazy on purpose: the index is built before + * guards run, so a guard that would refuse an event must get that chance before + * the stream is asked for its result. Both outcomes of both reads are kept — a + * refusal is remembered and re-raised rather than retried, so a source cannot + * refuse one phase and then answer the next. + */ +/** + * Present a retained member the way the event it stands for presents it. + * + * Own and enumerable, so a retained event spreads, serializes, and compares + * like the plain event a backend would have supplied. The settled cells stay + * genuinely private, which is what keeps them out of all of that. + */ +function present(target: object, name: string, read: () => T): void { + Object.defineProperty(target, name, { enumerable: true, get: read }); +} + +class RetainedYield implements Yield { + /** + * Settled by the classification that chose this wrapper, never re-read. A + * second read of the source's own discriminator is a second chance for it to + * answer differently, and an event that classifies as a Yield here and a + * Close there is the same substitution one level further out. + */ + declare readonly type: "yield"; + declare readonly coroutineId: CoroutineId; + declare readonly description: EffectDescription; + declare readonly result: Result; + #source: Yield; + #identity: Settled | undefined; + #settled: Settled | undefined; + + constructor(source: Yield) { + this.#source = source; + present(this, "type", () => "yield" as const); + present(this, "coroutineId", () => this.#stable().coroutineId); + present(this, "description", () => this.#stable().description); + present(this, "result", () => { + this.#settled ??= settle(() => detachResult(this.#source.result)); + return resolve(this.#settled); + }); + } + + #stable(): RetainedIdentity { + this.#identity ??= settle(() => readIdentity(this.#source)); + return resolve(this.#identity); + } +} + +/** + * One retained Close: settled identity, and one cell for its terminal result. + * + * A Close decides whether a coroutine has a terminal result to reuse, so it + * participates in admission exactly as a Yield does. Left as the backend's own + * object it could belong to a child coroutine while one phase asks and to the + * root while the next does — a history nobody could admit, reused as a result + * nobody asked for. + */ +class RetainedClose implements Close { + declare readonly type: "close"; + declare readonly coroutineId: CoroutineId; + declare readonly result: Result; + #identity: Settled; + #settled: Settled; + + constructor(source: Close) { + // Settled here, while the history is being retained, rather than at a first + // later read. A Close carries the result a completed run hands back, and + // deferring that read leaves an interval — between the moment a consumer's + // private admission accepts the history and the moment terminal reuse + // consumes it — in which the backend still owns the answer and can replace + // it. Reading once at a later getter closes repeated reads and leaves that + // window open. + // + // Settling cannot throw: a refusal is captured and re-raised from the + // getter, so retaining a history is never the thing that fails. + this.#identity = settle(() => readCoroutineId(source)); + this.#settled = settle(() => detachResult(source.result)); + present(this, "type", () => "close" as const); + present(this, "coroutineId", () => resolve(this.#identity)); + present(this, "result", () => resolve(this.#settled)); + } +} + +/** + * An event that would not say what it is. + * + * Classification is the one thing every later phase depends on, so an event + * that refuses it is not an event this history can describe. The refusal is + * remembered and re-raised from every member, rather than retried — a source + * that refuses one phase must not answer the next. + */ +class RetainedRefusal implements Yield { + declare readonly type: "yield"; + declare readonly coroutineId: CoroutineId; + declare readonly description: EffectDescription; + declare readonly result: Result; + #refusal: unknown; + + constructor(refusal: unknown) { + this.#refusal = refusal; + for (const name of ["type", "coroutineId", "description", "result"]) { + present(this, name, (): never => { + throw this.#refusal; + }); + } + } +} + +/** + * The retained form of a journal's events. + * + * Idempotent: retaining an already-retained event returns it, so a caller that + * has produced the stable history hands the same objects onward rather than a + * second wrapping of them. That is what lets one snapshot serve every phase. + * + * Only the event type is read here, which is the least a caller can read and + * still tell a Yield from a Close. Everything else is the retained event's own. + */ +export function retainEvents(events: readonly DurableEvent[]): DurableEvent[] { + return events.map((event) => { + if (isRetained(event)) { + return event; + } + // The one read of the source's discriminator. Whatever it says here is what + // the retained event reports from now on, to every phase. + const classified = settle(() => event.type); + if (classified.kind === "refusal") { + return new RetainedRefusal(classified.refusal); + } + if (classified.value === "yield") { + return new RetainedYield(event as Yield); + } + if (classified.value === "close") { + return new RetainedClose(event as Close); + } + return new RetainedRefusal(new TypeError("a retained event is a yield or a close")); + }); +} + +function isRetained(event: DurableEvent): boolean { + return ( + event instanceof RetainedYield || + event instanceof RetainedClose || + event instanceof RetainedRefusal + ); +} + +/** + * An isolated observation of a retained event, for public policy to read. + * + * A replay guard is composable policy, and composition means handlers read, + * annotate, and pass along. What it must never mean is that a handler edits the + * history the execution already validated: the authoritative graph is what + * admission accepted and what replay consumes, and a guard that could rewrite a + * root selection or an effect description after admission would hold exactly + * the authority the private gate exists to keep out of public hands. + * + * So policy reads a copy. It is deep and mutable, so middleware may compose over + * it as freely as it likes, and nothing it does reaches replay. + */ +export function observeEvent(event: DurableEvent): DurableEvent { + if (event.type === "close") { + return { type: "close", coroutineId: event.coroutineId, result: consumable(event.result) }; + } + return { + type: "yield", + coroutineId: event.coroutineId, + description: observeDescription(event.description), + result: consumable(event.result), + }; +} + +function observeDescription(description: EffectDescription): EffectDescription { + const copy: EffectDescription = { type: description.type, name: description.name }; + for (const [key, member] of Object.entries(description)) { + if (key === "type" || key === "name") { + continue; + } + Object.defineProperty(copy, key, { + value: detachJson(member), + enumerable: true, + writable: true, + configurable: true, + }); + } + return copy; +} + +/** + * A retained result as a consumer may hold it: ordinary mutable JSON. + * + * The authoritative copy is frozen so policy cannot rewrite it. A document + * that resumes on a restored binding writes to it, so what a workflow receives + * is a fresh copy taken from that authority rather than the authority itself. + */ +export function consumable(result: Result): Result { + if (result.status === "ok") { + return "value" in result && result.value !== undefined + ? { status: "ok", value: detachJson(result.value) } + : { status: "ok" }; + } + if (result.status === "err") { + return { status: "err", error: { ...result.error } }; + } + return { status: "cancelled" }; +} diff --git a/packages/durable-streams/run.ts b/packages/durable-streams/run.ts index 141ab862..31a55b88 100644 --- a/packages/durable-streams/run.ts +++ b/packages/durable-streams/run.ts @@ -19,6 +19,7 @@ import { DurableContext } from "./context.ts"; import { activeDurabilityFailure, appendDurableEvent } from "./durability.ts"; import { EarlyReturnDivergenceError, TerminalDivergenceError } from "./errors.ts"; import { ReplayGuard } from "./replay-guard.ts"; +import { consumable, observeEvent } from "./retained.ts"; import { ReplayIndex } from "./replay-index.ts"; import { deserializeError, serializeError } from "./serialize.ts"; import type { DurableStream } from "./stream.ts"; @@ -36,12 +37,22 @@ function unalignedReplay(replayIndex: ReplayIndex, coroutineId: string) { * phase to gather observations (hash files, check timestamps) and cache * results for the decide phase. * + * The events come from the index rather than from the stream, so a guard reads + * the same retained result the replay path will use. Handing over the stream's + * own events instead would make validation and consumption two separate reads, + * and a source that answered differently between them could have a guard + * approve one result while execution used another. + * * See replay-guard-spec.md §5.5. */ -function* runCheckPhase(events: DurableEvent[], scope: Scope): Operation { - for (const event of events) { - if (event.type === "yield") { - yield* ReplayGuard.invoke(scope, "check", [event]); +function* runCheckPhase(replayIndex: ReplayIndex, scope: Scope): Operation { + for (const event of replayIndex.retainedYields()) { + // An isolated observation, not the retained event. Guards compose by + // reading and passing along; what composition must not become is the power + // to edit a history the execution already validated. + const observed = observeEvent(event); + if (observed.type === "yield") { + yield* ReplayGuard.invoke(scope, "check", [observed]); } } } @@ -104,7 +115,23 @@ export function* durableRun( // files, make network requests) to gather observations for the decide // phase. The check loop iterates all Yield events in journal order. // See replay-guard-spec.md §5.5. - yield* runCheckPhase(events, scope); + yield* runCheckPhase(replayIndex, scope); + + // ── REPLAY GUARD: Admit phase ── + // The retained history has been offered in full and nothing has been reused + // yet. A guard that requires something of the history as a whole — that an + // event it validates is present, and present once — refuses here, before the + // recorded terminal result below can answer for history nobody validated. + yield* ReplayGuard.invoke(scope, "admit", [ + { + coroutineId, + yields: replayIndex.retainedYields().flatMap((event) => { + const observed = observeEvent(event); + return observed.type === "yield" ? [observed] : []; + }), + terminal: replayIndex.hasClose(coroutineId), + }, + ]); // If the root coroutine already has a Close event in the journal, // the workflow completed in a previous run. Return the stored result @@ -112,7 +139,12 @@ export function* durableRun( if (replayIndex.hasClose(coroutineId)) { const closeEvent = replayIndex.getClose(coroutineId)!; if (closeEvent.result.status === "ok") { - return closeEvent.result.value as T; + // A fresh consumer copy, exactly as a replayed Yield's result is. The + // retained settlement is frozen so policy cannot rewrite it; what a + // caller receives from a completed run is ordinary data it may hold and + // change, and changing it cannot reach the next replay. + const settled = consumable(closeEvent.result); + return (settled.status === "ok" ? settled.value : undefined) as T; } else if (closeEvent.result.status === "err") { throw deserializeError(closeEvent.result.error); } else { diff --git a/packages/durable-streams/specs/protocol-specification.md b/packages/durable-streams/specs/protocol-specification.md index 8892010a..d397864b 100644 --- a/packages/durable-streams/specs/protocol-specification.md +++ b/packages/durable-streams/specs/protocol-specification.md @@ -297,7 +297,103 @@ replay runs, even for children spawned during teardown. The replay index is a derived, in-memory structure built from the stream on startup. It provides per-coroutine cursored access to yield events -and keyed access to close events: +and keyed access to close events. + +Indexing reads each Yield's **identity** and not its result. A replay guard's +check phase runs after the index is built and before anything is replayed, so a +guard that would refuse an event has to get that chance before the stream is +asked to produce what the event settled to — an eager read hands a backend that +cannot produce a result the ability to fail past every guard, carrying its own +error. + +Each retained Yield owns one cell for what it settled to, and **that cell spans +the whole replay lifecycle**: the check phase is handed the retained Yields +rather than the stream's own events, so a guard that validates a result and the +replay path that later consumes it observe the same value. The stream's event is +consulted at most once. Reading once across phases is the property that makes +validation meaningful — with two reads, a source answering differently between +them has a guard approve one result while execution uses another, and nothing +downstream can detect the substitution. + +Both outcomes of that read are kept. The settlement is **detached**: the whole +result tree is rebuilt from members read once each, so nothing the stream still +owns remains reachable and a nested accessor cannot answer one thing to a guard +and another to replay. The detached copy is not frozen through — detaching is +the claim against the stream, while freezing would be a claim against the +consumer, and replayed values are legitimately mutable. A read that threw is +remembered and re-raised rather than retried, so a source cannot refuse the +guard and then answer replay. + +**Every** event that participates in admission, indexing, or terminal reuse is +retained — Close as well as Yield. A Close decides whether a coroutine has a +terminal result to reuse, so leaving it as the backend's own object lets it +belong to a child coroutine while one phase asks and to the root while the next +does: the phase that admits the history sees no terminal result, and the phase +that reuses one sees it. + +The discriminator is settled by the classification that chooses an event's +retained kind, and never read from the source again. Identity — the coroutine an +event belongs to, and a Yield's complete effect description — is settled once +too, so no phase can be shown a different event than the phase before it. An +event that refuses to say what it is is refused from every member. + +A Yield's settlement stays lazy, so a guard's check remains the first ordinary +consumer of a recorded result. + +A Close's result is settled while the history is retained. A Close carries what +a completed run hands back, and deferring that read leaves an interval — between +the moment a consumer's own admission accepts the history and the moment +terminal reuse consumes it — in which the backend still owns the answer and can +replace it. Memoizing on first access closes repeated reads and leaves that +window open. + +Every cell keeps both outcomes: a refusal is remembered and re-raised rather +than retried, so retaining a history never fails and a refusal reaches whichever +phase asks. + +The detached result is ordinary mutable JSON. Detaching is the claim against the +stream; making the copy immutable would be a claim against the consumer, and +replayed values are legitimately written to. + +After every retained event has been offered to `check` and before any recorded +terminal result is reused, guards receive the retained history once through +`admit`. A guard that requires something of the history as a whole — that an +event it validates is present, and present once — refuses there, because a +per-event check has nothing to object to in a journal that simply omits the +event. Its default is a no-op. + +A replay distinguishes three views of one history, and conflating any two hands +authority to whoever holds the wrong one: + +1. the **authoritative retained history**, which admission validated and replay + consumes — detached from the backend and immutable to policy, its + descriptions and results frozen through; +2. **isolated guard observations**, a deep mutable copy made per guard + invocation, over which middleware composes freely and from which nothing + flows back; and +3. **values delivered to workflow code**, a fresh mutable copy taken from the + authority at consumption — for a replayed effect and for a completed run's + own return value alike. This is the only thing "mutable replayed value" + means: writing to that copy reaches neither the authority nor the next + replay. + +The authority is frozen through in every settlement shape: a success with a +value, a `Result` with none, a failure, and a cancellation, on Yield and +Close alike. An envelope left writable is one a public observation could add a +value to before replay reads it. + +Handing policy the authoritative events would let a guard rename effect A to B — +so a workflow asking for B consumes A's result without B ever running — or +rewrite a recorded root selection after admission accepted it. + +Replay guards are **composable policy, not authority**. Guards compose through +middleware, and a handler installed further out may decline to delegate — which +is what composition is for. An invariant that must not be negotiable therefore +does not belong in a guard. A consumer whose durable identity depends on +validating retained history owns that validation itself, for example by wrapping +the `DurableStream` it passes to `durableRun` so the check happens inside the +read every later phase depends on. + ```typescript class ReplayIndex { @@ -309,10 +405,13 @@ class ReplayIndex { private closes = new Map(); constructor(events: DurableEvent[]) { - for (const event of events) { + // Retained first, idempotently: a caller that already produced the stable + // history hands the same objects on rather than a second wrapping of them. + for (const event of retainEvents(events)) { if (event.type === "yield") { const list = this.yields.get(event.coroutineId) ?? []; - list.push({ description: event.description, result: event.result }); + // Identity now; the result when a consumer asks. See below. + list.push(event); this.yields.set(event.coroutineId, list); } if (event.type === "close") { diff --git a/packages/durable-streams/tests/replay-guard.test.ts b/packages/durable-streams/tests/replay-guard.test.ts index 4baf074c..9a18fed6 100644 --- a/packages/durable-streams/tests/replay-guard.test.ts +++ b/packages/durable-streams/tests/replay-guard.test.ts @@ -11,12 +11,19 @@ */ import { describe, it } from "@executablemd/test-support/bdd"; -import { run } from "effection"; +import { run, scoped } from "effection"; import { expect } from "@executablemd/test-support/expect"; +import type { Operation } from "effection"; import { + createDurableOperation, type DurableEvent, + type DurableStream, InMemoryStream, + type Json, ReplayGuard, + ReplayIndex, + type Result, + retainEvents, type ReplayOutcome, StaleInputError, type Workflow, @@ -636,3 +643,486 @@ describe("replay guard", () => { } }); }); + +/** + * durableRun — a guard and the replay path read one settled result. + * + * The unit rows above prove repeated access through one `YieldEntry` is cached. + * They cannot prove the property that matters, which spans two phases: a guard + * validates a retained result in the check phase, and the replay path consumes + * it afterwards. If those are two reads of the stream's own event, a source + * answering differently between them has the guard approve one value while + * execution uses another, and nothing downstream can detect it. + * + * The journal here is partial — no Close — so replay consumes the retained + * Yield and then continues live, which is the shape that reaches both phases. + */ +describe("durableRun — one retained result across check and replay", () => { + it("hands the check phase and replay the same first answer", function* () { + const answers: Json[] = ["first", "second"]; + const reads = { count: 0 }; + + const event: Record = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + }; + const result: Record = { status: "ok" }; + Object.defineProperty(result, "value", { + enumerable: true, + get() { + const answer = answers[Math.min(reads.count, answers.length - 1)]; + reads.count++; + return answer; + }, + }); + event["result"] = result; + + const stream = new InMemoryStream(); + // Appended directly rather than through `append`, which clones and would + // resolve the accessor before the run ever starts. + const events = [event as unknown as DurableEvent]; + const reading: DurableStream = { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + append: (durable: DurableEvent) => stream.append(durable), + }; + + const checked: unknown[] = []; + const replayed = yield* scoped(function* () { + yield* ReplayGuard.around({ + *check([yielded], next) { + checked.push(yielded.result.status === "ok" ? yielded.result.value : undefined); + return yield* next(yielded); + }, + }); + return yield* durableRun( + function* () { + return (yield createDurableOperation({ type: "call", name: "work" }, function* () { + throw new Error("replay must not re-execute this effect"); + })) as Json; + }, + { stream: reading }, + ); + }); + + // The guard saw the first answer, replay used the first answer, and the + // stream was asked exactly once. + expect(checked).toEqual(["first"]); + expect(replayed).toBe("first"); + expect(reads.count).toBe(1); + }); +}); + +/** + * Detachment reaches the whole result, not its outermost object. + * + * Freezing `{ ...result }` stops nobody: `value` is still the journal's own + * object, and an accessor beneath it answers afresh on every read. The guard + * validates one tree and replay consumes another, with nothing in between able + * to notice. + */ +/** One member of a retained JSON object, when it really is one. */ +function member(value: Json | undefined, key: string): Json | undefined { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value[key] + : undefined; +} + +describe("durableRun — a retained result detaches completely", () => { + it("a nested accessor is read once and cannot answer twice", function* () { + const answers = ["first", "second"]; + const reads = { count: 0 }; + + const nested: Record = {}; + Object.defineProperty(nested, "target", { + enumerable: true, + get() { + const answer = answers[Math.min(reads.count, answers.length - 1)]; + reads.count++; + return answer; + }, + }); + + const event = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok", value: nested }, + }; + const events = [event as unknown as DurableEvent]; + const appended = new InMemoryStream(); + const reading: DurableStream = { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + append: (durable: DurableEvent) => appended.append(durable), + }; + + const checked: unknown[] = []; + const replayed = yield* scoped(function* () { + yield* ReplayGuard.around({ + *check([yielded], next) { + checked.push( + member(yielded.result.status === "ok" ? yielded.result.value : undefined, "target"), + ); + return yield* next(yielded); + }, + }); + return yield* durableRun( + function* () { + return (yield createDurableOperation({ type: "call", name: "work" }, function* () { + throw new Error("replay must not re-execute this effect"); + })) as Json; + }, + { stream: reading }, + ); + }); + + expect(checked).toEqual(["first"]); + expect(member(replayed, "target")).toBe("first"); + expect(reads.count).toBe(1); + }); + + it("a nested member that refuses is refused once, not retried", function* () { + const reads = { count: 0 }; + const nested: Record = {}; + Object.defineProperty(nested, "target", { + enumerable: true, + get() { + reads.count++; + throw new Error("the backend will not produce this member"); + }, + }); + + const index = new ReplayIndex([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok", value: nested }, + } as unknown as DurableEvent, + ]); + const entry = index.peekYield("root"); + + for (let attempt = 0; attempt < 3; attempt++) { + let caught: unknown; + try { + void entry?.result; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not produce this member", + ); + } + // Refused once and remembered, so a source cannot refuse the guard and then + // answer replay. + expect(reads.count).toBe(1); + }); + + it("a detached result shares nothing with the source it was read from", function* () { + // Detachment is the claim, not immutability: the copy is ordinary JSON and + // a consumer may write to it, which `updates an existing property` covers. + const list = ["a"]; + const nested = { list }; + const source = { nested }; + const index = new ReplayIndex([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok", value: source }, + } as unknown as DurableEvent, + ]); + const result = index.peekYield("root")?.result; + const value = result?.status === "ok" ? result.value : undefined; + + // Nothing beneath the settlement is the source's. + expect(value).not.toBe(source); + expect(member(value, "nested")).not.toBe(nested); + expect(member(member(value, "nested"), "list")).not.toBe(list); + + // So rewriting the source afterwards cannot change what replay will use. + list[0] = "rewritten"; + list.push("injected"); + expect(member(member(value, "nested"), "list")).toEqual(["a"]); + }); +}); + +/** + * Guard policy cannot rewrite the identity or result replay consumes. + * + * A replay guard is composable policy. Composition means reading, annotating, + * and passing along; it must not mean editing the history the execution already + * validated. Given the retained events themselves, a guard could rename effect + * A to B and have a workflow asking for B consume A's result, without B ever + * running. + */ +describe("replay guard — observation is isolated from replay authority", () => { + function journal(): DurableEvent[] { + return [ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "A" }, + result: { status: "ok", value: "A-result" }, + }, + ]; + } + + function reading(events: DurableEvent[], appended: DurableEvent[]): DurableStream { + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + } + + /** Run a workflow asking for `name`, with `install` in scope. */ + function* asking( + name: string, + events: DurableEvent[], + install: () => Operation, + ): Operation<{ value: unknown; error: unknown; ran: boolean; appended: DurableEvent[] }> { + const appended: DurableEvent[] = []; + let ran = false; + const outcome = yield* scoped(function* () { + yield* install(); + try { + const value = yield* durableRun( + function* () { + return (yield createDurableOperation({ type: "call", name }, function* () { + ran = true; + return `${name}-live`; + })) as Json; + }, + { stream: reading(events, appended) }, + ); + return { value, error: undefined }; + } catch (error) { + return { value: undefined, error }; + } + }); + return { ...outcome, ran, appended }; + } + + it("check cannot rename A to B and feed B the A result", function* () { + const outcome = yield* asking("B", journal(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + Object.assign(event.description, { name: "B" }); + return yield* next(event); + }, + }); + }); + // B never received A's result: either B ran live, or the run failed. + expect(outcome.value).not.toBe("A-result"); + if (outcome.error === undefined) { + expect(outcome.ran).toBe(true); + expect(outcome.value).toBe("B-live"); + } + }); + + it("admit cannot rename A to B and feed B the A result", function* () { + const outcome = yield* asking("B", journal(), function* () { + yield* ReplayGuard.around({ + *admit([history], next) { + for (const event of history.yields) { + Object.assign(event.description, { name: "B" }); + } + return yield* next(history); + }, + }); + }); + expect(outcome.value).not.toBe("A-result"); + }); + + it("decide cannot rewrite the result replay is about to feed", function* () { + const outcome = yield* asking("A", journal(), function* () { + yield* ReplayGuard.around({ + decide([event], next) { + if (event.result.status === "ok") { + Object.assign(event.result, { value: "rewritten-by-policy" }); + } + return next(event); + }, + }); + }); + expect(outcome.value).toBe("A-result"); + expect(outcome.ran).toBe(false); + }); + + it("a public ReplayIndex observation cannot change what replay decides", function* () { + const events = journal(); + const index = new ReplayIndex(retainEvents(events)); + const entry = index.peekYield("root"); + expect(entry).toBeDefined(); + + // Whatever a caller does with what the index hands back, the retained + // answer is unchanged. + try { + Object.assign(entry!.description, { name: "B" }); + } catch { + // A frozen description refuses the write outright, which is the same + // conclusion reached more directly. + } + expect(index.peekYield("root")?.description.name).toBe("A"); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: "A-result" }); + }); + + it("the control — an unmodified guard still replays the recorded result", function* () { + const seen: string[] = []; + const outcome = yield* asking("A", journal(), function* () { + yield* ReplayGuard.around({ + *check([event], next) { + seen.push(event.description.name); + return yield* next(event); + }, + }); + }); + expect(seen).toEqual(["A"]); + expect(outcome.value).toBe("A-result"); + expect(outcome.ran).toBe(false); + }); +}); + +/** + * Every Result shape is authority; every consumer value is a copy. + * + * The distinction has to hold for all four settlements, not only the one that + * carries data. A `Result` left writable is an envelope a public + * observation could add a value to before replay reads it, and a completed run + * that hands back the retained value itself gives a caller a frozen object + * where the live run gave ordinary JSON. + */ +describe("replay authority — every settlement, and its consumer copy", () => { + function completed(value: Json): DurableEvent[] { + return [{ type: "close", coroutineId: "root", result: { status: "ok", value } }]; + } + + function reading(events: DurableEvent[], appended: DurableEvent[]): DurableStream { + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return events; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; + } + + function* replayCompleted( + events: DurableEvent[], + appended: DurableEvent[], + ran: { live: boolean }, + ): Operation { + return yield* durableRun( + function* () { + ran.live = true; + return "live" as Json; + }, + { stream: reading(events, appended) }, + ); + } + + it("a completed run hands back ordinary mutable JSON", function* () { + const appended: DurableEvent[] = []; + const ran = { live: false }; + const events = completed({ count: 1, tags: ["a"], nested: { deep: true } }); + const value = yield* replayCompleted(events, appended, ran); + + const held = value as Record; + held["count"] = 2; + (held["tags"] as Json[]).push("b"); + (held["nested"] as Record)["deep"] = false; + expect(held).toEqual({ count: 2, tags: ["a", "b"], nested: { deep: false } }); + + // Completed replay ran nothing and wrote nothing. + expect(ran.live).toBe(false); + expect(appended).toEqual([]); + }); + + it("mutating a completed value cannot change the next replay", function* () { + const events = completed({ count: 1 }); + const first = yield* replayCompleted(events, [], { live: false }); + (first as Record)["count"] = 99; + + const second = yield* replayCompleted(events, [], { live: false }); + expect(second).toEqual({ count: 1 }); + }); + + it("a retained ok-without-value settlement is authority", function* () { + const index = new ReplayIndex( + retainEvents([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok" }, + }, + ]), + ); + const settled = index.peekYield("root")?.result; + expect(settled).toEqual({ status: "ok" }); + try { + Object.assign(settled as object, { value: "planted" }); + } catch { + // A frozen settlement refuses outright, which is the same conclusion. + } + expect(index.peekYield("root")?.result).toEqual({ status: "ok" }); + }); + + it("a retained void Close settlement is authority", function* () { + const index = new ReplayIndex( + retainEvents([{ type: "close", coroutineId: "root", result: { status: "ok" } }]), + ); + const settled = index.getClose("root")?.result; + try { + Object.assign(settled as object, { value: "planted" }); + } catch { + // As above. + } + expect(index.getClose("root")?.result).toEqual({ status: "ok" }); + }); + + it("ok(value), ok(void), err and cancelled are all frozen through", function* () { + const results: Result[] = [ + { status: "ok", value: { nested: ["a"] } }, + { status: "ok" }, + { status: "err", error: { message: "failed" } }, + { status: "cancelled" }, + ]; + for (const result of results) { + const index = new ReplayIndex( + retainEvents([ + { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result, + }, + ]), + ); + const settled = index.peekYield("root")!.result; + expect(Object.isFrozen(settled)).toBe(true); + if (settled.status === "ok" && "value" in settled && settled.value !== undefined) { + expect(Object.isFrozen(settled.value)).toBe(true); + expect(Object.isFrozen(member(settled.value, "nested"))).toBe(true); + } + if (settled.status === "err") { + expect(Object.isFrozen(settled.error)).toBe(true); + } + } + }); +}); diff --git a/packages/durable-streams/tests/replay-index.test.ts b/packages/durable-streams/tests/replay-index.test.ts index f9b15dc3..b046d999 100644 --- a/packages/durable-streams/tests/replay-index.test.ts +++ b/packages/durable-streams/tests/replay-index.test.ts @@ -298,3 +298,88 @@ describe("ReplayIndex", () => { }); }); }); + +/** + * Ordering — indexing reads identity, never a Yield's result. + * + * A replay guard's check phase runs after the index is built and before + * anything is replayed, so a guard that would refuse an event has to get its + * chance before the stream is asked what that event settled to. Reading + * eagerly took that chance away: a backend that could not produce a result + * failed inside construction, carrying its own error out past every guard. + * + * These are unit-level on purpose. The consequence for a document run is + * covered in `@executablemd/core`'s Tier TX; what belongs here is the ordering + * property itself. + */ +describe("ReplayIndex — result access ordering", () => { + /** A Yield whose result refuses to be read, counting attempts. */ + function refusingYield(coroutineId: string, reads: { count: number }): DurableEvent { + const event: Record = { + type: "yield", + coroutineId, + description: { type: "call", name: "work" }, + }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + reads.count++; + throw new Error("the backend will not produce this result"); + }, + }); + return event as unknown as DurableEvent; + } + + it("builds an index over a Yield whose result cannot be read", function* () { + const reads = { count: 0 }; + const index = new ReplayIndex([refusingYield("root", reads)]); + + // Construction completed, and it never asked. + expect(reads.count).toBe(0); + // Identity is indexed, because indexing is what identity is for. + expect(index.yieldCount("root")).toBe(1); + expect(index.peekYield("root")?.description).toEqual({ type: "call", name: "work" }); + expect(index.hasAnyUnconsumedYields()).toBe(true); + // Still never asked: peeking is not consuming. + expect(reads.count).toBe(0); + }); + + it("reads the result only when a consumer asks for it", function* () { + const reads = { count: 0 }; + const index = new ReplayIndex([refusingYield("root", reads)]); + const entry = index.peekYield("root"); + expect(entry).toBeDefined(); + + let caught: unknown; + try { + void entry?.result; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe("the backend will not produce this result"); + expect(reads.count).toBe(1); + }); + + it("reads a result once, so a changing source cannot change replay", function* () { + let answers = 0; + const event: Record = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + answers++; + return { status: "ok", value: answers }; + }, + }); + + const index = new ReplayIndex([event as unknown as DurableEvent]); + const entry = index.peekYield("root"); + expect(entry?.result).toEqual({ status: "ok", value: 1 }); + expect(entry?.result).toEqual({ status: "ok", value: 1 }); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: 1 }); + expect(answers).toBe(1); + }); +}); diff --git a/packages/durable-streams/tests/retained.test.ts b/packages/durable-streams/tests/retained.test.ts new file mode 100644 index 00000000..72f6251c --- /dev/null +++ b/packages/durable-streams/tests/retained.test.ts @@ -0,0 +1,445 @@ +/** + * Retained history — one settled answer per event, for every phase. + * + * A journal is data a backend supplies, and every phase of a replay reads the + * same events: a consumer's own admission gate, the replay index, public guard + * policy, and the replay path. Where those are separate reads of the backend's + * objects, a source that answers differently between them decides one thing for + * validation and another for execution. + * + * Classification is the read everything else rests on: an event that is a Yield + * to one phase and a Close to the next cannot be reasoned about at all. These + * go through `retainEvents`, which is how a run actually obtains its history, + * rather than constructing wrappers directly. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { detachJson, retainEvents } from "../retained.ts"; +import { ReplayIndex } from "../replay-index.ts"; +import type { DurableEvent, Json } from "../types.ts"; + +/** An event whose members answer from a list, counting reads per member. */ +function shifting( + members: Record, + fixed: Record = {}, +): { event: DurableEvent; reads: Record } { + const reads: Record = {}; + const event: Record = { ...fixed }; + for (const [key, answers] of Object.entries(members)) { + reads[key] = 0; + Object.defineProperty(event, key, { + enumerable: true, + get() { + const index = Math.min(reads[key]!, answers.length - 1); + reads[key] = reads[key]! + 1; + return answers[index]; + }, + }); + } + return { event: event as unknown as DurableEvent, reads }; +} + +function retain(event: DurableEvent): DurableEvent { + return retainEvents([event])[0]!; +} + +describe("retained history — classification settles once", () => { + it("an event that turns from Yield into Close stays a Yield", function* () { + const { event, reads } = shifting( + { type: ["yield", "close"] }, + { + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok" }, + }, + ); + const retained = retain(event); + expect(retained.type).toBe("yield"); + expect(retained.type).toBe("yield"); + // Classified once; the wrapper never asks the source again. + expect(reads["type"]).toBe(1); + }); + + it("an event that turns from Close into Yield stays a Close", function* () { + const { event, reads } = shifting( + { type: ["close", "yield"] }, + { coroutineId: "root", result: { status: "ok", value: "kept" } }, + ); + const retained = retain(event); + expect(retained.type).toBe("close"); + expect(retained.type).toBe("close"); + expect(reads["type"]).toBe(1); + }); + + it("a discriminator that refuses is refused from every member, once", function* () { + let asked = 0; + const event: Record = {}; + Object.defineProperty(event, "type", { + enumerable: true, + get() { + asked++; + throw new Error("the backend will not say what this event is"); + }, + }); + const retained = retain(event as unknown as DurableEvent); + for (const read of [() => retained.type, () => retained.coroutineId, () => retained.result]) { + let caught: unknown; + try { + read(); + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not say what this event is", + ); + } + expect(asked).toBe(1); + }); + + it("an event that is neither is refused rather than passed through", function* () { + const retained = retain({ type: "something-else" } as unknown as DurableEvent); + let caught: unknown; + try { + void retained.type; + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(TypeError); + }); + + it("retaining an already-retained event returns it unchanged", function* () { + const once = retain({ + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + result: { status: "ok" }, + }); + expect(retainEvents([once])[0]).toBe(once); + }); +}); + +describe("retained history — a Close settles once", () => { + it("a coroutine that moves from a child to the root stays the child's", function* () { + const { event, reads } = shifting( + { coroutineId: ["root.7", "root"] }, + { type: "close", result: { status: "ok", value: "alpha" } }, + ); + const retained = retain(event); + expect(retained.coroutineId).toBe("root.7"); + expect(retained.coroutineId).toBe("root.7"); + expect(reads["coroutineId"]).toBe(1); + + // And the index — the phase that decides whether a terminal result exists — + // sees exactly what the first reader saw. + const index = new ReplayIndex([retained]); + expect(index.hasClose("root")).toBe(false); + expect(index.hasClose("root.7")).toBe(true); + }); + + it("a coroutine that moves from the root to a child stays the root's", function* () { + const { event } = shifting( + { coroutineId: ["root", "root.7"] }, + { type: "close", result: { status: "ok", value: "alpha" } }, + ); + const index = new ReplayIndex([retain(event)]); + expect(index.hasClose("root")).toBe(true); + expect(index.hasClose("root.7")).toBe(false); + }); + + it("a successful terminal result that changes is answered once", function* () { + const { event, reads } = shifting( + { + result: [ + { status: "ok", value: "first" }, + { status: "ok", value: "second" }, + ], + }, + { type: "close", coroutineId: "root" }, + ); + const retained = retain(event); + expect(retained.result).toEqual({ status: "ok", value: "first" }); + expect(retained.result).toEqual({ status: "ok", value: "first" }); + expect(reads["result"]).toBe(1); + }); + + it("a failed terminal result that changes is answered once", function* () { + const { event } = shifting( + { + result: [ + { status: "err", error: { message: "first" } }, + { status: "err", error: { message: "second" } }, + ], + }, + { type: "close", coroutineId: "root" }, + ); + const retained = retain(event); + expect(retained.result).toEqual({ status: "err", error: { message: "first" } }); + expect(retained.result).toEqual({ status: "err", error: { message: "first" } }); + }); + + it("a terminal result that refuses and then answers stays refused", function* () { + let asked = 0; + const event: Record = { type: "close", coroutineId: "root" }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + asked++; + if (asked === 1) { + throw new Error("the backend will not produce this result"); + } + return { status: "ok", value: "answered later" }; + }, + }); + const retained = retain(event as unknown as DurableEvent); + for (let attempt = 0; attempt < 3; attempt++) { + let caught: unknown; + try { + void retained.result; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not produce this result", + ); + } + expect(asked).toBe(1); + }); + + /** + * Detached while the history is retained, not at a first later read. + * + * Memoizing on first access closes repeated reads and leaves the interval + * between a consumer's admission and terminal reuse open: nobody has touched + * the getter yet, so the backend still owns the answer. + */ + it("a terminal result is detached before anyone reads it", function* () { + const value: Record = { output: "original" }; + const retained = retain({ + type: "close", + coroutineId: "root", + result: { status: "ok", value }, + } as unknown as DurableEvent); + + // Mutated before the retained result has ever been read. + value["output"] = "planted after retention"; + expect(retained.result).toEqual({ status: "ok", value: { output: "original" } }); + }); + + it("a terminal result is detached from the source", function* () { + const value = { list: ["a"] }; + const retained = retain({ + type: "close", + coroutineId: "root", + result: { status: "ok", value }, + } as unknown as DurableEvent); + const held = retained.result; + value.list.push("injected"); + expect(held).toEqual({ status: "ok", value: { list: ["a"] } }); + }); + + it("a cancelled terminal result is retained as itself", function* () { + const retained = retain({ + type: "close", + coroutineId: "root", + result: { status: "cancelled" }, + } as unknown as DurableEvent); + expect(retained.result).toEqual({ status: "cancelled" }); + }); + + it("the control — an intact history classifies and settles normally", function* () { + const events: DurableEvent[] = [ + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository" } }, + }, + { type: "close", coroutineId: "root", result: { status: "ok", value: "done" } }, + ]; + const index = new ReplayIndex(retainEvents(events)); + expect(index.peekYield("root")?.description).toEqual({ + type: "import_component", + name: "__root__", + }); + expect(index.hasClose("root")).toBe(true); + expect(index.getClose("root")?.result).toEqual({ status: "ok", value: "done" }); + }); +}); + +describe("retained history — a Yield's identity settles once", () => { + const BASE = { result: { status: "ok", value: 1 } }; + + it("a name that turns into __root__ is seen the same way by every phase", function* () { + const { event } = shifting( + { name: ["unrelated", "__root__"] }, + { type: "yield", coroutineId: "root" }, + ); + const description = { type: "import_component" }; + Object.defineProperty(description, "name", { + enumerable: true, + get: Object.getOwnPropertyDescriptor(event, "name")!.get!, + }); + const retained = retain({ + type: "yield", + coroutineId: "root", + description, + ...BASE, + } as unknown as DurableEvent); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("unrelated"); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("unrelated"); + }); + + it("a name that turns away from __root__ keeps naming the root import", function* () { + const { event } = shifting( + { name: ["__root__", "unrelated"] }, + { type: "yield", coroutineId: "root" }, + ); + const description = { type: "import_component" }; + Object.defineProperty(description, "name", { + enumerable: true, + get: Object.getOwnPropertyDescriptor(event, "name")!.get!, + }); + const retained = retain({ + type: "yield", + coroutineId: "root", + description, + ...BASE, + } as unknown as DurableEvent); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("__root__"); + expect(retained.type === "yield" ? retained.description.name : undefined).toBe("__root__"); + }); + + it("a shifting coroutine id settles once", function* () { + const { event, reads } = shifting( + { coroutineId: ["root", "root.3"] }, + { type: "yield", description: { type: "call", name: "work" }, ...BASE }, + ); + const retained = retain(event); + expect(retained.coroutineId).toBe("root"); + expect(retained.coroutineId).toBe("root"); + expect(reads["coroutineId"]).toBe(1); + }); + + it("a shifting nested description member settles once", function* () { + const description: Record = { type: "call", name: "work" }; + let asked = 0; + Object.defineProperty(description, "marker", { + enumerable: true, + get() { + asked++; + return asked === 1 ? "stable" : "swapped"; + }, + }); + const retained = retain({ + type: "yield", + coroutineId: "root", + description, + ...BASE, + } as unknown as DurableEvent); + const read = () => (retained.type === "yield" ? retained.description["marker"] : undefined); + expect(read()).toBe("stable"); + expect(read()).toBe("stable"); + expect(asked).toBe(1); + }); + + it("an identity accessor that refuses is refused once, not retried", function* () { + let asked = 0; + const event: Record = { + type: "yield", + description: { type: "call", name: "work" }, + ...BASE, + }; + Object.defineProperty(event, "coroutineId", { + enumerable: true, + get() { + asked++; + throw new Error("the backend will not say which coroutine this is"); + }, + }); + const retained = retain(event as unknown as DurableEvent); + for (let attempt = 0; attempt < 3; attempt++) { + let caught: unknown; + try { + void retained.coroutineId; + } catch (error) { + caught = error; + } + expect((caught as Error | undefined)?.message).toBe( + "the backend will not say which coroutine this is", + ); + } + expect(asked).toBe(1); + }); + + it("a Yield's settlement stays lazy, so a guard can refuse before it is read", function* () { + let asked = 0; + const event: Record = { + type: "yield", + coroutineId: "root", + description: { type: "call", name: "work" }, + }; + Object.defineProperty(event, "result", { + enumerable: true, + get() { + asked++; + return { status: "ok", value: 1 }; + }, + }); + const index = new ReplayIndex(retainEvents([event as unknown as DurableEvent])); + expect(index.peekYield("root")?.description.name).toBe("work"); + expect(asked).toBe(0); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: 1 }); + expect(index.peekYield("root")?.result).toEqual({ status: "ok", value: 1 }); + expect(asked).toBe(1); + }); +}); + +describe("retained history — detached values stay ordinary JSON", () => { + it("`__proto__` is retained as an own data member", function* () { + const source: Record = {}; + Object.defineProperty(source, "__proto__", { + value: { polluted: true }, + enumerable: true, + writable: true, + configurable: true, + }); + const detached = detachJson(source as Json); + expect(Object.getPrototypeOf(detached)).toBe(Object.prototype); + expect(Object.getOwnPropertyNames(detached)).toEqual(["__proto__"]); + expect(({} as Record)["polluted"]).toBe(undefined); + }); + + it("nested objects and arrays detach from the journal's own", function* () { + const list = ["a"]; + const nested = { list }; + const detached = detachJson({ nested } as unknown as Json); + const held = (detached as Record)["nested"]; + expect(held).not.toBe(nested); + expect((held as Record)["list"]).not.toBe(list); + + list.push("injected"); + nested.list = ["replaced"]; + expect((held as Record)["list"]).toEqual(["a"]); + }); + + it("a detached member is writable and configurable", function* () { + const detached = detachJson({ count: 0, tags: ["a"] } as unknown as Json); + const held = detached as Record; + held["count"] = 1; + (held["tags"] as Json[]).push("b"); + expect(held).toEqual({ count: 1, tags: ["a", "b"] }); + }); + + it("a cycle is refused rather than followed", function* () { + const looped: Record = {}; + looped["self"] = looped; + let caught: unknown; + try { + detachJson(looped as Json); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(TypeError); + }); +}); diff --git a/packages/workflow/tests/workflow-run.test.ts b/packages/workflow/tests/workflow-run.test.ts index 5eda7f7c..b3c90bc6 100644 --- a/packages/workflow/tests/workflow-run.test.ts +++ b/packages/workflow/tests/workflow-run.test.ts @@ -400,6 +400,19 @@ describe("Tier WR — workflow runs", () => { description: { type: "workflow_run", name: "workflow_run", base: "main" }, result: { status: "ok", value: { runId: "seeded-run", base: "main", pinnedCommit: COMMIT } }, }); + // The root import a real completed run always records. A retained terminal + // result is reused on the strength of the selection its root import + // established, so a journal that carries one without the other describes a + // run that never happened and is refused. + yield* stream.append({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "", content: "# Hello\n" }, + }, + }); yield* stream.append({ type: "close", coroutineId: "root", diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index aa991474..be902cee 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2351,9 +2351,16 @@ operation, with the read it leads to. What the journal holds is serializable: a repository selection records the chosen path and its content, and a registration records its origin, never its function. +The root's selection carries one more member. A targeted root resolves its +selector here too, against the text this operation is about to record, and +records the **exact** target it resolved to — so the section the run executed is +part of the record rather than something a later read rediscovers (§5.4). An +untargeted root records no `target` member, which is what keeps journals written +before targets existed readable. + ```typescript type DurableSelection = - | { kind: "repository"; path: string; content: string } + | { kind: "repository"; path: string; content: string; target?: string } | { kind: "registered"; origin: string; reserved: boolean }; function* durableImportComponent( @@ -2606,6 +2613,448 @@ schema, an invalid value, a body error, and a failure raised after `` all complete `Err`, and body text emitted before the failure remains only on the output stream. +#### Document targets + +A root document addresses its own sections. A **document target** is an +addressable static heading in the document's root Markdown flow, named by the +canonical path of heading labels that reaches it. Selecting one executes: + +1. the document preamble; +2. the direct content of every ancestor needed to reach the target; and +3. the selected heading's complete subtree. + +Sibling subtrees do not execute. Retained headings stay in the projected body, +so the projection reads as a document rather than as an excerpt. + +##### Which headings are targets + +Only root-level Markdown heading nodes form the outline. A heading inside a +block quote, a list, a fenced block, raw HTML, or component children is not one. + +Heading discovery does not parse raw XMD with a Markdown parser. A component's +children are ordinary text to that parser, and a blank line among them ends the +HTML block it inferred, which surfaces a child heading as a root heading. +Discovery instead parses a copy of the body in which the boundary scanner's +top-level component spans are replaced by spaces of the same length. Newline +positions, offsets, and everything outside those spans are unchanged, so a +heading found in the masked copy sits where it sits in the original, and the +original supplies its text and its source. + +A heading's parent is the nearest preceding heading with a smaller depth. +Skipped depths are ordinary. **Outermost** means the smallest heading depth +present in the root flow, which need not be `h1`. + +When the document has exactly one outermost heading, that heading is the +document **title**: it takes no level in any target path, it is no target +itself, and its heading and direct content are retained in every projection +beneath it. When the document has more than one outermost heading, each of them +takes a path level. A document with no addressable heading has an empty +catalog. + +A heading is **not addressable** when its own source overlaps executable +component syntax, or contains an unescaped Executable MDX interpolation — +`{meta.key}`, `{props.key}`, `{binding}`, and the dotted forms of each. Escaped +interpolation (`\{meta.key\}`) is literal static text and stays addressable. A +heading that renders no text is not addressable. An unaddressable heading +required as a path level makes its whole subtree unaddressable; because the sole +title is not a path level, static sections beneath a computed title remain +addressable. + +##### Labels and canonical encoding + +A label is the statically rendered Markdown text of the heading: formatting and +link destinations are removed, while visible text, inline-code text, and image +alternative text are retained. The result is normalized to NFC, every run of +Unicode whitespace collapses to one ASCII space, leading and trailing +whitespace is trimmed, and case is preserved. There are no generated slugs, +suffixes, case folding, or punctuation removal. + +A canonical target is the sequence of labels from the target's outermost +addressable ancestor to the target, each percent-encoded and joined with raw +`/`. Encoding leaves the RFC 3986 unreserved characters (`A-Z a-z 0-9 - . _ ~`) +alone and escapes everything else as uppercase UTF-8 hexadecimal, so a `/`, +`*`, `#`, or `%` inside a heading becomes `%2F`, `%2A`, `%23`, or `%25` and +cannot be read as syntax. + +A fragment is an **exact** canonical target only when every level survives the +whole round trip: decoding it, normalizing the label, and re-encoding that label +reproduce the level byte for byte. That one rule rejects a wildcard operator, an +empty level, a lowercase escape, a raw `#`, an NFD spelling, a tab, and leading, +trailing, or uncollapsed whitespace, because none of them is what the encoder +writes. Two spellings of one section are therefore never two identities. + +The catalog is in source order and retains duplicates: two sections whose +canonical paths are equal stay two entries, so the ambiguity is observable. + +##### Selectors + +A document reference is: + +```text +# +``` + +The first raw `#` separates the two. Raw `/` separates target levels and raw +`*` and `**` are operators; the selector is split on those before its literal +chunks are percent-decoded, which is what keeps `%2F` a slash inside one label +and `%2A` a literal asterisk. Decoding is URI path decoding: `+` is a plus, not +a space. Malformed escapes, byte sequences that are not UTF-8, NUL, a leading +or trailing slash, an empty level, and a raw `#` — the reference's own +delimiter, written `%23` when a heading really contains one — are all refused. +Matching is case-sensitive. + +- A literal level matches one canonical label exactly, after decoding and label + normalization. +- `*` within a level matches zero or more characters of that one label, and may + appear more than once. +- A level that is exactly `**` matches zero or more complete path levels. + +There is no `?`, character class, brace, or backslash dialect. Within a +wildcard level only the literal chunks are decoded and normalized; whitespace +beside a wildcard is part of what the selector asked for, and only the beginning +of the first chunk and the end of the last are trimmed. Matching compares +Unicode code points and completes in time bounded by the product of the pattern +and label sizes. + +A selector must resolve to exactly one catalog entry. Zero matches and several +matches both fail. Diagnostics report canonical encoded references, so a +duplicate canonical path is reported as an ambiguity rather than resolved. + +##### Projection + +Source ranges are defined against the original, unprojected body: + +- the **preamble** runs from the body start to immediately before the first + root-flow heading, whatever its depth — a document may open at a deeper level + than the one supplying its title, and anchoring here to the outermost heading + would put an earlier addressable section inside the preamble, where every + other target would retain and run it; +- an **ancestor's direct content** runs from its heading start to its first + child heading's start, or to its subtree end when it has no child heading; + and +- the **selected subtree** runs from the selected heading's start to the next + heading of equal or smaller depth, or to the body end. + +The projected body is the preamble, each retained ancestor's direct content in +order, and the selected subtree. For a sole outermost title, the title is the +first retained ancestor even though it takes no level in the path. + +Each retained range is scanned separately, under the origin that range has in +the original file — its path, its offset, and its line. The ranges are not +concatenated and rescanned: skipped source must not renumber what follows it, +because a retained element's source position is what its expansion identifier +is derived from. A retained element therefore carries the same expansion ID in +a targeted run as in a full one, and two targets that retain it agree with each +other. The target string takes no part in expansion identity; a run's own +identity is what distinguishes the effects of two target runs. + +Frontmatter, root props, `returns`, the return mode, and `` behavior are +unchanged and apply to the projected body. Structural validation applies to the +projected body too: an invalid skipped sibling is irrelevant, while an invalid +retained range fails before any authored effect in the projection runs. + +##### Failure timing and durable identity + +Selection happens before the body expands. An invalid, unmatched, or ambiguous +selector runs no authored document effect. + +The live root import records the **exact canonical target**, never the caller's +selector. An untargeted import records no target member at all, so journals +written before targets existed stay readable by untargeted runs. + +Selection validation is **execution-owned and non-contextual**. Exact canonical +target is workflow-definition identity, and identity may not be decided by +anything a document, a component, or an enclosing scope can replace — a public +replay guard installed further out may decline to delegate, which is what +composable policy is for. The check therefore lives inside the journal the +execution hands to `durableRun`: it reads the retained history once, owns the +retained snapshot every later phase observes, validates the recorded selection +and the required root-import structure against it, and passes that same snapshot +on. It is also a trusted journal-provenance wrapping site. Provenance is not +transitive, so a wrapper is unproven unless its wrapping site carries the source +witness onto it, and a run whose journal is unproven is refused by a Workspace +provider before any transaction. This wrapper qualifies — core installs it +before any document code exists and it delegates every append to the exact +stream it was handed — and it transfers only the witness that source already +has, establishing none. + +It runs ahead of public guard policy, of any retained event reaching execution, +of a retained terminal result being reused, of authored work, and of any append. +The snapshot it owns covers **every** event that participates in admission, +indexing, or terminal reuse — a recorded completion as much as a recorded effect +— so a completion cannot belong to one coroutine while admission asks and to +another while the run reuses it, and cannot carry one result while admission +accepts it and another while the run consumes it. A recorded completion's result +is detached as the history is retained, and recognized during admission, so +nothing the backend still owns reaches a later phase. + +That snapshot is also immutable to public policy. Guard `check`, `admit`, and +`decide` receive an isolated copy rather than the retained events, so a handler +cannot rewrite a recorded target, the recorded content, a selection failure, an +effect description, or a result after admission accepted it. What a document +finally receives is a third thing again: a fresh mutable copy taken from the +authority at consumption, so a resumed binding is still ordinary JSON its own +continuation writes to. Public `ReplayGuard` policy remains composable and may short-circuit +other public guards; it cannot suppress this. + +Reusing a recorded terminal result additionally requires exactly one +recognizable root import in the retained history, belonging to the coroutine +whose terminal result is being reused. Reusing that result means standing behind +the selection its root import established; a history that recorded none, +recorded two, or recorded one belonging to another coroutine establishes nothing +to stand behind. A partial history is held to the same selection rule: a +recorded root import that names another section is refused before replay +continues into it. + +The validation itself proceeds as follows. It +resolves the current selector against the *recorded* content and requires the +same selection outcome; the recorded content is then what the projection is +taken from. A different selector naming the same section replays, and so does +the same failing selector. A different exact target, a targeted request against +an untargeted record, an untargeted request against a targeted record, and any +difference in a failed selection are all stale input (§6.11). Stale input is +reported as itself: the guard retains no failure object from the selection it +could not match. The check runs before a completed run's recorded terminal +result can be reused, so a finished journal cannot answer for a selection it +never made. + +##### One validation order + +`inspectDocument()`, a live `execute()`, and a replayed `execute()` validate a +root document in one order: + +1. parse the source far enough to obtain the body and its outline; +2. resolve the requested target; +3. compile frontmatter, props, and return schemas; +4. build and project the selected definition. + +Resolving the target before compiling schemas is what makes the answer the same +on all three paths, and it is the useful order: a caller who named a section the +document does not offer asked the wrong question, and should hear that rather +than a complaint about a schema they never reached. So an unresolvable target +with an invalid props schema raises `DocumentTargetError`; a resolvable target +with an invalid schema raises `PropsSchemaError`; an unresolvable target with a +valid schema raises `DocumentTargetError`. A recorded failed selection keeps the +same precedence on replay and is not replaced by the recorded terminal error. + +##### Naming a root document + +`@executablemd/core` exposes the shared shapes: + +```ts +interface FileRootDocument { + readonly path: string; + readonly source?: undefined; + readonly target?: string; +} + +interface InlineRootDocument { + readonly path: ""; + readonly source: string; + readonly target?: string; +} + +type RootDocumentSource = FileRootDocument | InlineRootDocument; + +function fileSource(reference: string): FileRootDocument; +function inlineSource(source: string, options?: { readonly target?: string }): InlineRootDocument; +function formatDocumentReference(path: string, target?: string): string; +``` + +`fileSource()` splits a document reference at the first raw `#`, percent-decodes +the path portion, and stores the fragment — still encoded — as `target`. It does +not decode the fragment as one string, because `%2F` must stay distinguishable +from a level separator. An empty path, a malformed escape, a byte sequence that +is not UTF-8, and NUL each fail with a cause-free `TypeError` whose message is +exactly `Invalid document reference`; the input is a command-line argument, and +echoing it back would put arbitrary bytes into a diagnostic. A filename +containing `#` is written `%23`, and one containing a literal `%HH` sequence is +written `%25HH`. + +`formatDocumentReference()` takes a decoded path and, optionally, an +already-canonical exact target. It encodes the path, validates the target rather +than encoding it again, and joins them with `#`. It is the one formatter +diagnostics, command output, and workflow handoff use. Making an authored glob +canonical is the selector parser's work, not this function's. + +It only formats what `fileSource()` reads back: the encoded path is decoded +again and must reproduce the path exactly. NUL, which the decoder refuses, and +an unpaired surrogate, which encodes lossily to the replacement character, are +therefore rejected rather than turned into a reference naming a different file. + +Existing programmatic `{ path }` values and `inlineSource(source)` remain valid +and untargeted. + +An unresolvable target raises `DocumentTargetError`. It is an ordinary +invocation failure, not a durability or `API.Files` failure, and it is the same +error on every public path: `inspectDocument()`, a live `execute()`, and a +replayed `execute()` all raise one carrying the same fields. + +```ts +interface DocumentTargetFailure { + readonly type: "executablemd.document-target-failure"; + readonly kind: "invalid-selector" | "no-match" | "multiple-matches"; + readonly selector: string; + readonly matches: readonly string[]; + readonly available: readonly string[]; +} + +class DocumentTargetError extends Error { + readonly data: DocumentTargetFailure; +} + +function isDocumentTargetError(error: unknown): boolean; +function asDocumentTargetError(error: unknown): DocumentTargetError | undefined; +function parseDocumentTargetFailure(value: unknown): DocumentTargetFailure | undefined; +``` + +`selector` is the fragment as it arrived. `matches` is the ambiguity list and is +empty for every other kind; `available` is the whole catalog. The message is +derived from the data, quotes the selector as JSON, and lists canonical encoded +references, so a heading holding a control character cannot reach a diagnostic +literally. + +Recognition is structural, total, and reconstructing. The data carries a stable +namespaced tag, so a failure built by a separately loaded copy of the package is +read on the same terms as one built locally — `instanceof` cannot answer that +question across two copies. + +`asDocumentTargetError()` never returns the candidate. It validates every field +and builds a **fresh local error** from the result, so nothing the candidate +owns is handed on: a nested list stays correct after its owner mutates it, and a +list reached through a revocable Proxy stays readable after the Proxy is +revoked. This is an ordinary invocation failure with no fail-stop reason to +preserve object identity, so rebuilding costs one allocation and removes every +way payload could travel. + +`isDocumentTargetError()` answers `boolean` and deliberately does not narrow. +A type predicate would say the candidate *is* the safe value, which is the one +thing it is not: the safe value is what `asDocumentTargetError()` builds. A +caller asking only "was this a target failure?" uses the predicate; a caller +that needs the typed error uses `asDocumentTargetError()` and reads its `data`. + +A candidate is read only when all of this holds: + +- the data carries exactly `type`, `kind`, `selector`, `matches`, and + `available`, with no other own member — enumerable, non-enumerable, or + symbol-keyed; +- `matches` and `available` are dense lists whose every entry is an exact + canonical target; +- `matches` is empty for `invalid-selector` and `no-match`, and holds more than + one entry for `multiple-matches`; and +- the fields describe an outcome selection could have reached: the selector is + parsed and matched against the catalog the data supplies, and the result must + be the matches it claims. + +The Error shell is checked as strictly: the fixed name, the message its own data +derives, no cause, and no enumerable member beyond the contract. Diagnostics are +derived from the reconstructed canonical data alone. + +##### A failed selection is recorded, not merely failed + +A selection that names no single section is an observation of the document: the +text was read, and it does not offer what was asked for. The root import records +that outcome structurally — its kind, the requested selector, the matches, and +the catalog — and the failure is then rebuilt from that record and raised. + +Recording it is what makes a resumed run correct. A journal is matched by effect +type and name alone, so without the record a run whose selector matched nothing +would leave a completed journal that answers a later request for a section that +does exist. The replay guard therefore compares whole selection outcomes — the +whole document, one exact target, or one failure — rather than target strings, +and reproduces a recorded failure with its fields intact before the recorded +terminal result can be reused. The same failing selector replays its own +failure; any difference in outcome, including a different selector that fails +the same way, is stale input. No authored effect runs in either case. + +The recorded selector is sanitized invocation metadata, retained only so an +ordinary failed execution can be reproduced. It never occupies the exact-target +field and never reaches a workflow definition. + +##### A recorded root selection is a closed protocol + +The recorded root import is parsed as a closed protocol with exactly two +supported shapes: a repository selection carrying the path, the content, and an +optional exact canonical target; and a failed selection carrying the path, the +content, and an exact failure record. An unknown kind, a missing or unreadable +member, a member of the wrong type, an extra member, a noncanonical target, and +a failure record that is not exactly this contract are each **malformed**. + +Malformed is not the same answer as "this event is not the root import". +Collapsing the two is what would let a corrupted record fall through to the +recorded terminal result, replaying an outcome the record no longer describes. + +Because the record carries the content it was taken from, the selection is +verified against it rather than merely parsed: a recorded exact target must +still resolve to itself in the recorded content, and a recorded failure must be +exactly the failure the recorded selector produces against that content. A +catalog, a match list, or a kind that the recorded document contradicts is +therefore malformed too. + +Reading a record is **total**, and identification is separate from reading. An +event that will not say what it is stays unrelated — it cannot be claimed as the +root import. Once it *is* identified as the root import, every way of not +producing a selection is malformed: + +- an absent, unreadable, or invalid result or settlement, except the ordinary + failed settlement, which is a root import that failed for reasons selection + knows nothing about and stays unrelated; +- a successful result whose value is absent or unreadable; +- an unreadable property, a key list a Proxy refuses to produce, recorded + markdown whose frontmatter no parser accepts, and any other exception raised + while reading or verifying the record. + +Absence and refusal are represented distinctly. Using one value for both is what +turns "the root import will not say what it settled to" into "this is not the +root import", which delegates and lets the recorded terminal result be reused +for a request nobody made. + +The boundary is synchronous throughout, so it can swallow no cancellation and no +durability failure — neither arises inside a synchronous parse. + +It covers the whole envelope, including a `result` that cannot be read at all. +That requires replay indexing to read a Yield's *identity* without reading what +it settled to: a guard's check phase runs after the index is built, so a guard +that would refuse an event has to get its chance before the stream is asked to +produce that event's result. + +The settlement is detached from the journal, not merely frozen at the top: the +whole result tree is rebuilt from members read once each, so a nested accessor +beneath a recorded value cannot answer one thing to a guard and another to +replay. Detaching is the stability claim; the copy is not frozen through, +because replayed values are legitimately mutable — an eval binding restored from +a journal is pushed to by the iteration that resumes on it. + +"Read once" spans the phases, not one accessor. The guard validates a recorded +selection and the replay path then projects it, and those observe one settled +value: the check phase is handed the retained Yields rather than the stream's +own events, and the stream is consulted at most once. Otherwise a source that +answered differently between validation and consumption would have the guard +approve one recorded target while the run executed another — exact-target +identity would say the section that ran is the one the record names, and nothing +would make that true. + +A malformed record fails before the recorded terminal result can be reused, with +one fixed, cause-free diagnostic and nothing else: no journal text, parser +message, path, selector, YAML fragment, or accessor message reaches it. It never +delegates, never replays the recorded terminal error, never executes authored +work, and never appends new history. + +A root import whose recorded result is not `ok` is left alone: a root can fail +for reasons that are not about selection, and those failures are not this +protocol's to interpret. + +A replay that reuses **retained terminal history** must first establish exactly +one recognizable root import. Reusing a recorded terminal result means standing +behind the selection its root import established, and a history that recorded +none — or recorded two — establishes nothing to stand behind; the per-event +check has nothing to object to, so the refusal belongs to the history as a +whole. A missing, duplicated, or unreadable required root import fails with the +same fixed cause-free diagnostic, executes no authored work, appends nothing, +and never returns the retained terminal result. A journal with no retained +terminal result is unaffected: it replays what it has and continues live, so a +root import it does not contain is one this run performs. + ### 5.5 The Component Api Expansion's context-dependent operations are exposed through one public @@ -5551,6 +6000,9 @@ workflow and returns a `DocumentExecution` handle. Options: - the root document source — either `path`, the path to the root markdown document, or an inline document built with `inlineSource(text)`, which carries the supplied text together with its `` identity +- `target?` — a document target selector, still encoded, resolved against the + root before its body expands (§5.4). `fileSource(reference)` builds a file + root and its selector from one document reference - `stream` — the durable stream that journals the run - `props?` — JSON values supplied to the root document (default: `{}`) - `componentDirs?` — component search directories (default: @@ -5675,9 +6127,18 @@ what it declares — without executing the document or creating a journal: - `returnMode` — `"text"` or `"value"`. An explicit `returns: { type: string }` produces the same effective schema as the default, so the mode is what tells the two apart. +- `targets` — every document target the root addresses, as canonical encoded + fragments without the document path or a leading `#`, in document order, + duplicates retained (§5.4). +- `target` — the exact canonical target the requested selector resolved to. + Present only when a target was requested and resolved, and never the caller's + glob. An invalid return schema fails inspection exactly as it fails execution: both -load the definition through the same path. +load the definition through the same path. So does an unresolvable target: +inspection discovers and selects targets without expanding the document, +evaluating a code block, importing a body component, or creating a journal, so +a host resolves a selector to one exact target before anything runs. `DocumentExecution` is an `Operation>`: `yield* execution` completes with `Ok(value)` on success and `Err(error)` on document, @@ -7099,6 +7560,73 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | WRR10/WRR10b | Outer rollback cache coherence | Failure and cancellation after an uncommitted removal and negative lookup roll back and invalidate both authoritative DOFS caches | | WRR11 | Historical file size | Every historical file entry's declared size agrees with its retained DOFS manifest during read-only recognition | +### Tier DT — Document target catalog, selectors, and projection + +| # | Test | Verify | +|---|------|--------| +| DT1–DT5 | Outline | ATX and Setext headings catalog in source order; a skipped depth still nests; the outermost depth is the smallest present; a sole outermost heading is the title and several are path levels | +| DT6 | Case | Matching is case-sensitive | +| DT7–DT9 | Labels | Formatting, link destinations, inline code, image alt text and passive HTML tags reduce to statically rendered text; a heading rendering no text is unaddressable | +| DT8 | Normalization | NFC-equivalent spellings are one label and Unicode whitespace collapses | +| DT10 | Encoding | `/`, `%`, `#` and `*` in a heading encode to `%2F`, `%25`, `%23` and `%2A` and never read as syntax | +| DT11 | Duplicates | Two sections with one canonical path stay two entries and report as ambiguous | +| DT12 | Nested flow | Headings in block quotes, lists, fences, exec fences and raw HTML are not targets | +| DT13 | Component children | A component child holding blank lines and `#` lines contributes no target — the regression that kills raw Remark discovery | +| DT14–DT17 | Addressability | A heading overlapping component syntax or carrying an interpolation is unaddressable and blocks its subtree; escaped interpolation stays static; a computed sole title still leaves its sections addressable | +| DT18/DT19 | Empty catalog | A document with no heading addresses nothing, and a sole title is no target | +| DT20–DT22 | Matching | Literal levels, embedded `*`, and `**` across zero or more levels | +| DT23 | Exactly one | Zero matches and several matches both fail, reporting matches and the catalog | +| DT24–DT26 | Selector syntax | Empty, leading/trailing slash, empty level, malformed escape, NUL and non-UTF-8 are refused; `+` is a plus | +| DT27 | Termination | A wildcard-dense selector against a long label completes without exponential search | +| DT28 | Wildcard whitespace | Whitespace beside a wildcard is matched; only the level's outer edges trim | +| DT29–DT32 | References | A reference splits at the first raw `#`; a path keeps separators and decodes escapes; an unreadable reference says only `Invalid document reference`, cause-free; formatting encodes the path and validates an exact target | +| DT33/DT48 | Canonical exactness | A level is canonical only through decode, normalize and re-encode — NFD, tabs, uncollapsed or edge whitespace, lowercase escapes, empty levels and raw operators are refused | +| DT49 | Raw `#` | A raw `#` is never a literal selector character; `%23` addresses a heading containing one | +| DT50/DT51 | Formatter totality | A path that cannot encode losslessly — NUL, an unpaired surrogate — is refused, and every formatted reference parses back to what it named | +| DT34–DT39 | Projection | Preamble, ancestor direct content and the selected subtree are retained; siblings are absent; a non-leaf keeps its descendants; a sole title stays | +| DT40–DT43 | Positions | A retained element keeps its authored offset and line, CRLF included; frontmatter, props and return mode survive; the untargeted parse still scans the whole body | +| DT44–DT47 | Inspection | The catalog is reported without selecting; a glob resolves to the exact target; an unresolvable target fails inspection; the failure's data is frozen and rebuilt | +| DT52/DT53 | Recognition | A failure from a separately loaded copy, and one built here, are read on the same terms | +| DT54–DT56 | Reconstruction | The result is a fresh local error, never the candidate; a mutable nested list is copied and later mutation changes nothing; a revoked Proxy cannot reach through a result already built | +| DT57 | Closed data | Enumerable, non-enumerable and symbol-keyed extras are refused | +| DT58 | Canonical lists | Raw spaces, tabs, no-break spaces, edge whitespace, lowercase escapes, NUL, non-string entries and sparse lists are refused — asserted against the data parser, so the derived message cannot mask the check | +| DT59 | Dotted heading paths | `.` and `..` are legal heading labels, so `../../etc/passwd` and `Alpha/../Beta` are canonical heading paths, never filesystem authority; structural parsing accepts them when the rest of the failure is consistent | +| DT60 | Semantic outcome | Fields no selection could have produced — a `no-match` whose selector matches, a single-match ambiguity, a match outside the catalog, an `invalid-selector` that parses — are refused | +| DT61/DT62 | Closed shell | A cause, an enumerable payload, and a message that does not derive from its data are refused; no planted payload survives stringification, spreading, symbol enumeration, or a journal round trip | + +### Tier TX — Targeted execution and replay + +| # | Test | Verify | +|---|------|--------| +| TX1–TX3 | Selection | Only the preamble, ancestors and subtree expand; a skipped sibling's components and code blocks never run; a non-leaf expands its descendants | +| TX4/TX5 | Identity | A retained element keeps the expansion ID it has in a full run, and two targets retaining it agree | +| TX6 | Sources | A file root and an inline root behave identically | +| TX7–TX9 | Root values | Root props, frontmatter interpolation, a declared `returns`, and `` apply to the projected body | +| TX10/TX11 | Structure | An invalid skipped sibling is irrelevant; an invalid retained range fails before any authored effect | +| TX12–TX14 | Failure timing | An unmatched or ambiguous target runs no authored effect and is structurally recognizable; inspection resolves without expanding a component | +| TX15/TX16 | Recording | The journal records the exact target, never the glob, and an untargeted run records no target member | +| TX17 | Compatibility | A different selector naming the same section replays | +| TX18–TX21 | Staleness | A different exact target, a targeted request against an untargeted journal, the reverse, and a selector the recorded content no longer resolves all fail stale before completed-Close reuse, carrying no foreign cause | +| TX22/TX23 | Recorded content | An untargeted journal replays untargeted; replay projects the recorded content, not the file on disk | +| TX24 | Failed selection | A journal from a selector that matched nothing never answers a later valid one | +| TX25–TX27 | Failed replay | The same failing selector replays its own recorded failure with no authored effect; a different failure kind or a different selector is stale; live and replayed failures are the same structural error | +| TX28–TX32 | Malformed records | Starting from a valid failed-selection journal and corrupting only the record: a missing or non-array catalog, an unknown kind, extra record or failure data, a catalog or target the recorded document does not derive, and inconsistent kind/matches data are each refused before completed-Close reuse, resumed with the failing selector and with a valid one, expanding nothing and appending nothing | +| TX33 | Not vacuous | An uncorrupted record still replays its recorded failure | +| TX34–TX37 | Totality, inside the value | Recorded markdown whose frontmatter no parser accepts, an unreadable member, a record refusing key enumeration, and a value that is not a record at all each become the one fixed diagnostic — cause-free, carrying no planted value, expanding nothing and appending nothing | +| DT63–DT66 | Preamble boundary | A section before the title is addressable and is not preamble; selecting a later section neither renders nor executes it; it stays independently addressable; real preamble text before the first heading is still retained | +| TX44/TX45 | Preamble execution | Selecting the later section runs no component of the earlier one, which still runs when it is the target | +| TX46 | Nested substitution | A nested `target` accessor answering Alpha then Beta cannot substitute a section: Alpha alone executes, the member is read once, and the appended Close describes Alpha | +| TX47–TX50 | Terminal history | Targeted and untargeted completed journals with the root import removed, and one with it duplicated, are refused before terminal reuse; an intact journal still replays | +| TX51–TX56 | Identity authority | A completed or partial Alpha journal resumed as Beta is refused with an enclosing `check` handler that never delegates, with the equivalent `admit` handler, and with a same-name guard from another loaded copy; same-target replay and ordinary guard composition are the controls | +| TX57–TX60 | Terminal binding | A root import on a child coroutine, a valid one plus a root-named child event, none at all, and two on the terminal coroutine each refuse before terminal reuse | +| TX61 | Detached but mutable | A partial replay restores a binding from the journal and the live continuation writes to it, matching the complete run exactly | +| TX68 | Shifting terminal coroutine | A Close that moves from a child to the root returns nothing: the fixed diagnostic, no retained output, nothing expanded, nothing appended, the coroutine asked once | +| TX69–TX71 | Post-admission replacement | Public guard policy rewriting the backend's own terminal result does not change what replay returns; a terminal result that refuses is the fixed diagnostic, read once, with nothing expanded and nothing appended; the intact control replays | +| TX64–TX67 | Validation order | An unresolvable target outranks an invalid schema on inspection, live execution, and replay; a resolvable target lets the schema failure be reported; the control runs | +| TX43 | One read across phases | Two valid recorded selections behind one accessor — Alpha then Beta — resume as Alpha: Alpha's section executes, Beta's never does, the source is read once, and the appended Close describes the Alpha execution | +| TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | +| TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | + ### Tier SL — Own-scope context updates | # | Test | Verify | diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index f6793b92..39b2e61c 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -482,8 +482,10 @@ capturing the root. The Deno provider binds an adapter-private proof operation to one exact WorkflowRun handle through module-private executor identity. It establishes the canonical durable-stream module's journal provenance for that run's journal and retains that exact witness. The generic pre-persistence guard -preserves nothing; the trusted secret-filter wrapping site preserves the -witness explicitly, including through nested trusted wrappers. The provider +preserves nothing. Two trusted wrapping sites preserve the witness explicitly, +and a run's journal passes through both: the secret filter, and the +execution-owned target-admission wrapper core installs before any document code +exists. Nesting them carries the same witness through each. The provider receives the invocation's exact executor identity and journal provenance from its execution-owned capability and validates them before it opens the caller-owned transaction. It refuses a foreign executor, foreign or absent