From f298715fa15b13af7718b65ed32f020a350d9a7a Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 10:14:23 -0400 Subject: [PATCH 01/10] =?UTF-8?q?=F0=9F=93=81=20Give=20a=20workflow=20run'?= =?UTF-8?q?s=20document=20its=20own=20filesystem?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A workflow run's `` and `` now reach the run's logical Workspace instead of the caller's filesystem. Each read, write and search is one durable Workspace effect, so the mutation, the immutable root it produces and the filtered journal result commit together, and a replay restores the recorded outcome without performing the mutation or asking what the file is now. An authored path is resolved by arithmetic on POSIX segments rooted at `/`, so no host path exists for a namespace race to replace. A documented DOFS refusal rolls its mutation savepoint back before the sanitized result is published and crosses the boundary as a `FilesReason` and nothing else; everything that is not a documented refusal stays an infrastructure failure. A temporary directory is refused rather than emulated. `useRetainedWorkflow(run)` is the other half: a host that has already created the run's storage record installs the exact frozen value, so the execution allocates no identifier and resolves no base, and every journal state requires the record to agree in run id, base and pinned commit. The CLI cannot reach any of this yet; `xmd run` and `xmd test` keep the host provider untouched. --- architecture.md | 53 +- packages/runtime/files.ts | 20 + packages/runtime/mod.ts | 3 + packages/workflow/deno.ts | 3 + packages/workflow/mod.ts | 2 +- .../workflow/src/deno/workspace/effect.ts | 2 +- .../workflow/src/deno/workspace/errors.ts | 13 + packages/workflow/src/deno/workspace/files.ts | 612 ++++++++++++++++++ packages/workflow/src/deno/workspace/host.ts | 63 ++ .../src/deno/workspace/logical-path.ts | 127 ++++ packages/workflow/src/journal.ts | 20 + packages/workflow/src/run.ts | 176 ++++- packages/workflow/tests/retained-run.test.ts | 210 ++++++ .../tests/support/workspace-crash-child.ts | 6 +- .../tests/support/workspace-restart-child.ts | 9 +- .../workspace-effect-transaction.test.ts | 4 +- .../workflow/tests/workspace-files.test.ts | 499 ++++++++++++++ scripts/runtime-test-exclusions.ts | 6 + specs/executable-mdx-spec.md | 31 + specs/workflow-spec.md | 89 ++- specs/workflow-workspace-spec.md | 8 +- 21 files changed, 1884 insertions(+), 72 deletions(-) create mode 100644 packages/workflow/src/deno/workspace/files.ts create mode 100644 packages/workflow/src/deno/workspace/host.ts create mode 100644 packages/workflow/src/deno/workspace/logical-path.ts create mode 100644 packages/workflow/tests/retained-run.test.ts create mode 100644 packages/workflow/tests/workspace-files.test.ts diff --git a/architecture.md b/architecture.md index 38c9cd71..45556034 100644 --- a/architecture.md +++ b/architecture.md @@ -129,6 +129,16 @@ or invoking Git. The supplied base must equal the recorded base. Git is not consulted to compare the current value of a moving branch with the pinned commit. +A host that has already created the run's storage record installs +`useRetainedWorkflow(run)` instead, with the exact frozen value. Nothing is left +for the execution to decide: it records that value through the same +`workflow_run` durable operation, allocates no identifier and resolves no base, +and every journal state — live, truncated and completed — requires the recorded +run to agree with the supplied one in run ID, base and pinned commit. A journal +that disagrees in any of them is not this run's journal, and the refusal names +the fields rather than their values. The programmatic `useWorkflow({ base })` +installation is unchanged. + `getWorkflowRun()` returns the frozen `WorkflowRun` for the current document execution. Every call in one live execution returns the same object. It throws outside a document execution associated with `useWorkflow()`, and it exposes no @@ -459,13 +469,14 @@ rather than invoking an untracked native Git side effect. Successful effect coordination finishes the mutation scope, including child cleanup, before capturing the resulting root. The Deno provider installs that -ordering for its adapter-private Workspace proof operation: the mutation +ordering for its Workspace effect operation: the mutation savepoint, root publication, filtered routed Yield, and caller-owned transaction -commit form one boundary. The proof filesystem uses the pinned synchronous DOFS +commit form one boundary. The Workspace filesystem uses the pinned synchronous DOFS entry points for its string and byte-array surface, so cancellation leaves no eager promise or stream pull able to reach the connection after transaction -authority ends. Public filesystem components and workflow lifecycle commands do -not yet select that operation. +authority ends. The transaction-bound Files provider selects that operation for +every document filesystem read, write and search; workflow lifecycle commands do +not select it yet. An external provider cannot join that transaction. Prompt, Git push and pull request effects derive a stable identity from the run and expansion, ask the @@ -538,11 +549,11 @@ after SQLite has restored the prior frontier. Retained roots, manifests and blobs remain indefinitely. Cloudflare garbage collection is not in the production closure and is never invoked. The provider -exposes no public Workspace mutation effect, history selection or fork -operation at this layer. Its adapter-private coordinator combines one mutation, -immutable-root publication and one filtered journal result atomically for the -provider-level proof; declarative `` and workflow start/resume do not -reach it yet. +exposes no public history selection or fork operation at this layer. Its +coordinator combines one mutation, immutable-root publication and one filtered +journal result atomically, and the transaction-bound Files provider is what +routes a document's `` and `` to it. Workflow start and resume do +not reach it yet. The coordinator treats only errors produced through its private filesystem adapter's documented path and mutation refusals as journalable operation @@ -925,6 +936,25 @@ than implied: Neither claim covers a native command a document runs. +The workflow provider's operations are durable effects. A read, a write and a +search each carry a description derived from the current expansion, the +operation and the resolved logical path, so one authored element is the same +effect across replays and a document edited to name another file is a different +one. `checkFilePath` is not among them: it is lexical admission, it performs no +effect, and it appends nothing. + +A write, the immutable root that results from it, and the filtered journal +result share the one caller-owned transaction. An ordinary refusal rolls its +mutation savepoint back before that result is published, so the retained +outcome describes a Workspace that is exactly what it was, and the reason that +crosses the boundary is selected from the shared vocabulary rather than derived +from anything the filesystem said. Replay restores those recorded outcomes +without performing the mutation, opening a transaction or consulting the current +frontier, which is why a create/delete/create history replays in order. A +temporary directory is refused outright: the provider has no host directory to +hand out, and falling through to the caller's would be the uncontained +filesystem the boundary exists to prevent. + Failure data crosses the boundary as a plain frozen object under a stable tag, carrying a reason from a fixed vocabulary and the phase it came from. No message, errno code, resolved path, temporary name, or symlink target crosses. @@ -1090,6 +1120,7 @@ Status is measured against main. | `xmd targets` | prints one document's catalog as full document references, by inspection alone | built on the #412 stack | | targeted `xmd run` | reads a file argument as a document reference and executes the one exact target its selector resolved to, replacing the selector before execution rereads the file | built on the #412 stack; the targeted workflow definition is unbuilt | | `useWorkflow()` / `getWorkflowRun()` | associates one document execution with a workflow run | built on main | +| `useRetainedWorkflow()` | associates one document execution with a run storage already created, requiring exact journal agreement | built on the #366 stack | | `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main | | workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; public workflow execution is unbuilt | | caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in | built on main | @@ -1099,7 +1130,7 @@ Status is measured against main. | `API.Service` / `startService()` | creates an authenticated, supervised loopback service attachment through a provider-neutral operation | built on main | | `API.Files` | routes every document filesystem operation to the installed provider, with no host default and structural failure data | built on the #227 stack | | host Files provider / `useHostFiles()` | resolves document paths in the caller's filesystem, containing them while the host namespace is stable; installed by all four CLI entrypoints | built on the #227 stack | -| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | unbuilt; the adapter is #227's second layer, and workflow effect coordination and CLI reachability remain unbuilt | +| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | built on the #366 stack; CLI reachability remains unbuilt | | `service=` | publishes the attachment's endpoint into the live binding overlay for its invocation | built on main | | `ephemeral eval` | reconstructs live middleware and bindings without a journal entry | built on main | | `useWorkflowServiceDenial()` | provides and tests a non-delegating workflow service denial provider; #366 will install it in future start and resume scopes | built on main; no workflow CLI execution branch exists yet | @@ -1108,7 +1139,7 @@ Status is measured against main. | Repository / Worktree / transactional Git effects | compose named checkouts and publish local mutations with their journal result | defined in `specs/workflow-workspace-spec.md`, unbuilt | | workflow inspection and history fork | reads status/history without advancing a run and creates a new run from a checkpoint | defined in `specs/workflow-workspace-spec.md`, unbuilt | | read-only workflow Agent / generated XMD | lets an Agent inspect a derived view and propose constrained executable changes | defined in `specs/workflow-workspace-spec.md`, unbuilt | -| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates an adapter-private mutation proof with its filtered Yield | built on the #365 stack; public mutation and workflow lifecycle reachability are unbuilt | +| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates one Workspace mutation with its filtered Yield | built on the #365 stack; public document filesystem effects route to it on the #366 stack, and workflow lifecycle reachability is unbuilt | | scoped Worker Shell | executes `just-bash` through the Workspace adapter inside a Deno Worker | containment and effect-transaction POCs complete (#351, #357); production integration unbuilt | | `` | retry a region until it completes | defined, unbuilt | | suspension effect | suspend durably | defined, unbuilt | diff --git a/packages/runtime/files.ts b/packages/runtime/files.ts index c3fbe1a0..769f1a50 100644 --- a/packages/runtime/files.ts +++ b/packages/runtime/files.ts @@ -443,6 +443,26 @@ function phaseOf(value: unknown): FilesPhase | undefined { return PHASES.find((phase) => phase === value); } +/** + * The vocabularies, for a provider that reads a failure back out of storage. + * + * A transaction-bound provider retains what it refused rather than a serialized + * error, so restoring one means turning stored text back into the vocabulary. + * Parsing it here is what keeps one list of reasons and phases: a provider that + * declared its own copy would be a second list to keep in agreement with this. + */ +export function parseFilesReason(value: unknown): FilesReason | undefined { + return reasonOf(value); +} + +export function parseFilesPhase(value: unknown): FilesPhase | undefined { + return phaseOf(value); +} + +export function parseFileWritePhase(value: unknown): FileWritePhase | undefined { + return writePhaseOf(value)?.[0]; +} + function invariantCategory(value: unknown): FilesInvariantCategory | undefined { return INVARIANT_CATEGORIES.find((category) => category === value); } diff --git a/packages/runtime/mod.ts b/packages/runtime/mod.ts index 4834d1ac..5d4ed9f6 100644 --- a/packages/runtime/mod.ts +++ b/packages/runtime/mod.ts @@ -84,7 +84,10 @@ export { fileWriteSuccess, filesFailure, isFilesFatal, + parseFilesPhase, + parseFilesReason, parseFileWriteFailure, + parseFileWritePhase, parseFileWriteSuccess, parseFilesFailure, parseFilesFatal, diff --git a/packages/workflow/deno.ts b/packages/workflow/deno.ts index fac983bd..9ab913e7 100644 --- a/packages/workflow/deno.ts +++ b/packages/workflow/deno.ts @@ -28,3 +28,6 @@ export { useWorkflowRunStorage } from "./src/deno/provider.ts"; export type { WorkflowRunStorageOptions } from "./src/deno/provider.ts"; export { hashRunId, workflowRunPath } from "./src/deno/path.ts"; export { APPLICATION_ID, SCHEMA_VERSION } from "./src/deno/schema.ts"; +export { useLogicalWorkspaceCwd, withWorkflowWorkspace } from "./src/deno/workspace/host.ts"; +export { useWorkflowFiles, WORKSPACE_FILE } from "./src/deno/workspace/files.ts"; +export { WORKSPACE_ROOT } from "./src/deno/workspace/logical-path.ts"; diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index f505148a..551a7680 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -26,7 +26,7 @@ export { Git, GitRevisionError, revParse } from "./src/git.ts"; export type { GitApi } from "./src/git.ts"; -export { getWorkflowRun, useWorkflow } from "./src/run.ts"; +export { getWorkflowRun, useRetainedWorkflow, useWorkflow } from "./src/run.ts"; export type { WorkflowRun } from "./src/run.ts"; export { useWorkflowServiceDenial, WorkflowServiceDeniedError } from "./src/service-denial.ts"; diff --git a/packages/workflow/src/deno/workspace/effect.ts b/packages/workflow/src/deno/workspace/effect.ts index 8c695ce2..e84ae526 100644 --- a/packages/workflow/src/deno/workspace/effect.ts +++ b/packages/workflow/src/deno/workspace/effect.ts @@ -218,7 +218,7 @@ export function withWorkspaceEffects( }); } -export function createWorkspaceProofEffect( +export function createWorkspaceEffect( database: WorkflowRunDatabase, description: EffectDescription, mutate: DenoWorkspaceMutation, diff --git a/packages/workflow/src/deno/workspace/errors.ts b/packages/workflow/src/deno/workspace/errors.ts index ebd6c156..723aa1c3 100644 --- a/packages/workflow/src/deno/workspace/errors.ts +++ b/packages/workflow/src/deno/workspace/errors.ts @@ -47,3 +47,16 @@ export function throwWorkspaceFilesystemFailure(error: unknown): never { export function isJournalableWorkspaceFailure(error: unknown): error is Error { return error instanceof JournalableWorkspaceFailure; } + +/** + * The documented filesystem condition this failure is, or `undefined` for one + * that is not documented. + * + * The code is the only part of a DOFS failure anything above this module reads. + * Its message, its cause and the paths either of them names stay here, so a + * consumer selecting a `FilesReason` from this receives a condition rather than + * platform text. + */ +export function journalableWorkspaceCode(error: unknown): string | undefined { + return error instanceof JournalableWorkspaceFailure ? error.code : undefined; +} diff --git a/packages/workflow/src/deno/workspace/files.ts b/packages/workflow/src/deno/workspace/files.ts new file mode 100644 index 00000000..e04d43f3 --- /dev/null +++ b/packages/workflow/src/deno/workspace/files.ts @@ -0,0 +1,612 @@ +/** + * The transaction-bound `API.Files` provider — a document's filesystem inside a + * workflow run. + * + * This is what `xmd workflow` installs where `xmd run` installs the host + * adapter. A document names the same paths and `` calls the same + * operations; what changes is where those paths land. Here they land in the + * run's own logical Workspace, and every read, write and search is one durable + * effect published by the run's effect transaction — the mutation, the + * resulting immutable Workspace root and the filtered journal result commit + * together or not at all. + * + * ## Why an authored path never reaches a host filesystem call + * + * Resolution is arithmetic on POSIX segments rooted at `/`, and the result is + * handed to the run's DOFS filesystem. No host path appears anywhere in it, so + * the containment claim needs no stable-namespace qualification: nothing + * outside the Workspace can be named, and no other process can replace part of + * a tree that lives inside one database. + * + * `checkFilePath` stays what the Api says it is — pure lexical admission that + * hands back nothing usable. It performs no effect and appends no journal + * entry, so a check that was skipped or answered elsewhere authorizes nothing; + * the write repeats the same admission from the same authored path. + * + * ## What replay does instead + * + * A recorded effect restores its recorded value. A read therefore answers with + * the bytes it read when it ran, even where the current frontier no longer + * holds them, and a write already recorded neither mutates nor captures a root + * again. Nothing here consults current state to decide whether an earlier + * effect happened, which is what lets a create/delete/create history replay in + * order rather than collapsing to whatever the file is now. + * + * ## What crosses the boundary + * + * A documented DOFS refusal selects a `FilesReason` and nothing else travels + * with it: no DOFS message, no errno payload, no SQLite text, no resolved path. + * A refusal is also a *rolled back* refusal — the mutation runs inside a + * savepoint of its own, so partial logical mutation is discarded before the + * sanitized result is durably published, and the run's current root is the one + * it was before. + * + * Everything that is not a documented refusal — connection, authority, + * savepoint, capture, publication, routing, teardown and commit failure — stays + * an infrastructure failure and fails the run. None of them is something a + * document did, and printing one would let the work after this file work run as + * though the file work had happened. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import { globToRegExp } from "@effectionx/fs"; +import { getExpansion } from "@executablemd/core"; +import { + Files, + FilesInvariantError, + FilesOperationDeniedError, + filesFailure, + fileWriteFailure, + fileWriteSuccess, + parseFilesPhase, + parseFilesReason, + parseFileWritePhase, +} from "@executablemd/runtime"; +import type { + FilePathInput, + FilesPhase, + FilesReason, + FileWriteInput, + FileWritePhase, + FileWriteSuccess, + GlobInput, +} from "@executablemd/runtime"; +import type { EffectDescription, Json, Workflow } from "@executablemd/durable-streams"; +import type { WorkflowRunDatabase } from "../../storage/api.ts"; +import { savepoint } from "../transaction.ts"; +import { createWorkspaceEffect } from "./effect.ts"; +import { journalableWorkspaceCode } from "./errors.ts"; +import type { DenoWorkspaceFilesystem, DenoWorkspaceStat } from "./filesystem.ts"; +import { + logicalDirectory, + logicalJoin, + logicalParent, + LogicalPathError, + resolveLogicalPath, + WORKSPACE_ROOT, +} from "./logical-path.ts"; + +/** The effect type every document filesystem operation in a run is recorded under. */ +export const WORKSPACE_FILE = "workspace_file"; + +/** + * The condition each documented DOFS code reports as. + * + * A `Map` rather than an object literal, because a lookup on one answers for + * inherited keys and the code comes from the filesystem rather than from here. + */ +const REASON_BY_CODE: ReadonlyMap = new Map([ + ["ENOENT", "missing"], + ["ENOTDIR", "not-directory"], + ["EISDIR", "directory"], + ["ENOTEMPTY", "directory-not-empty"], + ["EACCES", "permission-denied"], + ["EPERM", "permission-denied"], + ["EROFS", "read-only"], + ["ELOOP", "too-many-symlinks"], +]); + +/** A documented filesystem condition, carrying its reason and nothing else. */ +class WorkspaceRefusal extends Error { + override name = "WorkspaceRefusal"; + readonly reason: FilesReason; + + constructor(reason: FilesReason) { + super("workspace filesystem refused"); + this.reason = reason; + } +} + +/** + * The refusal this failure is, or a rethrow when it is not one. + * + * Rethrowing is what keeps infrastructure failures infrastructure failures: a + * condition DOFS never documented is not something a document did, and turning + * it into a printable reason would let the work after this file work run as + * though the file work had happened. + */ +function asRefusal(error: unknown): WorkspaceRefusal { + const code = journalableWorkspaceCode(error); + if (code === undefined) { + throw error; + } + return new WorkspaceRefusal(REASON_BY_CODE.get(code) ?? "operation-failed"); +} + +function refusalReason(error: Error): FilesReason { + return error instanceof WorkspaceRefusal ? error.reason : "operation-failed"; +} + +function lexicalReason(error: Error): FilesReason { + return error instanceof LogicalPathError ? error.reason : "operation-failed"; +} + +/** + * What one file effect recorded. + * + * A JSON value, because it is what the journal holds and what a replay hands + * back. A refusal is carried as a phase and a reason rather than as a + * serialized error, so nothing a filesystem said is retained and a restored + * refusal is rebuilt from the same vocabulary a live one is. + */ +type FileEffectOutcome = + | { readonly kind: "content"; readonly content: string } + | { readonly kind: "written" } + | { readonly kind: "paths"; readonly paths: string[] } + | { readonly kind: "refused"; readonly phase: string; readonly reason: string }; + +function refused(phase: FilesPhase | FileWritePhase, reason: FilesReason): FileEffectOutcome { + return { kind: "refused", phase, reason }; +} + +/** + * The outcome a journal record describes, or `undefined` when it describes none. + * + * The journal is parsed, never trusted. A record this cannot read has no + * printable reading, so the caller turns it into one fixed provider invariant + * rather than inventing a filesystem condition that was never reported. + */ +function parseOutcome(value: unknown): FileEffectOutcome | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + const record = Object.fromEntries(Object.entries(value)); + if (record.kind === "content" && typeof record.content === "string") { + return { kind: "content", content: record.content }; + } + if (record.kind === "written") { + return { kind: "written" }; + } + if ( + record.kind === "paths" && + Array.isArray(record.paths) && + record.paths.every((entry) => typeof entry === "string") + ) { + return { kind: "paths", paths: [...record.paths] }; + } + if ( + record.kind === "refused" && + typeof record.phase === "string" && + typeof record.reason === "string" + ) { + return { kind: "refused", phase: record.phase, reason: record.reason }; + } + return undefined; +} + +/** + * How one file effect is identified, deterministically. + * + * The expansion is what makes two `` elements different effects and one + * element the same effect across replays; the operation separates a read from a + * write performed by the same element; and the resolved logical target is what + * a changed authored path or a changed working directory moves. A document + * edited to name another file therefore diverges rather than quietly replaying + * the previous file's recorded bytes. + */ +function* describeFileEffect( + operation: string, + target: string, + detail: Record, +): Operation { + const expansion = yield* getExpansion(); + return { type: WORKSPACE_FILE, name: `${operation}:${expansion.id}:${target}`, ...detail }; +} + +function* fileEffect( + database: WorkflowRunDatabase, + description: EffectDescription, + perform: (filesystem: DenoWorkspaceFilesystem) => Operation, +): Workflow { + return yield createWorkspaceEffect(database, description, (filesystem) => perform(filesystem)); +} + +/** + * Run one file effect and read back what it recorded. + * + * The same path serves a live effect and a replayed one: live execution + * publishes the outcome and hands it back, replay hands back the outcome that + * was published. Neither branch is written twice here, which is what makes + * "replay restores the recorded result" a property of the code rather than a + * claim about it. + */ +function* performed( + database: WorkflowRunDatabase, + description: EffectDescription, + perform: (filesystem: DenoWorkspaceFilesystem) => Operation, +): Operation { + const outcome = parseOutcome(yield* fileEffect(database, description, perform)); + if (outcome === undefined) { + throw new FilesInvariantError("protocol"); + } + return outcome; +} + +function* statPath( + filesystem: DenoWorkspaceFilesystem, + path: string, +): Operation> { + try { + return Ok(yield* filesystem.stat(path)); + } catch (error) { + return Err(asRefusal(error)); + } +} + +function* readOutcome( + filesystem: DenoWorkspaceFilesystem, + path: string, +): Operation { + const info = yield* statPath(filesystem, path); + if (!info.ok) { + return refused("resolution", refusalReason(info.error)); + } + if (info.value.kind !== "file") { + return refused("target", "directory"); + } + try { + return { kind: "content", content: yield* filesystem.readTextFile(path) }; + } catch (error) { + return refused("access", refusalReason(asRefusal(error))); + } +} + +/** + * What the target already is, when that decides the write before it starts. + * + * A directory cannot become a file, and saying so before anything is attempted + * is what keeps the target claim `unchanged` rather than `rolled-back`. A path + * that does not exist yet is the ordinary case and answers `undefined`. + */ +function* classifyWriteTarget( + filesystem: DenoWorkspaceFilesystem, + path: string, +): Operation { + const info = yield* statPath(filesystem, path); + if (info.ok) { + return info.value.kind === "file" ? undefined : refused("target", "directory"); + } + const reason = refusalReason(info.error); + return reason === "missing" ? undefined : refused("target", reason); +} + +function* replace( + filesystem: DenoWorkspaceFilesystem, + parent: string, + path: string, + content: string, +): Operation { + if (parent !== WORKSPACE_ROOT) { + yield* filesystem.mkdir(parent, { recursive: true }); + } + yield* filesystem.writeFile(path, content); +} + +/** + * Replace one file, discarding every part of the attempt if any part refuses. + * + * The parents and the replacement share one savepoint, so a write that creates + * two directories and then cannot be written leaves neither behind. The refusal + * that comes back therefore describes a Workspace that is exactly what it was, + * which is the `rolled-back` target claim the write vocabulary already has. + */ +function* writeOutcome( + filesystem: DenoWorkspaceFilesystem, + path: string, + content: string, +): Operation { + const existing = yield* classifyWriteTarget(filesystem, path); + if (existing !== undefined) { + return existing; + } + + try { + yield* savepoint(replace(filesystem, logicalParent(path), path, content)); + } catch (error) { + return refused("transaction", refusalReason(asRefusal(error))); + } + return { kind: "written" }; +} + +const SUBTREE = "/**"; + +function toRegExp(pattern: string): RegExp { + return globToRegExp(pattern, { extended: true, globstar: true }); +} + +/** + * A matcher for directories whose entire subtree an exclusion covers. + * + * Only a trailing `/**` proves it: matching the directory itself says nothing + * about the files beneath it, so anything else is walked and filtered one file + * at a time. Descending a subtree whose files are all excluded costs reads; + * skipping one that holds a match loses the match. + */ +function pruneMatcher(pattern: string): RegExp | undefined { + if (pattern === "**") { + return toRegExp("**"); + } + if (!pattern.endsWith(SUBTREE)) { + return undefined; + } + return toRegExp(pattern.slice(0, -SUBTREE.length)); +} + +interface Traversal { + readonly include: RegExp[]; + readonly exclude: RegExp[]; + readonly prune: RegExp[]; + readonly matched: string[]; +} + +/** + * Collect matching files beneath one logical directory. + * + * Exclusion is decided per candidate: a file or symlink whose own relative path + * an exclusion matches is not reported. A directory is never a candidate, so its + * own path is not tested against exclusions at all — the only question it raises + * is whether walking it can still produce something. + * + * A symbolic link is reported by its own path and never followed, so traversal + * stays inside the directory it started in and cannot cycle. + */ +function* descend( + filesystem: DenoWorkspaceFilesystem, + directory: string, + prefix: string, + walk: Traversal, +): Operation { + for (const entry of yield* filesystem.readdir(directory)) { + const path = prefix === "" ? entry.name : `${prefix}/${entry.name}`; + + if (entry.kind === "directory") { + if (!walk.prune.some((expression) => expression.test(path))) { + yield* descend(filesystem, logicalJoin(directory, entry.name), path, walk); + } + continue; + } + + if (walk.exclude.some((expression) => expression.test(path))) { + continue; + } + if (walk.include.some((expression) => expression.test(path))) { + walk.matched.push(path); + } + } +} + +function byCodePoint(left: string, right: string): number { + return left < right ? -1 : left > right ? 1 : 0; +} + +function compile(include: string[], exclude: string[]): Result { + try { + return Ok({ + include: include.map(toRegExp), + exclude: exclude.map(toRegExp), + prune: exclude + .map(pruneMatcher) + .filter((expression): expression is RegExp => expression !== undefined), + matched: [], + }); + } catch (error) { + // The patterns are compiled before the walk, so an unusable one — an + // unterminated character class — arrives as a `SyntaxError` from `RegExp`. + // It is the one failure here a document can fix by editing what it wrote. + if (error instanceof SyntaxError) { + return Err(new WorkspaceRefusal("invalid-pattern")); + } + throw error; + } +} + +function* globOutcome( + filesystem: DenoWorkspaceFilesystem, + directory: string, + include: string[], + exclude: string[], +): Operation { + const info = yield* statPath(filesystem, directory); + if (!info.ok) { + return refused("target", refusalReason(info.error)); + } + if (info.value.kind !== "directory") { + return refused("target", "not-directory"); + } + + const walk = compile(include, exclude); + if (!walk.ok) { + return refused("pattern", refusalReason(walk.error)); + } + + try { + yield* descend(filesystem, directory, "", walk.value); + } catch (error) { + return refused("traversal", refusalReason(asRefusal(error))); + } + return { kind: "paths", paths: [...new Set(walk.value.matched)].sort(byCodePoint) }; +} + +/** + * Rebuild a failure from what the record described. + * + * The vocabulary is parsed rather than believed: a retained phase or reason + * that is not one becomes a provider invariant rather than a sentence about a + * filesystem condition nobody reported. + */ +function nonWriteFailure( + operation: "read" | "glob", + outcome: { readonly phase: string; readonly reason: string }, +): Result { + const phase = parseFilesPhase(outcome.phase); + const reason = parseFilesReason(outcome.reason); + if (phase === undefined || reason === undefined) { + throw new FilesInvariantError("protocol"); + } + return Err(filesFailure({ operation, phase, reason })); +} + +function writeFailure(outcome: { readonly phase: string; readonly reason: string }): Result { + const phase = parseFileWritePhase(outcome.phase); + const reason = parseFilesReason(outcome.reason); + if (phase === undefined || reason === undefined) { + throw new FilesInvariantError("protocol"); + } + return Err(fileWriteFailure({ phase, reason })); +} + +/** The document filesystem of one workflow run. */ +export interface WorkflowFilesHandler { + checkFilePath(input: FilePathInput): Operation>; + readTextFile(input: FilePathInput): Operation>; + writeTextFile(input: FileWriteInput): Operation>; + globFiles(input: GlobInput): Operation>; + temporaryDirectory(): Operation>; +} + +export function workflowFilesHandler(database: WorkflowRunDatabase): WorkflowFilesHandler { + return { + // deno-lint-ignore require-yield + *checkFilePath(input: FilePathInput): Operation> { + const resolved = resolveLogicalPath(input.cwd, input.path); + if (!resolved.ok) { + return Err( + filesFailure({ + operation: "check-file-path", + phase: "lexical", + reason: lexicalReason(resolved.error), + }), + ); + } + return Ok(undefined); + }, + + *readTextFile(input: FilePathInput): Operation> { + const resolved = resolveLogicalPath(input.cwd, input.path); + if (!resolved.ok) { + return Err( + filesFailure({ + operation: "read", + phase: "lexical", + reason: lexicalReason(resolved.error), + }), + ); + } + const path = resolved.value; + const outcome = yield* performed( + database, + yield* describeFileEffect("read", path, { path: input.path, cwd: input.cwd }), + (filesystem) => readOutcome(filesystem, path), + ); + if (outcome.kind === "refused") { + return nonWriteFailure("read", outcome); + } + if (outcome.kind !== "content") { + throw new FilesInvariantError("protocol"); + } + return Ok(outcome.content); + }, + + *writeTextFile(input: FileWriteInput): Operation> { + const resolved = resolveLogicalPath(input.cwd, input.path); + if (!resolved.ok) { + return Err(fileWriteFailure({ phase: "lexical", reason: lexicalReason(resolved.error) })); + } + const path = resolved.value; + const outcome = yield* performed( + database, + yield* describeFileEffect("write", path, { path: input.path, cwd: input.cwd }), + (filesystem) => writeOutcome(filesystem, path, input.content), + ); + if (outcome.kind === "refused") { + return writeFailure(outcome); + } + if (outcome.kind !== "written") { + throw new FilesInvariantError("protocol"); + } + return Ok(fileWriteSuccess("transaction-staged")); + }, + + *globFiles(input: GlobInput): Operation> { + const directory = logicalDirectory(input.cwd); + const include = [...input.include]; + const exclude = [...input.exclude]; + const outcome = yield* performed( + database, + yield* describeFileEffect("glob", directory, { include, exclude }), + (filesystem) => globOutcome(filesystem, directory, include, exclude), + ); + if (outcome.kind === "refused") { + return nonWriteFailure("glob", outcome); + } + if (outcome.kind !== "paths") { + throw new FilesInvariantError("protocol"); + } + return Ok(outcome.paths); + }, + + /** + * A workflow run has no host directory to hand out. + * + * Denied rather than emulated inside the Workspace: `` exists so a + * document can hand a path to a tool the caller already has, and a logical + * path is not one. Falling through to the host would give a run exactly the + * unretained, uncontained filesystem the boundary exists to keep it out of. + */ + // deno-lint-ignore require-yield + *temporaryDirectory(): Operation> { + throw new FilesOperationDeniedError("temporary-directory"); + }, + }; +} + +/** + * Install this run's document filesystem for the current scope and below. + * + * `{ at: "min" }` on the same terms as every other provider: an outer host + * adapter installed by the CLI entrypoint would otherwise answer ahead of this + * one, and the whole point of a workflow run is that it does not. + */ +export function useWorkflowFiles(database: WorkflowRunDatabase): Operation { + const handler = workflowFilesHandler(database); + return Files.around( + { + *checkFilePath([input]) { + return yield* handler.checkFilePath(input); + }, + *readTextFile([input]) { + return yield* handler.readTextFile(input); + }, + *writeTextFile([input]) { + return yield* handler.writeTextFile(input); + }, + *globFiles([input]) { + return yield* handler.globFiles(input); + }, + *temporaryDirectory() { + return yield* handler.temporaryDirectory(); + }, + }, + { at: "min" }, + ); +} diff --git a/packages/workflow/src/deno/workspace/host.ts b/packages/workflow/src/deno/workspace/host.ts new file mode 100644 index 00000000..e5df64c2 --- /dev/null +++ b/packages/workflow/src/deno/workspace/host.ts @@ -0,0 +1,63 @@ +/** + * What a host installs around one workflow document execution. + * + * Three installations, in one place, because they only make sense together: the + * run's effect coordinator decides how a Workspace effect commits, the Files + * provider is what turns a document's `` into one of those effects, and + * the logical working directory is what those paths are relative to. Installing + * any two without the third would leave a document resolving paths one provider + * cannot reach. + * + * They are installed **inside** the execution rather than at the entrypoint, so + * they sit beneath the host adapter `xmd run` installs and answer ahead of it. + * Ordinary `xmd run` keeps its host Files provider untouched; a workflow run's + * document never reaches it. + * + * This is the attachment path, and a completed run does not take it. A root + * result that is already recorded returns without expanding the document, so + * there is nothing to give a filesystem to — and attaching one anyway would + * open a transaction and capture a root for a run that is not going to perform + * an effect. + */ + +import { scoped, type Operation } from "effection"; +import { API } from "@executablemd/runtime"; +import type { WorkflowRunDatabase } from "../../storage/api.ts"; +import { withWorkspaceEffects } from "./effect.ts"; +import { useWorkflowFiles } from "./files.ts"; +import { WORKSPACE_ROOT } from "./logical-path.ts"; + +/** + * The working directory a workflow document starts in. + * + * The Workspace root, and a logical path rather than a host one. A document + * that resolves `notes.md` against it names an entry in the run's own + * filesystem, and nothing it can write reaches the directory the caller + * happened to invoke `xmd` from. + */ +export function useLogicalWorkspaceCwd(): Operation { + return API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return WORKSPACE_ROOT; + }, + }, + { at: "min" }, + ); +} + +/** Run `operation` with this run's Workspace attached to the document filesystem. */ +export function withWorkflowWorkspace( + database: WorkflowRunDatabase, + operation: Operation, +): Operation { + return withWorkspaceEffects( + database, + scoped(function* () { + yield* useLogicalWorkspaceCwd(); + yield* useWorkflowFiles(database); + return yield* operation; + }), + ); +} diff --git a/packages/workflow/src/deno/workspace/logical-path.ts b/packages/workflow/src/deno/workspace/logical-path.ts new file mode 100644 index 00000000..1b874ddf --- /dev/null +++ b/packages/workflow/src/deno/workspace/logical-path.ts @@ -0,0 +1,127 @@ +/** + * Where a document's path lands in a run's logical Workspace. + * + * Every path a workflow document writes is resolved here, and the result is an + * absolute POSIX path inside the run's own filesystem. Nothing this module + * produces is a host path: there is no drive, no separator to choose, and no + * outside for a resolution to reach — the Workspace root *is* the boundary, so + * containment is decided by arithmetic on segments rather than by observing a + * filesystem. + * + * That is why admission here is purely lexical. The host provider has to defer + * part of its judgement until it can see a symlink, because a host path can + * point anywhere; a logical path resolves inside a tree the run owns entirely, + * and `/..` is `/` the way POSIX says it is. + */ + +import { Err, Ok, type Result } from "effection"; +import type { FilesReason } from "@executablemd/runtime"; + +/** A path the document wrote that names nothing this Workspace can hold. */ +export class LogicalPathError extends Error { + override name = "LogicalPathError"; + readonly reason: FilesReason; + + constructor(reason: FilesReason) { + super("logical path refused"); + this.reason = reason; + } +} + +export const WORKSPACE_ROOT = "/"; + +/** No filesystem holds a name containing one, so no path here may carry one. */ +const NUL = "\u0000"; + +/** + * The segments of a directory this Workspace can be working in. + * + * A caller's working directory is arrangement rather than a document's own + * text, so it is clamped rather than refused: a leading `..` at the root stays + * at the root, exactly as it would in a POSIX filesystem, and a directory that + * is not written as an absolute path is read relative to the root. Neither can + * name anything outside the Workspace, which is the only property this needs. + */ +function directorySegments(cwd: string): string[] { + const segments: string[] = []; + for (const segment of cwd.split("/")) { + if (segment === "" || segment === ".") { + continue; + } + if (segment === "..") { + segments.pop(); + continue; + } + segments.push(segment); + } + return segments; +} + +function posix(segments: readonly string[]): string { + return segments.length === 0 ? WORKSPACE_ROOT : `/${segments.join("/")}`; +} + +/** The absolute logical directory a document is working in. */ +export function logicalDirectory(cwd: string): string { + return posix(directorySegments(cwd)); +} + +/** + * The absolute logical path an authored path names, or why it names none. + * + * The three lexical refusals are the ones a document can act on: it wrote + * nothing, it wrote somewhere absolute, or it wrote its way out of the + * directory it is working in. A NUL is none of those, so it is reported as an + * operation that cannot be carried out rather than described back to the + * document. + */ +export function resolveLogicalPath(cwd: string, path: string): Result { + if (path === "") { + return Err(new LogicalPathError("empty-path")); + } + if (path.startsWith("/")) { + return Err(new LogicalPathError("absolute-path")); + } + if (path.includes(NUL) || cwd.includes(NUL)) { + return Err(new LogicalPathError("operation-failed")); + } + + const base = directorySegments(cwd); + const segments = [...base]; + for (const segment of path.split("/")) { + if (segment === "" || segment === ".") { + continue; + } + if (segment === "..") { + if (segments.length <= base.length) { + return Err(new LogicalPathError("lexical-escape")); + } + segments.pop(); + continue; + } + segments.push(segment); + } + + // `.` and `a/..` normalize back onto the working directory. That is not an + // escape, and saying so would misdescribe it: the path names a directory, and + // target classification is what reports that. + return Ok(posix(segments)); +} + +/** The logical parent directory of an absolute logical path. */ +export function logicalParent(path: string): string { + const segments = path.split("/").filter((segment) => segment !== ""); + segments.pop(); + return posix(segments); +} + +/** The path of `entry` relative to `directory`, POSIX-separated. */ +export function logicalRelative(directory: string, entry: string): string { + const base = directory === WORKSPACE_ROOT ? "" : directory; + return entry.startsWith(`${base}/`) ? entry.slice(base.length + 1) : entry; +} + +/** One more segment beneath an absolute logical directory. */ +export function logicalJoin(directory: string, name: string): string { + return directory === WORKSPACE_ROOT ? `/${name}` : `${directory}/${name}`; +} diff --git a/packages/workflow/src/journal.ts b/packages/workflow/src/journal.ts index bc854d28..ac901ce2 100644 --- a/packages/workflow/src/journal.ts +++ b/packages/workflow/src/journal.ts @@ -56,6 +56,26 @@ export function malformedRecord(description: EffectDescription): StaleInputError ); } +/** + * The recorded run is not the retained run this execution was installed with. + * + * The differing fields are named and their values are not. A run id may be + * caller-selected and a base may be any revision expression, so both are + * external text on the same terms as retained props: naming a field says what + * disagrees without carrying the disagreement into logs and rendered output. + */ +export function retainedRunMismatch( + description: EffectDescription, + fields: readonly string[], +): StaleInputError { + return new StaleInputError( + `The journal records a workflow run whose ${fields.join(", ")} differs from the retained ` + + "run this execution was installed with. A retained run is replayed as itself rather " + + "than onto a different one. Resume the run the journal belongs to.", + { coroutineId: "root", description }, + ); +} + /** The recorded run started from a different base than this run supplied. */ export function baseMismatch( description: EffectDescription, diff --git a/packages/workflow/src/run.ts b/packages/workflow/src/run.ts index caef5dcb..c7a7244e 100644 --- a/packages/workflow/src/run.ts +++ b/packages/workflow/src/run.ts @@ -20,6 +20,12 @@ * is the only place a completed journal can restore its run — or refuse a * different base before the recorded result is handed back. * + * `useRetainedWorkflow(run)` is the same three states under a run that already + * exists. A workflow host has created the storage record before anything + * executes, so the live path records exactly the value it was given rather than + * allocating an id and resolving a base, and every state requires the journal to + * agree with that value in full. + * * All of it is operation-scoped. The value is installed in the scope that owns * the document execution, so every descendant of the expansion reads it, the * output emitted after the durable run still sees it, and ordinary teardown @@ -38,6 +44,7 @@ import { describeWorkflowRun, malformedRecord, readWorkflowRun, + retainedRunMismatch, WORKFLOW_RUN, } from "./journal.ts"; import type { WorkflowRun } from "./journal.ts"; @@ -69,19 +76,80 @@ export function* getWorkflowRun(): Operation { return run; } +/** + * How one installation decides what the run is, and what the journal is held to. + * + * Two hosts need different answers to both questions. A programmatic caller + * supplies a base and lets the first live execution allocate an id and resolve + * that base, so the only thing a record can disagree about is the base it was + * made from. A workflow host has already created the storage record, so the run + * is not the execution's to allocate: it arrives whole, and a journal that + * records a different one is not this run's journal. + */ +interface RunEstablishment { + readonly base: string; + /** The run this execution is of, reached only when nothing is recorded yet. */ + allocate(): Operation; + /** The recorded run, or a refusal naming what it disagrees about. */ + hold(description: EffectDescription, recorded: WorkflowRun): WorkflowRun; +} + /** Append the run to the journal, and answer with what the journal holds. */ -function* record(description: EffectDescription, base: string): Workflow { +function* record( + description: EffectDescription, + establishment: RunEstablishment, +): Workflow { return yield createDurableOperation(description, function* (): Operation { // Reached only when nothing is recorded yet: a replay hands the stored // value back without running this at all, so neither the identifier nor Git // is reached a second time. - const pinnedCommit = yield* revParse(`${base}^{commit}`); - // Web Crypto rather than `node:crypto`: a run id is allocated in shared - // code, which names no host. - return { runId: crypto.randomUUID(), base, pinnedCommit }; + const { runId, base, pinnedCommit } = yield* establishment.allocate(); + return { runId, base, pinnedCommit }; }); } +function allocating(base: string): RunEstablishment { + return { + base, + *allocate(): Operation { + const pinnedCommit = yield* revParse(`${base}^{commit}`); + // Web Crypto rather than `node:crypto`: a run id is allocated in shared + // code, which names no host. + return { runId: crypto.randomUUID(), base, pinnedCommit }; + }, + /** + * The description carries the base for a reader; divergence detection + * compares only type and name, so the base this run supplied is checked + * against the stored *value* rather than against the entry's identity. + */ + hold(description: EffectDescription, recorded: WorkflowRun): WorkflowRun { + if (recorded.base !== base) { + throw baseMismatch(description, recorded.base, base); + } + return recorded; + }, + }; +} + +function retaining(run: WorkflowRun): RunEstablishment { + return { + base: run.base, + // deno-lint-ignore require-yield + *allocate(): Operation { + return run; + }, + hold(description: EffectDescription, recorded: WorkflowRun): WorkflowRun { + const differing = (["runId", "base", "pinnedCommit"] as const).filter( + (field) => recorded[field] !== run[field], + ); + if (differing.length > 0) { + throw retainedRunMismatch(description, differing); + } + return recorded; + }, + }; +} + function same(left: WorkflowRun, right: WorkflowRun): boolean { return ( left.runId === right.runId && @@ -90,27 +158,22 @@ function same(left: WorkflowRun, right: WorkflowRun): boolean { ); } -/** - * Read the record this run is held to, refusing anything that is not it. - * - * The description carries the base for a reader; divergence detection compares - * only type and name, so the base this run supplied is checked against the - * stored *value* rather than against the entry's identity. - */ -function held(description: EffectDescription, stored: unknown, base: string): WorkflowRun { +/** Read the record this run is held to, refusing anything that is not it. */ +function held( + description: EffectDescription, + stored: unknown, + establishment: RunEstablishment, +): WorkflowRun { const run = readWorkflowRun(stored); if (run === undefined) { throw malformedRecord(description); } - if (run.base !== base) { - throw baseMismatch(description, run.base, base); - } - return run; + return establishment.hold(description, run); } -function* establish(base: string): Operation { - const description = describeWorkflowRun(base); - const run = held(description, yield* record(description, base), base); +function* establish(establishment: RunEstablishment): Operation { + const description = describeWorkflowRun(establishment.base); + const run = held(description, yield* record(description, establishment), establishment); const restored = yield* CurrentWorkflowRun.get(); // A truncated replay already restored this value in the check phase; keeping // that object is what makes every read in one execution the same one. @@ -120,35 +183,80 @@ function* establish(base: string): Operation { yield* CurrentWorkflowRun.set(run); } -function* restore(event: Yield, base: string): Operation { +function* restore(event: Yield, establishment: RunEstablishment): Operation { if (event.description.type !== WORKFLOW_RUN || event.result.status !== "ok") { return; } - yield* CurrentWorkflowRun.set(held(describeWorkflowRun(base), event.result.value, base)); + yield* CurrentWorkflowRun.set( + held(describeWorkflowRun(establishment.base), event.result.value, establishment), + ); } -/** - * Associate the document execution this scope owns with a workflow run. - * - * Installing this creates nothing. Executing a document under it does. - */ -export function* useWorkflow(options: { base: string }): Operation { - const { base } = options; - +function* install(establishment: RunEstablishment): Operation { yield* ReplayGuard.around({ *check([event], next) { // Runs before `durableRun` can short-circuit on a recorded root Close, so - // a completed journal restores its run here — and refuses a different - // base here, before the recorded result is returned. - yield* restore(event, base); + // a completed journal restores its run here — and refuses a record that + // is not this run's here, before the recorded result is returned. + yield* restore(event, establishment); return yield* next(event); }, }); yield* Execution.around({ *document([props], next) { - yield* establish(base); + yield* establish(establishment); return yield* next(props); }, }); } + +/** + * Associate the document execution this scope owns with a workflow run. + * + * Installing this creates nothing. Executing a document under it does. + */ +export function useWorkflow(options: { base: string }): Operation { + return install(allocating(options.base)); +} + +/** + * Associate the document execution this scope owns with a run that already + * exists. + * + * A workflow host creates the run's storage record before it executes anything, + * so by the time a document runs there is nothing left to allocate or resolve: + * the run id is the one storage answered with, and the pinned commit is the one + * the definition was established from. This installation records exactly that + * value and requires a journal to agree with it in every field, so a resumed + * execution can never continue under a run the storage record does not describe. + * + * Git is not consulted, and no identifier is generated. + */ +export function useRetainedWorkflow(run: WorkflowRun): Operation { + return install(retaining(retainedRun(run))); +} + +/** + * The retained run as a frozen value of its own. + * + * Parsed rather than believed: it arrives from a storage record a host read + * back, so a member that is missing or empty is a value that identifies no run + * rather than one to install and discover later. + */ +function retainedRun(run: WorkflowRun): WorkflowRun { + const parsed = readWorkflowRun(run); + if (parsed === undefined || parsed.runId === "" || parsed.base === "") { + throw new Error( + "useRetainedWorkflow() needs the retained run's id, base and pinned commit. A run " + + "installed without them identifies no workflow run.", + ); + } + if (parsed.pinnedCommit === "") { + throw new Error( + "useRetainedWorkflow() needs the retained run's pinned commit: an empty one pins the " + + "run to no repository state at all.", + ); + } + return parsed; +} diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts new file mode 100644 index 00000000..b0a85a52 --- /dev/null +++ b/packages/workflow/tests/retained-run.test.ts @@ -0,0 +1,210 @@ +/** + * Tier RR — installing a run that already exists. + * + * `useWorkflow({ base })` lets the first live execution decide what the run is: + * it allocates an identifier and resolves the base through Git. A workflow host + * has already done both by the time a document runs — storage answered with the + * run id, and the definition was established from a commit it pinned — so + * nothing is left to decide, and a journal that records a different run is not + * this run's journal. + * + * Every test here replaces `Git` with a provider that fails when it is + * consulted, so "the retained installation asks Git nothing" is asserted rather + * than assumed. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { scoped } from "effection"; +import type { Operation } from "effection"; +import { InMemoryStream, StaleInputError } from "@executablemd/durable-streams"; +import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import { collect, execute, inlineSource, registerComponents } from "@executablemd/core"; +import { Git } from "../src/git.ts"; +import { getWorkflowRun, useRetainedWorkflow } from "../src/run.ts"; +import type { WorkflowRun } from "../src/run.ts"; + +const COMMIT = "9fceb02d0ae598e95dc970b74767f19372d61af8"; +const OTHER_COMMIT = "1111111111111111111111111111111111111111"; + +const RETAINED: WorkflowRun = Object.freeze({ + runId: "release-1.4", + base: "main", + pinnedCommit: COMMIT, +}); + +/** A Git that fails the test if anything consults it. */ +function useForbiddenGit(): Operation { + return Git.around( + { + // deno-lint-ignore require-yield + *revParse([revision]) { + throw new Error(`Git was consulted for "${revision}"`); + }, + }, + { at: "min" }, + ); +} + +/** `` — reports the run it was expanded under. */ +function useProbe(seen: WorkflowRun[]): Operation { + return registerComponents([ + { + name: "Probe", + origin: "tier-rr", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + seen.push(yield* getWorkflowRun()); + return ""; + }, + }, + ]); +} + +interface Attempt { + readonly seen: WorkflowRun[]; + readonly thrown: unknown; +} + +function runRetained(run: WorkflowRun, stream: InMemoryStream): Operation { + return scoped(function* () { + const seen: WorkflowRun[] = []; + yield* useForbiddenGit(); + yield* useProbe(seen); + yield* useRetainedWorkflow(run); + try { + yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + return { seen, thrown: undefined }; + } catch (error) { + return { seen, thrown: error }; + } + }); +} + +function workflowEvents(stream: InMemoryStream): DurableEvent[] { + return stream + .snapshot() + .filter((event) => event.type === "yield" && event.description.type === "workflow_run"); +} + +function recordedRun(stream: InMemoryStream): Json | undefined { + const event = workflowEvents(stream)[0]; + if (event === undefined || event.type !== "yield" || event.result.status !== "ok") { + return undefined; + } + return event.result.value; +} + +/** The journal without the root's close, which is what makes the next run replay. */ +function partial(stream: InMemoryStream): InMemoryStream { + return new InMemoryStream( + stream.snapshot().filter((event) => !(event.type === "close" && event.coroutineId === "root")), + ); +} + +describe("Tier RR — retained workflow runs", () => { + it("RR1: records exactly the retained run, without allocating or resolving", function* () { + const stream = new InMemoryStream(); + const attempt = yield* runRetained(RETAINED, stream); + + expect(attempt.thrown).toBeUndefined(); + expect(attempt.seen).toEqual([RETAINED]); + expect(recordedRun(stream)).toEqual({ + runId: "release-1.4", + base: "main", + pinnedCommit: COMMIT, + }); + expect(workflowEvents(stream)).toHaveLength(1); + }); + + it("RR2: restores the retained run from a truncated journal without recording again", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const resumed = partial(first); + const attempt = yield* runRetained(RETAINED, resumed); + + expect(attempt.thrown).toBeUndefined(); + expect(attempt.seen).toEqual([RETAINED]); + expect(workflowEvents(resumed)).toHaveLength(1); + }); + + it("RR3: restores the retained run from a completed journal", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const replayed = new InMemoryStream(first.snapshot()); + const attempt = yield* runRetained(RETAINED, replayed); + + expect(attempt.thrown).toBeUndefined(); + expect(workflowEvents(replayed)).toHaveLength(1); + }); + + it("RR4: refuses a journal recording a different run id", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const attempt = yield* runRetained({ ...RETAINED, runId: "release-1.5" }, partial(first)); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + const message = attempt.thrown instanceof Error ? attempt.thrown.message : ""; + expect(message).toContain("runId"); + // The differing values are named nowhere: a run id is caller-selected text. + expect(message).not.toContain("release-1.4"); + expect(message).not.toContain("release-1.5"); + expect(attempt.seen).toEqual([]); + }); + + it("RR5: refuses a journal recording a different base or pinned commit", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const base = yield* runRetained({ ...RETAINED, base: "release/1.4" }, partial(first)); + expect(base.thrown).toBeInstanceOf(StaleInputError); + expect(base.thrown instanceof Error ? base.thrown.message : "").toContain("base"); + + const pinned = yield* runRetained({ ...RETAINED, pinnedCommit: OTHER_COMMIT }, partial(first)); + expect(pinned.thrown).toBeInstanceOf(StaleInputError); + expect(pinned.thrown instanceof Error ? pinned.thrown.message : "").toContain("pinnedCommit"); + }); + + it("RR6: refuses a record that does not describe a workflow run at all", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const damaged = new InMemoryStream( + partial(first) + .snapshot() + .map((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? { ...event, result: { status: "ok", value: { runId: 7 } } } + : event, + ), + ); + + const attempt = yield* runRetained(RETAINED, damaged); + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + }); + + it("RR7: refuses to install a run that identifies nothing", function* () { + const empty = yield* scoped(function* () { + try { + yield* useRetainedWorkflow({ runId: "", base: "main", pinnedCommit: COMMIT }); + return undefined; + } catch (error) { + return error; + } + }); + expect(empty).toBeInstanceOf(Error); + + const unpinned = yield* scoped(function* () { + try { + yield* useRetainedWorkflow({ runId: "r", base: "main", pinnedCommit: "" }); + return undefined; + } catch (error) { + return error; + } + }); + expect(unpinned).toBeInstanceOf(Error); + }); +}); diff --git a/packages/workflow/tests/support/workspace-crash-child.ts b/packages/workflow/tests/support/workspace-crash-child.ts index aa3d125b..fab3aa59 100644 --- a/packages/workflow/tests/support/workspace-crash-child.ts +++ b/packages/workflow/tests/support/workspace-crash-child.ts @@ -43,7 +43,7 @@ import { useJournalRouting } from "../../src/deno/journal-route.ts"; import { readTransaction } from "../../src/deno/reading.ts"; import { verifySchema } from "../../src/deno/schema.ts"; import { - createWorkspaceProofEffect, + createWorkspaceEffect, useWorkspaceEffects, withWorkspaceEffects, } from "../../src/deno/workspace/effect.ts"; @@ -146,7 +146,7 @@ function* crash(root: string, runId: string): Operation { // The run this process resumes already holds this effect's result, so it // replays. Executing it would mean the crash effect below is not the // first live work of the process, and the count says which happened. - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: BASELINE_EFFECT }, // deno-lint-ignore require-yield @@ -155,7 +155,7 @@ function* crash(root: string, runId: string): Operation { return null; }, ); - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: CRASH_EFFECT }, function* (selected) { diff --git a/packages/workflow/tests/support/workspace-restart-child.ts b/packages/workflow/tests/support/workspace-restart-child.ts index c3a94e04..4613c488 100644 --- a/packages/workflow/tests/support/workspace-restart-child.ts +++ b/packages/workflow/tests/support/workspace-restart-child.ts @@ -25,10 +25,7 @@ import { durableRun, type Workflow } from "@executablemd/durable-streams"; import { main, type Operation } from "effection"; import { WorkflowRunStorage, type WorkflowRunDatabase } from "../../mod.ts"; import { useWorkflowRunStorage } from "../../deno.ts"; -import { - createWorkspaceProofEffect, - withWorkspaceEffects, -} from "../../src/deno/workspace/effect.ts"; +import { createWorkspaceEffect, withWorkspaceEffects } from "../../src/deno/workspace/effect.ts"; import { setPrivateWorkspaceClock, transactWorkspaceRoots, @@ -62,7 +59,7 @@ const DEFINITION = { */ function workflow(database: WorkflowRunDatabase, marker: string, clock: { now: number }) { return function* (): Workflow { - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: "seed" }, function* (filesystem) { @@ -74,7 +71,7 @@ function workflow(database: WorkflowRunDatabase, marker: string, clock: { now: n return null; }, ); - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: "revise" }, function* (filesystem) { diff --git a/packages/workflow/tests/workspace-effect-transaction.test.ts b/packages/workflow/tests/workspace-effect-transaction.test.ts index 7b947ff7..1fe5ca9b 100644 --- a/packages/workflow/tests/workspace-effect-transaction.test.ts +++ b/packages/workflow/tests/workspace-effect-transaction.test.ts @@ -37,7 +37,7 @@ import { useJournalRouting } from "../src/deno/journal-route.ts"; import { SavepointObservation, type SavepointObserver } from "../src/deno/savepoints.ts"; import { initializeSchema } from "../src/deno/schema.ts"; import { - createWorkspaceProofEffect, + createWorkspaceEffect, useWorkspaceEffects, withWorkspaceEffects, } from "../src/deno/workspace/effect.ts"; @@ -132,7 +132,7 @@ function* workspaceStep( name: string, mutate: (filesystem: DenoWorkspaceFilesystem) => Operation, ): Workflow { - yield createWorkspaceProofEffect(database, { type: "workspace-proof", name }, mutate); + yield createWorkspaceEffect(database, { type: "workspace-proof", name }, mutate); } function* inspectWorkspace( diff --git a/packages/workflow/tests/workspace-files.test.ts b/packages/workflow/tests/workspace-files.test.ts new file mode 100644 index 00000000..29847d34 --- /dev/null +++ b/packages/workflow/tests/workspace-files.test.ts @@ -0,0 +1,499 @@ +/** + * Tier WF — the document filesystem of a workflow run. + * + * These drive the real `` and `` definitions through `execute()` + * against a real run database, because what is under test is where a document's + * paths land and what survives in the journal — neither of which a stand-in for + * DOFS or for SQLite could show. + * + * Two observations do most of the work. A second connection counts committed + * journal rows, which says whether a transaction has already published rather + * than whether a row is there now; and a host `API.Files` spy is installed + * *outside* the workflow provider, so any call that fell through to the caller's + * filesystem would be recorded rather than merely suspected. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { scoped, type Operation } from "effection"; +import { collect, execute, inlineSource } from "@executablemd/core"; +import type { Json } from "@executablemd/durable-streams"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { API, FILES_FATAL, parseFilesFatal, useHostFiles } from "@executablemd/runtime"; +import type { HostFilesEvent } from "@executablemd/runtime"; +import type { WorkflowRunDatabase } from "../mod.ts"; +import { withWorkflowWorkspace } from "../src/deno/workspace/host.ts"; +import { WORKSPACE_FILE } from "../src/deno/workspace/files.ts"; +import { transactWorkspaceRoots } from "../src/deno/workspace/private.ts"; +import type { PrivateWorkspaceTransaction } from "../src/deno/workspace/private.ts"; +import { + committedEventCount, + createRun, + runPath, + tamper, + useStorageRoot, + withStorage, +} from "./support/storage.ts"; + +/** What an operation threw, so a suite can assert on it rather than fail. */ +function* raised(operation: Operation): Operation { + try { + yield* operation; + return undefined; + } catch (error) { + return error; + } +} + +/** + * The infrastructure failure somewhere in this failure's causes. + * + * A denied operation is raised where `` acquired it and reaches the + * caller wrapped in whatever the document execution reported, so the assertion + * follows the chain the engine builds rather than the top of it. + */ +function fatalOf(error: unknown): unknown { + let current = error; + for (let depth = 0; depth < 16 && current instanceof Error; depth += 1) { + if (parseFilesFatal(current) !== undefined) { + return current; + } + current = current.cause; + } + return error; +} + +/** The current root pointer, as a second connection sees it. */ +function committedRoot(path: string): unknown { + let found: unknown; + tamper(path, (database) => { + found = database.prepare("SELECT current_root_id AS root FROM workspace_state").get()?.root; + }); + return found; +} + +/** The Workspace root the newest committed journal row is associated with. */ +function rootOfLastEvent(path: string): unknown { + let found: unknown; + tamper(path, (database) => { + found = database + .prepare( + "SELECT workspace_root_id AS root FROM journal_events ORDER BY sequence DESC LIMIT 1", + ) + .get()?.root; + }); + return found; +} + +/** Every host document-filesystem step this run performed. Must stay empty. */ +interface HostSpy { + readonly seen: HostFilesEvent[]; +} + +function* useHostSpy(): Operation { + const seen: HostFilesEvent[] = []; + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return "/nowhere-the-workflow-may-reach"; + }, + }, + { at: "min" }, + ); + yield* useHostFiles({ observe: (event) => seen.push(event) }); + return { seen }; +} + +interface Run { + readonly output: Json; + readonly host: HostSpy; +} + +/** + * Execute `source` as this run's root document, with the run's Workspace + * attached and a host provider installed outside it. + */ +function runDocument(database: WorkflowRunDatabase, source: string): Operation { + return scoped(function* () { + const host = yield* useHostSpy(); + const output = yield* withWorkflowWorkspace( + database, + scoped(function* () { + return yield* collect( + yield* execute({ ...inlineSource(source), stream: database.journal }), + ); + }), + ); + return { output, host }; + }); +} + +/** The same document again, replaying the journal the first execution wrote. */ +function replayDocument(database: WorkflowRunDatabase, source: string): Operation { + return runDocument(database, source); +} + +function* workspaceEvents(database: WorkflowRunDatabase): Operation { + const events = yield* database.journal.readAll(); + return events.filter( + (event) => event.type === "yield" && event.description.type === WORKSPACE_FILE, + ); +} + +/** + * The file effects this run recorded, as operation and outcome. + * + * What a document rendered is not evidence that a file effect happened: an + * element's own expansion is journaled too, so a provider that never recorded + * anything can still replay the text it produced. These rows are the provider's + * own history, which is what the durability claims are about. + */ +function* recordedFileEffects( + database: WorkflowRunDatabase, +): Operation> { + const events = yield* workspaceEvents(database); + return events.flatMap((event) => + event.type === "yield" ? [{ name: event.description.name, result: event.result }] : [], + ); +} + +function* workspaceText(database: WorkflowRunDatabase, path: string): Operation { + const read = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.readTextFile(path); + }); + if (!read.ok) { + throw read.error; + } + return read.value; +} + +function* mutateWorkspace( + database: WorkflowRunDatabase, + body: (workspace: PrivateWorkspaceTransaction) => Operation, +): Operation { + const changed = yield* transactWorkspaceRoots(database, function* (workspace) { + yield* body(workspace); + const root = yield* workspace.capture(); + yield* workspace.publish(root.rootId); + }); + if (!changed.ok) { + throw changed.error; + } +} + +describe("WF workflow document filesystem", () => { + it("writes a file into the run's own Workspace and records one effect", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument( + database, + ["# Release", "", 'Prepared', ""].join("\n"), + ); + + expect(run.host.seen).toEqual([]); + expect(yield* workspaceText(database, "/notes/release.md")).toEqual("Prepared"); + + const events = yield* workspaceEvents(database); + expect(events).toHaveLength(1); + expect(events[0]?.type === "yield" && events[0].description.name).toContain("write:"); + }); + }); + + it("reads a file back through the same logical Workspace", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument( + database, + [ + '{"channel":"stable"}', + "", + '', + "", + "Read: {config}", + ].join("\n"), + ); + + expect(run.host.seen).toEqual([]); + expect(String(run.output)).toContain('Read: {"channel":"stable"}'); + }); + }); + + it("restores a read's recorded content when the frontier no longer holds it", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const source = [ + 'first', + "", + '', + "", + "Seen: {seen}", + ].join("\n"); + const first = yield* runDocument(database, source); + expect(String(first.output)).toContain("Seen: first"); + + // The provider recorded the read itself, not merely the element that + // asked for it: a read that answered from the frontier would leave one + // effect here instead of two. + const recorded = yield* recordedFileEffects(database); + expect(recorded.map((effect) => effect.name.split(":")[0])).toEqual(["write", "read"]); + expect(recorded[1]?.result).toEqual({ + status: "ok", + value: { kind: "content", content: "first" }, + }); + + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/seed.txt", "replaced"); + }); + expect(yield* workspaceText(database, "/seed.txt")).toEqual("replaced"); + + const replayed = yield* replayDocument(database, source); + expect(String(replayed.output)).toContain("Seen: first"); + expect(replayed.host.seen).toEqual([]); + expect(yield* recordedFileEffects(database)).toEqual(recorded); + }); + }); + + it("refuses a path that leaves the working directory without touching the host", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument(database, 'no'); + + expect(String(run.output)).toContain("resolves outside the working directory"); + expect(run.host.seen).toEqual([]); + expect(yield* workspaceEvents(database)).toEqual([]); + }); + }); + + it("reports a missing file as missing rather than reaching the host for it", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument(database, ''); + + expect(String(run.output)).toContain("absent.txt"); + expect(run.host.seen).toEqual([]); + const events = yield* workspaceEvents(database); + expect(events).toHaveLength(1); + }); + }); + + it("searches the logical Workspace with ", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument( + database, + [ + 'a', + 'b', + 'c', + "", + '', + "", + "Found: {found}", + ].join("\n"), + ); + + expect(run.host.seen).toEqual([]); + expect(String(run.output)).toContain("docs/a.md"); + expect(String(run.output)).toContain("docs/b.md"); + expect(String(run.output)).not.toContain("skip.txt"); + }); + }); + + it("commits the bytes, the current root and the filtered result together", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + yield* runDocument(database, 'committed'); + + // Counted through a second connection, so what it reports is what the + // transaction published rather than what this handle is holding. + expect(committedEventCount(path)).toBeGreaterThan(0); + expect(committedRoot(path)).toEqual(rootOfLastEvent(path)); + expect(yield* workspaceText(database, "/atomic.txt")).toEqual("committed"); + + const events = yield* workspaceEvents(database); + const written = events[0]; + expect(written?.type === "yield" && written.result).toEqual({ + status: "ok", + value: { kind: "written" }, + }); + }); + }); + + it("replays a create/delete/create history without consulting the current file", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const source = [ + 'one', + "", + '', + "", + "Seen: {seen}", + ].join("\n"); + + const first = yield* runDocument(database, source); + expect(String(first.output)).toContain("Seen: one"); + + // The history the replay has to survive, built through the seam a + // provider owns rather than through a public delete component: the file + // the document created is removed, and then a different file is created + // at the same path. + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.remove("/x.txt"); + }); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/x.txt", "three"); + }); + + const before = yield* database.journal.readAll(); + const recorded = yield* recordedFileEffects(database); + expect(recorded).toEqual([ + { name: recorded[0]?.name ?? "", result: { status: "ok", value: { kind: "written" } } }, + { + name: recorded[1]?.name ?? "", + result: { status: "ok", value: { kind: "content", content: "one" } }, + }, + ]); + const frontier = yield* workspaceText(database, "/x.txt"); + expect(frontier).toEqual("three"); + + const replayed = yield* replayDocument(database, source); + + expect(String(replayed.output)).toContain("Seen: one"); + expect(replayed.host.seen).toEqual([]); + // The write did not run again, so the frontier is still what the private + // history left there rather than the document's own content. + expect(yield* workspaceText(database, "/x.txt")).toEqual(frontier); + expect((yield* database.journal.readAll()).length).toEqual(before.length); + expect(yield* recordedFileEffects(database)).toEqual(recorded); + }); + }); + + it("refuses to publish a workflow file effect into a stream that is not the run's", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const before = yield* database.journal.readAll(); + + const failure = yield* raised( + scoped(function* () { + yield* useHostSpy(); + return yield* withWorkflowWorkspace( + database, + scoped(function* () { + return yield* collect( + yield* execute({ + ...inlineSource('no'), + stream: new InMemoryStream(), + }), + ); + }), + ); + }), + ); + + expect(failure).toBeInstanceOf(Error); + expect((yield* database.journal.readAll()).length).toEqual(before.length); + const present = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.readTextFile("/smuggled.txt"); + }); + expect(present.ok).toEqual(false); + }); + }); + + it("publishes a refusal as rolled back, leaving the Workspace as it was", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/blocked", "a file, not a directory"); + }); + const before = committedRoot(path); + + const run = yield* runDocument(database, 'no'); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded).toHaveLength(1); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "refused", phase: "transaction", reason: "not-directory" }, + }); + // Nothing the attempt created survives, so the root the effect published + // is the one it started from. + expect(committedRoot(path)).toEqual(before); + + const created = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.stat("/blocked/deep"); + }); + expect(created.ok).toEqual(false); + }); + }); + + it("refuses to replace a directory before it changes anything", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.mkdir("/held", { recursive: true }); + yield* workspace.filesystem.writeFile("/held/inner.txt", "kept"); + }); + + const run = yield* runDocument(database, 'replacement'); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "refused", phase: "target", reason: "directory" }, + }); + expect(yield* workspaceText(database, "/held/inner.txt")).toEqual("kept"); + }); + }); + + it("denies a temporary directory instead of handing out a host one", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const failure = yield* raised( + runDocument( + database, + ["", 'x', ""].join("\n"), + ), + ); + + expect(failure).toBeInstanceOf(Error); + expect(parseFilesFatal(fatalOf(failure))).toEqual({ + type: FILES_FATAL, + kind: "operation-denied", + operation: "temporary-directory", + }); + }); + }); + + it("keeps an unrelated in-memory journal out of the run's storage", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const before = (yield* database.journal.readAll()).length; + yield* scoped(function* () { + yield* useHostSpy(); + yield* collect( + yield* execute({ ...inlineSource("# plain"), stream: new InMemoryStream() }), + ); + }); + expect((yield* database.journal.readAll()).length).toEqual(before); + }); + }); +}); diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index a1f35802..de99659f 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -138,6 +138,12 @@ const DENO_ONLY_TOOLING: RuntimeExclusion[] = [ "kills real Deno child processes with SIGKILL and reads the recovered node:sqlite WorkflowRun database they leave behind; the children run under the Deno executable and node:sqlite remains behind --experimental-sqlite on Node 22", issue: "https://github.com/taras/executable.md/issues/365", }, + { + path: "packages/workflow/tests/workspace-files.test.ts", + reason: + "drives and against a real node:sqlite WorkflowRun database through the Deno DOFS Workspace adapter; node:sqlite remains behind --experimental-sqlite on Node 22", + issue: "https://github.com/taras/executable.md/issues/366", + }, ]; /** diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index e23c2f42..eb39889e 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -1566,6 +1566,8 @@ run but are absent from the diagnostic trace. | `packages/cli/src/{deno,node,bun,compiled}-service.ts` | runtime-named service adapters for token, environment and stdio behavior | | `packages/cli/src/{deno,node,bun,compiled}.ts` | Entrypoints — each installs matching `API.Env` and `API.Service` adapters, then calls `runXmd` | | `packages/workflow/src/service-denial.ts` | `useWorkflowServiceDenial()`, the tested non-delegating provider for future workflow start and resume scopes (#366) | +| `packages/workflow/src/deno/workspace/files.ts` | the transaction-bound `API.Files` provider — one durable Workspace effect per document read, write and search | +| `packages/workflow/src/deno/workspace/host.ts` | `withWorkflowWorkspace()` — the run's effect coordinator, logical cwd `/`, and Files provider installed together inside one execution | | `packages/cli/src/file-stream.ts` | `FileStream` — JSONL-backed `DurableStream` implementation | Dependencies: `@effectionx/scope-eval`, `@effectionx/timebox`, @@ -7725,6 +7727,35 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX43 | One read across phases | Two valid recorded selections behind one accessor — Alpha then Beta — resume as Alpha: Alpha's section executes, Beta's never does, the source is read once, and the appended Close describes the Alpha execution | | TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | | TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | +### Tier RR — Retained workflow-run installation + +Defined in [Workflow runs](./workflow-spec.md) §3.1. + +| # | Test | Verify | +|---|------|--------| +| RR1 | Exact record | The retained run is recorded verbatim; no identifier is allocated and Git is never consulted | +| RR2/RR3 | Restoration | A truncated and a completed journal each restore the retained run without recording it again | +| RR4 | A different run id | Refused as `StaleInputError` naming the field, never either value, and the root document does not expand | +| RR5 | A different base or pinned commit | Refused on the same terms, naming the fields that differ | +| RR6 | A malformed record | Refused rather than coerced, without quoting what the journal held | +| RR7 | An unusable installation | A missing run id, base or pinned commit is refused before any document executes | + +### Tier WF — The workflow document filesystem + +Defined in [Workflow runs](./workflow-spec.md) §10. + +| # | Test | Verify | +|---|------|--------| +| WF1 | Public routing | `` and `` reach the run's logical Workspace, and a host `API.Files` spy installed outside the run observes nothing for any read, write, refusal or search | +| WF2 | Atomic write | File bytes, the current-root pointer and one filtered Yield are all visible to a second connection together, and the newest journal row names the published root | +| WF3 | Recorded read | A read is its own durable effect whose recorded value is the content it read | +| WF4 | Historical read | A read restores its recorded content where the current frontier holds something else | +| WF5 | Create/delete/create | A history built through the provider-private mutation seam replays recorded results in order, performing no mutation and consulting no current file | +| WF6 | Lexical refusal | An empty, absolute or escaping path is refused without an effect being recorded and without a host call | +| WF7 | Rolled-back refusal | A documented refusal publishes a `rolled-back` outcome, leaves the current root unchanged and creates none of the parents the attempt would have needed | +| WF8 | Target refusal | Replacing a directory is refused as `unchanged` before anything is attempted | +| WF9 | Foreign journal | A file effect published into a stream that is not the run's is refused before mutation, leaving the journal and the Workspace as they were | +| WF10 | Denied temporary directory | `` receives the operation-denied infrastructure failure and no host directory | ### Tier SL — Own-scope context updates diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 39b2e61c..14012d0a 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -2,8 +2,9 @@ * **Status:** Current * **Scope:** `@executablemd/workflow` — associating a document execution with a - workflow run whose starting repository state is pinned once, and retaining - that run so another process can find it. + workflow run whose starting repository state is pinned once, retaining that + run so another process can find it, and giving that run's document its own + transactional filesystem. --- @@ -73,6 +74,34 @@ middleware installation share one child scope. A later document execution — including one continuing the same workflow run — gets a new child scope and a new installation. +### 3.1 Installing a run that already exists + +A host that keeps runs in retained storage (§9) has decided what the run is +before anything executes: `create()` answered with the run id, and the +definition was established from a commit the host pinned. There is nothing left +for the execution to allocate or resolve, and a run id an execution invented +could not agree with the record storage already holds. + +```ts +yield* useRetainedWorkflow({ runId, base, pinnedCommit }); +``` + +This installs the same middleware in the same place and records through the same +`workflow_run` durable operation. What differs is both ends of it. The live path +writes exactly the value it was given: no identifier is generated and +`Git.revParse()` is never called. And every journal state holds the record to +that value in full — run id, base and pinned commit — rather than to the base +alone. + +A journal recording a different run is refused as `StaleInputError`, naming the +fields that differ and never their values: a run id may be caller-selected and a +base is any revision expression, so both are external text on the same terms as +retained props. A value installed without a run id, a base or a pinned commit +identifies no run and is refused before any document executes. + +`getWorkflowRun()`, the three journal states and the lifetime rules above are +otherwise identical under either installation. + ## 4. The three journal states The journal decides which middleware does the work. @@ -535,9 +564,9 @@ retained root from that state without the process that wrote it. The provider-neutral coordinator receives the failure-activation continuation needed for this boundary. The default live coordinator ignores it and preserves ordinary success/failure publication. Replay bypasses coordination, and only an -explicit Workspace operation selects the Workspace coordinator. The Deno proof -operation is adapter-private: public filesystem effects and workflow -start/resume do not reach it. +explicit Workspace operation selects the Workspace coordinator. The +transaction-bound Files provider of §10 selects it for every document +filesystem effect; workflow start and resume do not reach it. The private restoration materializer loads a fully validated retained root and rebuilds directories, files, chunks, modes, mtimes, symbolic links and hardlink @@ -613,10 +642,50 @@ Version 1 reads and writes version 1. Unsupported versions are refused without the file being touched; partial version-1 initialization is corruption and is also left unchanged. -## 10. Intentionally excluded +## 10. The document filesystem of a run + +A host attaches one run's Workspace to a document execution with +`withWorkflowWorkspace(database, operation)` from +`@executablemd/workflow/deno`. It installs three things together, inside the +execution rather than at an entrypoint, so they answer ahead of the host adapter +`xmd run` installs: the run's Workspace effect coordinator, the logical working +directory `/`, and the transaction-bound `API.Files` provider. + +Paths are absolute POSIX paths inside the run's own filesystem. An authored path +is resolved by arithmetic on segments and handed to the run's DOFS filesystem; +no host path exists anywhere in it, so containment needs no stable-namespace +qualification. An empty path, an absolute path and a lexical escape are refused +with the vocabulary `API.Files` already has. + +`readTextFile`, `writeTextFile` and `globFiles` are durable effects. +`checkFilePath` is not: it is lexical admission, it performs no effect, and it +appends nothing, so the write repeats the same admission from the same authored +path. Each effect's description is derived from the current expansion, the +operation and the resolved logical path, so one authored element is the same +effect across replays while an element edited to name another file is a +different one. + +One effect is one effect transaction. The mutation, the resulting immutable root +and the filtered journal result commit together. An ordinary filesystem refusal +rolls its mutation savepoint back before its result is published, so the +retained outcome describes a Workspace that is exactly what it was and the write +reports its target as rolled back. What crosses the boundary is a `FilesReason` +selected from the shared vocabulary — no DOFS message, errno payload, SQLite +text or resolved path. Everything that is not a documented refusal stays an +infrastructure failure and fails the run. + +Replay restores the recorded outcome: it performs no mutation, opens no +transaction and consults no current state, which is what lets a read answer with +the bytes it read at the time and a create/delete/create history replay in +order. + +`temporaryDirectory` is refused with the existing operation-denied failure. A +run has no host directory to hand out, and reaching the caller's would be the +uncontained filesystem this boundary exists to prevent. + +## 11. Intentionally excluded Public `xmd workflow` lifecycle commands; lifecycle transition policy, executor -leases and stale-owner recovery; public Workspace mutation and filesystem -effects; public root selection, history checkpoints and forks; `` integration; -workflow-owned worktrees; and deterministic Git and GitHub effects. Retained -roots and private restoration do not expose any of those behaviors. +leases and stale-owner recovery; public root selection, history checkpoints and +forks; workflow-owned worktrees; and deterministic Git and GitHub effects. +Retained roots and private restoration do not expose any of those behaviors. diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index a089e5ba..c4b3da41 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -844,9 +844,9 @@ and cannot return until the exact Result has been appended and recorded. Its supported filesystem calls use synchronous pinned DOFS primitives, leaving no asynchronous continuation after mutation teardown. It distinguishes a documented filesystem refusal from infrastructure failure and cancellation, and activates the durable -fail-stop fence for infrastructure failures. This foundation is not a public -filesystem effect: ``, `API.Files`, workflow start/resume and history -commands do not route to it in this slice. +fail-stop fence for infrastructure failures. ``, `` and every other +`API.Files` operation route to it through the transaction-bound Files provider. +Workflow start, resume and the history commands do not reach it in this slice. That boundary holds across processes as well as within one. A host killed between the mutation and the commit publishes nothing, and a process that @@ -867,7 +867,7 @@ delegated without changing the document language. | workflow-run and expansion identity | built by #289 / PR #341 | | retained run record and filtered journal | built by #291 | | caller-owned storage transaction | built by #291; Workspace mutations join it in #365 | -| provider-backed retained Workspace | defined here; unbuilt (#218) | +| provider-backed retained Workspace | document filesystem built by #366; repository, process and attachment capabilities unbuilt (#218) | | Repository, Worktree and transactional Git components | defined here; unbuilt | | lifecycle start/resume/status/history/fork/delete | defined here; unbuilt | | read-only Agent materialization | defined here; proof required | From baf0d7c794477a7f1066e14ead046697d0d147c7 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 11:13:03 -0400 Subject: [PATCH 02/10] =?UTF-8?q?=F0=9F=93=81=20Hold=20the=20workflow=20fi?= =?UTF-8?q?lesystem=20and=20its=20history=20to=20what=20they=20claim?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A search answers with regular files, on the contract the host provider already answers on: a symbolic link is neither a result nor a way into the tree it names. A recorded outcome is parsed rather than believed. A record must carry its variant's members and no others, and a refusal's phase and reason must both be words the operation's vocabulary holds; anything else is the one fixed cause-free provider invariant, carrying nothing the record happened to hold. A retained installation requires the retained history as a whole to hold exactly one successful workflow_run record that reads as a run and agrees with the retained one. Reading a record can only refuse a record the journal holds, so a completed journal recording a terminal result and no run at all had nothing to refuse. The document filesystem is installed through withWorkflowWorkspace() and nowhere else. The Files provider alone would resolve a document's paths against the surrounding host's working directory, and retain it. DOFS stops nowhere between creating a write's parents and writing the file, so the savepoint's rollback is observed through an adapter-private interposition on the filesystem a Workspace transaction hands its body. --- packages/workflow/deno.ts | 4 +- packages/workflow/src/deno/workspace/files.ts | 181 ++++++---- packages/workflow/src/deno/workspace/host.ts | 8 +- .../workflow/src/deno/workspace/private.ts | 41 ++- packages/workflow/src/journal.ts | 27 ++ packages/workflow/src/run.ts | 75 +++- packages/workflow/tests/retained-run.test.ts | 79 ++++- .../workflow/tests/workspace-files.test.ts | 330 +++++++++++++++++- specs/executable-mdx-spec.md | 5 + specs/workflow-spec.md | 43 ++- 10 files changed, 703 insertions(+), 90 deletions(-) diff --git a/packages/workflow/deno.ts b/packages/workflow/deno.ts index 9ab913e7..b09361ae 100644 --- a/packages/workflow/deno.ts +++ b/packages/workflow/deno.ts @@ -28,6 +28,6 @@ export { useWorkflowRunStorage } from "./src/deno/provider.ts"; export type { WorkflowRunStorageOptions } from "./src/deno/provider.ts"; export { hashRunId, workflowRunPath } from "./src/deno/path.ts"; export { APPLICATION_ID, SCHEMA_VERSION } from "./src/deno/schema.ts"; -export { useLogicalWorkspaceCwd, withWorkflowWorkspace } from "./src/deno/workspace/host.ts"; -export { useWorkflowFiles, WORKSPACE_FILE } from "./src/deno/workspace/files.ts"; +export { withWorkflowWorkspace } from "./src/deno/workspace/host.ts"; +export { WORKSPACE_FILE } from "./src/deno/workspace/files.ts"; export { WORKSPACE_ROOT } from "./src/deno/workspace/logical-path.ts"; diff --git a/packages/workflow/src/deno/workspace/files.ts b/packages/workflow/src/deno/workspace/files.ts index e04d43f3..199d8577 100644 --- a/packages/workflow/src/deno/workspace/files.ts +++ b/packages/workflow/src/deno/workspace/files.ts @@ -149,49 +149,100 @@ function lexicalReason(error: Error): FilesReason { * serialized error, so nothing a filesystem said is retained and a restored * refusal is rebuilt from the same vocabulary a live one is. */ -type FileEffectOutcome = +type FileEffectOutcome = | { readonly kind: "content"; readonly content: string } | { readonly kind: "written" } | { readonly kind: "paths"; readonly paths: string[] } - | { readonly kind: "refused"; readonly phase: string; readonly reason: string }; + | { readonly kind: "refused"; readonly phase: Phase; readonly reason: FilesReason }; -function refused(phase: FilesPhase | FileWritePhase, reason: FilesReason): FileEffectOutcome { +function refused( + phase: Phase, + reason: FilesReason, +): FileEffectOutcome { return { kind: "refused", phase, reason }; } +/** + * The whole of what each outcome carries. + * + * A `Map` rather than an object literal, because the discriminant is read from + * the journal and a lookup on an object answers for keys `Object.prototype` + * happens to hold. A record carrying anything beyond its variant's members — + * `written` with content, `content` with a reason — describes two outcomes at + * once and is therefore no outcome at all. + */ +const OUTCOME_MEMBERS: ReadonlyMap = new Map([ + ["content", ["kind", "content"]], + ["written", ["kind"]], + ["paths", ["kind", "paths"]], + ["refused", ["kind", "phase", "reason"]], +]); + +function carriesExactly(record: Record, kind: string): boolean { + const members = OUTCOME_MEMBERS.get(kind); + if (members === undefined) { + return false; + } + return ( + Object.keys(record).length === members.length && + members.every((member) => Object.hasOwn(record, member)) + ); +} + +function readPaths(value: unknown): string[] | undefined { + if (!Array.isArray(value)) { + return undefined; + } + const paths: string[] = []; + for (const entry of value) { + if (typeof entry !== "string") { + return undefined; + } + paths.push(entry); + } + return paths; +} + /** * The outcome a journal record describes, or `undefined` when it describes none. * - * The journal is parsed, never trusted. A record this cannot read has no - * printable reading, so the caller turns it into one fixed provider invariant - * rather than inventing a filesystem condition that was never reported. + * The journal is parsed, never trusted, and parsing here is total: a record must + * carry its variant's members and no others, each of the declared type, and a + * refusal's phase and reason must both be words this operation's vocabulary + * holds. A record this cannot read has no printable reading, so the caller turns + * it into one fixed provider invariant rather than inventing a filesystem + * condition that was never reported. */ -function parseOutcome(value: unknown): FileEffectOutcome | undefined { +function parseOutcome( + value: unknown, + parsePhase: (value: unknown) => Phase | undefined, +): FileEffectOutcome | undefined { if (typeof value !== "object" || value === null || Array.isArray(value)) { return undefined; } - const record = Object.fromEntries(Object.entries(value)); - if (record.kind === "content" && typeof record.content === "string") { - return { kind: "content", content: record.content }; + const record: Record = Object.fromEntries(Object.entries(value)); + const kind = record.kind; + if (typeof kind !== "string" || !carriesExactly(record, kind)) { + return undefined; + } + if (kind === "content") { + return typeof record.content === "string" + ? { kind: "content", content: record.content } + : undefined; } - if (record.kind === "written") { + if (kind === "written") { return { kind: "written" }; } - if ( - record.kind === "paths" && - Array.isArray(record.paths) && - record.paths.every((entry) => typeof entry === "string") - ) { - return { kind: "paths", paths: [...record.paths] }; + if (kind === "paths") { + const paths = readPaths(record.paths); + return paths === undefined ? undefined : { kind: "paths", paths }; } - if ( - record.kind === "refused" && - typeof record.phase === "string" && - typeof record.reason === "string" - ) { - return { kind: "refused", phase: record.phase, reason: record.reason }; + const phase = parsePhase(record.phase); + const reason = parseFilesReason(record.reason); + if (phase === undefined || reason === undefined) { + return undefined; } - return undefined; + return { kind: "refused", phase, reason }; } /** @@ -213,10 +264,10 @@ function* describeFileEffect( return { type: WORKSPACE_FILE, name: `${operation}:${expansion.id}:${target}`, ...detail }; } -function* fileEffect( +function* fileEffect( database: WorkflowRunDatabase, description: EffectDescription, - perform: (filesystem: DenoWorkspaceFilesystem) => Operation, + perform: (filesystem: DenoWorkspaceFilesystem) => Operation>, ): Workflow { return yield createWorkspaceEffect(database, description, (filesystem) => perform(filesystem)); } @@ -230,12 +281,13 @@ function* fileEffect( * "replay restores the recorded result" a property of the code rather than a * claim about it. */ -function* performed( +function* performed( database: WorkflowRunDatabase, description: EffectDescription, - perform: (filesystem: DenoWorkspaceFilesystem) => Operation, -): Operation { - const outcome = parseOutcome(yield* fileEffect(database, description, perform)); + parsePhase: (value: unknown) => Phase | undefined, + perform: (filesystem: DenoWorkspaceFilesystem) => Operation>, +): Operation> { + const outcome = parseOutcome(yield* fileEffect(database, description, perform), parsePhase); if (outcome === undefined) { throw new FilesInvariantError("protocol"); } @@ -256,7 +308,7 @@ function* statPath( function* readOutcome( filesystem: DenoWorkspaceFilesystem, path: string, -): Operation { +): Operation> { const info = yield* statPath(filesystem, path); if (!info.ok) { return refused("resolution", refusalReason(info.error)); @@ -281,7 +333,7 @@ function* readOutcome( function* classifyWriteTarget( filesystem: DenoWorkspaceFilesystem, path: string, -): Operation { +): Operation | undefined> { const info = yield* statPath(filesystem, path); if (info.ok) { return info.value.kind === "file" ? undefined : refused("target", "directory"); @@ -314,7 +366,7 @@ function* writeOutcome( filesystem: DenoWorkspaceFilesystem, path: string, content: string, -): Operation { +): Operation> { const existing = yield* classifyWriteTarget(filesystem, path); if (existing !== undefined) { return existing; @@ -360,15 +412,20 @@ interface Traversal { } /** - * Collect matching files beneath one logical directory. + * Collect matching regular files beneath one logical directory. + * + * A search answers with regular files, which is what `API.Files` says a search + * answers with wherever it runs. A symbolic link is neither reported nor + * descended through, so a link is not a result and no target is reached twice or + * reached at all through a name outside the walk — traversal stays inside the + * directory it started in and cannot cycle. `readdir` classifies a link by what + * the entry is rather than by what it points at, so a link to a directory is + * refused on the same terms as a link to a file. * - * Exclusion is decided per candidate: a file or symlink whose own relative path - * an exclusion matches is not reported. A directory is never a candidate, so its + * Exclusion is decided per candidate: a file whose own relative path an + * exclusion matches is not reported. A directory is never a candidate, so its * own path is not tested against exclusions at all — the only question it raises * is whether walking it can still produce something. - * - * A symbolic link is reported by its own path and never followed, so traversal - * stays inside the directory it started in and cannot cycle. */ function* descend( filesystem: DenoWorkspaceFilesystem, @@ -386,6 +443,9 @@ function* descend( continue; } + if (entry.kind !== "file") { + continue; + } if (walk.exclude.some((expression) => expression.test(path))) { continue; } @@ -425,7 +485,7 @@ function* globOutcome( directory: string, include: string[], exclude: string[], -): Operation { +): Operation> { const info = yield* statPath(filesystem, directory); if (!info.ok) { return refused("target", refusalReason(info.error)); @@ -447,34 +507,6 @@ function* globOutcome( return { kind: "paths", paths: [...new Set(walk.value.matched)].sort(byCodePoint) }; } -/** - * Rebuild a failure from what the record described. - * - * The vocabulary is parsed rather than believed: a retained phase or reason - * that is not one becomes a provider invariant rather than a sentence about a - * filesystem condition nobody reported. - */ -function nonWriteFailure( - operation: "read" | "glob", - outcome: { readonly phase: string; readonly reason: string }, -): Result { - const phase = parseFilesPhase(outcome.phase); - const reason = parseFilesReason(outcome.reason); - if (phase === undefined || reason === undefined) { - throw new FilesInvariantError("protocol"); - } - return Err(filesFailure({ operation, phase, reason })); -} - -function writeFailure(outcome: { readonly phase: string; readonly reason: string }): Result { - const phase = parseFileWritePhase(outcome.phase); - const reason = parseFilesReason(outcome.reason); - if (phase === undefined || reason === undefined) { - throw new FilesInvariantError("protocol"); - } - return Err(fileWriteFailure({ phase, reason })); -} - /** The document filesystem of one workflow run. */ export interface WorkflowFilesHandler { checkFilePath(input: FilePathInput): Operation>; @@ -516,10 +548,13 @@ export function workflowFilesHandler(database: WorkflowRunDatabase): WorkflowFil const outcome = yield* performed( database, yield* describeFileEffect("read", path, { path: input.path, cwd: input.cwd }), + parseFilesPhase, (filesystem) => readOutcome(filesystem, path), ); if (outcome.kind === "refused") { - return nonWriteFailure("read", outcome); + return Err( + filesFailure({ operation: "read", phase: outcome.phase, reason: outcome.reason }), + ); } if (outcome.kind !== "content") { throw new FilesInvariantError("protocol"); @@ -536,10 +571,11 @@ export function workflowFilesHandler(database: WorkflowRunDatabase): WorkflowFil const outcome = yield* performed( database, yield* describeFileEffect("write", path, { path: input.path, cwd: input.cwd }), + parseFileWritePhase, (filesystem) => writeOutcome(filesystem, path, input.content), ); if (outcome.kind === "refused") { - return writeFailure(outcome); + return Err(fileWriteFailure({ phase: outcome.phase, reason: outcome.reason })); } if (outcome.kind !== "written") { throw new FilesInvariantError("protocol"); @@ -554,10 +590,13 @@ export function workflowFilesHandler(database: WorkflowRunDatabase): WorkflowFil const outcome = yield* performed( database, yield* describeFileEffect("glob", directory, { include, exclude }), + parseFilesPhase, (filesystem) => globOutcome(filesystem, directory, include, exclude), ); if (outcome.kind === "refused") { - return nonWriteFailure("glob", outcome); + return Err( + filesFailure({ operation: "glob", phase: outcome.phase, reason: outcome.reason }), + ); } if (outcome.kind !== "paths") { throw new FilesInvariantError("protocol"); diff --git a/packages/workflow/src/deno/workspace/host.ts b/packages/workflow/src/deno/workspace/host.ts index e5df64c2..4857ebf0 100644 --- a/packages/workflow/src/deno/workspace/host.ts +++ b/packages/workflow/src/deno/workspace/host.ts @@ -18,6 +18,12 @@ * there is nothing to give a filesystem to — and attaching one anyway would * open a transaction and capture a root for a run that is not going to perform * an effect. + * + * `withWorkflowWorkspace()` is therefore the whole of what a host may install. + * The three pieces are not published separately: the Files provider alone would + * resolve a document's paths against whatever working directory the host adapter + * answers with, and a host path resolved that way is retained in the durable + * effects a run replays from. */ import { scoped, type Operation } from "effection"; @@ -35,7 +41,7 @@ import { WORKSPACE_ROOT } from "./logical-path.ts"; * filesystem, and nothing it can write reaches the directory the caller * happened to invoke `xmd` from. */ -export function useLogicalWorkspaceCwd(): Operation { +function useLogicalWorkspaceCwd(): Operation { return API.Env.around( { // deno-lint-ignore require-yield diff --git a/packages/workflow/src/deno/workspace/private.ts b/packages/workflow/src/deno/workspace/private.ts index 713087ef..501f1d86 100644 --- a/packages/workflow/src/deno/workspace/private.ts +++ b/packages/workflow/src/deno/workspace/private.ts @@ -42,6 +42,43 @@ function unavailable(): never { ); } +/** + * The filesystem a Workspace transaction hands its body, with anything installed + * around it. + * + * Nothing this package ships installs anything here, and no entrypoint exports + * it. It exists because a mutation that is discarded part-way through cannot + * otherwise be observed: a write creates the parent directories it needs and + * then writes the file, and DOFS has no condition that stops between the two — + * a parent chain that can be created is a chain the file can then be written + * into. Placing a failure there is the only way to watch the savepoint take the + * created parents back. + */ +interface WorkspaceFilesystemApi { + interpose(filesystem: DenoWorkspaceFilesystem): Operation; +} + +const WorkspaceFilesystem: Api = createApi( + "executablemd.workflow.deno.workspace.private.filesystem", + { + // deno-lint-ignore require-yield + *interpose(filesystem: DenoWorkspaceFilesystem): Operation { + return filesystem; + }, + }, +); + +/** Wrap the filesystem every Workspace transaction opened below this hands out. */ +export function interposeWorkspaceFilesystem( + wrap: (filesystem: DenoWorkspaceFilesystem) => DenoWorkspaceFilesystem, +): Operation { + return WorkspaceFilesystem.around({ + *interpose([filesystem], next) { + return wrap(yield* next(filesystem)); + }, + }); +} + const PrivateWorkspace: Api = createApi( "executablemd.workflow.deno.workspace.private", { @@ -85,7 +122,9 @@ export function usePrivateWorkspace(connections: WorkflowRunConnections): Operat connections.authorizeTransaction(database, transaction); }; const workspace: PrivateWorkspaceTransaction = { - filesystem: createDenoWorkspaceFilesystem(connection, authorize), + filesystem: yield* WorkspaceFilesystem.operations.interpose( + createDenoWorkspaceFilesystem(connection, authorize), + ), // deno-lint-ignore require-yield *currentRoot(): Operation { diff --git a/packages/workflow/src/journal.ts b/packages/workflow/src/journal.ts index ac901ce2..25edadaf 100644 --- a/packages/workflow/src/journal.ts +++ b/packages/workflow/src/journal.ts @@ -76,6 +76,33 @@ export function retainedRunMismatch( ); } +/** + * A completed journal offers a terminal result without the one record that says + * whose result it is. + * + * Reading a record can only refuse a record the journal holds. A completed + * journal holding no readable `workflow_run` for this run — none at all, one + * that failed, or more than one — leaves nothing to refuse, and the recorded + * terminal result would then be handed back on the strength of history that + * never identified this run. The tally is this module's own count rather than + * anything the journal said, so naming it carries nothing across. + */ +export function missingRunEvidence( + description: EffectDescription, + records: number, +): StaleInputError { + const problem = + records === 0 + ? "records no successful workflow run for it to be the result of" + : `records ${records} successful workflow runs where exactly one identifies a run`; + return new StaleInputError( + `The journal holds a completed result but ${problem}. A retained run is replayed only ` + + "from history that identifies it. Resume the run this journal belongs to, or re-run " + + "the document from the start.", + { coroutineId: "root", description }, + ); +} + /** The recorded run started from a different base than this run supplied. */ export function baseMismatch( description: EffectDescription, diff --git a/packages/workflow/src/run.ts b/packages/workflow/src/run.ts index c7a7244e..59459675 100644 --- a/packages/workflow/src/run.ts +++ b/packages/workflow/src/run.ts @@ -26,6 +26,13 @@ * allocating an id and resolving a base, and every state requires the journal to * agree with that value in full. * + * A completed journal is held to one thing more. Checking an event can only + * refuse an event the journal holds, so a journal that records a terminal + * result and no run at all offers nothing to refuse. The guard's admission + * phase runs once over the retained history, and a retained installation + * requires exactly one successful record there that reads as a workflow run and + * agrees with the retained one, before the stored result may answer for it. + * * All of it is operation-scoped. The value is installed in the scope that owns * the document execution, so every descendant of the expansion reads it, the * output emitted after the durable run still sees it, and ordinary teardown @@ -36,13 +43,20 @@ import { createContext } from "effection"; import type { Context, Operation } from "effection"; import { createDurableOperation } from "@executablemd/durable-streams"; import { ReplayGuard } from "@executablemd/durable-streams"; -import type { EffectDescription, Json, Workflow, Yield } from "@executablemd/durable-streams"; +import type { + EffectDescription, + Json, + RetainedHistory, + Workflow, + Yield, +} from "@executablemd/durable-streams"; import { Execution } from "@executablemd/core"; import { revParse } from "./git.ts"; import { baseMismatch, describeWorkflowRun, malformedRecord, + missingRunEvidence, readWorkflowRun, retainedRunMismatch, WORKFLOW_RUN, @@ -92,6 +106,8 @@ interface RunEstablishment { allocate(): Operation; /** The recorded run, or a refusal naming what it disagrees about. */ hold(description: EffectDescription, recorded: WorkflowRun): WorkflowRun; + /** Whether the retained history as a whole may answer for this run. */ + admit(history: RetainedHistory): void; } /** Append the run to the journal, and answer with what the journal holds. */ @@ -128,11 +144,17 @@ function allocating(base: string): RunEstablishment { } return recorded; }, + /** + * A base is all this installation knows before it runs, and a journal that + * records none is a journal this run has not started writing yet. Requiring + * a record here would refuse the ordinary live start. + */ + admit(_history: RetainedHistory): void {}, }; } function retaining(run: WorkflowRun): RunEstablishment { - return { + const establishment: RunEstablishment = { base: run.base, // deno-lint-ignore require-yield *allocate(): Operation { @@ -147,7 +169,49 @@ function retaining(run: WorkflowRun): RunEstablishment { } return recorded; }, + /** + * A completed journal answers with its recorded root result without ever + * reaching this run's middleware, so what makes that result this run's has + * to be required of the history rather than of an event. Exactly one + * successful record, readable as a workflow run and agreeing with the + * retained one in full, is what a run that got as far as closing left + * behind; anything else is another run's journal or a damaged one. + */ + admit(history: RetainedHistory): void { + if (!history.terminal) { + return; + } + const description = describeWorkflowRun(run.base); + const records = identifying(history, description, establishment); + if (records !== 1) { + throw missingRunEvidence(description, records); + } + }, }; + return establishment; +} + +/** + * How many retained records identify this run. + * + * Each candidate is read and held here rather than counted on the strength of + * the check phase having let it through, so admission proves for itself that + * what it counted describes a workflow run and describes this one. + */ +function identifying( + history: RetainedHistory, + description: EffectDescription, + establishment: RunEstablishment, +): number { + let records = 0; + for (const event of history.yields) { + if (event.description.type !== WORKFLOW_RUN || event.result.status !== "ok") { + continue; + } + held(description, event.result.value, establishment); + records += 1; + } + return records; } function same(left: WorkflowRun, right: WorkflowRun): boolean { @@ -201,6 +265,13 @@ function* install(establishment: RunEstablishment): Operation { yield* restore(event, establishment); return yield* next(event); }, + // Runs after every check and before a recorded terminal result may be + // reused, which is the only place a journal that records nothing at all can + // be refused. + *admit([history], next) { + establishment.admit(history); + return yield* next(history); + }, }); yield* Execution.around({ diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts index b0a85a52..6f62b593 100644 --- a/packages/workflow/tests/retained-run.test.ts +++ b/packages/workflow/tests/retained-run.test.ts @@ -18,7 +18,7 @@ import { expect } from "@executablemd/test-support/expect"; import { scoped } from "effection"; import type { Operation } from "effection"; import { InMemoryStream, StaleInputError } from "@executablemd/durable-streams"; -import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import type { DurableEvent, Json, Yield } from "@executablemd/durable-streams"; import { collect, execute, inlineSource, registerComponents } from "@executablemd/core"; import { Git } from "../src/git.ts"; import { getWorkflowRun, useRetainedWorkflow } from "../src/run.ts"; @@ -102,6 +102,29 @@ function partial(stream: InMemoryStream): InMemoryStream { ); } +/** + * The completed journal, with what it records about the run replaced. + * + * A completed journal is where the run record matters most and is read least: it + * answers with the recorded root result without expanding anything, so whatever + * these cases leave behind is the whole of the evidence that the result is this + * run's. + */ +function completedWith( + stream: InMemoryStream, + change: (event: Yield) => DurableEvent[], +): InMemoryStream { + return new InMemoryStream( + stream + .snapshot() + .flatMap((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? change(event) + : [event], + ), + ); +} + describe("Tier RR — retained workflow runs", () => { it("RR1: records exactly the retained run, without allocating or resolving", function* () { const stream = new InMemoryStream(); @@ -186,6 +209,60 @@ describe("Tier RR — retained workflow runs", () => { expect(attempt.thrown).toBeInstanceOf(StaleInputError); }); + // RR8: what a completed journal has to hold. `check` can only object to an + // event a journal contains, so a journal recording a terminal result and no + // readable run for it leaves nothing to object to — and the recorded result + // would answer for history that never identified this run. + it("RR8: refuses a completed journal that does not record exactly one run", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const cases: Array<{ says: string; stream: InMemoryStream }> = [ + { says: "no successful workflow run", stream: completedWith(first, () => []) }, + { + says: "no successful workflow run", + stream: completedWith(first, (event) => [ + { ...event, result: { status: "err", error: { message: "recorded failure" } } }, + ]), + }, + { + says: "2 successful workflow runs", + stream: completedWith(first, (event) => [event, event]), + }, + ]; + + for (const refused of cases) { + const before = refused.stream.snapshot().length; + const attempt = yield* runRetained(RETAINED, refused.stream); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain(refused.says); + // Refused before the recorded root result was handed back, and before + // anything expanded. + expect(attempt.seen).toEqual([]); + expect(refused.stream.snapshot().length).toEqual(before); + } + }); + + it("RR9: refuses a completed journal recording another run, or an unreadable one", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + const completed = () => new InMemoryStream(first.snapshot()); + + const foreign = yield* runRetained({ ...RETAINED, runId: "release-1.5" }, completed()); + expect(foreign.thrown).toBeInstanceOf(StaleInputError); + expect(foreign.thrown instanceof Error ? foreign.thrown.message : "").toContain("runId"); + + const damaged = yield* runRetained( + RETAINED, + completedWith(first, (event) => [ + { ...event, result: { status: "ok", value: { runId: 7 } } }, + ]), + ); + expect(damaged.thrown).toBeInstanceOf(StaleInputError); + expect(damaged.seen).toEqual([]); + }); + it("RR7: refuses to install a run that identifies nothing", function* () { const empty = yield* scoped(function* () { try { diff --git a/packages/workflow/tests/workspace-files.test.ts b/packages/workflow/tests/workspace-files.test.ts index 29847d34..bb4bdd35 100644 --- a/packages/workflow/tests/workspace-files.test.ts +++ b/packages/workflow/tests/workspace-files.test.ts @@ -25,7 +25,12 @@ import type { HostFilesEvent } from "@executablemd/runtime"; import type { WorkflowRunDatabase } from "../mod.ts"; import { withWorkflowWorkspace } from "../src/deno/workspace/host.ts"; import { WORKSPACE_FILE } from "../src/deno/workspace/files.ts"; -import { transactWorkspaceRoots } from "../src/deno/workspace/private.ts"; +import { throwWorkspaceFilesystemFailure } from "../src/deno/workspace/errors.ts"; +import type { DenoWorkspaceFilesystem } from "../src/deno/workspace/filesystem.ts"; +import { + interposeWorkspaceFilesystem, + transactWorkspaceRoots, +} from "../src/deno/workspace/private.ts"; import type { PrivateWorkspaceTransaction } from "../src/deno/workspace/private.ts"; import { committedEventCount, @@ -169,6 +174,128 @@ function* workspaceText(database: WorkflowRunDatabase, path: string): Operation< return read.value; } +/** + * A Workspace filesystem that refuses one write the way DOFS refuses one. + * + * The failure is raised through the adapter's own wrapping, so what reaches the + * provider is indistinguishable from a real `EACCES`: it selects a reason and + * carries nothing else. It exists because no ordinary DOFS condition stops a + * write between creating its parents and writing the file — a parent chain that + * can be created is a chain the file can then be written into — so the state a + * savepoint is there to discard cannot otherwise be produced. + */ +function refusingWrite( + target: string, +): (filesystem: DenoWorkspaceFilesystem) => DenoWorkspaceFilesystem { + return (filesystem) => ({ + ...filesystem, + *writeFile(path, content, mode) { + if (path === target) { + throwWorkspaceFilesystemFailure( + Object.assign(new Error("planted"), { name: "WorkspaceFsError", code: "EACCES" }), + ); + } + yield* filesystem.writeFile(path, content, mode); + }, + }); +} + +/** Whether one journal row is the recorded `operation` on `target`. */ +function namesEffect(record: unknown, operation: string, target: string): boolean { + if (typeof record !== "string") { + return false; + } + const parsed: unknown = JSON.parse(record); + if (typeof parsed !== "object" || parsed === null) { + return false; + } + const description = Reflect.get(parsed, "description"); + const name = + typeof description === "object" && description !== null + ? Reflect.get(description, "name") + : undefined; + return ( + Reflect.get(parsed, "type") === "yield" && + typeof name === "string" && + name.startsWith(`${operation}:`) && + name.endsWith(`:${target}`) + ); +} + +/** + * Replace what one recorded file effect settled to. + * + * Written through SQL rather than through the provider, because the point of + * these cases is a journal holding something the provider would never write. + */ +function plantOutcome(path: string, operation: string, target: string, value: Json): void { + tamper(path, (database) => { + let planted = 0; + for (const row of database.prepare("SELECT sequence, record FROM journal_events").all()) { + if (!namesEffect(row["record"], operation, target)) { + continue; + } + const record = JSON.parse(String(row["record"])); + record.result = { status: "ok", value }; + database + .prepare("UPDATE journal_events SET record = ? WHERE sequence = ?") + .run(`${JSON.stringify(record)}\n`, row["sequence"]); + planted += 1; + } + if (planted !== 1) { + throw new Error(`the journal records ${planted} ${operation} effects on ${target}`); + } + }); +} + +/** + * Take away the root's Close. + * + * A completed journal answers with its recorded root result without replaying + * anything, so a record planted in one of its effects is never read. Removing + * the Close is what makes the effects replay. + */ +function dropRootClose(path: string): void { + tamper(path, (database) => { + let dropped = 0; + for (const row of database.prepare("SELECT sequence, record FROM journal_events").all()) { + const parsed: unknown = JSON.parse(String(row["record"])); + if (typeof parsed !== "object" || parsed === null) { + continue; + } + if ( + Reflect.get(parsed, "type") !== "close" || + Reflect.get(parsed, "coroutineId") !== "root" + ) { + continue; + } + database.prepare("DELETE FROM journal_events WHERE sequence = ?").run(row["sequence"]); + dropped += 1; + } + if (dropped !== 1) { + throw new Error(`the journal records ${dropped} root closes`); + } + }); +} + +/** + * Drop the journal from the recorded effect on `target` onward. + * + * What is left replays up to that point and runs live after it, which is how a + * test observes what a document does *after* a replayed effect rather than only + * what that effect answers. + */ +function truncateFromEffect(path: string, operation: string, target: string): void { + tamper(path, (database) => { + const rows = database.prepare("SELECT sequence, record FROM journal_events").all(); + const found = rows.find((row) => namesEffect(row["record"], operation, target)); + if (found === undefined) { + throw new Error(`the journal records no ${operation} of ${target}`); + } + database.prepare("DELETE FROM journal_events WHERE sequence >= ?").run(found["sequence"]); + }); +} + function* mutateWorkspace( database: WorkflowRunDatabase, body: (workspace: PrivateWorkspaceTransaction) => Operation, @@ -258,15 +385,65 @@ describe("WF workflow document filesystem", () => { }); }); - it("refuses a path that leaves the working directory without touching the host", function* () { + // WF6: the three refusals a document can act on, each decided before any + // effect exists. Nothing is recorded and nothing outside the run is asked, + // which is what "lexical" means here. + it("refuses an empty, absolute or escaping path without an effect or a host call", function* () { + const cases: Array<{ path: string; says: string }> = [ + { path: "", says: "path is empty" }, + { path: "/etc/passwd", says: "an absolute path is not accepted" }, + { path: "../escape.txt", says: "resolves outside the working directory" }, + ]; + + for (const refused of cases) { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + const before = committedRoot(path); + const run = yield* runDocument(database, `no`); + + expect(String(run.output)).toContain(refused.says); + expect(run.host.seen).toEqual([]); + expect(yield* workspaceEvents(database)).toEqual([]); + expect(committedRoot(path)).toEqual(before); + }); + } + }); + + // WF11: the search's document-facing shape, on the same contract the host + // provider answers on (HF3). A link is not a file, so it is neither a result + // nor a way into the tree it names. + it("searches regular files only, reporting no symbolic link and following none", function* () { const root = yield* useStorageRoot(); yield* withStorage(root, function* () { const database = yield* createRun(); - const run = yield* runDocument(database, 'no'); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.mkdir("/docs", { recursive: true }); + yield* workspace.filesystem.writeFile("/docs/a.md", "a"); + yield* workspace.filesystem.mkdir("/hidden", { recursive: true }); + yield* workspace.filesystem.writeFile("/hidden/b.md", "b"); + yield* workspace.filesystem.symlink("/docs/a.md", "/link.md"); + // Named so that walking *through* it would produce a second, matching + // path for a file the walk already reaches by its own name. + yield* workspace.filesystem.symlink("/hidden", "/mirror.md"); + }); + + const run = yield* runDocument( + database, + ['', "", "Found: {found}"].join("\n"), + ); - expect(String(run.output)).toContain("resolves outside the working directory"); expect(run.host.seen).toEqual([]); - expect(yield* workspaceEvents(database)).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "paths", paths: ["docs/a.md", "hidden/b.md"] }, + }); + // The file link is not a result, and the directory link is neither a + // result nor a second route to `b.md`. + expect(String(run.output)).not.toContain("link.md"); + expect(String(run.output)).not.toContain("mirror.md"); }); }); @@ -441,6 +618,149 @@ describe("WF workflow document filesystem", () => { }); }); + // WF12: the savepoint around parent creation and the write together. The + // write fails after two directories exist, so what the assertion below sees + // is the rollback rather than an attempt that never started. + it("discards the parent directories a refused write already created", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/kept.txt", "kept"); + }); + + const run = yield* scoped(function* () { + yield* interposeWorkspaceFilesystem(refusingWrite("/made/deep/x.txt")); + return yield* runDocument( + database, + ['no', "", 'yes'].join( + "\n", + ), + ); + }); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "refused", phase: "transaction", reason: "permission-denied" }, + }); + + // Both directories the attempt created are gone. + for (const created of ["/made", "/made/deep"]) { + const stat = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.stat(created); + }); + expect(stat.ok).toEqual(false); + } + // What the Workspace already held is what it still holds. + expect(yield* workspaceText(database, "/kept.txt")).toEqual("kept"); + // The savepoint took back the mutation rather than the transaction, so + // the next effect still commits. + expect(recorded[1]?.result).toEqual({ status: "ok", value: { kind: "written" } }); + expect(yield* workspaceText(database, "/after.txt")).toEqual("yes"); + }); + }); + + // WF13: durable history is parsed, not believed. A record carrying more than + // its variant carries, or a word the vocabulary does not hold, describes no + // outcome — and nothing it happens to hold is repeated back. + it("refuses a recorded outcome carrying extra or contradictory members", function* () { + const source = [ + 'first', + "", + '', + "", + '', + "", + "Seen: {seen}", + ].join("\n"); + + const cases: Array<{ operation: string; target: string; value: Json }> = [ + // content, carrying a refusal's members as well as its own + { + operation: "read", + target: "/seed.txt", + value: { kind: "content", content: "first", reason: "missing" }, + }, + // written, which carries nothing but its kind + { operation: "write", target: "/seed.txt", value: { kind: "written", content: "first" } }, + // paths, carrying content + { + operation: "glob", + target: "/", + value: { kind: "paths", paths: ["seed.txt"], content: "x" }, + }, + // refused, missing the reason it is refused for + { operation: "read", target: "/seed.txt", value: { kind: "refused", phase: "target" } }, + // refused, in a vocabulary this provider does not speak + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "target", reason: "unspeakable" }, + }, + // refused, with planted text riding along beside the vocabulary + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "target", reason: "missing", detail: "PLANTED-SECRET" }, + }, + ]; + + for (const planted of cases) { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + yield* runDocument(database, source); + + dropRootClose(path); + plantOutcome(path, planted.operation, planted.target, planted.value); + const before = (yield* workspaceEvents(database)).length; + + const failure = yield* raised(replayDocument(database, source)); + + expect(failure).toBeInstanceOf(Error); + expect(String(failure)).not.toContain("PLANTED-SECRET"); + // The failed run performed no file effect of its own — the history it + // could not read is the whole of what it has. + expect((yield* workspaceEvents(database)).length).toEqual(before); + }); + } + }); + + it("performs no later file effect once the history it replays is malformed", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + const source = [ + 'one', + "", + 'two', + ].join("\n"); + yield* runDocument(database, source); + + // The journal now replays the first write and runs everything from the + // second one live, and the first write's record describes no outcome. + truncateFromEffect(path, "write", "/second.txt"); + plantOutcome(path, "write", "/first.txt", { kind: "written", content: "one" }); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.remove("/second.txt"); + }); + const before = (yield* workspaceEvents(database)).length; + + const failure = yield* raised(replayDocument(database, source)); + + expect(failure).toBeInstanceOf(Error); + expect((yield* workspaceEvents(database)).length).toEqual(before); + const second = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.stat("/second.txt"); + }); + expect(second.ok).toEqual(false); + }); + }); + it("refuses to replace a directory before it changes anything", function* () { const root = yield* useStorageRoot(); yield* withStorage(root, function* () { diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index eb39889e..68f7b1f3 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -7739,6 +7739,8 @@ Defined in [Workflow runs](./workflow-spec.md) §3.1. | RR5 | A different base or pinned commit | Refused on the same terms, naming the fields that differ | | RR6 | A malformed record | Refused rather than coerced, without quoting what the journal held | | RR7 | An unusable installation | A missing run id, base or pinned commit is refused before any document executes | +| RR8 | A completed journal recording no run, or more than one | A terminal result whose journal holds no successful `workflow_run` record, holds one that failed, or holds two is refused at admission — before the recorded root result is handed back, and without appending anything | +| RR9 | A completed journal recording another run | A completed journal whose record names a different run, or holds a value that does not read as a run at all, is refused on the same terms as a truncated one | ### Tier WF — The workflow document filesystem @@ -7756,6 +7758,9 @@ Defined in [Workflow runs](./workflow-spec.md) §10. | WF8 | Target refusal | Replacing a directory is refused as `unchanged` before anything is attempted | | WF9 | Foreign journal | A file effect published into a stream that is not the run's is refused before mutation, leaving the journal and the Workspace as they were | | WF10 | Denied temporary directory | `` receives the operation-denied infrastructure failure and no host directory | +| WF11 | The search's shape | Sorted, deduplicated, POSIX-relative regular files, on HF3's contract: neither a file symlink nor a directory symlink is a result, and a file reachable through a directory symlink is reported once, by its own path | +| WF12 | Discarded partial mutation | A write that refuses after creating two parent directories leaves neither behind, leaves what the Workspace already held untouched, records the sanitized refusal, and does not stop the next effect from committing | +| WF13 | Unreadable history | A recorded outcome carrying a member its variant does not have, or a phase or reason the vocabulary does not hold, is refused as the fixed cause-free provider invariant — nothing the record held is repeated back, and no later file effect is performed | ### Tier SL — Own-scope context updates diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 14012d0a..9a1bd540 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -99,6 +99,18 @@ base is any revision expression, so both are external text on the same terms as retained props. A value installed without a run id, a base or a pinned commit identifies no run and is refused before any document executes. +A completed journal is held to one thing more. Reading a record can only refuse +a record the journal holds, and a completed journal answers with its recorded +root result without expanding anything — so a journal recording a terminal +result and no run at all offers nothing to refuse. A retained installation +therefore requires, of the retained history as a whole, exactly one successful +`workflow_run` record that reads as a workflow run and agrees with the retained +one in full. None, one that failed, more than one, one that cannot be read, and +one naming another run are each refused before the recorded result is returned +and before anything expands. A journal with no history at all is the ordinary +live start and is unaffected, as is `useWorkflow({ base })`, which has no +retained value to hold a record to. + `getWorkflowRun()`, the three journal states and the lifetime rules above are otherwise identical under either installation. @@ -651,6 +663,12 @@ execution rather than at an entrypoint, so they answer ahead of the host adapter `xmd run` installs: the run's Workspace effect coordinator, the logical working directory `/`, and the transaction-bound `API.Files` provider. +That composed helper is the whole of what the entrypoint publishes. The three +pieces are not installable separately, because the Files provider alone would +resolve a document's paths against whatever working directory the surrounding +host adapter answers with, and a host path resolved that way is what the run +then retains in the durable effects it replays from. + Paths are absolute POSIX paths inside the run's own filesystem. An authored path is resolved by arithmetic on segments and handed to the run's DOFS filesystem; no host path exists anywhere in it, so containment needs no stable-namespace @@ -665,19 +683,30 @@ operation and the resolved logical path, so one authored element is the same effect across replays while an element edited to name another file is a different one. +A search answers with sorted, deduplicated, POSIX-relative regular files, which +is the contract `API.Files` holds wherever it runs. A symbolic link is neither a +result nor a way into the tree it names, so a file reachable through a directory +link is reported once, under its own path. + One effect is one effect transaction. The mutation, the resulting immutable root and the filtered journal result commit together. An ordinary filesystem refusal -rolls its mutation savepoint back before its result is published, so the -retained outcome describes a Workspace that is exactly what it was and the write -reports its target as rolled back. What crosses the boundary is a `FilesReason` -selected from the shared vocabulary — no DOFS message, errno payload, SQLite -text or resolved path. Everything that is not a documented refusal stays an -infrastructure failure and fails the run. +rolls its mutation savepoint back before its result is published, so a write +that created two parent directories and was then refused leaves neither behind, +the retained outcome describes a Workspace that is exactly what it was, the +write reports its target as rolled back, and the next effect still commits. What +crosses the boundary is a `FilesReason` selected from the shared vocabulary — no +DOFS message, errno payload, SQLite text or resolved path. Everything that is +not a documented refusal stays an infrastructure failure and fails the run. Replay restores the recorded outcome: it performs no mutation, opens no transaction and consults no current state, which is what lets a read answer with the bytes it read at the time and a create/delete/create history replay in -order. +order. What it restores is parsed rather than believed. A record must carry its +variant's members and no others, each of the declared type, and a refusal's +phase and reason must both be words the operation's vocabulary holds. Anything +else describes no outcome, and becomes the one fixed cause-free provider +invariant — carrying nothing the record happened to hold, and performing no +further file effect. `temporaryDirectory` is refused with the existing operation-denied failure. A run has no host directory to hand out, and reaching the caller's would be the From 3c41ecf1594e4f1f40aa437e9927650c4f858f7d Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 11:48:35 -0400 Subject: [PATCH 03/10] =?UTF-8?q?=F0=9F=94=92=20Decide=20workflow-run=20id?= =?UTF-8?q?entity=20where=20nothing=20can=20decline=20to=20ask?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A ReplayGuard is composable policy: a handler installed further out may answer without delegating. Identity decided there depended on middleware ordering, and a completed journal reached under a suppressed guard handed its recorded root result to whichever run asked. The comparison is now a step inside the journal's own readAll, on the terms core holds a resumed run to its recorded root selection: reachable through no context, replaceable by nothing, ahead of every guard phase, of terminal reuse, of live execution and of any append. It carries the witness its source stream already had and establishes none. A record identifies a run only as the root coroutine's successfully settled Yield under the canonical type and the canonical name, holding a closed value of exactly the three members a run has. Any history with events must carry exactly one, so a same-typed Yield written elsewhere cannot stand in for the record that was removed. The filesystem a Workspace transaction hands its body is injected where the provider is installed and kept in its closure. A stable Api name is composition, and a component that reconstructs one reached the authoritative filesystem through the seam this replaces. Refusals retain a description holding the effect's type and name alone, so nothing about the run stays reachable on the error object. --- architecture.md | 14 +- packages/workflow/src/deno/provider.ts | 20 +- .../workflow/src/deno/workspace/private.ts | 58 ++-- packages/workflow/src/journal.ts | 203 +++++++++--- packages/workflow/src/run.ts | 214 +++++++------ packages/workflow/tests/retained-run.test.ts | 289 +++++++++++++++++- packages/workflow/tests/support/storage.ts | 21 +- .../workflow/tests/workspace-files.test.ts | 160 +++++++--- specs/executable-mdx-spec.md | 14 +- specs/workflow-spec.md | 68 ++++- 10 files changed, 807 insertions(+), 254 deletions(-) diff --git a/architecture.md b/architecture.md index 45556034..5ac6aee9 100644 --- a/architecture.md +++ b/architecture.md @@ -439,9 +439,9 @@ one operation does not enlist unrelated durable operations in the same scope. The Deno adapter establishes the canonical module's journal provenance for each WorkflowRun journal and retains that exact witness. The generic pre-persistence guard is policy-neutral and returns an unproven wrapper; the trusted -secret-filter wrapping site preserves provenance explicitly, so a filtered -journal — including one wrapped more than once — still carries the witness its -source carried. Before it opens a transaction, the adapter requires both the +secret-filter and workflow-run-admission wrapping sites preserve provenance +explicitly, so a filtered journal — including one wrapped more than once — still +carries the witness its source carried. Before it opens a transaction, the adapter requires both the exact proof executor and the journal provenance from the consumed invocation to belong to the selected WorkflowRun. An in-memory stream, another run's journal, a copied property, an ordinary guard, a custom look-alike, or a wrapper another @@ -1057,9 +1057,11 @@ It qualifies only while every one of these holds: witness per stream, with duplicate establishment refused; - transfer happens only at a trusted wrapping site — one installed before any code the journal's content could influence, delegating to the exact stream it - was handed. A document execution's journal passes through two: the secret - filter and the execution-owned target-admission wrapper. Each transfers only - what its source already had, so an unproven journal stays unproven; and + was handed. A document execution's journal passes through up to three: the + secret filter, the execution-owned target-admission wrapper, and — when a + workflow installation is present — the workflow-run admission wrapper, which + holds a retained history to the run it belongs to. Each transfers only what + its source already had, so an unproven journal stays unproven; and - it retains no execution, lifecycle, journal content or provider state. The exception exists because the exact-object, anti-forgery and loaded-copy diff --git a/packages/workflow/src/deno/provider.ts b/packages/workflow/src/deno/provider.ts index 5d5379ae..6661c827 100644 --- a/packages/workflow/src/deno/provider.ts +++ b/packages/workflow/src/deno/provider.ts @@ -69,6 +69,7 @@ import { readTransaction } from "./reading.ts"; import { initializeSchema, isUninitialized, translateSqliteError, verifySchema } from "./schema.ts"; import { SavepointObservation } from "./savepoints.ts"; import { usePrivateWorkspace } from "./workspace/private.ts"; +import type { PrivateWorkspaceOptions } from "./workspace/private.ts"; import { useWorkspaceEffects } from "./workspace/effect.ts"; const INSERT_RUN = `INSERT INTO workflow_run @@ -107,14 +108,29 @@ export const WorkflowRunRecognition = createContext * long as the scope that installed the provider and nothing accumulates * between runs. */ -export function* useWorkflowRunStorage(options: WorkflowRunStorageOptions): Operation { +export function useWorkflowRunStorage(options: WorkflowRunStorageOptions): Operation { + return installWorkflowRunStorage(options, {}); +} + +/** + * The same installation, with what only this adapter's own suites supply. + * + * Kept apart from the published entrypoint on purpose: `internal` carries a + * decorator for the authoritative Workspace filesystem, which is provider + * authority rather than host arrangement. It is captured in the provider's + * closure here and never handed to a scope, a context or a descendant. + */ +export function* installWorkflowRunStorage( + options: WorkflowRunStorageOptions, + internal: PrivateWorkspaceOptions, +): Operation { const root = authorizedRoot(options.root); const connections = createWorkflowRunConnections(yield* SavepointObservation.get()); yield* ensure(() => { connections.close(); }); yield* useJournalRouting(connections); - yield* usePrivateWorkspace(connections); + yield* usePrivateWorkspace(connections, internal); yield* useWorkspaceEffects(connections); yield* WorkflowRunStorage.around( diff --git a/packages/workflow/src/deno/workspace/private.ts b/packages/workflow/src/deno/workspace/private.ts index 501f1d86..44e04e5d 100644 --- a/packages/workflow/src/deno/workspace/private.ts +++ b/packages/workflow/src/deno/workspace/private.ts @@ -43,40 +43,26 @@ function unavailable(): never { } /** - * The filesystem a Workspace transaction hands its body, with anything installed - * around it. + * What decorates the filesystem a Workspace transaction hands its body. * - * Nothing this package ships installs anything here, and no entrypoint exports - * it. It exists because a mutation that is discarded part-way through cannot - * otherwise be observed: a write creates the parent directories it needs and - * then writes the file, and DOFS has no condition that stops between the two — - * a parent chain that can be created is a chain the file can then be written - * into. Placing a failure there is the only way to watch the savepoint take the - * created parents back. + * Dependency injection, deliberately, rather than anything a scope can reach. + * It is supplied once when the storage provider is installed — before any + * document exists — and lives in that provider's closure. There is no context + * name to reconstruct, nothing is handed to a descendant, and no module-scoped + * hook accumulates: a component cannot observe the authoritative filesystem, + * and cannot put anything in front of it. + * + * It exists because a mutation discarded part-way through cannot otherwise be + * observed. A write creates the parent directories it needs and then writes the + * file, and DOFS has no condition that stops between the two — a parent chain + * that can be created is a chain the file can then be written into. */ -interface WorkspaceFilesystemApi { - interpose(filesystem: DenoWorkspaceFilesystem): Operation; -} - -const WorkspaceFilesystem: Api = createApi( - "executablemd.workflow.deno.workspace.private.filesystem", - { - // deno-lint-ignore require-yield - *interpose(filesystem: DenoWorkspaceFilesystem): Operation { - return filesystem; - }, - }, -); +export type WorkspaceFilesystemDecorator = ( + filesystem: DenoWorkspaceFilesystem, +) => DenoWorkspaceFilesystem; -/** Wrap the filesystem every Workspace transaction opened below this hands out. */ -export function interposeWorkspaceFilesystem( - wrap: (filesystem: DenoWorkspaceFilesystem) => DenoWorkspaceFilesystem, -): Operation { - return WorkspaceFilesystem.around({ - *interpose([filesystem], next) { - return wrap(yield* next(filesystem)); - }, - }); +export interface PrivateWorkspaceOptions { + readonly decorateFilesystem?: WorkspaceFilesystemDecorator; } const PrivateWorkspace: Api = createApi( @@ -105,7 +91,11 @@ const PrivateWorkspace: Api = createApi { +export function usePrivateWorkspace( + connections: WorkflowRunConnections, + options: PrivateWorkspaceOptions = {}, +): Operation { + const decorate = options.decorateFilesystem ?? ((filesystem) => filesystem); return PrivateWorkspace.around( { *transact([database, transaction, body]: [ @@ -122,9 +112,7 @@ export function usePrivateWorkspace(connections: WorkflowRunConnections): Operat connections.authorizeTransaction(database, transaction); }; const workspace: PrivateWorkspaceTransaction = { - filesystem: yield* WorkspaceFilesystem.operations.interpose( - createDenoWorkspaceFilesystem(connection, authorize), - ), + filesystem: decorate(createDenoWorkspaceFilesystem(connection, authorize)), // deno-lint-ignore require-yield *currentRoot(): Operation { diff --git a/packages/workflow/src/journal.ts b/packages/workflow/src/journal.ts index 25edadaf..fae29eb9 100644 --- a/packages/workflow/src/journal.ts +++ b/packages/workflow/src/journal.ts @@ -1,15 +1,23 @@ /** - * The durable `workflow_run` record. + * The durable `workflow_run` record, and what a history has to hold to be a + * run's own. * * One immutable value per workflow run, written before the root document is * imported. The journal is parsed, never trusted: a record that does not * describe a workflow run is refused rather than coerced, and a record made * from a different base is refused rather than quietly standing in for this * run's. + * + * Recognition is deliberately narrow. A record identifies a run only when it is + * the root coroutine's own successfully settled Yield, under the canonical type + * *and* the canonical name, holding a closed value of exactly the three members + * a run has. Anything looser lets a same-typed Yield written under another name, + * or by a child coroutine, stand in for the record that was removed — and a + * recorded terminal result would then be reused on its authority. */ import { StaleInputError } from "@executablemd/durable-streams"; -import type { EffectDescription } from "@executablemd/durable-streams"; +import type { DurableEvent, EffectDescription } from "@executablemd/durable-streams"; /** * One workflow run: an opaque identifier, the base that was asked for, and the @@ -23,23 +31,139 @@ export interface WorkflowRun { export const WORKFLOW_RUN = "workflow_run"; +/** The coroutine a document execution's own history belongs to. */ +const ROOT_COROUTINE = "root"; + +const RUN_MEMBERS: readonly string[] = ["runId", "base", "pinnedCommit"]; + /** How the record identifies itself. `base` is for a reader, never for matching. */ export function describeWorkflowRun(base: string): EffectDescription { return { type: WORKFLOW_RUN, name: WORKFLOW_RUN, base }; } -/** The workflow run a stored value describes, or `undefined` if it describes none. */ +/** + * What a refusal is allowed to carry about the run. + * + * `StaleInputError` retains the description it is given, so handing it the + * recording description would keep the base reachable on the error object even + * though no message prints it. A fresh value each time, holding the two members + * that name the effect and nothing else. + */ +function refusalDescription(): EffectDescription { + return { type: WORKFLOW_RUN, name: WORKFLOW_RUN }; +} + +/** + * The workflow run a stored value describes, or `undefined` if it describes none. + * + * Closed over its three members: a value carrying a fourth is not a run with + * something extra, it is a value this version cannot account for. + */ export function readWorkflowRun(value: unknown): WorkflowRun | undefined { if (typeof value !== "object" || value === null || Array.isArray(value)) { return undefined; } - const { runId, base, pinnedCommit } = Object.fromEntries(Object.entries(value)); + const record: Record = Object.fromEntries(Object.entries(value)); + if ( + Object.keys(record).length !== RUN_MEMBERS.length || + !RUN_MEMBERS.every((member) => Object.hasOwn(record, member)) + ) { + return undefined; + } + const { runId, base, pinnedCommit } = record; if (typeof runId !== "string" || typeof base !== "string" || typeof pinnedCommit !== "string") { return undefined; } return Object.freeze({ runId, base, pinnedCommit }); } +function attempt(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +/** What one retained event claims about the run, when it claims anything. */ +interface RunClaim { + readonly settled: boolean; + readonly value: unknown; +} + +/** + * The claim this event makes, read totally. + * + * Every discriminator is forced here, so a record that refuses to be read is a + * record this history does not contain rather than one a later phase trips + * over. A Yield that is not the root coroutine's, or not under both the + * canonical type and the canonical name, makes no claim at all. + */ +function runClaim(event: DurableEvent): RunClaim | undefined { + return attempt(() => { + if (event.type !== "yield" || event.coroutineId !== ROOT_COROUTINE) { + return undefined; + } + if (event.description.type !== WORKFLOW_RUN || event.description.name !== WORKFLOW_RUN) { + return undefined; + } + return event.result.status === "ok" + ? { settled: true, value: event.result.value } + : { settled: false, value: undefined }; + }); +} + +/** + * Hold a retained history to the run it belongs to. + * + * An empty history is the ordinary live start and is held to nothing. Any other + * history has to carry exactly one canonical record, because the record is + * written before the root document is imported: a history with events and no + * readable record of its own describes work this run never authorized, and a + * recorded terminal result would otherwise be reused on that history's word. + * + * `agree` decides what "its own" means for the installation — a base for a + * programmatic run, all three members for a retained one — and refuses by + * throwing. + */ +export function admitWorkflowRunHistory( + retained: readonly DurableEvent[], + agree: (recorded: WorkflowRun) => WorkflowRun, +): WorkflowRun | undefined { + if (retained.length === 0) { + return undefined; + } + + const claims: RunClaim[] = []; + for (const event of retained) { + const claim = runClaim(event); + if (claim !== undefined) { + claims.push(claim); + } + } + + const runs: WorkflowRun[] = []; + for (const claim of claims) { + const run = claim.settled ? readWorkflowRun(claim.value) : undefined; + if (run !== undefined) { + runs.push(run); + } + } + + if (runs.length !== 1) { + // One settled claim that will not read is a damaged record rather than an + // absent one, and saying so is what tells a corrupted journal from a + // journal belonging to something else. + if (claims.length === 1 && claims[0]?.settled === true && runs.length === 0) { + throw malformedRecord(); + } + throw missingRunEvidence(runs.length); + } + + const only = runs[0]; + return only === undefined ? undefined : agree(only); +} + /** * The journal holds something that is not a workflow run. * @@ -47,12 +171,31 @@ export function readWorkflowRun(value: unknown): WorkflowRun | undefined { * reporting it would carry whatever it happened to hold into logs and rendered * output. */ -export function malformedRecord(description: EffectDescription): StaleInputError { +export function malformedRecord(): StaleInputError { return new StaleInputError( - `The journal records "${description.name}" holding a value that does not describe a ` + + `The journal records "${WORKFLOW_RUN}" holding a value that does not describe a ` + "workflow run. A record that cannot be read is refused rather than replayed. Re-run " + "the document from the start rather than resuming from this journal.", - { coroutineId: "root", description }, + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, + ); +} + +/** + * A journal with history offers no one record that says whose history it is. + * + * The tally is this module's own count rather than anything the journal said, + * so naming it carries nothing across. + */ +export function missingRunEvidence(records: number): StaleInputError { + const problem = + records === 0 + ? "records no successful workflow run of its own" + : `records ${records} successful workflow runs where exactly one identifies a run`; + return new StaleInputError( + `The journal holds recorded history but ${problem}. A workflow run replays only from ` + + "history that identifies it. Resume the run this journal belongs to, or re-run " + + "the document from the start.", + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, ); } @@ -64,56 +207,30 @@ export function malformedRecord(description: EffectDescription): StaleInputError * external text on the same terms as retained props: naming a field says what * disagrees without carrying the disagreement into logs and rendered output. */ -export function retainedRunMismatch( - description: EffectDescription, - fields: readonly string[], -): StaleInputError { +export function retainedRunMismatch(fields: readonly string[]): StaleInputError { return new StaleInputError( `The journal records a workflow run whose ${fields.join(", ")} differs from the retained ` + "run this execution was installed with. A retained run is replayed as itself rather " + "than onto a different one. Resume the run the journal belongs to.", - { coroutineId: "root", description }, + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, ); } /** - * A completed journal offers a terminal result without the one record that says - * whose result it is. + * The recorded run started from a different base than this run supplied. * - * Reading a record can only refuse a record the journal holds. A completed - * journal holding no readable `workflow_run` for this run — none at all, one - * that failed, or more than one — leaves nothing to refuse, and the recorded - * terminal result would then be handed back on the strength of history that - * never identified this run. The tally is this module's own count rather than - * anything the journal said, so naming it carries nothing across. + * Both bases are named, because here they are the two things a caller has to + * compare to understand the refusal, and both came from that caller rather than + * from the journal's opaque content. They are named in the sentence only: the + * description this error retains still holds nothing but the effect's type and + * name. */ -export function missingRunEvidence( - description: EffectDescription, - records: number, -): StaleInputError { - const problem = - records === 0 - ? "records no successful workflow run for it to be the result of" - : `records ${records} successful workflow runs where exactly one identifies a run`; - return new StaleInputError( - `The journal holds a completed result but ${problem}. A retained run is replayed only ` + - "from history that identifies it. Resume the run this journal belongs to, or re-run " + - "the document from the start.", - { coroutineId: "root", description }, - ); -} - -/** The recorded run started from a different base than this run supplied. */ -export function baseMismatch( - description: EffectDescription, - recorded: string, - supplied: string, -): StaleInputError { +export function baseMismatch(recorded: string, supplied: string): StaleInputError { return new StaleInputError( `The journal records this workflow run starting from "${recorded}", but this run ` + `supplied "${supplied}". A recorded base cannot be replayed onto a run that asked ` + "for a different one. Re-run the document from the start rather than resuming from " + "this journal.", - { coroutineId: "root", description }, + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, ); } diff --git a/packages/workflow/src/run.ts b/packages/workflow/src/run.ts index 59459675..e8268948 100644 --- a/packages/workflow/src/run.ts +++ b/packages/workflow/src/run.ts @@ -6,19 +6,18 @@ * reaches its first durable operation, which resolves the base once, records * one immutable value, and only then lets the root document be imported. * - * Two middlewares are needed because a journal can be in three states. + * A journal can be in three states, and two of them are held to the run. * * - **Live** — no record yet. `Execution.document` allocates the run id, * resolves `${base}^{commit}` through `Git.revParse()`, and records the value * before `next()` imports the root. - * - **Truncated** — the record is there but the root never closed. Both - * middlewares run: the guard restores the value, and the durable operation - * still runs so the journal cursor advances past its own entry. + * - **Truncated** — the record is there but the root never closed. The durable + * operation replays the stored value, so neither the identifier nor Git is + * reached a second time, and the journal cursor still advances past its own + * entry. * - **Completed** — the root `Close` is recorded, and `durableRun` returns the * stored result without ever invoking the workflow, so `Execution.document` - * is never reached. The guard's check phase runs before that shortcut, which - * is the only place a completed journal can restore its run — or refuse a - * different base before the recorded result is handed back. + * is never reached. * * `useRetainedWorkflow(run)` is the same three states under a run that already * exists. A workflow host has created the storage record before anything @@ -26,12 +25,22 @@ * allocating an id and resolving a base, and every state requires the journal to * agree with that value in full. * - * A completed journal is held to one thing more. Checking an event can only - * refuse an event the journal holds, so a journal that records a terminal - * result and no run at all offers nothing to refuse. The guard's admission - * phase runs once over the retained history, and a retained installation - * requires exactly one successful record there that reads as a workflow run and - * agrees with the retained one, before the stored result may answer for it. + * ## Where identity is decided + * + * Not in `ReplayGuard`. A guard is composable policy: a handler installed + * further out may decline to call `next`, which is what composition is for and + * exactly why durable identity cannot live there. A completed journal reached + * under a suppressed guard would hand back its recorded root result as though it + * were this run's. + * + * The comparison is a step inside the journal's own `readAll`, on the same terms + * core holds a resumed run to its recorded root selection. It runs where a + * journal first becomes readable — ahead of public guard policy, ahead of any + * retained Yield reaching execution, ahead of a retained `Close` being reused, + * ahead of authored work, and ahead of any append — and it is reachable through + * no context and replaceable by nothing. Public `ReplayGuard` handlers still + * observe and may still reject the history this admits; none of them can widen + * it. * * All of it is operation-scoped. The value is installed in the scope that owns * the document execution, so every descendant of the expansion reads it, the @@ -41,25 +50,23 @@ import { createContext } from "effection"; import type { Context, Operation } from "effection"; -import { createDurableOperation } from "@executablemd/durable-streams"; -import { ReplayGuard } from "@executablemd/durable-streams"; +import { createDurableOperation, preserveJournalProvenance } from "@executablemd/durable-streams"; import type { + DurableEvent, + DurableStream, EffectDescription, Json, - RetainedHistory, Workflow, - Yield, } from "@executablemd/durable-streams"; import { Execution } from "@executablemd/core"; import { revParse } from "./git.ts"; import { + admitWorkflowRunHistory, baseMismatch, describeWorkflowRun, malformedRecord, - missingRunEvidence, readWorkflowRun, retainedRunMismatch, - WORKFLOW_RUN, } from "./journal.ts"; import type { WorkflowRun } from "./journal.ts"; @@ -105,9 +112,7 @@ interface RunEstablishment { /** The run this execution is of, reached only when nothing is recorded yet. */ allocate(): Operation; /** The recorded run, or a refusal naming what it disagrees about. */ - hold(description: EffectDescription, recorded: WorkflowRun): WorkflowRun; - /** Whether the retained history as a whole may answer for this run. */ - admit(history: RetainedHistory): void; + hold(recorded: WorkflowRun): WorkflowRun; } /** Append the run to the journal, and answer with what the journal holds. */ @@ -138,80 +143,41 @@ function allocating(base: string): RunEstablishment { * compares only type and name, so the base this run supplied is checked * against the stored *value* rather than against the entry's identity. */ - hold(description: EffectDescription, recorded: WorkflowRun): WorkflowRun { + hold(recorded: WorkflowRun): WorkflowRun { if (recorded.base !== base) { - throw baseMismatch(description, recorded.base, base); + throw baseMismatch(recorded.base, base); } return recorded; }, - /** - * A base is all this installation knows before it runs, and a journal that - * records none is a journal this run has not started writing yet. Requiring - * a record here would refuse the ordinary live start. - */ - admit(_history: RetainedHistory): void {}, }; } function retaining(run: WorkflowRun): RunEstablishment { - const establishment: RunEstablishment = { + return { base: run.base, // deno-lint-ignore require-yield *allocate(): Operation { return run; }, - hold(description: EffectDescription, recorded: WorkflowRun): WorkflowRun { + hold(recorded: WorkflowRun): WorkflowRun { const differing = (["runId", "base", "pinnedCommit"] as const).filter( (field) => recorded[field] !== run[field], ); if (differing.length > 0) { - throw retainedRunMismatch(description, differing); + throw retainedRunMismatch(differing); } return recorded; }, - /** - * A completed journal answers with its recorded root result without ever - * reaching this run's middleware, so what makes that result this run's has - * to be required of the history rather than of an event. Exactly one - * successful record, readable as a workflow run and agreeing with the - * retained one in full, is what a run that got as far as closing left - * behind; anything else is another run's journal or a damaged one. - */ - admit(history: RetainedHistory): void { - if (!history.terminal) { - return; - } - const description = describeWorkflowRun(run.base); - const records = identifying(history, description, establishment); - if (records !== 1) { - throw missingRunEvidence(description, records); - } - }, }; - return establishment; } -/** - * How many retained records identify this run. - * - * Each candidate is read and held here rather than counted on the strength of - * the check phase having let it through, so admission proves for itself that - * what it counted describes a workflow run and describes this one. - */ -function identifying( - history: RetainedHistory, - description: EffectDescription, - establishment: RunEstablishment, -): number { - let records = 0; - for (const event of history.yields) { - if (event.description.type !== WORKFLOW_RUN || event.result.status !== "ok") { - continue; - } - held(description, event.result.value, establishment); - records += 1; +/** Read the record this run is held to, refusing anything that is not it. */ +function held(stored: unknown, establishment: RunEstablishment): WorkflowRun { + const run = readWorkflowRun(stored); + if (run === undefined) { + throw malformedRecord(); } - return records; + return establishment.hold(run); } function same(left: WorkflowRun, right: WorkflowRun): boolean { @@ -222,57 +188,75 @@ function same(left: WorkflowRun, right: WorkflowRun): boolean { ); } -/** Read the record this run is held to, refusing anything that is not it. */ -function held( - description: EffectDescription, - stored: unknown, - establishment: RunEstablishment, -): WorkflowRun { - const run = readWorkflowRun(stored); - if (run === undefined) { - throw malformedRecord(description); - } - return establishment.hold(description, run); -} - function* establish(establishment: RunEstablishment): Operation { const description = describeWorkflowRun(establishment.base); - const run = held(description, yield* record(description, establishment), establishment); + const run = held(yield* record(description, establishment), establishment); const restored = yield* CurrentWorkflowRun.get(); - // A truncated replay already restored this value in the check phase; keeping - // that object is what makes every read in one execution the same one. + // A resumed journal already installed this value when it was admitted; + // keeping that object is what makes every read in one execution the same one. if (restored !== undefined && same(restored, run)) { return; } yield* CurrentWorkflowRun.set(run); } -function* restore(event: Yield, establishment: RunEstablishment): Operation { - if (event.description.type !== WORKFLOW_RUN || event.result.status !== "ok") { - return; - } - yield* CurrentWorkflowRun.set( - held(describeWorkflowRun(establishment.base), event.result.value, establishment), - ); +/** + * The journal this execution reads and appends through, with the workflow-run + * identity check built into the read. + * + * **This authority is not middleware.** It is a step inside `readAll`, owned by + * the installation, reachable through no context and replaceable by nothing — + * so a public `ReplayGuard` handler that declines to delegate cannot reach past + * it, and neither can a same-named guard loaded from another copy of this + * package. + * + * It also owns the retained snapshot: the events it validates are the events it + * returns, so what every later phase observes is what identity was decided on. + * + * It is a trusted wrapping site, and says so explicitly. Journal provenance is + * not transitive, and a run whose journal is unproven is refused by the + * Workspace provider before any transaction. This wrapper qualifies because the + * installation puts it in place before any document code exists and it delegates + * every append to the exact stream it was handed. What it transfers is only the + * witness that exact source already has — it establishes none, so an unproven + * source stays unproven. + */ +function admittingJournal(stream: DurableStream, establishment: RunEstablishment): DurableStream { + const admitting: DurableStream = { + *readAll(): Operation { + const retained = yield* stream.readAll(); + const admitted = admitWorkflowRunHistory(retained, (recorded) => + establishment.hold(recorded), + ); + // Installed here rather than from a guard, because a completed journal + // never reaches `Execution.document`: this is the only place inside the + // execution where the run a recorded result belongs to is known, and it is + // the one place nothing composed around this installation can skip. + if (admitted !== undefined) { + yield* CurrentWorkflowRun.set(admitted); + } + return retained; + }, + append: (event: DurableEvent) => stream.append(event), + }; + return preserveJournalProvenance(stream, admitting); } function* install(establishment: RunEstablishment): Operation { - yield* ReplayGuard.around({ - *check([event], next) { - // Runs before `durableRun` can short-circuit on a recorded root Close, so - // a completed journal restores its run here — and refuses a record that - // is not this run's here, before the recorded result is returned. - yield* restore(event, establishment); - return yield* next(event); - }, - // Runs after every check and before a recorded terminal result may be - // reused, which is the only place a journal that records nothing at all can - // be refused. - *admit([history], next) { - establishment.admit(history); - return yield* next(history); + // `{ at: "min" }` so nothing sits between this and the execution that reads + // the stream: whatever journal arrives is the journal this wraps, and the + // wrapper is what reaches the durable run. + yield* Execution.around( + { + *execute([options], next) { + return yield* next({ + ...options, + stream: admittingJournal(options.stream, establishment), + }); + }, }, - }); + { at: "min" }, + ); yield* Execution.around({ *document([props], next) { @@ -313,10 +297,16 @@ export function useRetainedWorkflow(run: WorkflowRun): Operation { * * Parsed rather than believed: it arrives from a storage record a host read * back, so a member that is missing or empty is a value that identifies no run - * rather than one to install and discover later. + * rather than one to install and discover later. The three members are named + * here, so a host handing over a wider record installs the run it describes + * rather than being refused for carrying its own bookkeeping. */ function retainedRun(run: WorkflowRun): WorkflowRun { - const parsed = readWorkflowRun(run); + const parsed = readWorkflowRun({ + runId: run?.runId, + base: run?.base, + pinnedCommit: run?.pinnedCommit, + }); if (parsed === undefined || parsed.runId === "" || parsed.base === "") { throw new Error( "useRetainedWorkflow() needs the retained run's id, base and pinned commit. A run " + diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts index 6f62b593..24607a08 100644 --- a/packages/workflow/tests/retained-run.test.ts +++ b/packages/workflow/tests/retained-run.test.ts @@ -17,9 +17,16 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { scoped } from "effection"; import type { Operation } from "effection"; -import { InMemoryStream, StaleInputError } from "@executablemd/durable-streams"; +import { type Api, createApi } from "@effectionx/context-api"; +import { InMemoryStream, ReplayGuard, StaleInputError } from "@executablemd/durable-streams"; import type { DurableEvent, Json, Yield } from "@executablemd/durable-streams"; -import { collect, execute, inlineSource, registerComponents } from "@executablemd/core"; +import { + collect, + DocumentOutput, + execute, + inlineSource, + registerComponents, +} from "@executablemd/core"; import { Git } from "../src/git.ts"; import { getWorkflowRun, useRetainedWorkflow } from "../src/run.ts"; import type { WorkflowRun } from "../src/run.ts"; @@ -63,24 +70,105 @@ function useProbe(seen: WorkflowRun[]): Operation { interface Attempt { readonly seen: WorkflowRun[]; + readonly emitted: string[]; readonly thrown: unknown; } -function runRetained(run: WorkflowRun, stream: InMemoryStream): Operation { +/** + * Run `` under a retained installation, watching everything a refusal + * has to prevent. + * + * `seen` is non-empty only if the document expanded; `emitted` is non-empty only + * if a recorded result reached a consumer. `install` is where a test puts the + * policy it wants to prove cannot widen the boundary — installed *outside* the + * workflow installation, which is the position a suppressing handler wants. + */ +function runRetained( + run: WorkflowRun, + stream: InMemoryStream, + install: Operation = ok(), +): Operation { return scoped(function* () { const seen: WorkflowRun[] = []; + const emitted: string[] = []; yield* useForbiddenGit(); yield* useProbe(seen); + yield* install; yield* useRetainedWorkflow(run); + yield* DocumentOutput.around({ + *output([text], next) { + emitted.push(text); + yield* next(text); + }, + }); try { yield* collect(yield* execute({ ...inlineSource("\n"), stream })); - return { seen, thrown: undefined }; + return { seen, emitted, thrown: undefined }; } catch (error) { - return { seen, thrown: error }; + return { seen, emitted, thrown: error }; } }); } +// deno-lint-ignore require-yield +function* ok(): Operation {} + +/** + * A `ReplayGuard` that answers one phase without delegating. + * + * This is what composable policy is allowed to do, and the whole reason durable + * identity may not live behind it. + */ +function useSuppressingGuard(phase: "check" | "admit" | "decide"): Operation { + return ReplayGuard.around({ + *check([event], next) { + if (phase !== "check") { + yield* next(event); + } + }, + *admit([history], next) { + if (phase !== "admit") { + yield* next(history); + } + }, + decide([event], next) { + return phase === "decide" ? { outcome: "replay" } : next(event); + }, + }); +} + +/** + * The same suppression, through a descriptor this test built for itself. + * + * A contextual Api composes by stable name across loaded copies, so a second + * copy of `durable-streams` is exactly this: the same name, a handler nothing + * here imported. + */ +function useForeignGuard(): Operation { + const foreign: Api<{ + check(event: unknown): Operation; + admit(history: unknown): Operation; + decide(event: unknown): { outcome: "replay" }; + }> = createApi("DurableEffection.ReplayGuard", { + // deno-lint-ignore require-yield + *check(_event: unknown): Operation {}, + // deno-lint-ignore require-yield + *admit(_history: unknown): Operation {}, + decide(_event: unknown): { outcome: "replay" } { + return { outcome: "replay" }; + }, + }); + return foreign.around({ + // deno-lint-ignore require-yield + *check() {}, + // deno-lint-ignore require-yield + *admit() {}, + decide() { + return { outcome: "replay" }; + }, + }); +} + function workflowEvents(stream: InMemoryStream): DurableEvent[] { return stream .snapshot() @@ -263,6 +351,197 @@ describe("Tier RR — retained workflow runs", () => { expect(damaged.seen).toEqual([]); }); + // RR10: the reproduction the architecture review ran. Under a guard that + // answers without delegating, run-a's completed journal used to hand its + // recorded root result back to run-b. Identity is not policy, so it does not. + it("RR10: refuses another run's completed journal beneath a suppressing guard", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + for (const phase of ["check", "admit", "decide"] as const) { + const completed = new InMemoryStream(first.snapshot()); + const before = completed.snapshot().length; + + const attempt = yield* runRetained( + { ...RETAINED, runId: "release-1.5" }, + completed, + useSuppressingGuard(phase), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain("runId"); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(completed.snapshot().length).toEqual(before); + } + }); + + it("RR11: refuses it beneath a same-named guard this suite built itself", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const completed = new InMemoryStream(first.snapshot()); + const before = completed.snapshot().length; + + const attempt = yield* runRetained( + { ...RETAINED, runId: "release-1.5" }, + completed, + useForeignGuard(), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(completed.snapshot().length).toEqual(before); + }); + + it("RR12: a valid completed journal still replays beneath a suppressing guard", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + for (const phase of ["check", "admit", "decide"] as const) { + const completed = new InMemoryStream(first.snapshot()); + const attempt = yield* runRetained(RETAINED, completed, useSuppressingGuard(phase)); + + expect(attempt.thrown).toBeUndefined(); + // Zero live execution: the recorded result answered. + expect(attempt.seen).toEqual([]); + expect(workflowEvents(completed)).toHaveLength(1); + } + }); + + // RR13: the guard surface still works as policy. It observes what admission + // let through, and it may still refuse — it simply cannot widen. + it("RR13: a public guard still observes admitted history and may reject it", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const observed: string[] = []; + const watched = yield* runRetained( + RETAINED, + new InMemoryStream(first.snapshot()), + ReplayGuard.around({ + *check([event], next) { + observed.push(event.description.type); + yield* next(event); + }, + }), + ); + expect(watched.thrown).toBeUndefined(); + expect(observed).toContain("workflow_run"); + + const rejected = yield* runRetained( + RETAINED, + new InMemoryStream(first.snapshot()), + ReplayGuard.around({ + // deno-lint-ignore require-yield + *check([event]) { + if (event.description.type === "workflow_run") { + throw new Error("this guard says no"); + } + }, + }), + ); + expect(rejected.thrown).toBeInstanceOf(Error); + expect(rejected.emitted).toEqual([]); + }); + + // RR14: the record has to be the canonical one. A same-typed Yield under + // another name, or under a child coroutine, establishes nothing — otherwise + // removing the genuine record and adding one of these would authorize reuse. + it("RR14: refuses a record that is not the root coroutine's canonical one", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const counterfeits: Array<{ says: string; change: (event: Yield) => DurableEvent[] }> = [ + { + says: "wrong name", + change: (event) => [ + { ...event, description: { ...event.description, name: "workflow_run_v2" } }, + ], + }, + { + says: "child coroutine", + change: (event) => [{ ...event, coroutineId: "root.0" }], + }, + { + says: "extra member", + change: (event) => [ + { + ...event, + result: { + status: "ok", + value: { ...RETAINED, executor: "someone-else" }, + }, + }, + ], + }, + ]; + + for (const counterfeit of counterfeits) { + const damaged = completedWith(first, counterfeit.change); + const before = damaged.snapshot().length; + const attempt = yield* runRetained(RETAINED, damaged); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(damaged.snapshot().length).toEqual(before); + // Nothing the journal held is quoted back. + expect(String(attempt.thrown)).not.toContain("someone-else"); + expect(String(attempt.thrown)).not.toContain("workflow_run_v2"); + } + }); + + // RR15: a truncated journal is held to the same rule. A recorded + // establishment failure is not a licence to establish the run again. + it("RR15: refuses a truncated journal whose run record failed", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const failed = new InMemoryStream( + partial(first) + .snapshot() + .map((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? { ...event, result: { status: "err", error: { message: "planted-establishment" } } } + : event, + ), + ); + const before = failed.snapshot().length; + + const attempt = yield* runRetained(RETAINED, failed); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(failed.snapshot().length).toEqual(before); + // The recorded failure text is not replayed as this run's diagnostic. + expect(String(attempt.thrown)).not.toContain("planted-establishment"); + }); + + // RR16: nothing about the run reaches the error object, only the message. + it("RR16: a refusal retains no run id, base, pinned commit or planted member", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const attempt = yield* runRetained( + { ...RETAINED, runId: "release-1.5" }, + new InMemoryStream(first.snapshot()), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + // The whole error, not only its sentence: `StaleInputError` keeps what it + // is handed, so the description it retains is inspected too. + const whole = JSON.stringify(attempt.thrown, (_key, value) => + value instanceof Error ? { ...value, message: value.message, name: value.name } : value, + ); + for (const secret of ["release-1.4", "release-1.5", COMMIT, "main"]) { + expect(whole).not.toContain(secret); + } + expect(whole).toContain("runId"); + }); + it("RR7: refuses to install a run that identifies nothing", function* () { const empty = yield* scoped(function* () { try { diff --git a/packages/workflow/tests/support/storage.ts b/packages/workflow/tests/support/storage.ts index 94b09b23..0914488d 100644 --- a/packages/workflow/tests/support/storage.ts +++ b/packages/workflow/tests/support/storage.ts @@ -23,7 +23,9 @@ import { type WorkflowRunDatabase, WorkflowRunStorage, } from "../../mod.ts"; -import { useWorkflowRunStorage, workflowRunPath } from "../../deno.ts"; +import { workflowRunPath } from "../../deno.ts"; +import { installWorkflowRunStorage } from "../../src/deno/provider.ts"; +import type { PrivateWorkspaceOptions } from "../../src/deno/workspace/private.ts"; export const SHA1 = "9fceb02d0ae598e95dc970b74767f19372d61af8"; @@ -67,10 +69,21 @@ export function request( }; } -/** Run `body` with this host's storage installed for its scope only. */ -export function withStorage(root: string, body: () => Operation): Operation { +/** + * Run `body` with this host's storage installed for its scope only. + * + * `internal` is the provider's own installation option, supplied here and + * nowhere a document could reach: the decorator it may carry replaces the + * authoritative Workspace filesystem, and that decision belongs to whoever + * installs the provider. + */ +export function withStorage( + root: string, + body: () => Operation, + internal: PrivateWorkspaceOptions = {}, +): Operation { return scoped(function* () { - yield* useWorkflowRunStorage({ root }); + yield* installWorkflowRunStorage({ root }, internal); return yield* body(); }); } diff --git a/packages/workflow/tests/workspace-files.test.ts b/packages/workflow/tests/workspace-files.test.ts index bb4bdd35..18193493 100644 --- a/packages/workflow/tests/workspace-files.test.ts +++ b/packages/workflow/tests/workspace-files.test.ts @@ -15,8 +15,9 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { scoped, type Operation } from "effection"; -import { collect, execute, inlineSource } from "@executablemd/core"; +import { createContext, scoped, type Operation } from "effection"; +import { type Api, createApi } from "@effectionx/context-api"; +import { collect, execute, inlineSource, registerComponents } from "@executablemd/core"; import type { Json } from "@executablemd/durable-streams"; import { InMemoryStream } from "@executablemd/durable-streams"; import type { DurableEvent } from "@executablemd/durable-streams"; @@ -27,10 +28,7 @@ import { withWorkflowWorkspace } from "../src/deno/workspace/host.ts"; import { WORKSPACE_FILE } from "../src/deno/workspace/files.ts"; import { throwWorkspaceFilesystemFailure } from "../src/deno/workspace/errors.ts"; import type { DenoWorkspaceFilesystem } from "../src/deno/workspace/filesystem.ts"; -import { - interposeWorkspaceFilesystem, - transactWorkspaceRoots, -} from "../src/deno/workspace/private.ts"; +import { transactWorkspaceRoots } from "../src/deno/workspace/private.ts"; import type { PrivateWorkspaceTransaction } from "../src/deno/workspace/private.ts"; import { committedEventCount, @@ -623,42 +621,103 @@ describe("WF workflow document filesystem", () => { // is the rollback rather than an attempt that never started. it("discards the parent directories a refused write already created", function* () { const root = yield* useStorageRoot(); - yield* withStorage(root, function* () { - const database = yield* createRun(); - yield* mutateWorkspace(database, function* (workspace) { - yield* workspace.filesystem.writeFile("/kept.txt", "kept"); - }); + yield* withStorage( + root, + function* () { + const database = yield* createRun(); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/kept.txt", "kept"); + }); - const run = yield* scoped(function* () { - yield* interposeWorkspaceFilesystem(refusingWrite("/made/deep/x.txt")); - return yield* runDocument( + const run = yield* runDocument( database, ['no', "", 'yes'].join( "\n", ), ); - }); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "refused", phase: "transaction", reason: "permission-denied" }, + }); + + // Both directories the attempt created are gone. + for (const created of ["/made", "/made/deep"]) { + const stat = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.stat(created); + }); + expect(stat.ok).toEqual(false); + } + // What the Workspace already held is what it still holds. + expect(yield* workspaceText(database, "/kept.txt")).toEqual("kept"); + // The savepoint took back the mutation rather than the transaction, so + // the next effect still commits. + expect(recorded[1]?.result).toEqual({ status: "ok", value: { kind: "written" } }); + expect(yield* workspaceText(database, "/after.txt")).toEqual("yes"); + }, + { decorateFilesystem: refusingWrite("/made/deep/x.txt") }, + ); + }); + + // WF14: the transaction filesystem is the provider's, and a document is not + // where it is decided. Nothing a component can install — including the exact + // shapes a contextual seam would answer to — reaches it. + it("keeps document-scope middleware away from the transaction filesystem", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const reached: string[] = []; + yield* registerComponents([ + { + name: "Tamper", + origin: "tier-wf", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + // Every name a filesystem seam has ever answered to here, rebuilt + // from a component's own scope. A contextual Api composes by name + // across loaded copies, so if one still existed this would be it. + for (const name of [ + "executablemd.workflow.deno.workspace.private.filesystem", + "executablemd.workflow.deno.workspace.private", + "executablemd.workflow.deno.workspace.effect.mutation", + ]) { + const impostor: Api<{ interpose(value: unknown): Operation }> = createApi( + name, + { + // deno-lint-ignore require-yield + *interpose(value: unknown): Operation { + return value; + }, + }, + ); + yield* impostor.around({ + *interpose([value], next) { + reached.push(name); + return yield* next(value); + }, + }); + const alias = createContext(name, undefined); + yield* alias.set({ seized: true }); + } + return ""; + }, + }, + ]); + + const run = yield* runDocument( + database, + ["", "", 'written'].join("\n"), + ); expect(run.host.seen).toEqual([]); + // The write went to the run's own Workspace, untouched. + expect(yield* workspaceText(database, "/guarded.txt")).toEqual("written"); const recorded = yield* recordedFileEffects(database); - expect(recorded[0]?.result).toEqual({ - status: "ok", - value: { kind: "refused", phase: "transaction", reason: "permission-denied" }, - }); - - // Both directories the attempt created are gone. - for (const created of ["/made", "/made/deep"]) { - const stat = yield* transactWorkspaceRoots(database, function* (workspace) { - return yield* workspace.filesystem.stat(created); - }); - expect(stat.ok).toEqual(false); - } - // What the Workspace already held is what it still holds. - expect(yield* workspaceText(database, "/kept.txt")).toEqual("kept"); - // The savepoint took back the mutation rather than the transaction, so - // the next effect still commits. - expect(recorded[1]?.result).toEqual({ status: "ok", value: { kind: "written" } }); - expect(yield* workspaceText(database, "/after.txt")).toEqual("yes"); + expect(recorded[0]?.result).toEqual({ status: "ok", value: { kind: "written" } }); + // Nothing a component installed was ever consulted. + expect(reached).toEqual([]); }); }); @@ -705,6 +764,26 @@ describe("WF workflow document filesystem", () => { target: "/seed.txt", value: { kind: "refused", phase: "target", reason: "missing", detail: "PLANTED-SECRET" }, }, + // Every member, holding the wrong kind of value. + { operation: "read", target: "/seed.txt", value: { kind: "content", content: 7 } }, + { operation: "glob", target: "/", value: { kind: "paths", paths: "seed.txt" } }, + { operation: "glob", target: "/", value: { kind: "paths", paths: ["seed.txt", 7] } }, + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: 7, reason: "missing" }, + }, + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "target", reason: 7 }, + }, + // A word from the other operation's vocabulary is not this one's. + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "commit", reason: "missing" }, + }, ]; for (const planted of cases) { @@ -721,6 +800,15 @@ describe("WF workflow document filesystem", () => { const failure = yield* raised(replayDocument(database, source)); expect(failure).toBeInstanceOf(Error); + // Exactly the fixed provider invariant — not merely "something failed". + const fatal = fatalOf(failure); + expect(parseFilesFatal(fatal)).toEqual({ + type: FILES_FATAL, + kind: "invariant", + category: "protocol", + }); + // Cause-free: nothing the journal held is carried along underneath it. + expect(fatal instanceof Error ? fatal.cause : "not an error").toBeUndefined(); expect(String(failure)).not.toContain("PLANTED-SECRET"); // The failed run performed no file effect of its own — the history it // could not read is the whole of what it has. @@ -752,7 +840,11 @@ describe("WF workflow document filesystem", () => { const failure = yield* raised(replayDocument(database, source)); - expect(failure).toBeInstanceOf(Error); + expect(parseFilesFatal(fatalOf(failure))).toEqual({ + type: FILES_FATAL, + kind: "invariant", + category: "protocol", + }); expect((yield* workspaceEvents(database)).length).toEqual(before); const second = yield* transactWorkspaceRoots(database, function* (workspace) { return yield* workspace.filesystem.stat("/second.txt"); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 68f7b1f3..2a9a8359 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -1568,6 +1568,8 @@ run but are absent from the diagnostic trace. | `packages/workflow/src/service-denial.ts` | `useWorkflowServiceDenial()`, the tested non-delegating provider for future workflow start and resume scopes (#366) | | `packages/workflow/src/deno/workspace/files.ts` | the transaction-bound `API.Files` provider — one durable Workspace effect per document read, write and search | | `packages/workflow/src/deno/workspace/host.ts` | `withWorkflowWorkspace()` — the run's effect coordinator, logical cwd `/`, and Files provider installed together inside one execution | +| `packages/workflow/src/journal.ts` | the `workflow_run` record, canonical-record recognition, and the refusals that name differing fields without their values | +| `packages/workflow/src/run.ts` | `useWorkflow()` / `useRetainedWorkflow()`, and the trusted journal wrapper that decides workflow-run identity inside `readAll` | | `packages/cli/src/file-stream.ts` | `FileStream` — JSONL-backed `DurableStream` implementation | Dependencies: `@effectionx/scope-eval`, `@effectionx/timebox`, @@ -7739,8 +7741,15 @@ Defined in [Workflow runs](./workflow-spec.md) §3.1. | RR5 | A different base or pinned commit | Refused on the same terms, naming the fields that differ | | RR6 | A malformed record | Refused rather than coerced, without quoting what the journal held | | RR7 | An unusable installation | A missing run id, base or pinned commit is refused before any document executes | -| RR8 | A completed journal recording no run, or more than one | A terminal result whose journal holds no successful `workflow_run` record, holds one that failed, or holds two is refused at admission — before the recorded root result is handed back, and without appending anything | +| RR8 | A completed journal recording no run, or more than one | A terminal result whose journal holds no successful `workflow_run` record, holds one that failed, or holds two is refused before the recorded root result is handed back, and without appending anything | | RR9 | A completed journal recording another run | A completed journal whose record names a different run, or holds a value that does not read as a run at all, is refused on the same terms as a truncated one | +| RR10 | Suppressed guard policy | A completed run-a journal resumed as run-b is refused beneath a `ReplayGuard` that answers check, admit or decide without delegating — nothing expands, nothing is emitted, nothing is appended | +| RR11 | A guard from another loaded copy | The same, beneath a suppressing handler installed through an independently constructed descriptor of the guard's stable name | +| RR12 | Valid replay under the same suppression | A completed journal that does agree still returns its recorded result with zero live execution | +| RR13 | Policy still composes | A public guard observes the admitted history, and one that rejects it still refuses | +| RR14 | Non-canonical records | A same-typed Yield under another name, under a child coroutine, or holding a value with an extra member establishes nothing, and none of what it held is quoted back | +| RR15 | Failed establishment in a truncated history | A recorded `workflow_run` failure fails closed rather than replaying its planted text for a run storage already describes | +| RR16 | What a refusal retains | Inspecting the whole error object — not only its message — finds no run id, base, pinned commit or planted description member | ### Tier WF — The workflow document filesystem @@ -7760,7 +7769,8 @@ Defined in [Workflow runs](./workflow-spec.md) §10. | WF10 | Denied temporary directory | `` receives the operation-denied infrastructure failure and no host directory | | WF11 | The search's shape | Sorted, deduplicated, POSIX-relative regular files, on HF3's contract: neither a file symlink nor a directory symlink is a result, and a file reachable through a directory symlink is reported once, by its own path | | WF12 | Discarded partial mutation | A write that refuses after creating two parent directories leaves neither behind, leaves what the Workspace already held untouched, records the sanitized refusal, and does not stop the next effect from committing | -| WF13 | Unreadable history | A recorded outcome carrying a member its variant does not have, or a phase or reason the vocabulary does not hold, is refused as the fixed cause-free provider invariant — nothing the record held is repeated back, and no later file effect is performed | +| WF13 | Unreadable history | A recorded outcome carrying a member its variant does not have, a member of the wrong type, or a phase or reason the operation's vocabulary does not hold, is refused as exactly the cause-free `protocol` provider invariant — nothing the record held is repeated back, and no later file effect is performed | +| WF14 | The transaction filesystem is the provider's | Contextual middleware installed from inside the document — including descriptors rebuilt for every name a filesystem seam has used — neither observes nor replaces the filesystem a Workspace transaction hands its body | ### Tier SL — Own-scope context updates diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 9a1bd540..4418a803 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -99,17 +99,57 @@ base is any revision expression, so both are external text on the same terms as retained props. A value installed without a run id, a base or a pinned commit identifies no run and is refused before any document executes. -A completed journal is held to one thing more. Reading a record can only refuse -a record the journal holds, and a completed journal answers with its recorded -root result without expanding anything — so a journal recording a terminal -result and no run at all offers nothing to refuse. A retained installation -therefore requires, of the retained history as a whole, exactly one successful -`workflow_run` record that reads as a workflow run and agrees with the retained -one in full. None, one that failed, more than one, one that cannot be read, and -one naming another run are each refused before the recorded result is returned -and before anything expands. A journal with no history at all is the ordinary -live start and is unaffected, as is `useWorkflow({ base })`, which has no -retained value to hold a record to. +### 3.2 Where workflow-run identity is decided + +**Workflow-run identity is execution-owned, not `ReplayGuard` policy.** A guard +is composable policy by design: a handler installed further out may answer +without delegating, and that is what composition is for. Identity decided there +would depend on middleware ordering — a completed journal reached under a +suppressed guard would hand back its recorded root result as though it belonged +to the run asking for it. + +The comparison is therefore a step inside the journal's own `readAll`, on the +same terms core holds a resumed run to its recorded root selection. It is +reachable through no context and replaceable by nothing, including by a +same-named guard from another loaded copy of `durable-streams`, and it runs: + +- before any public `ReplayGuard` check, admit or decide; +- before a recorded root `Close` can be reused; +- before live execution, Workspace mutation, or any append. + +Public `ReplayGuard` handlers still observe the history this admits and may +still reject it. None of them can widen it. + +The wrapper is a trusted wrapping site: it is installed before any document code +exists, delegates every append to the exact stream it was handed, and carries +that stream's journal-provenance witness onto itself without establishing one. + +What a history is held to depends on the installation. `useWorkflow({ base })` +requires a recorded run's base to match; `useRetainedWorkflow(run)` requires +`runId`, `base` and `pinnedCommit` to match exactly. + +A record identifies a run only when it is all of these at once: + +- a Yield owned by the root coroutine; +- under the canonical effect type **and** the canonical effect name; +- successfully settled; +- holding a closed value of exactly `runId`, `base` and `pinnedCommit`, each a + string; and +- in agreement with the identity the installation supplied. + +An empty journal is the ordinary live start and is held to nothing. **Any other +history must carry exactly one such record**, because the record is written +before the root document is imported — so a history with events and no readable +record of its own describes work no run authorized. Missing, failed, duplicated, +malformed, carrying an extra member, written under another name, written by a +child coroutine, and naming another run are each refused, whether the history is +truncated or completed. A recorded establishment failure is refused rather than +replayed as this run's diagnostic. + +A refusal is a `StaleInputError` that names the fields that differ and never +their values, and the description it retains carries only the effect's type and +name — so nothing about the run, and nothing the journal held, is reachable on +the error object. `getWorkflowRun()`, the three journal states and the lifetime rules above are otherwise identical under either installation. @@ -669,6 +709,12 @@ resolve a document's paths against whatever working directory the surrounding host adapter answers with, and a host path resolved that way is what the run then retains in the durable effects it replays from. +The filesystem a Workspace transaction hands its body is the provider's, decided +where the provider is installed and held in its closure. It is reached through no +context and no contextual Api, so a document cannot observe it and cannot put +anything in front of it — a stable name is composition, and composition is not +where authority belongs. + Paths are absolute POSIX paths inside the run's own filesystem. An authored path is resolved by arithmetic on segments and handed to the run's DOFS filesystem; no host path exists anywhere in it, so containment needs no stable-namespace From 5445bbbe9d2955c9a7810b8e2484859009b9f62b Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 11:57:10 -0400 Subject: [PATCH 04/10] =?UTF-8?q?=F0=9F=94=92=20Put=20the=20filesystem=20a?= =?UTF-8?q?dversary=20where=20composition=20actually=20reaches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../workflow/tests/workspace-files.test.ts | 117 +++++++++++------- 1 file changed, 71 insertions(+), 46 deletions(-) diff --git a/packages/workflow/tests/workspace-files.test.ts b/packages/workflow/tests/workspace-files.test.ts index 18193493..5dea6744 100644 --- a/packages/workflow/tests/workspace-files.test.ts +++ b/packages/workflow/tests/workspace-files.test.ts @@ -198,6 +198,58 @@ function refusingWrite( }); } +/** + * Every name a Workspace filesystem decorator has answered to, rebuilt here. + * + * A contextual Api composes by stable name across loaded copies, so an + * independently constructed descriptor of the same name *is* the second copy. + * Each handler records that it was consulted and then delegates, so a seam that + * still existed would show up as a name in `reached` rather than as a broken + * run. + */ +const SEAM_NAMES: readonly string[] = [ + "executablemd.workflow.deno.workspace.private.filesystem", + "executablemd.workflow.deno.workspace.private", + "executablemd.workflow.deno.workspace.effect.mutation", +]; + +interface SeamShape { + interpose(value: unknown): Operation; +} + +function* useImpostorSeams(reached: string[]): Operation { + for (const name of SEAM_NAMES) { + const impostor: Api = createApi(name, { + // deno-lint-ignore require-yield + *interpose(value: unknown): Operation { + return value; + }, + }); + yield* impostor.around({ + *interpose([value], next) { + reached.push(name); + return yield* next(value); + }, + }); + yield* createContext(name, undefined).set({ seized: true }); + } +} + +/** `` — the same impostors, installed from inside the document. */ +function useTamper(reached: string[]): Operation { + return registerComponents([ + { + name: "Tamper", + origin: "tier-wf", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + yield* useImpostorSeams(reached); + return ""; + }, + }, + ]); +} + /** Whether one journal row is the recorded `operation` on `target`. */ function namesEffect(record: unknown, operation: string, target: string): boolean { if (typeof record !== "string") { @@ -661,62 +713,35 @@ describe("WF workflow document filesystem", () => { ); }); - // WF14: the transaction filesystem is the provider's, and a document is not - // where it is decided. Nothing a component can install — including the exact - // shapes a contextual seam would answer to — reaches it. - it("keeps document-scope middleware away from the transaction filesystem", function* () { + // WF14: the transaction filesystem is the provider's, decided where the + // provider was installed. The adversary here sits in the strongest position + // any composed code can occupy — a scope that encloses the whole document and + // was installed *after* the provider — and rebuilds, by name, every seam a + // filesystem decorator has answered to. A stable name is composition; this is + // what it means for authority not to travel through one. + it("keeps composed middleware away from the transaction filesystem", function* () { const root = yield* useStorageRoot(); yield* withStorage(root, function* () { const database = yield* createRun(); const reached: string[] = []; - yield* registerComponents([ - { - name: "Tamper", - origin: "tier-wf", - props: { type: "object", properties: {}, additionalProperties: false }, - *fn() { - // Every name a filesystem seam has ever answered to here, rebuilt - // from a component's own scope. A contextual Api composes by name - // across loaded copies, so if one still existed this would be it. - for (const name of [ - "executablemd.workflow.deno.workspace.private.filesystem", - "executablemd.workflow.deno.workspace.private", - "executablemd.workflow.deno.workspace.effect.mutation", - ]) { - const impostor: Api<{ interpose(value: unknown): Operation }> = createApi( - name, - { - // deno-lint-ignore require-yield - *interpose(value: unknown): Operation { - return value; - }, - }, - ); - yield* impostor.around({ - *interpose([value], next) { - reached.push(name); - return yield* next(value); - }, - }); - const alias = createContext(name, undefined); - yield* alias.set({ seized: true }); - } - return ""; - }, - }, - ]); - const run = yield* runDocument( - database, - ["", "", 'written'].join("\n"), - ); + const run = yield* scoped(function* () { + yield* useImpostorSeams(reached); + yield* useTamper(reached); + return yield* runDocument( + database, + ["", "", 'written'].join("\n"), + ); + }); expect(run.host.seen).toEqual([]); // The write went to the run's own Workspace, untouched. expect(yield* workspaceText(database, "/guarded.txt")).toEqual("written"); const recorded = yield* recordedFileEffects(database); - expect(recorded[0]?.result).toEqual({ status: "ok", value: { kind: "written" } }); - // Nothing a component installed was ever consulted. + expect(recorded.at(-1)?.result).toEqual({ status: "ok", value: { kind: "written" } }); + // Neither position reached the filesystem: not the enclosing scope, and + // not the component that installed the same names from inside the + // document. expect(reached).toEqual([]); }); }); From 6b306212ebad7a637f55e3f2f0e68334a9800218 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 13:37:56 -0400 Subject: [PATCH 05/10] =?UTF-8?q?=F0=9F=94=92=20Let=20core=20own=20the=20r?= =?UTF-8?q?ead=20that=20decides=20workflow-run=20identity?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- architecture.md | 15 +- packages/core/mod.ts | 2 + packages/core/src/execute.ts | 16 +- packages/core/src/journal-admission.ts | 48 ++++++ packages/workflow/src/journal.ts | 109 +++++++------ packages/workflow/src/run.ts | 124 +++++++-------- packages/workflow/tests/retained-run.test.ts | 151 ++++++++++++++++++- packages/workflow/tests/workflow-run.test.ts | 49 ++++++ specs/executable-mdx-spec.md | 6 +- specs/workflow-spec.md | 81 +++++++--- 10 files changed, 448 insertions(+), 153 deletions(-) create mode 100644 packages/core/src/journal-admission.ts diff --git a/architecture.md b/architecture.md index 5ac6aee9..ccd371be 100644 --- a/architecture.md +++ b/architecture.md @@ -439,7 +439,7 @@ one operation does not enlist unrelated durable operations in the same scope. The Deno adapter establishes the canonical module's journal provenance for each WorkflowRun journal and retains that exact witness. The generic pre-persistence guard is policy-neutral and returns an unproven wrapper; the trusted -secret-filter and workflow-run-admission wrapping sites preserve provenance +secret-filter and execution-owned admission wrapping sites preserve provenance explicitly, so a filtered journal — including one wrapped more than once — still carries the witness its source carried. Before it opens a transaction, the adapter requires both the exact proof executor and the journal provenance from the consumed invocation to @@ -1057,11 +1057,14 @@ It qualifies only while every one of these holds: witness per stream, with duplicate establishment refused; - transfer happens only at a trusted wrapping site — one installed before any code the journal's content could influence, delegating to the exact stream it - was handed. A document execution's journal passes through up to three: the - secret filter, the execution-owned target-admission wrapper, and — when a - workflow installation is present — the workflow-run admission wrapper, which - holds a retained history to the run it belongs to. Each transfers only what - its source already had, so an unproven journal stays unproven; and + was handed. A document execution's journal passes through two: the secret + filter and the execution-owned admission wrapper. That second wrapper holds a + resumed run to its recorded root selection and, in the same read, applies + whatever an installation required of the history through `admitJournal()` — + workflow-run identity among them. Requirements are contributed to it rather + than wrapped around it, so no wrapping site is added and none of them is + reachable by middleware. Each transfers only what its source already had, so + an unproven journal stays unproven; and - it retains no execution, lifecycle, journal content or provider state. The exception exists because the exact-object, anti-forgery and loaded-copy diff --git a/packages/core/mod.ts b/packages/core/mod.ts index 45af4b8a..5ccd3b1e 100644 --- a/packages/core/mod.ts +++ b/packages/core/mod.ts @@ -94,6 +94,8 @@ export { walkSchema } from "./src/schema-walk.ts"; export type { NameKind, SchemaVisitor } from "./src/schema-walk.ts"; export { hasContent, useContent } from "./src/content-context.ts"; +export { admitJournal } from "./src/journal-admission.ts"; +export type { JournalAdmission } from "./src/journal-admission.ts"; export { ContentError } from "./src/errors.ts"; export { getExpansion } from "./src/expansion.ts"; export type { Expansion } from "./src/expansion.ts"; diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index c8fd4a83..763d5a2e 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -29,6 +29,8 @@ import { exec, readTextFile, cwd } from "@executablemd/runtime"; import { cwd as processCwd } from "@effectionx/fs"; import type { Workflow, Json } from "@executablemd/durable-streams"; import { createReplayStream } from "./replay-stream.ts"; +import { requiredJournalAdmissions } from "./journal-admission.ts"; +import type { JournalAdmission } from "./journal-admission.ts"; import { createContext } from "effection"; import type { Context } from "effection"; import type { @@ -612,10 +614,17 @@ function guardedJournal( stream: DurableStream, root: RootDocumentSource, coroutineId: CoroutineId, + required: readonly JournalAdmission[], ): DurableStream { const admitting: DurableStream = { *readAll(): Operation { const retained = retainEvents(yield* stream.readAll()); + // What an installation requires runs first, because "whose history is + // this" precedes "what did it select": a journal belonging to another run + // is refused as another run's rather than as a changed selection. + for (const admission of required) { + yield* admission(retained); + } admitRootHistory(retained, root, coroutineId); return retained; }, @@ -1377,9 +1386,12 @@ function* executeDocument(options: ExecuteOptions): Operation // The journal is wrapped before it reaches `durableRun`, so the identity // check happens inside the read that every phase downstream depends on - // rather than in middleware anything could replace. + // rather than in middleware anything could replace. What an installation + // requires of the history is read here too — after every `Execution` + // handler has run, so none of them can replace the stream it applies to. + const required = yield* requiredJournalAdmissions(); const returned = yield* durableRun(() => Execution.operations.document(props), { - stream: guardedJournal(journal, root, ROOT_COROUTINE), + stream: guardedJournal(journal, root, ROOT_COROUTINE, required), }); // Taken rather than read, so the handoff belongs to the run that made it. const live = yield* takeLiveFailure(liveFailure); diff --git a/packages/core/src/journal-admission.ts b/packages/core/src/journal-admission.ts new file mode 100644 index 00000000..fcd57b64 --- /dev/null +++ b/packages/core/src/journal-admission.ts @@ -0,0 +1,48 @@ +/** + * What an installation requires of a retained history, decided inside the + * execution's own journal read. + * + * A workflow installation has to hold a journal to the run it belongs to, and + * that decision may not be middleware. `ReplayGuard` is composable policy — a + * handler further out may answer without delegating — and so is `Execution`: a + * handler registered at the same position can replace the options a later one + * built, including the stream. Either one would make durable identity depend on + * registration order. + * + * So an installation contributes the requirement rather than the wrapper. Core + * reads what is installed once, in the scope that owns the execution and before + * any document code exists, and runs it inside the same trusted `readAll` that + * already holds a resumed run to its recorded root selection. Nothing composed + * around the execution can suppress, replace or reorder it, because by the time + * any of it runs the read has already happened. + * + * Admissions are additive. Installing one keeps the ones already installed, so + * two installations in one scope both apply and neither can drop the other. + */ + +import { createContext } from "effection"; +import type { Context, Operation } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; + +/** + * One requirement, offered the exact retained history the run will replay. + * + * It refuses by throwing. The events it receives are the retained snapshot — + * every discriminator already settled — and they are the same objects every + * later phase reads, so what it decided on is what gets replayed. + */ +export type JournalAdmission = (retained: readonly DurableEvent[]) => Operation; + +const JournalAdmissions: Context = createContext< + readonly JournalAdmission[] | undefined +>("executablemd.core.journal-admission", undefined); + +/** Require `admission` of the journal every document execution in this scope reads. */ +export function* admitJournal(admission: JournalAdmission): Operation { + yield* JournalAdmissions.set([...(yield* requiredJournalAdmissions()), admission]); +} + +/** What this execution was installed to require. Read once, before the document. */ +export function* requiredJournalAdmissions(): Operation { + return (yield* JournalAdmissions.get()) ?? []; +} diff --git a/packages/workflow/src/journal.ts b/packages/workflow/src/journal.ts index fae29eb9..fb99e09f 100644 --- a/packages/workflow/src/journal.ts +++ b/packages/workflow/src/journal.ts @@ -77,14 +77,6 @@ export function readWorkflowRun(value: unknown): WorkflowRun | undefined { return Object.freeze({ runId, base, pinnedCommit }); } -function attempt(read: () => T): T | undefined { - try { - return read(); - } catch { - return undefined; - } -} - /** What one retained event claims about the run, when it claims anything. */ interface RunClaim { readonly settled: boolean; @@ -92,15 +84,27 @@ interface RunClaim { } /** - * The claim this event makes, read totally. + * The claim this event makes, read totally and fail-closed. * - * Every discriminator is forced here, so a record that refuses to be read is a - * record this history does not contain rather than one a later phase trips - * over. A Yield that is not the root coroutine's, or not under both the - * canonical type and the canonical name, makes no claim at all. + * Every member a decision rests on is forced here — the discriminator, the + * coroutine, both halves of the description, the settlement and a successful + * settlement's value — and an event that refuses any of them is a history this + * run cannot describe rather than an unrelated event to step past. Skipping it + * is what would let a record that will not read stand in for one that is + * absent, and a recorded terminal result be reused on the difference. + * + * A Yield that reads cleanly but is not the root coroutine's, or is not under + * both the canonical type and the canonical name, makes no claim at all. */ function runClaim(event: DurableEvent): RunClaim | undefined { - return attempt(() => { + try { + if (event.type === "close") { + // Forced, not read for a value: a Close this history cannot classify is + // the same refusal as a Yield that will not read. + void event.coroutineId; + void event.result.status; + return undefined; + } if (event.type !== "yield" || event.coroutineId !== ROOT_COROUTINE) { return undefined; } @@ -110,58 +114,73 @@ function runClaim(event: DurableEvent): RunClaim | undefined { return event.result.status === "ok" ? { settled: true, value: event.result.value } : { settled: false, value: undefined }; - }); + } catch { + throw malformedRecord(); + } } /** - * Hold a retained history to the run it belongs to. + * What one installation requires of a history that is not empty. * - * An empty history is the ordinary live start and is held to nothing. Any other - * history has to carry exactly one canonical record, because the record is - * written before the root document is imported: a history with events and no - * readable record of its own describes work this run never authorized, and a - * recorded terminal result would otherwise be reused on that history's word. + * The two installations differ in one thing, and it is not strictness. A + * retained run was created by a host before anything executed, so a history of + * its own is something it must have: none, or one that failed, means the + * recorded work is not this run's. A programmatic run allocates itself on first + * execution, and §6 records a base that would not resolve as a *failed* effect — + * so a history whose only record is that failure is this run's history, and + * replaying it reproduces the failure rather than asking Git again. + */ +export interface RunHistoryRules { + /** Whether a non-empty history must carry a successful record. */ + readonly required: boolean; + /** The recorded run, or a refusal naming what it disagrees about. */ + agree(recorded: WorkflowRun): WorkflowRun; +} + +/** + * Hold a retained history to the run it belongs to. * - * `agree` decides what "its own" means for the installation — a base for a - * programmatic run, all three members for a retained one — and refuses by - * throwing. + * An empty history is the ordinary live start and is held to nothing. Otherwise + * every canonical record it carries must read as a workflow run and agree with + * this installation, and it may carry at most one — the record is written before + * the root document is imported, so two describe two runs. */ export function admitWorkflowRunHistory( retained: readonly DurableEvent[], - agree: (recorded: WorkflowRun) => WorkflowRun, + rules: RunHistoryRules, ): WorkflowRun | undefined { if (retained.length === 0) { return undefined; } - const claims: RunClaim[] = []; + const runs: WorkflowRun[] = []; + let settled = 0; for (const event of retained) { const claim = runClaim(event); - if (claim !== undefined) { - claims.push(claim); - } - } - - const runs: WorkflowRun[] = []; - for (const claim of claims) { - const run = claim.settled ? readWorkflowRun(claim.value) : undefined; - if (run !== undefined) { - runs.push(run); + if (claim === undefined || !claim.settled) { + continue; } - } - - if (runs.length !== 1) { - // One settled claim that will not read is a damaged record rather than an - // absent one, and saying so is what tells a corrupted journal from a - // journal belonging to something else. - if (claims.length === 1 && claims[0]?.settled === true && runs.length === 0) { + settled += 1; + const run = readWorkflowRun(claim.value); + // A settled record that will not read is damaged rather than absent, and + // saying so is what tells a corrupted journal from somebody else's. + if (run === undefined) { throw malformedRecord(); } - throw missingRunEvidence(runs.length); + runs.push(run); } + if (settled > 1) { + throw missingRunEvidence(settled); + } const only = runs[0]; - return only === undefined ? undefined : agree(only); + if (only === undefined) { + if (rules.required) { + throw missingRunEvidence(0); + } + return undefined; + } + return rules.agree(only); } /** diff --git a/packages/workflow/src/run.ts b/packages/workflow/src/run.ts index e8268948..9ba09547 100644 --- a/packages/workflow/src/run.ts +++ b/packages/workflow/src/run.ts @@ -27,20 +27,26 @@ * * ## Where identity is decided * - * Not in `ReplayGuard`. A guard is composable policy: a handler installed - * further out may decline to call `next`, which is what composition is for and - * exactly why durable identity cannot live there. A completed journal reached - * under a suppressed guard would hand back its recorded root result as though it - * were this run's. + * In no middleware at all. `ReplayGuard` is composable policy — a handler + * installed further out may decline to call `next` — and so is `Execution`: a + * handler registered at the same position can rebuild the options a later one + * produced, stream included. A journal held to its run from either place would + * be held to it by registration order. * - * The comparison is a step inside the journal's own `readAll`, on the same terms - * core holds a resumed run to its recorded root selection. It runs where a - * journal first becomes readable — ahead of public guard policy, ahead of any - * retained Yield reaching execution, ahead of a retained `Close` being reused, - * ahead of authored work, and ahead of any append — and it is reachable through - * no context and replaceable by nothing. Public `ReplayGuard` handlers still - * observe and may still reject the history this admits; none of them can widen - * it. + * So this installation contributes the *requirement* and core owns the read. + * `admitJournal()` records what a history has to satisfy; core reads what is + * installed before any document code exists and applies it inside the same + * trusted `readAll` that already holds a resumed run to its recorded root + * selection. That read happens before any middleware runs, on the retained + * snapshot every later phase consumes — ahead of public guard policy, of any + * retained Yield reaching execution, of a retained `Close` being reused, of + * authored work, and of any append. Public `ReplayGuard` handlers still observe + * and may still reject the history this admits; none of them can widen it. + * + * The two installations differ in what they require, not in how strictly it is + * enforced. See `RunHistoryRules`: a base that would not resolve is recorded as + * a failed effect (§6), so a programmatic run replays that failure rather than + * demanding a successful record it never wrote. * * All of it is operation-scoped. The value is installed in the scope that owns * the document execution, so every descendant of the expansion reads it, the @@ -50,15 +56,15 @@ import { createContext } from "effection"; import type { Context, Operation } from "effection"; -import { createDurableOperation, preserveJournalProvenance } from "@executablemd/durable-streams"; +import { createDurableOperation } from "@executablemd/durable-streams"; import type { DurableEvent, - DurableStream, EffectDescription, Json, Workflow, } from "@executablemd/durable-streams"; -import { Execution } from "@executablemd/core"; +import { admitJournal, Execution } from "@executablemd/core"; +import type { JournalAdmission } from "@executablemd/core"; import { revParse } from "./git.ts"; import { admitWorkflowRunHistory, @@ -68,7 +74,7 @@ import { readWorkflowRun, retainedRunMismatch, } from "./journal.ts"; -import type { WorkflowRun } from "./journal.ts"; +import type { RunHistoryRules, WorkflowRun } from "./journal.ts"; export type { WorkflowRun } from "./journal.ts"; @@ -107,12 +113,10 @@ export function* getWorkflowRun(): Operation { * is not the execution's to allocate: it arrives whole, and a journal that * records a different one is not this run's journal. */ -interface RunEstablishment { +interface RunEstablishment extends RunHistoryRules { readonly base: string; /** The run this execution is of, reached only when nothing is recorded yet. */ allocate(): Operation; - /** The recorded run, or a refusal naming what it disagrees about. */ - hold(recorded: WorkflowRun): WorkflowRun; } /** Append the run to the journal, and answer with what the journal holds. */ @@ -132,6 +136,10 @@ function* record( function allocating(base: string): RunEstablishment { return { base, + // A base that would not resolve is recorded as a failed effect (§6), and a + // history whose only record is that failure is this run's own. Requiring a + // successful one would retry Git instead of replaying what happened. + required: false, *allocate(): Operation { const pinnedCommit = yield* revParse(`${base}^{commit}`); // Web Crypto rather than `node:crypto`: a run id is allocated in shared @@ -143,7 +151,7 @@ function allocating(base: string): RunEstablishment { * compares only type and name, so the base this run supplied is checked * against the stored *value* rather than against the entry's identity. */ - hold(recorded: WorkflowRun): WorkflowRun { + agree(recorded: WorkflowRun): WorkflowRun { if (recorded.base !== base) { throw baseMismatch(recorded.base, base); } @@ -155,11 +163,15 @@ function allocating(base: string): RunEstablishment { function retaining(run: WorkflowRun): RunEstablishment { return { base: run.base, + // The host created this run before anything executed, so a history of its + // own is something it must have: none, or one that only failed, means the + // recorded work is not this run's. + required: true, // deno-lint-ignore require-yield *allocate(): Operation { return run; }, - hold(recorded: WorkflowRun): WorkflowRun { + agree(recorded: WorkflowRun): WorkflowRun { const differing = (["runId", "base", "pinnedCommit"] as const).filter( (field) => recorded[field] !== run[field], ); @@ -177,7 +189,7 @@ function held(stored: unknown, establishment: RunEstablishment): WorkflowRun { if (run === undefined) { throw malformedRecord(); } - return establishment.hold(run); + return establishment.agree(run); } function same(left: WorkflowRun, right: WorkflowRun): boolean { @@ -201,62 +213,30 @@ function* establish(establishment: RunEstablishment): Operation { } /** - * The journal this execution reads and appends through, with the workflow-run - * identity check built into the read. - * - * **This authority is not middleware.** It is a step inside `readAll`, owned by - * the installation, reachable through no context and replaceable by nothing — - * so a public `ReplayGuard` handler that declines to delegate cannot reach past - * it, and neither can a same-named guard loaded from another copy of this - * package. + * What this installation requires of the history a document execution replays. * - * It also owns the retained snapshot: the events it validates are the events it - * returns, so what every later phase observes is what identity was decided on. + * Contributed to core rather than wrapped around core. The comparison runs + * inside the execution's own trusted journal read, on the retained snapshot + * every later phase consumes — so no `ReplayGuard` handler that declines to + * delegate, and no `Execution` handler that rebuilds the options a later one + * produced, can suppress, replace or reorder it. By the time any middleware + * runs, the read has already happened. * - * It is a trusted wrapping site, and says so explicitly. Journal provenance is - * not transitive, and a run whose journal is unproven is refused by the - * Workspace provider before any transaction. This wrapper qualifies because the - * installation puts it in place before any document code exists and it delegates - * every append to the exact stream it was handed. What it transfers is only the - * witness that exact source already has — it establishes none, so an unproven - * source stays unproven. + * Setting the run here is not incidental: a completed journal never reaches + * `Execution.document`, so this is the only place inside the execution where the + * run a recorded result belongs to is known. */ -function admittingJournal(stream: DurableStream, establishment: RunEstablishment): DurableStream { - const admitting: DurableStream = { - *readAll(): Operation { - const retained = yield* stream.readAll(); - const admitted = admitWorkflowRunHistory(retained, (recorded) => - establishment.hold(recorded), - ); - // Installed here rather than from a guard, because a completed journal - // never reaches `Execution.document`: this is the only place inside the - // execution where the run a recorded result belongs to is known, and it is - // the one place nothing composed around this installation can skip. - if (admitted !== undefined) { - yield* CurrentWorkflowRun.set(admitted); - } - return retained; - }, - append: (event: DurableEvent) => stream.append(event), +function admits(establishment: RunEstablishment): JournalAdmission { + return function* (retained: readonly DurableEvent[]): Operation { + const admitted = admitWorkflowRunHistory(retained, establishment); + if (admitted !== undefined) { + yield* CurrentWorkflowRun.set(admitted); + } }; - return preserveJournalProvenance(stream, admitting); } function* install(establishment: RunEstablishment): Operation { - // `{ at: "min" }` so nothing sits between this and the execution that reads - // the stream: whatever journal arrives is the journal this wraps, and the - // wrapper is what reaches the durable run. - yield* Execution.around( - { - *execute([options], next) { - return yield* next({ - ...options, - stream: admittingJournal(options.stream, establishment), - }); - }, - }, - { at: "min" }, - ); + yield* admitJournal(admits(establishment)); yield* Execution.around({ *document([props], next) { diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts index 24607a08..2ff2629f 100644 --- a/packages/workflow/tests/retained-run.test.ts +++ b/packages/workflow/tests/retained-run.test.ts @@ -19,11 +19,18 @@ import { scoped } from "effection"; import type { Operation } from "effection"; import { type Api, createApi } from "@effectionx/context-api"; import { InMemoryStream, ReplayGuard, StaleInputError } from "@executablemd/durable-streams"; -import type { DurableEvent, Json, Yield } from "@executablemd/durable-streams"; +import type { + DurableEvent, + DurableStream, + Json, + Result, + Yield, +} from "@executablemd/durable-streams"; import { collect, DocumentOutput, execute, + Execution, inlineSource, registerComponents, } from "@executablemd/core"; @@ -85,8 +92,9 @@ interface Attempt { */ function runRetained( run: WorkflowRun, - stream: InMemoryStream, + stream: DurableStream, install: Operation = ok(), + after: Operation = ok(), ): Operation { return scoped(function* () { const seen: WorkflowRun[] = []; @@ -95,6 +103,7 @@ function runRetained( yield* useProbe(seen); yield* install; yield* useRetainedWorkflow(run); + yield* after; yield* DocumentOutput.around({ *output([text], next) { emitted.push(text); @@ -190,6 +199,73 @@ function partial(stream: InMemoryStream): InMemoryStream { ); } +/** An `Execution` handler that hands the durable run a journal of its choosing. */ +function useStreamHijack(raw: DurableStream): Operation { + return Execution.around( + { + *execute([options], next) { + return yield* next({ ...options, stream: raw }); + }, + }, + { at: "min" }, + ); +} + +/** The same, through a descriptor of the Api's stable name built in this suite. */ +function useForeignStreamHijack(raw: DurableStream): Operation { + const foreign: Api<{ + execute(options: { stream: DurableStream }): Operation; + }> = createApi("Execution", { + // deno-lint-ignore require-yield + *execute(_options: { stream: DurableStream }): Operation { + return undefined; + }, + }); + return foreign.around( + { + *execute([options], next) { + return yield* next({ ...options, stream: raw }); + }, + }, + { at: "min" }, + ); +} + +/** + * A journal that answers differently every time it is read. + * + * The run id shifts on the second read of the recorded value, which is what a + * backend handing out live objects can do. Admission and replay must therefore + * be reading one retained snapshot rather than each taking their own look — + * otherwise a history admitted as one run is replayed as another. + */ +function shiftingJournal(events: readonly DurableEvent[], reads: string[]): DurableStream { + const appended: DurableEvent[] = []; + const shift = (event: DurableEvent): DurableEvent => { + if (event.type !== "yield" || event.description.type !== "workflow_run") { + return event; + } + return { + ...event, + get result(): Result { + const runId = reads.length === 0 ? "release-1.4" : "release-1.5"; + reads.push(runId); + return { status: "ok", value: { runId, base: "main", pinnedCommit: COMMIT } }; + }, + }; + }; + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return [...events.map(shift), ...appended]; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; +} + /** * The completed journal, with what it records about the run replaced. * @@ -542,6 +618,77 @@ describe("Tier RR — retained workflow runs", () => { expect(whole).toContain("runId"); }); + // RR17: the second bypass the architecture review found. `Execution` is + // composable too — a handler registered at the same position can rebuild the + // options a later one produced, stream included — so an installation that + // wrapped the stream itself was still held to registration order. Nothing is + // wrapped now: core reads the requirement and applies it to the journal it + // built, after every handler has had its turn. + it("RR17: refuses another run's journal however Execution middleware is ordered", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + const other = { ...RETAINED, runId: "release-1.5" }; + + const hijacks: Array<{ says: string; install: (raw: DurableStream) => Operation }> = [ + { says: "the package's own descriptor", install: useStreamHijack }, + { says: "a descriptor built here", install: useForeignStreamHijack }, + ]; + + for (const hijack of hijacks) { + // Registered before the workflow installation, and registered after it. + // One of these was the order that used to succeed. + for (const order of ["before", "after"] as const) { + const raw = new InMemoryStream(first.snapshot()); + const handler = hijack.install(raw); + const attempt = yield* runRetained( + other, + new InMemoryStream(first.snapshot()), + order === "before" ? handler : ok(), + order === "after" ? handler : ok(), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain("runId"); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(raw.snapshot().length).toEqual(first.snapshot().length); + } + } + }); + + // RR18: admission and replay read one history, not two. The journal below + // answers differently on a second read; what makes that harmless is that the + // history is retained once, before anything is decided, and every phase is + // handed those same objects. + it("RR18: identity, guard observation and replay consume one retained snapshot", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const reads: string[] = []; + const observed: Json[] = []; + const attempt = yield* runRetained( + RETAINED, + shiftingJournal(first.snapshot(), reads), + ReplayGuard.around({ + *check([event], next) { + if (event.description.type === "workflow_run" && event.result.status === "ok") { + observed.push(event.result.value ?? null); + } + yield* next(event); + }, + }), + ); + + // The backend was asked once and never again, which is what makes the + // shift unreachable: the history is retained before anything is decided. + expect(reads).toEqual(["release-1.4"]); + // Admission agreed with that settlement, so nothing was refused... + expect(attempt.thrown).toBeUndefined(); + // ...and the guard was handed the same settled value, not a second look. + expect(observed).toEqual([{ runId: "release-1.4", base: "main", pinnedCommit: COMMIT }]); + expect(attempt.seen).toEqual([]); + }); + it("RR7: refuses to install a run that identifies nothing", function* () { const empty = yield* scoped(function* () { try { diff --git a/packages/workflow/tests/workflow-run.test.ts b/packages/workflow/tests/workflow-run.test.ts index b3c90bc6..d842609c 100644 --- a/packages/workflow/tests/workflow-run.test.ts +++ b/packages/workflow/tests/workflow-run.test.ts @@ -294,6 +294,55 @@ describe("Tier WR — workflow runs", () => { expect(recordedRun(stream)).toBeUndefined(); }); + // WR18: a base that would not resolve is journaled as a failed effect, and + // that history is this run's own — so a programmatic installation contributes + // no refusal about it. Requiring a *successful* record here would refuse a + // journal this run wrote, and retry Git on the way to doing so. + // + // What this does not assert is that the recorded Git failure is what the + // caller sees. It is not, and it was not before this PR either: the journal + // holds a root Close and no root import, which core's target admission refuses + // on its own terms (verified against `main` at b324b97). That contradiction + // between core's rule and workflow-spec §6 is recorded in §6 and is not this + // PR's to settle. + it("WR18: a recorded base-resolution failure is not refused as missing evidence", function* () { + const stream = new InMemoryStream(); + + const first = yield* scoped(function* () { + yield* Git.around( + { + // deno-lint-ignore require-yield + *revParse() { + throw new Error("fatal: not a git repository"); + }, + }, + { at: "min" }, + ); + yield* useWorkflow({ base: "main" }); + return yield* yield* execute({ ...inlineSource("\n"), stream }); + }); + expect(first.ok).toBe(false); + expect(recordedRun(stream)).toBeUndefined(); + + const before = stream.snapshot().length; + const expanded: WorkflowRun[] = []; + const replayed = yield* scoped(function* () { + yield* useForbiddenGit(); + yield* useProbe(expanded); + yield* useWorkflow({ base: "main" }); + return yield* yield* execute({ ...inlineSource("\n"), stream }); + }); + + expect(replayed.ok).toBe(false); + const message = replayed.ok ? "" : replayed.error.message; + // Nothing about workflow-run evidence: this installation had no objection. + expect(message).not.toContain("workflow run"); + expect(message).not.toContain("identifies"); + // Git was never asked, the root never expanded, and nothing was appended. + expect(expanded).toEqual([]); + expect(stream.snapshot().length).toEqual(before); + }); + it("WR8: a run survives a document that fails after it was recorded", function* () { const stream = new InMemoryStream(); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 2a9a8359..630dd6ab 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -1569,7 +1569,8 @@ run but are absent from the diagnostic trace. | `packages/workflow/src/deno/workspace/files.ts` | the transaction-bound `API.Files` provider — one durable Workspace effect per document read, write and search | | `packages/workflow/src/deno/workspace/host.ts` | `withWorkflowWorkspace()` — the run's effect coordinator, logical cwd `/`, and Files provider installed together inside one execution | | `packages/workflow/src/journal.ts` | the `workflow_run` record, canonical-record recognition, and the refusals that name differing fields without their values | -| `packages/workflow/src/run.ts` | `useWorkflow()` / `useRetainedWorkflow()`, and the trusted journal wrapper that decides workflow-run identity inside `readAll` | +| `packages/workflow/src/run.ts` | `useWorkflow()` / `useRetainedWorkflow()`, and the workflow-run requirement they contribute through `admitJournal()` | +| `packages/core/src/journal-admission.ts` | `admitJournal()` — what an installation requires of a retained history, applied inside the execution's own trusted journal read | | `packages/cli/src/file-stream.ts` | `FileStream` — JSONL-backed `DurableStream` implementation | Dependencies: `@effectionx/scope-eval`, `@effectionx/timebox`, @@ -7487,6 +7488,7 @@ Defined in [Workflow runs](./workflow-spec.md). | WR11 | A malformed record | Refused, and the refusal never quotes what the journal held | | WR12 | A slow base | Resolving one run's base does not stall a sibling execution | | WR13/WR14 | Seeded journals | A completed and a truncated journal written by hand restore without any live run having happened | +| WR18 | Recorded base-resolution failure | A programmatic installation raises no evidence objection to the journal §6 describes: Git is not consulted, the root does not expand, and nothing is appended | ### Tier WD — Workflow definitions and storage contracts @@ -7750,6 +7752,8 @@ Defined in [Workflow runs](./workflow-spec.md) §3.1. | RR14 | Non-canonical records | A same-typed Yield under another name, under a child coroutine, or holding a value with an extra member establishes nothing, and none of what it held is quoted back | | RR15 | Failed establishment in a truncated history | A recorded `workflow_run` failure fails closed rather than replaying its planted text for a run storage already describes | | RR16 | What a refusal retains | Inspecting the whole error object — not only its message — finds no run id, base, pinned commit or planted description member | +| RR17 | Suppressed or reordered `Execution` policy | A handler that hands the durable run a different stream — the package's own descriptor or one built elsewhere, registered before or after the workflow installation — cannot make another run's journal replay | +| RR18 | One retained snapshot | A journal whose recorded value shifts between reads is settled once: identity admission, guard observation and replay are handed the same objects, and the second answer is never reached | ### Tier WF — The workflow document filesystem diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 4418a803..baa844f3 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -101,17 +101,22 @@ identifies no run and is refused before any document executes. ### 3.2 Where workflow-run identity is decided -**Workflow-run identity is execution-owned, not `ReplayGuard` policy.** A guard -is composable policy by design: a handler installed further out may answer -without delegating, and that is what composition is for. Identity decided there -would depend on middleware ordering — a completed journal reached under a -suppressed guard would hand back its recorded root result as though it belonged -to the run asking for it. - -The comparison is therefore a step inside the journal's own `readAll`, on the -same terms core holds a resumed run to its recorded root selection. It is -reachable through no context and replaceable by nothing, including by a -same-named guard from another loaded copy of `durable-streams`, and it runs: +**Workflow-run identity is execution-owned, and it is not middleware of any +kind.** `ReplayGuard` is composable policy by design: a handler installed +further out may answer without delegating. So is `Execution`: a handler +registered at the same position may rebuild the options a later one produced, +stream included. Identity decided from either place is identity decided by +registration order — a completed journal reached under a suppressing guard, or +under a handler that swapped the stream back, would hand its recorded root +result to whichever run asked. + +An installation therefore contributes the *requirement* rather than a wrapper. +`admitJournal()` records what a history has to satisfy; core reads what is +installed before any document code exists and applies it inside the same trusted +`readAll` that already holds a resumed run to its recorded root selection. No +wrapping site is added, nothing is reachable through a context a document can +rebind, and by the time any middleware runs the read has already happened. It +runs: - before any public `ReplayGuard` check, admit or decide; - before a recorded root `Close` can be reused; @@ -124,9 +129,20 @@ The wrapper is a trusted wrapping site: it is installed before any document code exists, delegates every append to the exact stream it was handed, and carries that stream's journal-provenance witness onto itself without establishing one. -What a history is held to depends on the installation. `useWorkflow({ base })` -requires a recorded run's base to match; `useRetainedWorkflow(run)` requires -`runId`, `base` and `pinnedCommit` to match exactly. +What a history is held to depends on the installation, and they differ in one +thing beyond which fields must agree. + +`useRetainedWorkflow(run)` requires `runId`, `base` and `pinnedCommit` to match +exactly, and requires the record to be *there*: a host created the run before +anything executed, so a non-empty history carrying no successful record — none +at all, or only one that failed — is not this run's history. + +`useWorkflow({ base })` requires a recorded run's base to match, and requires +nothing to be present. It allocates its run on first execution, and §6 records a +base that would not resolve as a failed effect; a history whose only record is +that failure is this run's own, and refusing it would refuse a journal this run +wrote. Both installations refuse a history carrying more than one successful +record, and both refuse one that cannot be read. A record identifies a run only when it is all of these at once: @@ -137,14 +153,21 @@ A record identifies a run only when it is all of these at once: string; and - in agreement with the identity the installation supplied. -An empty journal is the ordinary live start and is held to nothing. **Any other -history must carry exactly one such record**, because the record is written -before the root document is imported — so a history with events and no readable -record of its own describes work no run authorized. Missing, failed, duplicated, -malformed, carrying an extra member, written under another name, written by a -child coroutine, and naming another run are each refused, whether the history is -truncated or completed. A recorded establishment failure is refused rather than -replayed as this run's diagnostic. +An empty journal is the ordinary live start and is held to nothing. Otherwise a +history may carry **at most one** such record — the record is written before the +root document is imported, so two describe two runs — and under a retained +installation it must carry exactly one. Duplicated, malformed, carrying an extra +member, written under another name, written by a child coroutine, and naming +another run are refused under either installation, whether the history is +truncated or completed. A missing or failed record is refused under a retained +installation and permitted under a programmatic one, for the reason above. + +The history admission reads is the retained history: every discriminator settled +once, before anything is decided, and the same objects every later phase +consumes. An event that refuses any member a decision rests on — its type, its +coroutine, either half of its description, its settlement, or a successful +settlement's value — is a history this run cannot describe and is refused, never +stepped past as unrelated. A refusal is a `StaleInputError` that names the fields that differ and never their values, and the description it retains carries only the effect's type and @@ -193,9 +216,17 @@ cannot be invoked, the working directory is not a Git repository, or the base does not resolve to a commit. Such a failure is journaled the way every durable effect's failure is — as a -recorded failed effect. Resuming the same journal therefore reproduces the -failure rather than retrying Git. No `WorkflowRun` value exists in either case, -which is what "records no workflow run" means. +recorded failed effect, and no `WorkflowRun` value exists, which is what +"records no workflow run" means. Resuming that journal does not retry Git, and +the workflow installation raises no objection to it (§3.2). + +**What resuming it reports is currently core's, not this failure.** The journal +holds a root `Close` and no root import, and core's target admission refuses any +terminal history in that shape — so the caller sees that refusal rather than the +recorded Git failure. This contradicts the sentence this paragraph used to make +and is recorded rather than resolved here: it predates the workflow document +filesystem, reproduces on `main`, and settling it means changing core's rule +about a `Close` without the import that authorized it. ## 7. The Git capability From 2adfc455265755f08707c2479c2c61eed77948b9 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 13:39:33 -0400 Subject: [PATCH 06/10] =?UTF-8?q?=F0=9F=A7=AA=20Pin=20the=20unreadable-his?= =?UTF-8?q?tory=20refusal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- packages/workflow/tests/retained-run.test.ts | 43 ++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts index 2ff2629f..d11bc84d 100644 --- a/packages/workflow/tests/retained-run.test.ts +++ b/packages/workflow/tests/retained-run.test.ts @@ -266,6 +266,21 @@ function shiftingJournal(events: readonly DurableEvent[], reads: string[]): Dura }; } +/** A journal handing back exactly these events, accessors and all. */ +function unreadableJournal(events: readonly DurableEvent[]): DurableStream { + const appended: DurableEvent[] = []; + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return [...events, ...appended]; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; +} + /** * The completed journal, with what it records about the run replaced. * @@ -689,6 +704,34 @@ describe("Tier RR — retained workflow runs", () => { expect(attempt.seen).toEqual([]); }); + // RR19: an event that refuses to be read is a history this run cannot + // describe, not one to step past on the way to a record that does read. + // + // Coverage rather than proof: core's own target admission forces the same + // discriminator and refuses a history like this too, so removing this + // installation's fail-closed handling does not make the test pass. What it + // pins is that the refusal happens and carries nothing. + it("RR19: refuses a history holding an event that will not read", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const refusing: DurableEvent = { + get type(): never { + throw new Error("planted-unreadable-discriminator"); + }, + } as unknown as DurableEvent; + + const attempt = yield* runRetained( + RETAINED, + unreadableJournal([refusing, ...first.snapshot()]), + ); + + expect(attempt.thrown).toBeInstanceOf(Error); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(String(attempt.thrown)).not.toContain("planted-unreadable-discriminator"); + }); + it("RR7: refuses to install a run that identifies nothing", function* () { const empty = yield* scoped(function* () { try { From 6b57fb7b3ba6dda3c98886d2ab18aa4221438337 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 14:14:26 -0400 Subject: [PATCH 07/10] =?UTF-8?q?=F0=9F=94=92=20Parse=20a=20recorded=20run?= =?UTF-8?q?=20totally,=20and=20count=20a=20second=20entry=20however=20it?= =?UTF-8?q?=20settled?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- packages/workflow/src/journal.ts | 79 +++++++--- packages/workflow/src/run.ts | 29 +++- packages/workflow/tests/retained-run.test.ts | 158 ++++++++++++++++++- specs/executable-mdx-spec.md | 2 + specs/workflow-spec.md | 24 ++- 5 files changed, 256 insertions(+), 36 deletions(-) diff --git a/packages/workflow/src/journal.ts b/packages/workflow/src/journal.ts index fb99e09f..4dd6781e 100644 --- a/packages/workflow/src/journal.ts +++ b/packages/workflow/src/journal.ts @@ -53,17 +53,46 @@ function refusalDescription(): EffectDescription { return { type: WORKFLOW_RUN, name: WORKFLOW_RUN }; } +/** + * Read one value the journal supplied, or answer that reading it refused. + * + * Deliberately narrow: exactly one read of exactly one journal-controlled + * value is inside. A hostile `ownKeys` trap, a `getOwnPropertyDescriptor` trap, + * a throwing getter and a revoked proxy all raise from here, and all of them + * mean the same thing — this value does not describe a run. Widening it would + * start converting programmer and infrastructure errors into "malformed + * journal", which is the opposite of a total parse. + */ +function readingJournalValue(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + /** * The workflow run a stored value describes, or `undefined` if it describes none. * * Closed over its three members: a value carrying a fourth is not a run with * something extra, it is a value this version cannot account for. + * + * Total over anything the journal can hold. Classification and enumeration are + * both the value's to refuse — `Array.isArray` throws for a revoked proxy, and + * `Object.entries` runs the traps and the getters — so both happen inside one + * guarded read and a refusal is an answer rather than an exception carrying the + * journal's own text out with it. */ export function readWorkflowRun(value: unknown): WorkflowRun | undefined { - if (typeof value !== "object" || value === null || Array.isArray(value)) { + if (typeof value !== "object" || value === null) { + return undefined; + } + const record = readingJournalValue(() => + Array.isArray(value) ? undefined : Object.fromEntries(Object.entries(value)), + ); + if (record === undefined) { return undefined; } - const record: Record = Object.fromEntries(Object.entries(value)); if ( Object.keys(record).length !== RUN_MEMBERS.length || !RUN_MEMBERS.every((member) => Object.hasOwn(record, member)) @@ -154,13 +183,20 @@ export function admitWorkflowRunHistory( } const runs: WorkflowRun[] = []; - let settled = 0; + let claimed = 0; for (const event of retained) { const claim = runClaim(event); - if (claim === undefined || !claim.settled) { + if (claim === undefined) { + continue; + } + // Counted whether or not it settled successfully. The record is written + // once, before the root document is imported, so a second entry under the + // canonical identity describes a second run however it ended — and a failed + // one beside a successful one is two runs, not one run with a stumble. + claimed += 1; + if (!claim.settled) { continue; } - settled += 1; const run = readWorkflowRun(claim.value); // A settled record that will not read is damaged rather than absent, and // saying so is what tells a corrupted journal from somebody else's. @@ -170,13 +206,13 @@ export function admitWorkflowRunHistory( runs.push(run); } - if (settled > 1) { - throw missingRunEvidence(settled); + if (claimed > 1) { + throw duplicateRunRecords(claimed); } const only = runs[0]; if (only === undefined) { if (rules.required) { - throw missingRunEvidence(0); + throw missingRunEvidence(); } return undefined; } @@ -199,21 +235,28 @@ export function malformedRecord(): StaleInputError { ); } +/** A journal with history offers no record that says whose history it is. */ +export function missingRunEvidence(): StaleInputError { + return new StaleInputError( + "The journal holds recorded history but records no successful workflow run of its " + + "own. A workflow run replays only from history that identifies it. Resume the run " + + "this journal belongs to, or re-run the document from the start.", + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, + ); +} + /** - * A journal with history offers no one record that says whose history it is. + * A journal carries more than one entry under the canonical run identity. * * The tally is this module's own count rather than anything the journal said, - * so naming it carries nothing across. + * so naming it carries nothing across. Failed entries count: the question is + * how many runs the history describes, not how many of them finished. */ -export function missingRunEvidence(records: number): StaleInputError { - const problem = - records === 0 - ? "records no successful workflow run of its own" - : `records ${records} successful workflow runs where exactly one identifies a run`; +export function duplicateRunRecords(records: number): StaleInputError { return new StaleInputError( - `The journal holds recorded history but ${problem}. A workflow run replays only from ` + - "history that identifies it. Resume the run this journal belongs to, or re-run " + - "the document from the start.", + `The journal records ${records} workflow run entries where at most one describes a ` + + "run. A history recording more than one run is not one run's history. Resume the " + + "run this journal belongs to, or re-run the document from the start.", { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, ); } diff --git a/packages/workflow/src/run.ts b/packages/workflow/src/run.ts index 9ba09547..759b7a42 100644 --- a/packages/workflow/src/run.ts +++ b/packages/workflow/src/run.ts @@ -183,6 +183,20 @@ function retaining(run: WorkflowRun): RunEstablishment { }; } +/** + * Read one member set off a value a host supplied, or answer that it refused. + * + * One read, nothing else inside — the same narrowness the journal's own reads + * are held to, for the same reason. + */ +function readingRetainedValue(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + /** Read the record this run is held to, refusing anything that is not it. */ function held(stored: unknown, establishment: RunEstablishment): WorkflowRun { const run = readWorkflowRun(stored); @@ -282,11 +296,16 @@ export function useRetainedWorkflow(run: WorkflowRun): Operation { * rather than being refused for carrying its own bookkeeping. */ function retainedRun(run: WorkflowRun): WorkflowRun { - const parsed = readWorkflowRun({ - runId: run?.runId, - base: run?.base, - pinnedCommit: run?.pinnedCommit, - }); + // Named through the same total read as a journal value: a host that hands + // over a record whose members refuse to be read has supplied a value that + // identifies no run, which is the sentence below rather than its exception. + const parsed = readWorkflowRun( + readingRetainedValue(() => ({ + runId: run?.runId, + base: run?.base, + pinnedCommit: run?.pinnedCommit, + })), + ); if (parsed === undefined || parsed.runId === "" || parsed.base === "") { throw new Error( "useRetainedWorkflow() needs the retained run's id, base and pinned commit. A run " + diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts index d11bc84d..8e52ad6f 100644 --- a/packages/workflow/tests/retained-run.test.ts +++ b/packages/workflow/tests/retained-run.test.ts @@ -266,10 +266,16 @@ function shiftingJournal(events: readonly DurableEvent[], reads: string[]): Dura }; } -/** A journal handing back exactly these events, accessors and all. */ +/** + * A journal handing back exactly these events, accessors and all. + * + * `InMemoryStream` clones what it is given, which is the right thing for it and + * the wrong thing here: a value that refuses to be read cannot survive being + * copied. These events reach the execution as written. + */ function unreadableJournal(events: readonly DurableEvent[]): DurableStream { const appended: DurableEvent[] = []; - return { + const journal: DurableStream = { // deno-lint-ignore require-yield *readAll(): Operation { return [...events, ...appended]; @@ -279,6 +285,28 @@ function unreadableJournal(events: readonly DurableEvent[]): DurableStream { appended.push(event); }, }; + appendedEvents.set(journal, appended); + return journal; +} + +const appendedEvents = new WeakMap(); + +/** What a refused run managed to append, which must be nothing. */ +function appendedTo(journal: DurableStream): DurableEvent[] { + return appendedEvents.get(journal) ?? []; +} + +/** + * The whole error, not only its sentence. + * + * `StaleInputError` retains what it is handed, so a refusal is inspected as an + * object: anything reachable on it is something a log or a rendered document + * could carry. + */ +function wholeError(thrown: unknown): string { + return JSON.stringify(thrown, (_key, value) => + value instanceof Error ? { ...value, message: value.message, name: value.name } : value, + ); } /** @@ -405,7 +433,7 @@ describe("Tier RR — retained workflow runs", () => { ]), }, { - says: "2 successful workflow runs", + says: "2 workflow run entries", stream: completedWith(first, (event) => [event, event]), }, ]; @@ -624,9 +652,7 @@ describe("Tier RR — retained workflow runs", () => { expect(attempt.thrown).toBeInstanceOf(StaleInputError); // The whole error, not only its sentence: `StaleInputError` keeps what it // is handed, so the description it retains is inspected too. - const whole = JSON.stringify(attempt.thrown, (_key, value) => - value instanceof Error ? { ...value, message: value.message, name: value.name } : value, - ); + const whole = wholeError(attempt.thrown); for (const secret of ["release-1.4", "release-1.5", COMMIT, "main"]) { expect(whole).not.toContain(secret); } @@ -732,6 +758,126 @@ describe("Tier RR — retained workflow runs", () => { expect(String(attempt.thrown)).not.toContain("planted-unreadable-discriminator"); }); + // RR20: nothing a journal holds gets to raise its own exception. Each value + // below refuses a different way — enumeration, descriptors, a getter, and + // classification of a revoked proxy — and every one of them has to arrive as + // the same fixed refusal, carrying none of its own text. + it("RR20: a hostile recorded value becomes the fixed refusal and leaks nothing", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const hostile: Array<{ says: string; value: () => unknown }> = [ + { + says: "ownKeys refuses", + value: () => + new Proxy( + { ...RETAINED }, + { + ownKeys() { + throw new Error("PLANTED-OWNKEYS"); + }, + }, + ), + }, + { + says: "getOwnPropertyDescriptor refuses", + value: () => + new Proxy( + { ...RETAINED }, + { + getOwnPropertyDescriptor() { + throw new Error("PLANTED-DESCRIPTOR"); + }, + }, + ), + }, + { + says: "a getter refuses", + value: () => ({ + base: "main", + pinnedCommit: COMMIT, + get runId(): never { + throw new Error("PLANTED-GETTER"); + }, + }), + }, + { + says: "classification refuses", + value: () => { + const revoked = Proxy.revocable({ ...RETAINED }, {}); + revoked.revoke(); + return revoked.proxy; + }, + }, + ]; + + for (const planted of hostile) { + const journal = unreadableJournal( + first + .snapshot() + .map((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? { ...event, result: { status: "ok", value: planted.value() as Json } } + : event, + ), + ); + + const attempt = yield* runRetained(RETAINED, journal); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + // No terminal result reused, no document code, no output, no append. + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(appendedTo(journal)).toEqual([]); + // The whole error object, not only its sentence. + const whole = wholeError(attempt.thrown); + for (const secret of [ + "PLANTED-OWNKEYS", + "PLANTED-DESCRIPTOR", + "PLANTED-GETTER", + "release-1.4", + COMMIT, + ]) { + expect(whole).not.toContain(secret); + } + } + }); + + // RR21: the record is written once, before the root import, so a second entry + // under the canonical identity describes a second run — however either of them + // ended. A failure beside a success is two runs, not one run with a stumble. + it("RR21: refuses a duplicate canonical record in every settlement combination", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + const failure = (event: Yield): DurableEvent => ({ + ...event, + result: { status: "err", error: { message: "planted-second-establishment" } }, + }); + + const combinations: Array<{ + says: string; + change: (event: Yield) => DurableEvent[]; + }> = [ + { says: "successful + successful", change: (event) => [event, event] }, + { says: "successful + failed", change: (event) => [event, failure(event)] }, + { says: "failed + successful", change: (event) => [failure(event), event] }, + { says: "failed + failed", change: (event) => [failure(event), failure(event)] }, + ]; + + for (const combination of combinations) { + const journal = completedWith(first, combination.change); + const before = journal.snapshot().length; + const attempt = yield* runRetained(RETAINED, journal); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain("2"); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(journal.snapshot().length).toEqual(before); + expect(wholeError(attempt.thrown)).not.toContain("planted-second-establishment"); + } + }); + it("RR7: refuses to install a run that identifies nothing", function* () { const empty = yield* scoped(function* () { try { diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 630dd6ab..e3557575 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -7754,6 +7754,8 @@ Defined in [Workflow runs](./workflow-spec.md) §3.1. | RR16 | What a refusal retains | Inspecting the whole error object — not only its message — finds no run id, base, pinned commit or planted description member | | RR17 | Suppressed or reordered `Execution` policy | A handler that hands the durable run a different stream — the package's own descriptor or one built elsewhere, registered before or after the workflow installation — cannot make another run's journal replay | | RR18 | One retained snapshot | A journal whose recorded value shifts between reads is settled once: identity admission, guard observation and replay are handed the same objects, and the second answer is never reached | +| RR20 | A hostile recorded value | A value whose `ownKeys`, `getOwnPropertyDescriptor` or getter refuses, and one whose classification refuses, each become the same fixed refusal — nothing reused, executed, emitted or appended, and the whole error object carries none of the planted text | +| RR21 | A duplicate canonical record | Two entries under the canonical run identity are refused in every settlement combination — successful+successful, successful+failed, failed+successful, failed+failed — because the record is written once and a second entry describes a second run however it ended | ### Tier WF — The workflow document filesystem diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index baa844f3..1338e21b 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -154,13 +154,23 @@ A record identifies a run only when it is all of these at once: - in agreement with the identity the installation supplied. An empty journal is the ordinary live start and is held to nothing. Otherwise a -history may carry **at most one** such record — the record is written before the -root document is imported, so two describe two runs — and under a retained -installation it must carry exactly one. Duplicated, malformed, carrying an extra -member, written under another name, written by a child coroutine, and naming -another run are refused under either installation, whether the history is -truncated or completed. A missing or failed record is refused under a retained -installation and permitted under a programmatic one, for the reason above. +history may carry **at most one** entry under the canonical run identity — the +record is written before the root document is imported, so a second entry +describes a second run, *however either of them settled*. Two successful +records, a successful one beside a failed one in either order, and two failed +ones are all refused. Under a retained installation the history must carry +exactly one, and it must have succeeded. + +Malformed, carrying an extra member, written under another name, written by a +child coroutine, and naming another run are refused under either installation, +whether the history is truncated or completed. A single missing or failed record +is refused under a retained installation and permitted under a programmatic one, +for the reason above. + +Reading a recorded value is total. A value whose enumeration, property +descriptors, getters or classification refuse describes no run, and becomes the +same fixed refusal every other unreadable record becomes — it never escapes +carrying its own text. The history admission reads is the retained history: every discriminator settled once, before anything is decided, and the same objects every later phase From 7f6e8565b40d8686562dd89583e3490df27a9c48 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 14:15:57 -0400 Subject: [PATCH 08/10] =?UTF-8?q?=F0=9F=A7=AA=20Pin=20the=20total=20read?= =?UTF-8?q?=20on=20the=20host-supplied=20side=20too?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- packages/workflow/tests/retained-run.test.ts | 32 +++++++++++++++++--- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts index 8e52ad6f..35327e3c 100644 --- a/packages/workflow/tests/retained-run.test.ts +++ b/packages/workflow/tests/retained-run.test.ts @@ -732,11 +732,6 @@ describe("Tier RR — retained workflow runs", () => { // RR19: an event that refuses to be read is a history this run cannot // describe, not one to step past on the way to a record that does read. - // - // Coverage rather than proof: core's own target admission forces the same - // discriminator and refuses a history like this too, so removing this - // installation's fail-closed handling does not make the test pass. What it - // pins is that the refusal happens and carries nothing. it("RR19: refuses a history holding an event that will not read", function* () { const first = new InMemoryStream(); yield* runRetained(RETAINED, first); @@ -878,6 +873,33 @@ describe("Tier RR — retained workflow runs", () => { } }); + // RR22: the same totality on the other side of the boundary. A host hands the + // retained run over directly, so its members are read here rather than out of + // a journal — and a member that refuses is a value identifying no run, which + // is a sentence rather than the host's own exception. + it("RR22: refuses a retained run whose members refuse to be read", function* () { + const hostile = { + base: "main", + pinnedCommit: COMMIT, + get runId(): never { + throw new Error("PLANTED-INSTALL-GETTER"); + }, + }; + + const thrown = yield* scoped(function* () { + try { + yield* useRetainedWorkflow(hostile as unknown as WorkflowRun); + return undefined; + } catch (error) { + return error; + } + }); + + expect(thrown).toBeInstanceOf(Error); + expect(String(thrown)).toContain("identifies no workflow run"); + expect(wholeError(thrown)).not.toContain("PLANTED-INSTALL-GETTER"); + }); + it("RR7: refuses to install a run that identifies nothing", function* () { const empty = yield* scoped(function* () { try { From 15c24ff27bef93da498522dac751f140e51f2052 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 15:30:05 -0400 Subject: [PATCH 09/10] =?UTF-8?q?=F0=9F=90=9B=20Scope=20the=20appended-eve?= =?UTF-8?q?nt=20watcher=20to=20the=20test=20that=20owns=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- packages/workflow/tests/retained-run.test.ts | 48 ++++++++++---------- 1 file changed, 23 insertions(+), 25 deletions(-) diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts index 35327e3c..d1d78ec0 100644 --- a/packages/workflow/tests/retained-run.test.ts +++ b/packages/workflow/tests/retained-run.test.ts @@ -273,27 +273,27 @@ function shiftingJournal(events: readonly DurableEvent[], reads: string[]): Dura * the wrong thing here: a value that refuses to be read cannot survive being * copied. These events reach the execution as written. */ -function unreadableJournal(events: readonly DurableEvent[]): DurableStream { +interface WatchedJournal { + readonly journal: DurableStream; + /** What a refused run managed to append, which must stay empty. */ + readonly appended: DurableEvent[]; +} + +function unreadableJournal(events: readonly DurableEvent[]): WatchedJournal { const appended: DurableEvent[] = []; - const journal: DurableStream = { - // deno-lint-ignore require-yield - *readAll(): Operation { - return [...events, ...appended]; - }, - // deno-lint-ignore require-yield - *append(event: DurableEvent): Operation { - appended.push(event); + return { + appended, + journal: { + // deno-lint-ignore require-yield + *readAll(): Operation { + return [...events, ...appended]; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, }, }; - appendedEvents.set(journal, appended); - return journal; -} - -const appendedEvents = new WeakMap(); - -/** What a refused run managed to append, which must be nothing. */ -function appendedTo(journal: DurableStream): DurableEvent[] { - return appendedEvents.get(journal) ?? []; } /** @@ -742,10 +742,8 @@ describe("Tier RR — retained workflow runs", () => { }, } as unknown as DurableEvent; - const attempt = yield* runRetained( - RETAINED, - unreadableJournal([refusing, ...first.snapshot()]), - ); + const watched = unreadableJournal([refusing, ...first.snapshot()]); + const attempt = yield* runRetained(RETAINED, watched.journal); expect(attempt.thrown).toBeInstanceOf(Error); expect(attempt.seen).toEqual([]); @@ -807,7 +805,7 @@ describe("Tier RR — retained workflow runs", () => { ]; for (const planted of hostile) { - const journal = unreadableJournal( + const watched = unreadableJournal( first .snapshot() .map((event) => @@ -817,13 +815,13 @@ describe("Tier RR — retained workflow runs", () => { ), ); - const attempt = yield* runRetained(RETAINED, journal); + const attempt = yield* runRetained(RETAINED, watched.journal); expect(attempt.thrown).toBeInstanceOf(StaleInputError); // No terminal result reused, no document code, no output, no append. expect(attempt.seen).toEqual([]); expect(attempt.emitted).toEqual([]); - expect(appendedTo(journal)).toEqual([]); + expect(watched.appended).toEqual([]); // The whole error object, not only its sentence. const whole = wholeError(attempt.thrown); for (const secret of [ From 31f7d3ac744c06d92781ac8f6f1510a24940a458 Mon Sep 17 00:00:00 2001 From: Taras Mankovski <74687+taras@users.noreply.github.com> Date: Mon, 10 Aug 2026 11:07:48 -0400 Subject: [PATCH 10/10] =?UTF-8?q?=F0=9F=9A=80=20Start=20and=20resume=20a?= =?UTF-8?q?=20workflow=20run=20from=20the=20CLI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `xmd workflow start [--id] [--props-*] ` and `xmd workflow resume ` run a document as a retained workflow run: one implicit logical Workspace and one journal in a database that outlives the process, so an interrupted procedure continues from its journal frontier instead of from the beginning. `start` names a document and `resume` names a run, and that asymmetry is the lifecycle rule — a path locates a definition and never selects a previous run, so two starts without `--id` are two runs. What executes is the *committed* document: `start` resolves HEAD once and stores that commit as the run's identity, so uncommitted edits do not change what a run is a run of, and a resume loads the same object through retained, credential-free retrieval metadata rather than the current HEAD or a same-named working-tree file. Identity and outcome go to standard error as two stable lines, leaving stdout to the document. Only a completed run exits zero; failed, suspended, cancelled and interrupted are distinguishable so automation cannot mistake an incomplete workflow for a finished one. The capability lives on one host and the grammar on all of them: the Deno entrypoints own the local run store, and Node and Bun refuse before creating or executing anything. Resolving the host happens before the props phase, because that phase establishes the definition from Git in order to read what the pinned document declares — a host without the capability would otherwise answer with whatever Git said about the caller's directory instead of the reason the command is not going to run. The shared CLI module imports no SQLite, no DOFS and no runtime detection: it asks a host adapter to open storage and attach the run's Workspace. Until #367 supplies durable ownership, a run left running because its host disappeared is closed as an orphaned interrupted execution by the next resume. Nothing here claims concurrent resume is safe. --- .github/workflows/publish-packages.yml | 18 +- README.md | 89 ++++ architecture.md | 99 ++++- bun.lock | 2 + packages/cli/package.json | 1 + packages/cli/src/bun.ts | 3 +- packages/cli/src/cli.ts | 273 +++++++++++- packages/cli/src/compiled.ts | 3 +- packages/cli/src/deno-workflow.ts | 43 ++ packages/cli/src/deno.ts | 3 +- packages/cli/src/node.ts | 3 +- packages/cli/src/workflow-definition.ts | 253 ++++++++++++ packages/cli/src/workflow.ts | 481 ++++++++++++++++++++++ packages/cli/tests/workflow-cli.test.ts | 384 +++++++++++++++++ packages/cli/tests/workflow-crash.test.ts | 253 ++++++++++++ packages/cli/tests/workflow-host.test.ts | 107 +++++ packages/core/mod.ts | 2 + packages/core/src/root-source.ts | 48 ++- packages/runtime/apis.ts | 19 +- packages/runtime/mod.ts | 1 + packages/test-support/launch.ts | 14 + packages/workflow/mod.ts | 13 +- packages/workflow/src/git.ts | 133 +++++- pnpm-lock.yaml | 3 + scripts/runtime-test-exclusions.ts | 12 + site/routes/docs/index.tsx | 10 +- specs/executable-mdx-spec.md | 35 +- specs/workflow-workspace-spec.md | 61 ++- 28 files changed, 2309 insertions(+), 57 deletions(-) create mode 100644 packages/cli/src/deno-workflow.ts create mode 100644 packages/cli/src/workflow-definition.ts create mode 100644 packages/cli/src/workflow.ts create mode 100644 packages/cli/tests/workflow-cli.test.ts create mode 100644 packages/cli/tests/workflow-crash.test.ts create mode 100644 packages/cli/tests/workflow-host.test.ts diff --git a/.github/workflows/publish-packages.yml b/.github/workflows/publish-packages.yml index 8afc866c..c5297191 100644 --- a/.github/workflows/publish-packages.yml +++ b/.github/workflows/publish-packages.yml @@ -30,7 +30,7 @@ jobs: - name: Validate the manifests declare this version run: | VERSION="${{ steps.resolve.outputs.value }}" - for f in packages/durable-streams/deno.json packages/runtime/deno.json packages/core/deno.json packages/acp/deno.json packages/testing/deno.json packages/test-agent/deno.json packages/web/deno.json packages/cli/deno.json packages/code-review-agent/deno.json packages/workflow/deno.json; do + for f in packages/durable-streams/deno.json packages/runtime/deno.json packages/core/deno.json packages/acp/deno.json packages/testing/deno.json packages/test-agent/deno.json packages/web/deno.json packages/workflow/deno.json packages/cli/deno.json packages/code-review-agent/deno.json; do declared="$(jq -r .version "$f")" if [ "$declared" != "$VERSION" ]; then echo "::error::$f declares $declared, not $VERSION — the tag does not match the manifests" @@ -110,8 +110,15 @@ jobs: package: packages/web version: ${{ needs.version.outputs.value }} + workflow: + needs: [version, core, durable-streams, runtime] + uses: ./.github/workflows/publish-one.yml + with: + package: packages/workflow + version: ${{ needs.version.outputs.value }} + cli: - needs: [version, acp, core, durable-streams, runtime, test-agent, testing, web] + needs: [version, acp, core, durable-streams, runtime, test-agent, testing, web, workflow] uses: ./.github/workflows/publish-one.yml with: package: packages/cli @@ -124,13 +131,6 @@ jobs: package: packages/code-review-agent version: ${{ needs.version.outputs.value }} - workflow: - needs: [version, core, durable-streams, runtime] - uses: ./.github/workflows/publish-one.yml - with: - package: packages/workflow - version: ${{ needs.version.outputs.value }} - jsr: needs: [version] runs-on: ubuntu-latest diff --git a/README.md b/README.md index dee889f9..08b20685 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,95 @@ Useful flags: - `--verbose`, `-V` - print durable journal entries to stderr while running. - `--component-dir` - add component search directories. Defaults to `components` and `.`. +## Run a document as a workflow + +`xmd run` executes against your own filesystem and promises nothing afterwards. +`xmd workflow` executes against a **run**: one retained Workspace and one +journal, in a database that outlives the process, so an interrupted procedure +continues from where it stopped rather than from the beginning. + +```bash +xmd workflow start flows/prepare-release.md +xmd workflow start --id=release-1.4 --props-channel=stable flows/prepare-release.md +xmd workflow resume release-1.4 +``` + +`start` names a document; `resume` names a run. A document path locates a +definition and never selects a previous run, so starting the same document twice +makes two runs. `resume` takes no document and no properties: it uses the ones +its run retained. + +What the run executes is the **committed** document. `start` resolves `HEAD` +once and stores that commit as the run's identity, so uncommitted edits in your +working tree do not change what a run is a run of, and a resume months later +loads the same object rather than whatever the file says now. + +Inside a run, `` and `` name entries in the run's own logical +filesystem rather than yours. Each read, write and search is one durable effect: +the change, the Workspace version it produces and the journal entry commit +together, so a crash leaves all three or none, and a resume restores what +already happened instead of doing it again. Operations a run does not have — +a temporary directory, a native service — fail explicitly rather than reaching +your machine. + +Identity and outcome go to standard error, so piping stdout still gives you the +document: + +```text +workflow run: release-1.4 +workflow status: completed +``` + +Only a completed run exits `0`. Failed exits `1`, suspended `2`, cancelled `3` +and interrupted `130`, so a script cannot mistake an incomplete workflow for a +finished one. + +Runs live under `~/.xmd/runs`; set `XMD_WORKFLOW_RUNS` to an absolute path to +keep them somewhere else. The command is available through the Deno entrypoint +and the compiled binary; under Node and Bun it reports that and does nothing. + +Status, list, history, cancel and fork are designed but not yet shipped. + +## Run a workflow + +`xmd run` executes against the directory you are in and promises nothing +afterwards. `xmd workflow` executes against a *run*: one retained Workspace and +one filtered journal, in a database that outlives the process, so an interrupted +procedure resumes from where it stopped instead of starting again. + +```bash +xmd workflow start flows/prepare-release.md +xmd workflow start --id=release-1.4 --props-channel=stable flows/prepare-release.md +xmd workflow resume release-1.4 +``` + +`start` names a document; `resume` names a run. A document path locates a +definition and never selects a previous run, so starting the same document twice +without `--id` creates two runs. Reusing an `--id` addresses the same run when +the definition, base and normalized properties all agree, and is refused when +any of them differ. + +What a run is, is a Git object: the repository containing the document, the +commit `HEAD` resolves to, and the document's path inside it. **The committed +document runs**, so uncommitted edits in your working tree do not change what a +run is a run of, and a resume months later means the same thing it did. + +Inside a run, `` and `` name entries in the run's own logical +filesystem rather than yours. Each read, write and search is one durable effect: +the mutation, the Workspace root it produces and the journal result commit +together, and a resume restores what was recorded instead of doing it again. +Operations a run does not have — a temporary directory, a native service — fail +explicitly rather than reaching your machine. + +Runs live under `~/.xmd/runs`; set `XMD_WORKFLOW_RUNS` to an absolute directory +to keep them somewhere else. Only a completed run exits `0` — failed exits `1`, +suspended `2`, cancelled `3` and interrupted `130` — and the run id and final +status are written to standard error as `workflow run: ` and +`workflow status: `, so standard output stays the document's own. + +`xmd workflow` is available through the Deno entrypoint and the compiled binary. +Under Node and Bun the command exists and refuses before creating anything. + ## Coding agents Run ACP-compatible coding agents directly from a document with ``, diff --git a/architecture.md b/architecture.md index ccd371be..27942252 100644 --- a/architecture.md +++ b/architecture.md @@ -149,9 +149,8 @@ The `@executablemd/workflow` package owns `WorkflowRun`, `useWorkflow()`, `getWorkflowRun()` and the Git capability. It depends on `@executablemd/core`, `@executablemd/durable-streams` and `@executablemd/runtime`, whose contextual `exec()` and `cwd()` the Git provider invokes; core never imports workflow or -Git. The future CLI lifecycle is `xmd workflow start` and `xmd workflow resume`; -there is no workflow CLI execution branch yet. The durable lookup that resume -will require is the run storage below. Ordinary `xmd run` remains unchanged. +Git. `xmd workflow start` and `xmd workflow resume` are the CLI lifecycle, and +they resume through the run storage below. Ordinary `xmd run` remains unchanged. ## Workflow run storage @@ -347,6 +346,68 @@ describe what failed without repeating retained props or journal payloads — including their member *names*, which can carry a credential as readily as a member value can. +## The workflow lifecycle + +`xmd workflow start [--id=] [--props-*=…] ` and +`xmd workflow resume ` are the two commands. Both run in the foreground, +stream the document's own output to standard output, and report identity and +outcome on standard error as two stable lines — `workflow run: ` once +the run has been created or found, and `workflow status: ` once the +execution settles. Only a completed run exits zero: failed exits 1, suspended 2, +cancelled 3 and interrupted 130, so shell automation cannot mistake an +incomplete workflow for a finished one. A request the command refuses — bad +grammar, a missing run, an incompatible reuse, damaged storage, an unsupported +host — exits 1. + +`start` establishes an immutable definition from Git rather than identifying a +working-tree file. It locates the repository containing the supplied path, +resolves `HEAD^{commit}` once because the command has no base option, reads the +repository's object format, and stores version 1 of the descriptor with that +format, the lowercase commit ID and the normalized repository-relative POSIX +path. **The bytes that execute are the ones that commit holds**, so a working +tree with uncommitted edits runs the committed document. Where the repository is +checked out is retrieval metadata: replaceable, credential-free, excluded from +identity, and reauthorized before it is used again. `resume` loads exactly the +retained object through that locator and never substitutes the current `HEAD` or +a same-named working-tree file. A missing object, missing or unreadable +retrieval metadata, a path outside the repository, or a root that is not +Markdown fails explicitly; none of them creates a replacement run or an empty +definition. A workflow definition is one immutable object, so the component +search path is empty and a repository component fails to resolve rather than +resolving to content beside the definition in a mutable checkout. + +Every actual execution opens or creates the run's database, begins a +document-execution record, installs the exact retained WorkflowRun, installs the +service denial before the root is imported, and — for a live or partial +execution — installs the logical working directory `/`, the transaction-bound +Files provider and that database's Workspace effect coordinator. It executes +against the database's own journal with the retained props and secret detection, +then finishes the execution record and publishes the run's status. A completed +run still replays, so its retained output and result are emitted, but it +attaches no Workspace provider or coordinator and performs no filesystem +mutation. + +A failure retains `failed` and uses a journal stop reason when a retained event +identifies it, and a categorical host code otherwise; no exception text is +retained beside the journal that filtered it. Graceful foreground interruption +finishes the execution `interrupted` and exits 130. `suspended` is resumable; +`failed` and `cancelled` are refused by `resume`, and `completed` replays under +either command. + +The initial host is Deno-local. The Deno entrypoints — source and the compiled +binary — install the local run store, beneath `~/.xmd/runs` unless +`XMD_WORKFLOW_RUNS` names another absolute directory. Node and Bun expose the +same grammar and refuse before creating or executing anything. The shared CLI +module imports no SQLite, no DOFS and no runtime detection: it asks a host +adapter to open storage and to attach a run's Workspace, and the entrypoints +decide which adapter exists. + +Durable ownership and concurrent-executor enforcement belong to #367. Until it +lands, a run left `running` because its host disappeared is treated as an +orphaned interrupted execution by the next resume, which closes that unfinished +execution record as `interrupted` before beginning its own. Nothing here claims +concurrent resume is safe. + ## Workflow Workspace The command selects the environment; the document describes the procedure. @@ -475,8 +536,8 @@ commit form one boundary. The Workspace filesystem uses the pinned synchronous D entry points for its string and byte-array surface, so cancellation leaves no eager promise or stream pull able to reach the connection after transaction authority ends. The transaction-bound Files provider selects that operation for -every document filesystem read, write and search; workflow lifecycle commands do -not select it yet. +every document filesystem read, write and search, and the workflow lifecycle +commands install that provider. An external provider cannot join that transaction. Prompt, Git push and pull request effects derive a stable identity from the run and expansion, ask the @@ -552,8 +613,8 @@ collection is not in the production closure and is never invoked. The provider exposes no public history selection or fork operation at this layer. Its coordinator combines one mutation, immutable-root publication and one filtered journal result atomically, and the transaction-bound Files provider is what -routes a document's `` and `` to it. Workflow start and resume do -not reach it yet. +routes a document's `` and `` to it. `xmd workflow start` and +`xmd workflow resume` install that provider around each execution. The coordinator treats only errors produced through its private filesystem adapter's documented path and mutation refusals as journalable operation @@ -1022,12 +1083,12 @@ consumed. If collision handling terminates immediately, receives no terminal event; restoring the compatible definition can still replay it. -#390 provides and tests the non-delegating `useWorkflowServiceDenial()` provider. -#366 will install it in the future `xmd workflow start` and `xmd workflow resume` -scopes. No workflow CLI execution branch exists yet. The provider prevents a -workflow from reaching an inherited host adapter, because a run-owned durable -service requires stable identity and reconciliation rather than an -execution-owned live process. +`xmd workflow start` and `xmd workflow resume` install the non-delegating +`useWorkflowServiceDenial()` provider inside each execution scope, before the +root document is imported — in the same place `xmd run` installs its host +service adapter. The provider prevents a workflow from reaching an inherited +host adapter, because a run-owned durable service requires stable identity and +reconciliation rather than an execution-owned live process. ## State ownership @@ -1127,7 +1188,7 @@ Status is measured against main. | `useWorkflow()` / `getWorkflowRun()` | associates one document execution with a workflow run | built on main | | `useRetainedWorkflow()` | associates one document execution with a run storage already created, requiring exact journal agreement | built on the #366 stack | | `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main | -| workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; public workflow execution is unbuilt | +| workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; the CLI lifecycle reaches it on the #366 stack | | caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in | built on main | | live durable-operation coordinator | explicitly coordinates structured live execution with existing Yield publication while leaving replay and callback effects unchanged | built on the #365 stack | | Workspace coordination API | fails closed by default; replaceable context routes only a one-use provider selection, while the selected provider directly invokes an execution-owned credentialed capability for execution, publication and failure activation | built on the #365 stack; the Deno provider installs an adapter-private atomic handler | @@ -1135,16 +1196,16 @@ Status is measured against main. | `API.Service` / `startService()` | creates an authenticated, supervised loopback service attachment through a provider-neutral operation | built on main | | `API.Files` | routes every document filesystem operation to the installed provider, with no host default and structural failure data | built on the #227 stack | | host Files provider / `useHostFiles()` | resolves document paths in the caller's filesystem, containing them while the host namespace is stable; installed by all four CLI entrypoints | built on the #227 stack | -| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | built on the #366 stack; CLI reachability remains unbuilt | +| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | built on the #366 stack | | `service=` | publishes the attachment's endpoint into the live binding overlay for its invocation | built on main | | `ephemeral eval` | reconstructs live middleware and bindings without a journal entry | built on main | -| `useWorkflowServiceDenial()` | provides and tests a non-delegating workflow service denial provider; #366 will install it in future start and resume scopes | built on main; no workflow CLI execution branch exists yet | -| `xmd workflow start` / `xmd workflow resume` | starts or resumes a workflow run from the CLI | defined in `specs/workflow-workspace-spec.md`, unbuilt; the lookup it resumes through is built | -| implicit workflow Workspace | retains provider-neutral filesystem, repository and attachment state by run ID | defined in `specs/workflow-workspace-spec.md`, unbuilt (#218) | +| `useWorkflowServiceDenial()` | provides a non-delegating workflow service denial provider, installed inside every start and resume execution scope | built on the #366 stack | +| `xmd workflow start` / `xmd workflow resume` | starts or resumes a workflow run from the CLI, under the Deno entrypoints only | built on the #366 stack; status, list, history, cancel, fork and delete are unbuilt | +| implicit workflow Workspace | retains provider-neutral filesystem, repository and attachment state by run ID | document filesystem built on the #366 stack; repository, process and attachment capabilities unbuilt (#218) | | Repository / Worktree / transactional Git effects | compose named checkouts and publish local mutations with their journal result | defined in `specs/workflow-workspace-spec.md`, unbuilt | | workflow inspection and history fork | reads status/history without advancing a run and creates a new run from a checkpoint | defined in `specs/workflow-workspace-spec.md`, unbuilt | | read-only workflow Agent / generated XMD | lets an Agent inspect a derived view and propose constrained executable changes | defined in `specs/workflow-workspace-spec.md`, unbuilt | -| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates one Workspace mutation with its filtered Yield | built on the #365 stack; public document filesystem effects route to it on the #366 stack, and workflow lifecycle reachability is unbuilt | +| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates one Workspace mutation with its filtered Yield | built on the #365 stack; public document filesystem effects and the CLI lifecycle route to it on the #366 stack | | scoped Worker Shell | executes `just-bash` through the Workspace adapter inside a Deno Worker | containment and effect-transaction POCs complete (#351, #357); production integration unbuilt | | `` | retry a region until it completes | defined, unbuilt | | suspension effect | suspend durably | defined, unbuilt | diff --git a/bun.lock b/bun.lock index 69f14894..794de04a 100644 --- a/bun.lock +++ b/bun.lock @@ -74,6 +74,7 @@ "@executablemd/test-agent": "workspace:*", "@executablemd/testing": "workspace:*", "@executablemd/web": "workspace:*", + "@executablemd/workflow": "workspace:*", "@standard-schema/spec": "^1.0.0", "configliere": "^0.4.0", "effection": "4.1.0", @@ -211,6 +212,7 @@ "dependencies": { "@effectionx/context-api": "0.6.0", "@effectionx/fs": "0.3.0", + "@effectionx/process": "0.8.1", "@executablemd/core": "workspace:*", "@executablemd/durable-streams": "workspace:*", "@executablemd/runtime": "workspace:*", diff --git a/packages/cli/package.json b/packages/cli/package.json index af34a288..fe1954f5 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -19,6 +19,7 @@ "@executablemd/test-agent": "workspace:*", "@executablemd/testing": "workspace:*", "@executablemd/web": "workspace:*", + "@executablemd/workflow": "workspace:*", "@standard-schema/spec": "^1.0.0", "configliere": "^0.4.0", "effection": "4.1.0", diff --git a/packages/cli/src/bun.ts b/packages/cli/src/bun.ts index 6f8ba15f..3a76f161 100644 --- a/packages/cli/src/bun.ts +++ b/packages/cli/src/bun.ts @@ -11,6 +11,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileDataUri } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { unsupportedWorkflowHost } from "./workflow.ts"; import { useBunService } from "./bun-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -35,5 +36,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useBunService); + yield* runXmd(args, useBunService, unsupportedWorkflowHost); }); diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index afb66366..9b48b3a0 100755 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -5,9 +5,12 @@ * xmd run [options] * xmd [options] (run is the default command) * xmd targets + * xmd workflow start [options] + * xmd workflow resume * * A document reference is a path, optionally followed by `#` and one target - * selector naming a section of the document (spec §5.4). + * selector naming a section of the document (spec §5.4). `workflow start` takes + * a plain path: a definition descriptor cannot record a target yet. * * Examples: * xmd run packages/core/examples/hello-world.md @@ -15,6 +18,8 @@ * xmd run packages/core/examples/hello-world.md --journal events.jsonl * xmd targets README.md * xmd run README.md#Release/Publish + * xmd workflow start --id=release-1.4 flows/prepare-release.md + * xmd workflow resume release-1.4 */ import { @@ -55,6 +60,7 @@ import { installAgentComponents, installPermissionMode, registerAgentProvider, + retainedSource, rootSourcePath, useNormalizedOutput, useTerminalOutput, @@ -82,6 +88,17 @@ import type { Binding, Extraction } from "./props.ts"; import { componentSearchPath, resolveTestTarget } from "./test-target.ts"; import { EVAL_ALIAS, EVAL_OPTION, evalGrammarError, readEvalFlags } from "./eval-source.ts"; import type { EvalFlags } from "./eval-source.ts"; +import { + parseWorkflowRequest, + runWorkflow, + UNSUPPORTED_WORKFLOW_HOST, + unsupportedWorkflowHost, + workflowConfig, +} from "./workflow.ts"; +import type { HostWorkflowInstaller, WorkflowHost, WorkflowStart } from "./workflow.ts"; +import { establishDefinition } from "./workflow-definition.ts"; +import type { EstablishedDefinition } from "./workflow-definition.ts"; +import { useWorkflowServiceDenial } from "@executablemd/workflow"; import denoJson from "../deno.json" with { type: "json" }; const SECRET_DETECTION_OPTION = "--secret-detection"; @@ -228,6 +245,7 @@ const xmd = program({ test: testConfig, targets: targetsConfig, "test-agent": testAgentConfig, + workflow: workflowConfig, }, { default: "run" }, ), @@ -427,12 +445,22 @@ interface DocumentConfig { raw: boolean; /** Whether this document's durable events are scanned before they persist. */ secretDetection: boolean; + /** + * The journal this execution reads and appends, when the caller owns one. + * + * A workflow run does: its journal is the run's retained history, and + * replacing it with a fresh stream would make every execution a first one. + * `xmd run` supplies none and gets the empty stream below. + */ + stream?: DurableStream; } interface DocumentMode { testing: boolean; agent?: AgentFlags; props?: Record; + /** Installed inside the execution scope, before the root document is imported. */ + install?: () => Operation; } export type HostServiceInstaller = () => Operation; @@ -452,11 +480,14 @@ function* runDocument( ): Operation> { const { root, componentDir, verbose, journal, raw, secretDetection } = config; - // Every CLI invocation starts from an empty stream. --journal writes - // current-run diagnostics only; existing traces are never loaded. + // Every CLI invocation starts from an empty stream unless the caller owns + // one. --journal writes current-run diagnostics only; existing traces are + // never loaded. let stream: DurableStream; - if (journal) { + if (config.stream) { + stream = config.stream; + } else if (journal) { yield* createJournalFile(journal); stream = new FileStream(journal); } else { @@ -531,6 +562,13 @@ function* runDocument( // alone. Reading the mode costs no document effects. const valueRoot = !mode.testing && (yield* readsValue(root)); + // Installed in the execution's own scope, before the root is imported: a + // workflow run's retained identity has to be readable by everything the + // document reaches, and has to be there before the first durable effect. + if (mode.install) { + yield* mode.install(); + } + // Native service authority belongs only to document execution. Help, // document inspection, and the agent worker never enter this scope. // @@ -961,6 +999,15 @@ interface PropsPhase { propsSchema?: unknown; declared?: string[]; error?: string; + /** + * The immutable definition a `workflow start` established, when it did. + * + * Established here rather than later because the props a run is created with + * are the ones the *pinned* document declares: reading the working tree to + * build the bindings and then executing the commit would let help and parsing + * describe a document that is not the one running. + */ + established?: EstablishedDefinition; } /** @@ -980,6 +1027,10 @@ function* preparePropsPhase(args: string[], evalFlags: EvalFlags): Operation { + if (inlineDocument !== undefined) { + return { + args, + bindings: [], + error: `unrecognized option for xmd workflow: ${EVAL_OPTION} — inline documents are exclusive to xmd run`, + }; + } + + const stray = findPropsFlag(args); + if (config.action !== "start") { + if (stray) { + return { + args, + bindings: [], + error: + `unrecognized option for xmd workflow ${config.action ?? "resume"}: ${stray} — a resume ` + + "runs the props its run retained", + }; + } + return { args, bindings: [] }; + } + + if (config.target === undefined || config.target === "") { + return { args, bindings: [] }; + } + + const established = yield* establishDefinition(config.target); + if (!established.ok) { + return { args, bindings: [], error: established.error.message }; + } + + const root = retainedSource( + established.value.definition.rootDocumentPath, + established.value.source, + ); + try { + const document = yield* inspectDocument(root); + const bindings = buildBindings(document.props); + const extraction = extractPropsArgs(args, bindings); + return { + args: extraction.rest, + root, + bindings, + extraction, + propsSchema: document.props, + declared: declaredProperties(document.props), + established: established.value, + }; + } catch (error) { + return { args, bindings: [], root, error: describeError(error) }; + } +} + +/** + * Whether these arguments select the `workflow` command. + * + * Read from argv rather than from a parse, because the answer is needed before + * the props phase and the props phase is what makes a parse meaningful. + * Whatever the command turns out to be, only `workflow` names it first. + */ +function namesWorkflow(args: string[]): boolean { + return args[0] === "workflow"; +} + +/** + * A third positional argument to `xmd workflow`, when one was written. + * + * The parser stops at the first token it does not define rather than rejecting + * it, so `xmd workflow resume ` would otherwise run the resume + * and silently ignore the document — exactly the confusion the lifecycle rule + * exists to prevent, since a document never selects a run. + * + * Read from the argv the props phase already stripped, so a generated property + * value is not mistaken for an argument. `--id` is the one option that takes a + * separated value, and tokens after `--` belong to the document. + */ +function extraWorkflowArgument(args: string[]): string | undefined { + const start = args.indexOf("workflow"); + if (start === -1) { + return undefined; + } + let positionals = 0; + let skip = false; + for (const arg of args.slice(start + 1)) { + if (arg === "--") { + return undefined; + } + if (skip) { + skip = false; + continue; + } + if (arg.startsWith("-")) { + skip = arg === "--id"; + continue; + } + positionals += 1; + if (positionals > 2) { + return arg; + } + } + return undefined; +} + +const COMMAND_NAMES = ["run", "test", "targets", "test-agent", "workflow"]; /** * What a caller has to know to write a filename that contains reference @@ -1174,7 +1341,15 @@ function* resolveRunProps( * `process.stdout` and `node:fs/promises`. Routing those through contextual * APIs is #156. */ -export function* runXmd(args: string[], installService: HostServiceInstaller): Operation { +export function* runXmd( + args: string[], + installService: HostServiceInstaller, + // Defaults to the host that refuses. A caller driving this without naming a + // workflow host has no run store, and inheriting one by omission is the + // failure mode the whole boundary exists to prevent — so the default is the + // one that creates and executes nothing. + installWorkflowHost: HostWorkflowInstaller = unsupportedWorkflowHost, +): Operation { // First, so that no later scanner — help, properties, agent flags — can // mistake the inline document's own text for an option. const evalFlags = readEvalFlags(args); @@ -1186,6 +1361,24 @@ export function* runXmd(args: string[], installService: HostServiceInstaller): O } const helpRequest = takeHelpFlag(evalFlags.rest); + + // Before the props phase, because that phase establishes a workflow start's + // definition from Git in order to read what the *pinned* document declares. + // On a host without workflow support the first thing a caller would otherwise + // see is whatever Git said about their directory, which is not the reason the + // command is not going to run. Help is exempt: the grammar is the same + // everywhere, and describing it costs nothing. + let workflowHost: WorkflowHost | undefined; + if (!helpRequest.requested && namesWorkflow(helpRequest.args)) { + try { + workflowHost = yield* installWorkflowHost(); + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + yield* exit(1); + return; + } + } + const propsPhase = yield* preparePropsPhase(helpRequest.args, evalFlags); if (propsPhase.error) { @@ -1320,5 +1513,73 @@ export function* runXmd(args: string[], installService: HostServiceInstaller): O case "test-agent": yield* runTestAgentWorker({ connect: command.config.connect }); break; + case "workflow": { + const config = command.config; + const agentFlag = findAgentOnlyFlag(evalFlags.rest); + if (agentFlag) { + console.error( + `unrecognized option for xmd workflow: ${agentFlag} — agent options are exclusive to xmd run`, + ); + yield* exit(1); + break; + } + const extra = extraWorkflowArgument(propsPhase.args); + if (extra !== undefined) { + console.error( + `unrecognized argument for xmd workflow: ${extra} — start names one definition and ` + + "resume names one run", + ); + yield* exit(1); + break; + } + if (workflowHost === undefined) { + console.error(UNSUPPORTED_WORKFLOW_HOST); + yield* exit(1); + break; + } + const request = parseWorkflowRequest(config); + if (!request.ok) { + console.error(request.error.message); + yield* exit(1); + break; + } + const props = yield* resolveRunProps(propsPhase); + if (props.error) { + console.error(props.error); + yield* exit(1); + break; + } + const start: WorkflowStart | undefined = + propsPhase.established === undefined + ? undefined + : { established: propsPhase.established, props: props.value ?? {} }; + announceSecretDetection(config.secretDetection); + const outcome = yield* runWorkflow(request.value, start, workflowHost, (execution) => + execution.around( + runScopedDocument( + { + root: execution.root, + // A workflow definition is one immutable object. A component + // search path would read the mutable checkout beside it, so a + // repository component fails to resolve rather than resolving + // to content the definition does not describe. + componentDir: [], + verbose: request.value.verbose, + journal: undefined, + raw: request.value.raw, + secretDetection: request.value.secretDetection, + stream: execution.stream, + }, + { testing: false, props: execution.props, install: execution.install }, + // The workflow authority boundary sits exactly where a host + // service adapter would: installed inside the execution scope, + // before the root document is imported. + useWorkflowServiceDenial, + ), + ), + ); + yield* exit(outcome.exitCode); + break; + } } } diff --git a/packages/cli/src/compiled.ts b/packages/cli/src/compiled.ts index e1abf66a..7c1f2377 100644 --- a/packages/cli/src/compiled.ts +++ b/packages/cli/src/compiled.ts @@ -10,6 +10,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileDataUri } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { useDenoWorkflowHost } from "./deno-workflow.ts"; import { useCompiledService } from "./compiled-service.ts"; await main(function* (args) { @@ -32,5 +33,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useCompiledService); + yield* runXmd(args, useCompiledService, useDenoWorkflowHost); }); diff --git a/packages/cli/src/deno-workflow.ts b/packages/cli/src/deno-workflow.ts new file mode 100644 index 00000000..c86e8373 --- /dev/null +++ b/packages/cli/src/deno-workflow.ts @@ -0,0 +1,43 @@ +/** + * The Deno workflow host — where `xmd workflow` keeps runs, and what it attaches. + * + * This is the only module in the CLI that names the local run store. SQLite, + * DOFS and the path a run lives at are all behind `@executablemd/workflow/deno`, + * and the shared command module reaches none of them: it asks a `WorkflowHost` + * to open storage and to attach a run's Workspace, and this is what Deno and the + * compiled binary supply. The binary is Deno too, so both entrypoints install + * this one rather than each carrying a copy. + * + * Runs live beneath `~/.xmd/runs` by default. `XMD_WORKFLOW_RUNS` names a + * different absolute directory, which is how a test — or a caller keeping one + * project's runs apart from another's — works without the real user state + * directory being involved at all. + */ + +import { homedir } from "node:os"; +import { join } from "node:path"; +import type { Operation } from "effection"; +import { env as readEnv } from "@executablemd/runtime"; +import { useWorkflowRunStorage, withWorkflowWorkspace } from "@executablemd/workflow/deno"; +import type { WorkflowRunDatabase } from "@executablemd/workflow"; +import type { WorkflowHost } from "./workflow.ts"; + +/** Where a run lives when nothing says otherwise. */ +export const DEFAULT_RUN_STORAGE_ROOT: string = join(homedir(), ".xmd", "runs"); + +/** The variable that names a different run store. Absolute, or it is refused. */ +export const RUN_STORAGE_ROOT_ENV = "XMD_WORKFLOW_RUNS"; + +export function* useDenoWorkflowHost(): Operation { + const configured = yield* readEnv(RUN_STORAGE_ROOT_ENV); + const root = + configured === undefined || configured === "" ? DEFAULT_RUN_STORAGE_ROOT : configured; + return { + useStorage(): Operation { + return useWorkflowRunStorage({ root }); + }, + attach(database: WorkflowRunDatabase, operation: Operation): Operation { + return withWorkflowWorkspace(database, operation); + }, + }; +} diff --git a/packages/cli/src/deno.ts b/packages/cli/src/deno.ts index 4415aebc..b0ed4e73 100644 --- a/packages/cli/src/deno.ts +++ b/packages/cli/src/deno.ts @@ -13,6 +13,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileDataUri } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { useDenoWorkflowHost } from "./deno-workflow.ts"; import { useDenoService } from "./deno-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -37,5 +38,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useDenoService); + yield* runXmd(args, useDenoService, useDenoWorkflowHost); }); diff --git a/packages/cli/src/node.ts b/packages/cli/src/node.ts index 878ab6fc..6a34daa7 100755 --- a/packages/cli/src/node.ts +++ b/packages/cli/src/node.ts @@ -17,6 +17,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileTempFile } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { unsupportedWorkflowHost } from "./workflow.ts"; import { useNodeService } from "./node-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -42,5 +43,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useNodeService); + yield* runXmd(args, useNodeService, unsupportedWorkflowHost); }); diff --git a/packages/cli/src/workflow-definition.ts b/packages/cli/src/workflow-definition.ts new file mode 100644 index 00000000..2e8159b0 --- /dev/null +++ b/packages/cli/src/workflow-definition.ts @@ -0,0 +1,253 @@ +/** + * What a workflow run is a run of, established from Git. + * + * `xmd workflow start notes.md` names a file in a working tree. A working tree + * changes, so it cannot be a run's identity — a resume months later has to mean + * the same document. What becomes identity is the object: the repository's + * object format, the full commit id `HEAD` resolved to once, and the document's + * repository-relative path inside it. + * + * The consequence is the part worth stating plainly. **The bytes that execute + * come from that commit, not from the file the caller pointed at.** A working + * tree with uncommitted edits runs the committed document, because running the + * edited one while recording the commit as identity would make the record a + * claim about something that never ran. + * + * Where the repository is *checked out* is not identity. It is retrieval + * metadata: replaceable, credential-free, excluded from the comparison that + * decides whether a reused run id addresses the same run, and reauthorized + * before it is used again. A run that moves between machines is the same run. + * + * Everything here goes through the contextual `Git` capability, so nothing + * below runs a command of its own or names a host. + */ + +import { isAbsolute, relative, resolve, sep } from "node:path"; +import { Err, Ok, scoped } from "effection"; +import type { Operation, Result } from "effection"; +import type { Json } from "@executablemd/durable-streams"; +import { API } from "@executablemd/runtime"; +import { + gitObjectFormat, + parseWorkflowDefinition, + readGitObject, + repositoryRoot, + revParse, +} from "@executablemd/workflow"; +import type { GitWorkflowDefinitionV1, WorkflowDefinition } from "@executablemd/workflow"; + +/** The base a `start` records. The command has no base option, so it is this. */ +export const DEFINITION_BASE = "HEAD"; + +/** How this host will find the definition again. Replaceable, never a credential. */ +export const RETRIEVAL_KIND = "local-checkout"; + +/** Everything one `start` establishes before a run can exist. */ +export interface EstablishedDefinition { + readonly definition: WorkflowDefinition; + readonly base: string; + readonly pinnedCommit: string; + readonly retrieval: Json; + /** The document as the pinned commit holds it. */ + readonly source: string; +} + +/** A definition that cannot be established, or a retained one that cannot be loaded. */ +export class WorkflowDefinitionUnavailableError extends Error { + override name = "WorkflowDefinitionUnavailableError"; +} + +function unavailable(message: string, cause?: unknown): WorkflowDefinitionUnavailableError { + return new WorkflowDefinitionUnavailableError(message, cause === undefined ? {} : { cause }); +} + +/** + * Run `body` with the repository as the contextual working directory. + * + * Git answers about the directory it is asked in, so every question about one + * repository is asked from the same place rather than from wherever the process + * started. Keeping Git's own output out of the caller's is the capability's + * own business and is done there. + */ +function inRepository(directory: string, body: () => Operation): Operation { + return scoped(function* () { + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return directory; + }, + }, + { at: "min" }, + ); + return yield* body(); + }); +} + +/** + * The document's path inside its repository, as a definition may hold it. + * + * Repository-relative, POSIX-separated, and refused rather than repaired when + * it leaves the working tree: a path outside the repository names no object in + * the commit, and normalizing one would silently run a different document. + */ +function repositoryRelativePath(root: string, documentPath: string): Result { + const absolute = resolve(documentPath); + const within = relative(resolve(root), absolute); + if (within === "" || within.startsWith("..") || isAbsolute(within)) { + return Err( + unavailable( + "the document is not inside the repository this command resolved, so no commit in it " + + "holds the document. Run the command from the repository the document belongs to.", + ), + ); + } + return Ok(within.split(sep).join("/")); +} + +/** + * Establish the immutable definition of a run that is starting. + * + * The order matters: the repository is located from the document's own + * directory, `HEAD` is resolved once, and the object format is read from the + * same repository — so a definition never mixes one repository's commit with + * another's format. + */ +export function* establishDefinition( + documentPath: string, +): Operation> { + const absolute = resolve(documentPath); + try { + const root = yield* inRepository(absolute.slice(0, absolute.lastIndexOf(sep)) || sep, () => + repositoryRoot(), + ); + + return yield* inRepository(root, function* (): Operation> { + const rootDocumentPath = repositoryRelativePath(root, absolute); + if (!rootDocumentPath.ok) { + return rootDocumentPath; + } + + const pinnedCommit = yield* revParse(`${DEFINITION_BASE}^{commit}`); + const objectFormat = yield* gitObjectFormat(); + + const definition = parseWorkflowDefinition({ + version: 1, + kind: "git", + objectFormat, + objectId: pinnedCommit.toLowerCase(), + rootDocumentPath: rootDocumentPath.value, + }); + if (!definition.ok) { + return definition; + } + + const source = yield* readGitObject(pinnedCommit, rootDocumentPath.value); + return Ok({ + definition: definition.value, + base: DEFINITION_BASE, + pinnedCommit, + retrieval: { version: 1, kind: RETRIEVAL_KIND, checkout: root }, + source, + }); + }); + } catch (error) { + return Err( + unavailable( + `the workflow definition could not be established from ${documentPath}: ` + + (error instanceof Error ? error.message : String(error)), + error, + ), + ); + } +} + +/** + * The checkout a retained locator names, reauthorized before it is used. + * + * A retained path is replaceable metadata rather than permission a host already + * has, so it is checked against the repository it claims to be: a directory + * that is no longer a working tree, or is now a different one, fails rather + * than quietly resolving the definition somewhere else. + */ +function parseRetrieval(metadata: Json | undefined): Result { + if (typeof metadata !== "object" || metadata === null || Array.isArray(metadata)) { + return Err( + unavailable( + "this run retains no usable retrieval metadata, so its definition cannot be located. " + + "The run is left exactly as it is.", + ), + ); + } + const record = Object.fromEntries(Object.entries(metadata)); + if (record.kind !== RETRIEVAL_KIND || typeof record.checkout !== "string") { + return Err( + unavailable( + "this run's retrieval metadata does not describe a local checkout this host can " + + "reach. The run is left exactly as it is.", + ), + ); + } + return Ok(record.checkout); +} + +/** + * Load the exact object a retained run's definition names. + * + * It never substitutes the current `HEAD` or a same-named file in the working + * tree. A resume that could do either would silently continue a different + * document under the same run id. + */ +export function* loadRetainedDefinition( + definition: WorkflowDefinition, + metadata: Json | undefined, +): Operation> { + const checkout = parseRetrieval(metadata); + if (!checkout.ok) { + return checkout; + } + + try { + return yield* inRepository(checkout.value, function* (): Operation> { + const root = yield* repositoryRoot(); + if (resolve(root) !== resolve(checkout.value)) { + return Err( + unavailable( + "the checkout this run retains is no longer the root of the repository it names. " + + "The run is left exactly as it is.", + ), + ); + } + const format = yield* gitObjectFormat(); + if (format !== definition.objectFormat) { + return Err( + unavailable( + "the retained checkout names its objects with a different format than this run's " + + "definition. The run is left exactly as it is.", + ), + ); + } + return Ok(yield* readGitObject(definition.objectId, definition.rootDocumentPath)); + }); + } catch (error) { + return Err( + unavailable( + "this run's retained definition could not be loaded: " + + (error instanceof Error ? error.message : String(error)), + error, + ), + ); + } +} + +/** Whether this definition names a document this slice can execute. */ +export function supportedRootDocument(definition: GitWorkflowDefinitionV1): Result { + if (definition.rootDocumentPath.endsWith(".md")) { + return Ok(undefined); + } + return Err( + unavailable( + "xmd workflow runs Markdown definitions. A function-component root is not supported yet.", + ), + ); +} diff --git a/packages/cli/src/workflow.ts b/packages/cli/src/workflow.ts new file mode 100644 index 00000000..534237e7 --- /dev/null +++ b/packages/cli/src/workflow.ts @@ -0,0 +1,481 @@ +/** + * `xmd workflow start` and `xmd workflow resume` — running a document as a + * retained workflow run. + * + * The command selects the environment; the document describes the procedure. + * `xmd run` executes against the caller's own filesystem and promises nothing + * afterwards. These two execute against a run: one implicit logical Workspace, + * one filtered journal, both in one database that outlives the process, so an + * interrupted procedure continues from its journal frontier rather than from + * the beginning. + * + * ```sh + * xmd workflow start [--id=] [--props-*=…] + * xmd workflow resume + * ``` + * + * `start` names a document; `resume` names a run. That asymmetry is the whole + * lifecycle rule: a document path locates a definition and never selects a + * previous run, and a run id addresses a run whose definition and props are + * already retained. Starting the same document twice without `--id` therefore + * makes two runs, and `resume` accepts no definition, no props and no generated + * prop arguments at all — supplying them would ask this run to be a different + * one. + * + * ## Where the boundary is + * + * This module is runtime-neutral: it names no SQLite, no DOFS and no host. What + * it cannot do itself — open a run's storage, and attach that run's Workspace — + * it asks a `WorkflowHost` for, and the entrypoints decide which one exists. + * Deno and the compiled binary supply the local one; Node and Bun supply a host + * that refuses, so the grammar is the same everywhere and the refusal arrives + * before anything is created or executed. + * + * ## Exit status + * + * Only a completed run exits zero. The rest are distinguishable, because shell + * automation that could not tell a suspended run from a finished one would + * treat every incomplete workflow as success. + */ + +import { Err, Ok, ensure, scoped } from "effection"; +import type { Operation, Result } from "effection"; +import { field, object, cli } from "configliere"; +import { z } from "zod"; +import type { DurableStream, Json } from "@executablemd/durable-streams"; +import { retainedSource } from "@executablemd/core"; +import type { RootDocumentSource } from "@executablemd/core"; +import { useRetainedWorkflow, WorkflowRunStorage } from "@executablemd/workflow"; +import type { + WorkflowRunDatabase, + WorkflowRunStatus, + WorkflowStopReason, +} from "@executablemd/workflow"; +import { loadRetainedDefinition, supportedRootDocument } from "./workflow-definition.ts"; +import type { EstablishedDefinition } from "./workflow-definition.ts"; + +/** + * What this module cannot do without knowing the host. + * + * Two operations, both of which reach storage. Everything else about the + * lifecycle — the grammar, the definition, the props, the statuses, the exit + * codes — is the same wherever a run lives. + */ +export interface WorkflowHost { + /** Install this host's run storage for the current scope and its descendants. */ + useStorage(): Operation; + /** + * Attach one run's Workspace around a live or partial document execution. + * + * A completed run does not take this path: its root result is already + * recorded, so there is nothing to give a filesystem to. + */ + attach(database: WorkflowRunDatabase, operation: Operation): Operation; +} + +export type HostWorkflowInstaller = () => Operation; + +/** The one sentence a host without workflow support says. */ +export const UNSUPPORTED_WORKFLOW_HOST = + "xmd workflow is available only through the Deno entrypoint or compiled xmd binary"; + +export class WorkflowHostUnsupportedError extends Error { + override name = "WorkflowHostUnsupportedError"; + + constructor() { + super(UNSUPPORTED_WORKFLOW_HOST); + } +} + +/** The installer Node and Bun supply: the grammar exists, the capability does not. */ +// deno-lint-ignore require-yield +export function* unsupportedWorkflowHost(): Operation { + throw new WorkflowHostUnsupportedError(); +} + +/** Only a completed run exits zero. */ +const EXIT_BY_STATUS: Readonly> = Object.freeze({ + completed: 0, + failed: 1, + suspended: 2, + cancelled: 3, + interrupted: 130, + // A run this process is still holding has not reported an outcome. Reaching + // here with one is a defect, and zero would call it success. + running: 1, +}); + +/** A failure the host classified, rather than an exception message it retained. */ +const HOST_FAILURE_CODE = "document-execution-failed"; +const HOST_INTERRUPTED_CODE = "executor-interrupted"; +const HOST_ORPHANED_CODE = "executor-disappeared"; + +export const WORKFLOW_ACTIONS = ["start", "resume"]; + +export const workflowConfig = object({ + action: { + description: "start or resume", + ...field(z.string().optional(), cli.argument()), + }, + target: { + description: "markdown definition to start, or the run id to resume", + ...field(z.string().optional(), cli.argument()), + }, + id: { + description: "run id to create or address (start only; generated when absent)", + ...field(z.string().optional()), + }, + verbose: { + description: "log journal entries to stderr", + aliases: ["-V"], + ...field(z.boolean(), field.default(false)), + }, + raw: { + description: "output raw markdown without normalization or terminal formatting", + ...field(z.boolean(), field.default(false)), + }, + secretDetection: { + description: + "scan durable events for credentials before they persist; " + + "disable with --no-secret-detection", + ...field(z.boolean(), field.default(true)), + }, +}); + +/** What one invocation asks for, after the grammar has been read. */ +export interface WorkflowRequest { + readonly action: "start" | "resume"; + readonly target: string; + readonly id: string | undefined; + readonly verbose: boolean; + readonly raw: boolean; + readonly secretDetection: boolean; +} + +/** What the shared CLI needs in order to execute this run's document. */ +export interface WorkflowExecution { + readonly root: RootDocumentSource; + readonly props: Record; + readonly stream: DurableStream; + /** Installed inside the execution scope, before the root document is imported. */ + install(): Operation; + /** Wraps the whole document execution, or passes it through on completed replay. */ + around(operation: Operation): Operation; +} + +/** How one `start` or `resume` ended. */ +export interface WorkflowOutcome { + readonly exitCode: number; +} + +/** + * The run id `start` uses when the caller named none. + * + * Opaque and cryptographically random, so two starts of one document are two + * runs and neither id says anything about what it runs. A caller who wants a + * stable id supplies one; the local caller is authorized to use any + * storage-valid id, and hashing is what keeps that id from becoming a path. + */ +function generatedRunId(): string { + return crypto.randomUUID(); +} + +function report(message: string): void { + console.error(message); +} + +function reportRun(runId: string): void { + report(`workflow run: ${runId}`); +} + +function reportStatus(status: WorkflowRunStatus): void { + report(`workflow status: ${status}`); +} + +/** Whether this journal already holds the root's terminal event. */ +function* isCompleted(stream: DurableStream): Operation { + const events = yield* stream.readAll(); + return events.some((event) => event.type === "close" && event.coroutineId === "root"); +} + +/** + * The stop reason a failure gets. + * + * A retained event that already crossed the secret filter is preferable to a + * code, because it says which effect failed. Anything else becomes one + * categorical host code: the alternative is retaining an exception message + * beside the journal that filtered it, which is history nothing has filtered. + */ +function* failureReason(database: WorkflowRunDatabase): Operation { + const entries = yield* database.readJournalEntries(); + if (entries.ok) { + for (let index = entries.value.length - 1; index >= 0; index -= 1) { + const entry = entries.value[index]; + if (entry !== undefined && entry.event.result.status === "err") { + return { kind: "journal", eventId: entry.eventId }; + } + } + } + return { kind: "host", code: HOST_FAILURE_CODE }; +} + +/** + * Close an execution record this process did not start and cannot finish. + * + * A run left `running` by a host that disappeared has an execution record with + * no end. Closing it as interrupted before a new one begins is what keeps the + * records a history of executions rather than a history with a hole in it. + * Concurrent ownership is #367's; nothing here claims a second executor is safe. + */ +function* closeOrphanedExecutions(database: WorkflowRunDatabase): Operation> { + const executions = yield* database.readDocumentExecutions(); + if (!executions.ok) { + return executions; + } + for (const execution of executions.value) { + if (execution.stoppedAt !== undefined) { + continue; + } + const finished = yield* database.finishDocumentExecution({ + executionId: execution.executionId, + status: "interrupted", + reason: { kind: "host", code: HOST_ORPHANED_CODE }, + }); + if (!finished.ok) { + return finished; + } + } + return Ok(undefined); +} + +/** The grammar this command accepts, refusing what it does not. */ +export function parseWorkflowRequest(config: { + action?: string; + target?: string; + id?: string; + verbose: boolean; + raw: boolean; + secretDetection: boolean; +}): Result { + const { action, target } = config; + if (action === undefined) { + return Err( + new Error( + "xmd workflow requires a subcommand — `xmd workflow start ` or " + + "`xmd workflow resume `", + ), + ); + } + if (action !== "start" && action !== "resume") { + return Err( + new Error( + `unrecognized subcommand for xmd workflow: ${action} — ` + + `expected ${WORKFLOW_ACTIONS.join(" or ")}`, + ), + ); + } + if (target === undefined || target === "") { + return Err( + new Error( + action === "start" + ? "xmd workflow start requires a markdown definition — `xmd workflow start `" + : "xmd workflow resume requires a run id — `xmd workflow resume `", + ), + ); + } + if (action === "resume" && config.id !== undefined) { + return Err( + new Error( + "unrecognized option for xmd workflow resume: --id — a resume names its run as its " + + "only argument", + ), + ); + } + return Ok({ + action, + target, + id: config.id, + verbose: config.verbose, + raw: config.raw, + secretDetection: config.secretDetection, + }); +} + +/** What the props phase already established for a `start`. */ +export interface WorkflowStart { + readonly established: EstablishedDefinition; + readonly props: Record; +} + +/** + * Open the run this request addresses, creating it when `start` describes a new + * one. + * + * `create()` is also how a run is found: a request describing the stored run + * answers with it, and one differing in any immutable field is refused with the + * conflict diagnostics storage already has. Nothing here repairs, replaces or + * initializes anything it did not create, and a lookup that finds nothing + * creates nothing. + */ +function* openRun( + request: WorkflowRequest, + start: WorkflowStart | undefined, +): Operation> { + if (request.action === "resume") { + const found = yield* WorkflowRunStorage.operations.lookup(request.target); + if (!found.ok) { + return found; + } + const database = found.value; + const source = yield* loadRetainedDefinition( + database.record.definition, + database.retrieval?.metadata, + ); + if (!source.ok) { + return source; + } + return Ok({ database, source: source.value }); + } + + if (start === undefined) { + return Err(new Error("xmd workflow start has no definition to run")); + } + + const supported = supportedRootDocument(start.established.definition); + if (!supported.ok) { + return supported; + } + + const created = yield* WorkflowRunStorage.operations.create({ + runId: request.id ?? generatedRunId(), + definition: start.established.definition, + base: start.established.base, + props: start.props, + }); + if (!created.ok) { + return created; + } + + const replaced = yield* created.value.replaceRetrievalMetadata(start.established.retrieval); + if (!replaced.ok) { + return replaced; + } + return Ok({ database: created.value, source: start.established.source }); +} + +/** + * Run one `start` or `resume` to completion, and answer with its exit status. + * + * `execute` is the shared CLI's own document machinery, handed everything this + * run decided: the pinned source, the retained props, the run's journal, the + * installations that belong inside the execution scope, and the attachment that + * wraps it. + */ +export function runWorkflow( + request: WorkflowRequest, + start: WorkflowStart | undefined, + host: WorkflowHost, + execute: (execution: WorkflowExecution) => Operation>, +): Operation { + return scoped(function* () { + yield* host.useStorage(); + + const opened = yield* openRun(request, start); + if (!opened.ok) { + report(opened.error.message); + return { exitCode: 1 }; + } + + const { database, source } = opened.value; + const { record } = database; + reportRun(record.runId); + + const orphaned = yield* closeOrphanedExecutions(database); + if (!orphaned.ok) { + report(orphaned.error.message); + return { exitCode: 1 }; + } + + const begun = yield* database.beginDocumentExecution(); + if (!begun.ok) { + report(begun.error.message); + return { exitCode: 1 }; + } + const executionId = begun.value.executionId; + + // Interruption is the outcome nothing else publishes. Registered before the + // execution starts, so a scope torn down by Ctrl-C settles the run rather + // than leaving a record with no end and a status of `running`. + let settled = false; + yield* ensure(function* () { + if (settled) { + return; + } + const reason: WorkflowStopReason = { kind: "host", code: HOST_INTERRUPTED_CODE }; + yield* database.finishDocumentExecution({ executionId, status: "interrupted", reason }); + yield* database.updateRunState({ status: "interrupted", reason }); + reportStatus("interrupted"); + }); + + const completed = yield* isCompleted(database.journal); + const execution: WorkflowExecution = { + root: retainedSource(record.definition.rootDocumentPath, source), + props: record.props, + stream: database.journal, + *install(): Operation { + // Installed into the execution's own scope rather than a scope of its + // own: a `scoped()` here would tear it down before the root was + // imported. Service denial is installed beside it, through the same + // host-service slot `xmd run` fills with a real adapter. + yield* useRetainedWorkflow({ + runId: record.runId, + base: record.base, + pinnedCommit: record.definition.objectId, + }); + }, + around(operation: Operation): Operation { + // A completed run replays its retained output and result. Attaching a + // Workspace for it would open a transaction and capture a root for + // work that is not going to happen. + return completed ? operation : host.attach(database, operation); + }, + }; + + const result = yield* attempt(execution, execute); + settled = true; + + const status: WorkflowRunStatus = result.ok ? "completed" : "failed"; + const reason = result.ok ? undefined : yield* failureReason(database); + yield* database.finishDocumentExecution({ executionId, status, reason }); + yield* database.updateRunState({ status, reason }); + reportStatus(status); + + if (!result.ok) { + report(result.error.message); + } + return { exitCode: EXIT_BY_STATUS[status] }; + }); +} + +/** + * Run the document, converting a failure the shared machinery did not catch. + * + * Whatever escapes is still this execution's outcome rather than an + * interruption, so it is published as a failure and the interruption finalizer + * stays out of the way. + */ +function* attempt( + execution: WorkflowExecution, + execute: (execution: WorkflowExecution) => Operation>, +): Operation> { + try { + return yield* execute(execution); + } catch (error) { + return Err(error instanceof Error ? error : new Error(String(error))); + } +} + +/** The exit code a status reports, for a caller composing its own outcome. */ +export function workflowExitCode(status: WorkflowRunStatus): number { + return EXIT_BY_STATUS[status]; +} diff --git a/packages/cli/tests/workflow-cli.test.ts b/packages/cli/tests/workflow-cli.test.ts new file mode 100644 index 00000000..1b1d58e4 --- /dev/null +++ b/packages/cli/tests/workflow-cli.test.ts @@ -0,0 +1,384 @@ +/** + * Tier WFC — `xmd workflow start` and `xmd workflow resume`. + * + * Every run here shells out, so exit status, the two stderr metadata lines and + * the document's own stdout are observed exactly as a caller sees them. Each + * fixture is a real Git repository in a temporary directory with a real commit, + * because what the definition *is* comes from Git, and each run store is an + * isolated absolute directory named by `XMD_WORKFLOW_RUNS` — nothing here goes + * near `~/.xmd`. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped } from "effection"; +import type { Operation } from "effection"; +import { ensureDir, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { exec } from "@effectionx/process"; +import { randomUUID } from "node:crypto"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { runCli } from "@executablemd/test-support/launch"; +import { workflowRunPath } from "@executablemd/workflow/deno"; + +interface Fixture { + /** The repository the definition lives in. */ + readonly repository: string; + /** The isolated run store. */ + readonly runs: string; + /** An isolated HOME, so nothing reaches the developer's own configuration. */ + readonly home: string; +} + +const RELEASE = [ + "---", + "props:", + " channel:", + " type: string", + " default: stable", + "---", + "", + "# Release", + "", + 'channel={props.channel}', + "", + '', + "", + "Wrote: {notes}", + "", +].join("\n"); + +/** + * A document that fails rather than printing. + * + * A `` refusal is a *printed* error — data, and the run still completes — + * so a failed run needs both an error nothing prints and a region that decides + * one. `` is that region: it makes an undecided error the document + * execution's own outcome. The unresolvable component doubles as the evidence for the other claim this + * fixture carries: a workflow definition is one immutable object, so the + * component search path is empty and a repository component fails to resolve + * rather than resolving to content beside the definition in a mutable checkout. + */ +const REFUSING = [ + "# Refusing", + "", + "", + "", + "", + "the run's result", + "", + "", +].join("\n"); + +function* git(repository: string, args: string[]): Operation { + const result = yield* exec("git", { arguments: args, cwd: repository }).expect(); + if (result.code !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr}`); + } +} + +/** A committed repository, an empty run store, and an isolated HOME. */ +function useFixture( + files: Record, + body: (fixture: Fixture) => Operation, +): Operation { + return scoped(function* () { + const root = join(tmpdir(), `xmd-wfc-${randomUUID()}`); + const fixture: Fixture = { + repository: join(root, "repository"), + runs: join(root, "runs"), + home: join(root, "home"), + }; + yield* ensure(() => rm(root, { recursive: true, force: true })); + yield* ensureDir(fixture.repository); + yield* ensureDir(fixture.home); + + for (const [name, content] of Object.entries(files)) { + const path = join(fixture.repository, name); + yield* ensureDir(join(path, "..")); + yield* writeTextFile(path, content); + } + + yield* git(fixture.repository, ["init", "-q", "--initial-branch=main", "."]); + yield* git(fixture.repository, ["config", "user.email", "tier-wfc@example.test"]); + yield* git(fixture.repository, ["config", "user.name", "Tier WFC"]); + yield* git(fixture.repository, ["add", "-A"]); + yield* git(fixture.repository, ["commit", "-q", "-m", "definition"]); + + return yield* body(fixture); + }); +} + +function xmd(fixture: Fixture, args: string[]) { + return runCli(args, { + cwd: fixture.repository, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }); +} + +/** The run id the `workflow run:` line reported. */ +function reportedRunId(stderr: string): string | undefined { + const line = stderr.split("\n").find((entry) => entry.startsWith("workflow run: ")); + return line?.slice("workflow run: ".length).trim(); +} + +/** The status the `workflow status:` line reported. */ +function reportedStatus(stderr: string): string | undefined { + const line = stderr.split("\n").find((entry) => entry.startsWith("workflow status: ")); + return line?.slice("workflow status: ".length).trim(); +} + +describe("Tier WFC — xmd workflow start and resume", () => { + it("WFC1: two starts without an id make two runs", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const first = yield* xmd(fixture, ["workflow", "start", "flows/release.md"]).join(); + const second = yield* xmd(fixture, ["workflow", "start", "flows/release.md"]).join(); + + expect(first.code).toBe(0); + expect(second.code).toBe(0); + expect(reportedStatus(first.stderr)).toBe("completed"); + + const one = reportedRunId(first.stderr); + const other = reportedRunId(second.stderr); + expect(one).toBeDefined(); + expect(other).toBeDefined(); + expect(one).not.toBe(other); + expect(first.stdout).toContain("Wrote: channel=stable"); + }); + }); + + it("WFC2: reusing an id compatibly finds the run, and incompatibly is refused", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const created = yield* xmd(fixture, [ + "workflow", + "start", + "--id=release-1", + "flows/release.md", + ]).join(); + expect(created.code).toBe(0); + expect(reportedRunId(created.stderr)).toBe("release-1"); + + const reused = yield* xmd(fixture, [ + "workflow", + "start", + "--id=release-1", + "flows/release.md", + ]).join(); + expect(reused.code).toBe(0); + expect(reportedRunId(reused.stderr)).toBe("release-1"); + + const conflicting = yield* xmd(fixture, [ + "workflow", + "start", + "--id=release-1", + "flows/release.md", + "--props-channel=beta", + ]).join(); + expect(conflicting.code).toBe(1); + expect(conflicting.stderr).toContain("release-1"); + expect(conflicting.stderr).toContain("props"); + // A refused reuse creates nothing: the run that is there is the one that + // was there, still reporting the props it was created with. + expect(reportedStatus(conflicting.stderr)).toBeUndefined(); + + const unchanged = yield* xmd(fixture, ["workflow", "resume", "release-1"]).join(); + expect(unchanged.stdout).toContain("Wrote: channel=stable"); + }); + }); + + it("WFC3: generated prop arguments belong to start alone", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const started = yield* xmd(fixture, [ + "workflow", + "start", + "--id=props-1", + "flows/release.md", + "--props-channel=beta", + ]).join(); + expect(started.code).toBe(0); + expect(started.stdout).toContain("Wrote: channel=beta"); + + const help = yield* xmd(fixture, ["workflow", "start", "flows/release.md", "--help"]).join(); + expect(help.code).toBe(0); + expect(help.stdout).toContain("--props-channel"); + + const refused = yield* xmd(fixture, [ + "workflow", + "resume", + "props-1", + "--props-channel=stable", + ]).join(); + expect(refused.code).toBe(1); + expect(refused.stderr).toContain("--props-channel"); + expect(reportedStatus(refused.stderr)).toBeUndefined(); + + const aggregate = yield* xmd(fixture, [ + "workflow", + "resume", + "props-1", + "--props", + '{"channel":"stable"}', + ]).join(); + expect(aggregate.code).toBe(1); + }); + }); + + it("WFC4: the definition is the committed object, not the working tree", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + yield* writeTextFile( + join(fixture.repository, "flows/release.md"), + `${RELEASE}\nUNCOMMITTED\n`, + ); + + const started = yield* xmd(fixture, [ + "workflow", + "start", + "--id=pinned-1", + "flows/release.md", + ]).join(); + + expect(started.code).toBe(0); + expect(started.stdout).toContain("Wrote: channel=stable"); + expect(started.stdout).not.toContain("UNCOMMITTED"); + }); + }); + + it("WFC5: resume names only a run, and finds its definition again", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + yield* xmd(fixture, ["workflow", "start", "--id=resume-1", "flows/release.md"]).expect(); + + const resumed = yield* xmd(fixture, ["workflow", "resume", "resume-1"]).join(); + expect(resumed.code).toBe(0); + expect(reportedRunId(resumed.stderr)).toBe("resume-1"); + expect(resumed.stdout).toContain("Wrote: channel=stable"); + + const withDocument = yield* xmd(fixture, [ + "workflow", + "resume", + "resume-1", + "flows/release.md", + ]).join(); + expect(withDocument.code).toBe(1); + }); + }); + + it("WFC6: a failing document reports failed and exits 1, and replays as failed", function* () { + yield* useFixture( + { + "flows/refusing.md": REFUSING, + "components/ComponentBesideTheDefinition.md": "resolved from the checkout\n", + }, + function* (fixture) { + const failed = yield* xmd(fixture, [ + "workflow", + "start", + "--id=failing-1", + "flows/refusing.md", + ]).join(); + expect(failed.code).toBe(1); + expect(reportedStatus(failed.stderr)).toBe("failed"); + // Nothing beside the definition was searched, so nothing beside it could + // have been read. + expect(failed.stderr).toContain("ComponentBesideTheDefinition"); + expect(failed.stderr).toContain("searched: "); + + // A compatible reuse of retained failed history replays that failure + // rather than retrying the work. + const replayed = yield* xmd(fixture, [ + "workflow", + "start", + "--id=failing-1", + "flows/refusing.md", + ]).join(); + expect(replayed.code).toBe(1); + expect(reportedStatus(replayed.stderr)).toBe("failed"); + expect(replayed.stdout).not.toContain("resolved from the checkout"); + }, + ); + }); + + it("WFC7: absent, foreign and unreadable storage is reported and left alone", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const missing = yield* xmd(fixture, ["workflow", "resume", "never-started"]).join(); + expect(missing.code).toBe(1); + expect(missing.stderr).toContain("never-started"); + expect(reportedRunId(missing.stderr)).toBeUndefined(); + + yield* xmd(fixture, ["workflow", "start", "--id=corrupt-1", "flows/release.md"]).expect(); + const runPath = workflowRunPath(fixture.runs, "corrupt-1"); + const before = yield* readTextFile(runPath); + yield* writeTextFile(runPath, "this is not a workflow run database"); + + const corrupt = yield* xmd(fixture, ["workflow", "resume", "corrupt-1"]).join(); + expect(corrupt.code).toBe(1); + expect(reportedStatus(corrupt.stderr)).toBeUndefined(); + // Described, never replaced: the bytes are exactly what the test wrote. + expect(yield* readTextFile(runPath)).toBe("this is not a workflow run database"); + expect(before).not.toBe("this is not a workflow run database"); + }); + }); + + it("WFC8: the grammar refuses what the command does not have", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const noAction = yield* xmd(fixture, ["workflow"]).join(); + expect(noAction.code).toBe(1); + expect(noAction.stderr).toContain("start"); + + const unknown = yield* xmd(fixture, ["workflow", "cancel", "release-1"]).join(); + expect(unknown.code).toBe(1); + expect(unknown.stderr).toContain("cancel"); + + const noTarget = yield* xmd(fixture, ["workflow", "start"]).join(); + expect(noTarget.code).toBe(1); + + const resumeId = yield* xmd(fixture, [ + "workflow", + "resume", + "release-1", + "--id=other", + ]).join(); + expect(resumeId.code).toBe(1); + expect(resumeId.stderr).toContain("--id"); + + const agent = yield* xmd(fixture, [ + "workflow", + "start", + "flows/release.md", + "--approve-all", + ]).join(); + expect(agent.code).toBe(1); + expect(agent.stderr).toContain("--approve-all"); + + const inline = yield* xmd(fixture, ["workflow", "start", "-e", "# hi"]).join(); + expect(inline.code).toBe(1); + }); + }); + + it("WFC9: a definition outside a repository, and one that is not Markdown", function* () { + yield* useFixture( + { "flows/release.md": RELEASE, "flows/root.ts": "export default 1;\n" }, + function* (fixture) { + const notMarkdown = yield* xmd(fixture, ["workflow", "start", "flows/root.ts"]).join(); + expect(notMarkdown.code).toBe(1); + expect(notMarkdown.stderr).toMatch(/markdown/i); + expect(reportedRunId(notMarkdown.stderr)).toBeUndefined(); + + const outside = yield* xmd(fixture, ["workflow", "start", "../elsewhere.md"]).join(); + expect(outside.code).toBe(1); + expect(reportedRunId(outside.stderr)).toBeUndefined(); + }, + ); + }); + + it("WFC10: ordinary xmd run is unchanged by any of this", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const run = yield* xmd(fixture, ["run", "flows/release.md", "--props-channel=beta"]).join(); + expect(run.code).toBe(0); + expect(run.stdout).toContain("Wrote: channel=beta"); + expect(run.stderr).not.toContain("workflow run:"); + // `xmd run` writes into the caller's own filesystem, which is exactly + // what a workflow run does not do. + expect(yield* readTextFile(join(fixture.repository, "notes.md"))).toBe("channel=beta"); + }); + }); +}); diff --git a/packages/cli/tests/workflow-crash.test.ts b/packages/cli/tests/workflow-crash.test.ts new file mode 100644 index 00000000..541f931c --- /dev/null +++ b/packages/cli/tests/workflow-crash.test.ts @@ -0,0 +1,253 @@ +/** + * Tier WFX — what a killed `xmd workflow start` leaves, and what a resume does + * with it. + * + * The point of a retained workflow is that losing the host is survivable, so + * this is made of real processes and a real `SIGKILL`. Nothing runs after the + * signal — no cleanup, no commit, no rollback — and what the next process finds + * is whatever the last committed transaction left. + * + * The document writes many files, one durable effect each, and the parent kills + * the child once a second connection can see that some of them have committed. + * Where the kill lands is deliberately not controlled: it may fall between two + * effects, or inside one whose transaction has not committed. Both are real, and + * the invariant that has to hold either way is the one asserted — every effect + * appears exactly once, in order, with no duplicate and no gap, and the effects + * that committed before the kill are not performed again. + * + * The stricter in-transaction kill, where the child is stopped at a point it has + * announced from inside an open transaction, is Tier WAC's + * (`packages/workflow/tests/workspace-crash-recovery.test.ts`). This suite is + * about the CLI lifecycle built on top of it. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped, spawn } from "effection"; +import type { Operation } from "effection"; +import { ensureDir, rm, writeTextFile } from "@effectionx/fs"; +import { exec } from "@effectionx/process"; +import { when } from "@effectionx/converge"; +import { randomUUID } from "node:crypto"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import process from "node:process"; +import { DatabaseSync } from "node:sqlite"; +import { cliCommand, runCli } from "@executablemd/test-support/launch"; +import { workflowRunPath } from "@executablemd/workflow/deno"; + +/** Enough effects that a kill lands part-way through rather than after. */ +const EFFECTS = 60; + +const RUN_ID = "crashed-run"; + +function definition(): string { + const lines = ["# Many effects", ""]; + for (let index = 0; index < EFFECTS; index += 1) { + lines.push(`effect ${index}`, ""); + } + return lines.join("\n"); +} + +interface Fixture { + readonly repository: string; + readonly runs: string; + readonly home: string; +} + +function* git(repository: string, args: string[]): Operation { + const result = yield* exec("git", { arguments: args, cwd: repository }).expect(); + if (result.code !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr}`); + } +} + +function useFixture(body: (fixture: Fixture) => Operation): Operation { + return scoped(function* () { + const root = join(tmpdir(), `xmd-wfx-${randomUUID()}`); + const fixture: Fixture = { + repository: join(root, "repository"), + runs: join(root, "runs"), + home: join(root, "home"), + }; + yield* ensure(() => rm(root, { recursive: true, force: true })); + yield* ensureDir(join(fixture.repository, "flows")); + yield* ensureDir(fixture.home); + yield* writeTextFile(join(fixture.repository, "flows/many.md"), definition()); + + yield* git(fixture.repository, ["init", "-q", "--initial-branch=main", "."]); + yield* git(fixture.repository, ["config", "user.email", "tier-wfx@example.test"]); + yield* git(fixture.repository, ["config", "user.name", "Tier WFX"]); + yield* git(fixture.repository, ["add", "-A"]); + yield* git(fixture.repository, ["commit", "-q", "-m", "definition"]); + + return yield* body(fixture); + }); +} + +interface FileEffect { + readonly eventId: string; + readonly name: string; +} + +/** + * The file effects a second connection can see, in append order. + * + * Read from outside the running process on purpose: rows inside an open + * transaction are invisible here until that transaction commits, so this + * reports what has been *published* rather than what some handle is holding. + */ +function committedEffects(path: string): FileEffect[] { + const database = new DatabaseSync(path, { readOnly: true }); + try { + const rows = database + .prepare("SELECT event_id AS id, record FROM journal_events ORDER BY sequence") + .all(); + const effects: FileEffect[] = []; + for (const row of rows) { + const record = typeof row["record"] === "string" ? row["record"] : ""; + const parsed = JSON.parse(record); + const description = parsed?.description; + if (description?.type === "workspace_file" && typeof description.name === "string") { + effects.push({ eventId: String(row["id"]), name: description.name }); + } + } + return effects; + } finally { + database.close(); + } +} + +/** The run's retained status, as a second connection sees it. */ +function committedStatus(path: string): string { + const database = new DatabaseSync(path, { readOnly: true }); + try { + const row = database.prepare("SELECT status FROM workflow_run WHERE id = 1").get(); + return String(row?.["status"]); + } finally { + database.close(); + } +} + +/** The current Workspace root, as a second connection sees it. */ +function committedRoot(path: string): string { + const database = new DatabaseSync(path, { readOnly: true }); + try { + const row = database.prepare("SELECT current_root_id AS root FROM workspace_state").get(); + return String(row?.["root"]); + } finally { + database.close(); + } +} + +function exists(path: string): boolean { + try { + new DatabaseSync(path, { readOnly: true }).close(); + return true; + } catch { + return false; + } +} + +describe("Tier WFX — a killed workflow run resumes from its frontier", () => { + it("WFX1: SIGKILL mid-run, then resume: every effect exactly once", function* () { + yield* useFixture(function* (fixture) { + const path = workflowRunPath(fixture.runs, RUN_ID); + + const killed = yield* scoped(function* () { + const cli = cliCommand(["workflow", "start", `--id=${RUN_ID}`, "flows/many.md"]); + const child = yield* exec(cli.command, { + arguments: cli.arguments, + cwd: fixture.repository, + env: { + ...inherited(), + HOME: fixture.home, + XMD_WORKFLOW_RUNS: fixture.runs, + }, + }); + // Read the child's streams so a full pipe cannot stall it before it has + // committed anything. + yield* spawn(function* () { + const subscription = yield* child.stdout; + let next = yield* subscription.next(); + while (!next.done) { + next = yield* subscription.next(); + } + }); + yield* spawn(function* () { + const subscription = yield* child.stderr; + let next = yield* subscription.next(); + while (!next.done) { + next = yield* subscription.next(); + } + }); + + // The condition is what the database has published, never a sleep. + yield* when( + function* () { + expect(exists(path)).toBe(true); + expect(committedEffects(path).length).toBeGreaterThanOrEqual(3); + }, + { timeout: 60_000 }, + ); + const before = committedEffects(path); + const rootBefore = committedRoot(path); + process.kill(child.pid, "SIGKILL"); + const status = yield* child.join(); + return { before, rootBefore, status }; + }); + + expect(killed.status.signal).toBe("SIGKILL"); + expect(killed.before.length).toBeGreaterThanOrEqual(3); + expect(killed.before.length).toBeLessThan(EFFECTS); + + // Nothing ran after the signal, so the run is still `running`: no status + // was published and no execution record was closed. + expect(committedStatus(path)).toBe("running"); + + const resumed = yield* runCli(["workflow", "resume", RUN_ID], { + cwd: fixture.repository, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + timeout: 180_000, + }).join(); + + expect(resumed.code).toBe(0); + expect(resumed.stderr).toContain(`workflow run: ${RUN_ID}`); + expect(resumed.stderr).toContain("workflow status: completed"); + expect(committedStatus(path)).toBe("completed"); + + const after = committedEffects(path); + + // Every effect the kill left committed is still there, by its own event + // id and in the same order: the resume replayed them rather than + // performing them again. + expect(after.slice(0, killed.before.length)).toEqual(killed.before); + + // And the whole history is exactly one effect per authored element — no + // duplicate from an interrupted transaction that had already published, + // and no gap from one that had not. + expect(after).toHaveLength(EFFECTS); + const targets = after.map((effect) => effect.name.split(":").at(-1)); + expect(new Set(targets).size).toBe(EFFECTS); + expect(targets[0]).toBe("/out/f0.txt"); + expect(targets.at(-1)).toBe(`/out/f${EFFECTS - 1}.txt`); + + // The resume continued from the root the last committed effect left, so + // the frontier moved on rather than starting again. + expect(committedRoot(path)).not.toBe(killed.rootBefore); + }); + }); +}); + +/** What a child needs from this process, without a developer's own HOME. */ +function inherited(): Record { + const names = ["PATH", "DENO_DIR", "DENO_INSTALL_ROOT", "XDG_CACHE_HOME", "TMPDIR"]; + const env: Record = {}; + for (const name of names) { + const value = process.env[name]; + if (typeof value === "string") { + env[name] = value; + } + } + return env; +} diff --git a/packages/cli/tests/workflow-host.test.ts b/packages/cli/tests/workflow-host.test.ts new file mode 100644 index 00000000..2b12748a --- /dev/null +++ b/packages/cli/tests/workflow-host.test.ts @@ -0,0 +1,107 @@ +/** + * Tier WFH — which hosts run a workflow. + * + * `xmd workflow` has the same grammar everywhere and the capability in one + * place. Deno and the compiled binary own the local run store; Node and Bun + * expose the command and refuse it before anything is created or executed, so + * a caller learns the boundary from one sentence rather than from a run that + * half-happened. + * + * Which entrypoint is under test is asked of `@executablemd/test-support`, + * which is the one place runtime detection belongs. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped } from "effection"; +import type { Operation } from "effection"; +import { ensureDir, exists, rm, writeTextFile } from "@effectionx/fs"; +import { randomUUID } from "node:crypto"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { cliRuntime, runCli } from "@executablemd/test-support/launch"; + +/** The one sentence a host without workflow support says. */ +const UNSUPPORTED = + "xmd workflow is available only through the Deno entrypoint or compiled xmd binary"; + +const DOCUMENT = ["# Nothing", "", "no effects at all", ""].join("\n"); + +interface Fixture { + readonly dir: string; + readonly runs: string; + readonly home: string; +} + +function useFixture(body: (fixture: Fixture) => Operation): Operation { + return scoped(function* () { + const root = join(tmpdir(), `xmd-wfh-${randomUUID()}`); + const fixture: Fixture = { + dir: join(root, "work"), + runs: join(root, "runs"), + home: join(root, "home"), + }; + yield* ensure(() => rm(root, { recursive: true, force: true })); + yield* ensureDir(fixture.dir); + yield* ensureDir(fixture.home); + yield* writeTextFile(join(fixture.dir, "flow.md"), DOCUMENT); + return yield* body(fixture); + }); +} + +describe("Tier WFH — workflow host boundary", () => { + it("WFH1: an unsupported host refuses before creating or executing anything", function* () { + yield* useFixture(function* (fixture) { + const result = yield* runCli(["workflow", "start", "flow.md"], { + cwd: fixture.dir, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }).join(); + + if (cliRuntime() === "deno") { + // The Deno entrypoint has the capability. What it refuses here is the + // definition — the fixture is not in a repository — which is a + // different sentence and a different reason. + expect(result.stderr).not.toContain(UNSUPPORTED); + return; + } + + expect(result.code).toBe(1); + expect(result.stderr).toContain(UNSUPPORTED); + // Nothing was created: no run store, and no `workflow run:` line, so no + // run id was ever allocated. + expect(result.stderr).not.toContain("workflow run:"); + expect(result.stderr).not.toContain("workflow status:"); + expect(yield* exists(fixture.runs)).toBe(false); + }); + }); + + it("WFH2: every host reads the same grammar", function* () { + yield* useFixture(function* (fixture) { + const help = yield* runCli(["workflow", "--help"], { + cwd: fixture.dir, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }).join(); + + expect(help.code).toBe(0); + expect(help.stdout).toContain("xmd workflow"); + expect(help.stdout).toContain("--id"); + }); + }); + + it("WFH3: an unsupported host refuses a resume too, and reads no store", function* () { + yield* useFixture(function* (fixture) { + const result = yield* runCli(["workflow", "resume", "any-run"], { + cwd: fixture.dir, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }).join(); + + expect(result.code).toBe(1); + if (cliRuntime() === "deno") { + expect(result.stderr).toContain("any-run"); + return; + } + expect(result.stderr).toContain(UNSUPPORTED); + expect(yield* exists(fixture.runs)).toBe(false); + }); + }); +}); diff --git a/packages/core/mod.ts b/packages/core/mod.ts index 5ccd3b1e..d116e5e9 100644 --- a/packages/core/mod.ts +++ b/packages/core/mod.ts @@ -137,11 +137,13 @@ export { formatDocumentReference, INLINE_SOURCE_PATH, inlineSource, + retainedSource, rootSourcePath, } from "./src/root-source.ts"; export type { FileRootDocument, InlineRootDocument, + RetainedRootDocument, RootDocumentSource, } from "./src/root-source.ts"; export { diff --git a/packages/core/src/root-source.ts b/packages/core/src/root-source.ts index b651f448..96c1680a 100644 --- a/packages/core/src/root-source.ts +++ b/packages/core/src/root-source.ts @@ -28,8 +28,54 @@ export interface InlineRootDocument { readonly target?: string; } +/** + * A root document whose text came from somewhere the engine cannot read again. + * + * A workflow definition is the case this exists for: the bytes are the ones a + * pinned Git object held, and the identity they report is the document's path + * inside that object. Text and identity travel together, as they do for an + * inline document, so nothing can execute one document while reporting another. + */ +export interface RetainedRootDocument { + readonly path: string; + readonly source: string; +} + /** Where a root document's text comes from: a path, or supplied text. */ -export type RootDocumentSource = FileRootDocument | InlineRootDocument; +/** + * A root document whose text came from somewhere the engine cannot read again. + * + * A workflow definition is the case this exists for: the bytes are the ones a + * pinned Git object held, and the identity they report is the document's path + * inside that object. + * + * `retained` is what keeps supplied text from travelling under an identity + * nobody vouched for. An inline document is branded by its path — there is only + * one `` — but a retained document's path is an ordinary one, so without + * this member `{ path, source }` would type-check for any path at all and the + * engine would report a document as having come from a file it never read. + * Constructing one is therefore a deliberate act, the way `inlineSource()` is. + */ +export interface RetainedRootDocument { + readonly path: string; + readonly source: string; + readonly retained: true; + /** The requested target selector, still encoded and possibly a glob. */ + readonly target?: string; +} + +/** Where a root document's text comes from: a path, or supplied text. */ +export type RootDocumentSource = FileRootDocument | InlineRootDocument | RetainedRootDocument; + +/** Supplied text as a root document reported by the path it came from. */ +export function retainedSource( + path: string, + source: string, + options?: { readonly target?: string }, +): RetainedRootDocument { + const target = options?.target; + return { path, source, retained: true, ...(target === undefined ? {} : { target }) }; +} /** Supplied text as a root document carrying the `` identity. */ export function inlineSource( diff --git a/packages/runtime/apis.ts b/packages/runtime/apis.ts index e3c07474..a25f1abb 100644 --- a/packages/runtime/apis.ts +++ b/packages/runtime/apis.ts @@ -85,7 +85,7 @@ import { stat as fsStat, writeTextFile as fsWriteTextFile, } from "@effectionx/fs"; -import { exec as processExec } from "@effectionx/process"; +import { exec as processExec, Stdio } from "@effectionx/process"; import { race, sleep, until } from "effection"; import type { Operation } from "effection"; import { timeout as contextualTimeout } from "./config.ts"; @@ -586,3 +586,20 @@ export const platform: typeof API.Env.operations.platform = API.Env.operations.p export const command: typeof API.Env.operations.command = API.Env.operations.command; export const compile: typeof API.Env.operations.compile = API.Env.operations.compile; + +/** + * Discard the standard output of subprocesses started in this scope. + * + * For a caller whose subprocess output is an *answer* rather than something to + * show: a command whose stdout is parsed and returned would otherwise also + * print itself into whatever the process was rendering. `stderr` is left alone, + * because that is where a failing command explains itself and a diagnostic is + * worth seeing. + * + * It lives here because reaching the process Api's stdio directly is host + * behavior, and modules held to the runtime-neutral boundary may not import a + * host process module of their own. + */ +export function useQuietProcessOutput(): Operation { + return Stdio.around({ *stdout() {} }); +} diff --git a/packages/runtime/mod.ts b/packages/runtime/mod.ts index 5d4ed9f6..ad7126f3 100644 --- a/packages/runtime/mod.ts +++ b/packages/runtime/mod.ts @@ -39,6 +39,7 @@ export { platform, command, compile, + useQuietProcessOutput, } from "./apis.ts"; export type { EvalBlock, ResponseHeaders, RuntimeFetchResponse, StatResult } from "./apis.ts"; export { diff --git a/packages/test-support/launch.ts b/packages/test-support/launch.ts index 61689e0a..c5b48f0b 100644 --- a/packages/test-support/launch.ts +++ b/packages/test-support/launch.ts @@ -13,6 +13,20 @@ function entry(runtime: string): string { return join(ROOT, "packages", "cli", "src", `${runtime}.ts`); } +/** + * Which entrypoint `runCli` launches. + * + * Runtime detection belongs to this package and nowhere else (Code Rule 12), so + * a suite whose subject differs by host — `xmd workflow`, which only the Deno + * entrypoints support — asks here rather than reading a global of its own. + */ +export function cliRuntime(): "deno" | "bun" | "node" { + if (Reflect.has(globalThis, "Deno")) { + return "deno"; + } + return Reflect.has(globalThis, "Bun") ? "bun" : "node"; +} + export function cliBase(): string[] { if (Reflect.has(globalThis, "Deno")) { return [process.execPath, "run", "--allow-all", entry("deno")]; diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index 551a7680..576b36cb 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -24,8 +24,17 @@ * imports SQLite, Deno or any other host. */ -export { Git, GitRevisionError, revParse } from "./src/git.ts"; -export type { GitApi } from "./src/git.ts"; +export { + Git, + gitObjectFormat, + GitObjectError, + GitRepositoryError, + GitRevisionError, + readGitObject, + repositoryRoot, + revParse, +} from "./src/git.ts"; +export type { GitApi, GitObjectFormat } from "./src/git.ts"; export { getWorkflowRun, useRetainedWorkflow, useWorkflow } from "./src/run.ts"; export type { WorkflowRun } from "./src/run.ts"; export { useWorkflowServiceDenial, WorkflowServiceDeniedError } from "./src/service-denial.ts"; diff --git a/packages/workflow/src/git.ts b/packages/workflow/src/git.ts index 06aab2a0..e33db796 100644 --- a/packages/workflow/src/git.ts +++ b/packages/workflow/src/git.ts @@ -1,16 +1,46 @@ /** * The Git capability. * - * Workflow infrastructure asks one question of the repository — resolve this - * revision expression to a commit — and asks it through a contextual Api, so a - * host or a test replaces the answer lexically rather than by arranging a - * repository on disk. Core never reaches Git at all: ordinary `execute()` and - * `xmd run` stay Git-independent. + * Workflow infrastructure asks a small, fixed set of questions of the + * repository, and asks them through a contextual Api, so a host or a test + * replaces the answers lexically rather than by arranging a repository on disk. + * Core never reaches Git at all: ordinary `execute()` and `xmd run` stay + * Git-independent. + * + * All four questions are about *one* repository — the one containing the + * contextual working directory — and none of them mutates it. Together they are + * what an immutable workflow definition is made of: which repository, which + * commit, in which object format, and what the root document held in it. */ import { type Api, createApi, type Operations } from "@effectionx/context-api"; +import { scoped } from "effection"; import type { Operation } from "effection"; -import { cwd, exec } from "@executablemd/runtime"; +import { cwd, exec, useQuietProcessOutput } from "@executablemd/runtime"; + +/** + * What Git said, without Git having said it to the terminal. + * + * Every answer here is a payload this module reads and returns, so echoing it + * would put a commit id and a document's own bytes into whatever the caller was + * rendering. `stderr` is left alone: it is where a failing Git explains itself, + * and that is a diagnostic rather than a payload. + */ +function asked(command: string[], directory: string): Operation { + return scoped(function* () { + yield* useQuietProcessOutput(); + return yield* exec({ command, cwd: directory }); + }); +} + +interface ExecResult { + readonly exitCode: number; + readonly stdout: string; + readonly stderr: string; +} + +/** The hash algorithm a repository names its objects with. */ +export type GitObjectFormat = "sha1" | "sha256"; export interface GitApi { /** @@ -20,6 +50,33 @@ export interface GitApi { * contextual working directory. */ revParse(revision: string): Operation; + + /** + * The absolute path of the working tree containing the contextual working + * directory. + * + * The semantics of `git rev-parse --show-toplevel`. A directory that is not + * inside a working tree is an error rather than an empty answer. + */ + repositoryRoot(): Operation; + + /** + * The repository's object format. + * + * The semantics of `git rev-parse --show-object-format`. It is part of a + * definition's identity: two hosts that agree about a commit only agree about + * the run if they agree about which algorithm named it. + */ + objectFormat(): Operation; + + /** + * The bytes one path held in one commit, as text. + * + * The semantics of `git cat-file blob :`. What comes back is the + * pinned object rather than whatever the working tree holds now, which is what + * lets a run claim a commit as its identity and mean it. + */ + readObject(commit: string, path: string): Operation; } /** Git could not answer for this revision. Carries what Git reported, not a guess. */ @@ -35,13 +92,43 @@ export class GitRevisionError extends Error { } } +/** Git could not answer a question about the repository itself. */ +export class GitRepositoryError extends Error { + override name = "GitRepositoryError"; + + constructor(question: string, result: { exitCode: number; stderr: string }) { + const reported = result.stderr.trim(); + super( + `git could not answer ${question}: exited ${result.exitCode}` + + (reported === "" ? " with no output" : ` — ${reported}`), + ); + } +} + +/** The object a definition names is not in the repository, or is not a file. */ +export class GitObjectError extends Error { + override name = "GitObjectError"; + + constructor(commit: string, path: string, result: { exitCode: number; stderr: string }) { + const reported = result.stderr.trim(); + super( + `git could not read "${path}" from commit ${commit}: exited ${result.exitCode}` + + (reported === "" ? " with no output" : ` — ${reported}`), + ); + } +} + +function objectFormat(value: string): GitObjectFormat | undefined { + return value === "sha1" || value === "sha256" ? value : undefined; +} + export const Git: Api = createApi("Git", { *revParse(revision: string): Operation { // `--verify` makes an unresolvable revision an error rather than an echo, // and `--end-of-options` stops a revision that looks like a flag from being // read as one. The command is an array, so nothing is ever parsed by a shell. const command = ["git", "rev-parse", "--verify", "--end-of-options", revision]; - const result = yield* exec({ command, cwd: yield* cwd() }); + const result = yield* asked(command, yield* cwd()); if (result.exitCode !== 0) { throw new GitRevisionError(revision, result); } @@ -53,6 +140,38 @@ export const Git: Api = createApi("Git", { } return objectId; }, + + *repositoryRoot(): Operation { + const result = yield* asked(["git", "rev-parse", "--show-toplevel"], yield* cwd()); + const root = result.stdout.trim(); + if (result.exitCode !== 0 || root === "") { + throw new GitRepositoryError("which working tree this directory is in", result); + } + return root; + }, + + *objectFormat(): Operation { + const result = yield* asked(["git", "rev-parse", "--show-object-format"], yield* cwd()); + const format = objectFormat(result.stdout.trim()); + if (result.exitCode !== 0 || format === undefined) { + throw new GitRepositoryError("which object format it uses", result); + } + return format; + }, + + *readObject(commit: string, path: string): Operation { + // `cat-file blob` rather than `show`: it refuses a tree or a commit instead + // of rendering one, so a root document path that names a directory fails + // here rather than executing as whatever `show` chose to print. + const result = yield* asked(["git", "cat-file", "blob", `${commit}:${path}`], yield* cwd()); + if (result.exitCode !== 0) { + throw new GitObjectError(commit, path, result); + } + return result.stdout; + }, }); export const revParse: Operations["revParse"] = Git.operations.revParse; +export const repositoryRoot: Operations["repositoryRoot"] = Git.operations.repositoryRoot; +export const gitObjectFormat: Operations["objectFormat"] = Git.operations.objectFormat; +export const readGitObject: Operations["readObject"] = Git.operations.readObject; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a18cf6a8..f5982842 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -174,6 +174,9 @@ importers: '@executablemd/web': specifier: workspace:* version: link:../web + '@executablemd/workflow': + specifier: workspace:* + version: link:../workflow '@standard-schema/spec': specifier: ^1.0.0 version: 1.1.0 diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index de99659f..87619b99 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -144,6 +144,18 @@ const DENO_ONLY_TOOLING: RuntimeExclusion[] = [ "drives and against a real node:sqlite WorkflowRun database through the Deno DOFS Workspace adapter; node:sqlite remains behind --experimental-sqlite on Node 22", issue: "https://github.com/taras/executable.md/issues/366", }, + { + path: "packages/cli/tests/workflow-cli.test.ts", + reason: + "drives `xmd workflow start` and `resume` against a real node:sqlite run store, which only the Deno entrypoints open; under Node and Bun the command refuses, and workflow-host.test.ts asserts that refusal on every runtime", + issue: "https://github.com/taras/executable.md/issues/366", + }, + { + path: "packages/cli/tests/workflow-crash.test.ts", + reason: + "kills a real `xmd workflow start` child with SIGKILL and reads the recovered node:sqlite run database it leaves behind; the command only exists on the Deno entrypoints", + issue: "https://github.com/taras/executable.md/issues/366", + }, ]; /** diff --git a/site/routes/docs/index.tsx b/site/routes/docs/index.tsx index 71a26eec..65bcfcf0 100644 --- a/site/routes/docs/index.tsx +++ b/site/routes/docs/index.tsx @@ -57,13 +57,15 @@ export default define.page(function GettingStarted({ url }) { xmd workflow - Not yet shipped + start · resume - Will run a document in a retained Workspace so an interrupted - workflow can reattach and continue. Unsupported imperative - operations fail explicitly instead of falling back to the host. + Runs a document in a retained Workspace, so an interrupted workflow + resumes from its journal frontier instead of starting again. + Unsupported operations fail explicitly instead of falling back to + the host. Available through the Deno entrypoint and the compiled + binary. diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index e3557575..f8e05a23 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -1565,7 +1565,10 @@ run but are absent from the diagnostic trace. | `packages/cli/src/service-host.ts` | shared XMD service handshake observer and supervised host-process adapter | | `packages/cli/src/{deno,node,bun,compiled}-service.ts` | runtime-named service adapters for token, environment and stdio behavior | | `packages/cli/src/{deno,node,bun,compiled}.ts` | Entrypoints — each installs matching `API.Env` and `API.Service` adapters, then calls `runXmd` | -| `packages/workflow/src/service-denial.ts` | `useWorkflowServiceDenial()`, the tested non-delegating provider for future workflow start and resume scopes (#366) | +| `packages/workflow/src/service-denial.ts` | `useWorkflowServiceDenial()`, the non-delegating provider `xmd workflow` installs in every start and resume scope | +| `packages/cli/src/workflow.ts` | the runtime-neutral `xmd workflow` lifecycle: grammar, run opening, execution records, statuses and exit codes | +| `packages/cli/src/workflow-definition.ts` | establishing an immutable Git definition from a working-tree path, and loading a retained one again | +| `packages/cli/src/deno-workflow.ts` | the Deno run store and Workspace attachment; Node and Bun install the refusing host instead | | `packages/workflow/src/deno/workspace/files.ts` | the transaction-bound `API.Files` provider — one durable Workspace effect per document read, write and search | | `packages/workflow/src/deno/workspace/host.ts` | `withWorkflowWorkspace()` — the run's effect coordinator, logical cwd `/`, and Files provider installed together inside one execution | | `packages/workflow/src/journal.ts` | the `workflow_run` record, canonical-record recognition, and the refusals that name differing fields without their values | @@ -7778,6 +7781,36 @@ Defined in [Workflow runs](./workflow-spec.md) §10. | WF13 | Unreadable history | A recorded outcome carrying a member its variant does not have, a member of the wrong type, or a phase or reason the operation's vocabulary does not hold, is refused as exactly the cause-free `protocol` provider invariant — nothing the record held is repeated back, and no later file effect is performed | | WF14 | The transaction filesystem is the provider's | Contextual middleware installed from inside the document — including descriptors rebuilt for every name a filesystem seam has used — neither observes nor replaces the filesystem a Workspace transaction hands its body | +### Tier WFC — `xmd workflow start` and `xmd workflow resume` + +Defined in [Workflow workspaces](./workflow-workspace-spec.md) §3. + +| # | Test | Verify | +|---|------|--------| +| WFC1 | Two starts | Two starts without `--id` report two different run ids and both complete | +| WFC2 | Reuse | A compatible reuse of an explicit id addresses the same run; an incompatible one is refused, publishes no status and leaves the stored run as it was | +| WFC3 | Properties | Generated `--props-*` arguments and their help come from the pinned definition; every property form is refused on `resume` | +| WFC4 | Pinned bytes | An uncommitted working-tree edit does not reach the run: the committed document executes | +| WFC5 | Resume | A resume names only a run and finds its definition through retained metadata; a document argument is refused | +| WFC6 | Failure | A failing document exits 1 and retains `failed`; a compatible reuse replays that failure. The component search path is empty, so a component beside the definition never resolves | +| WFC7 | Storage | A missing run and an unreadable database are reported without a status line, and the database is left byte-identical | +| WFC8 | Grammar | A missing or unknown subcommand, a missing target, a third argument, `--id` on resume, an agent option and an inline document are each refused | +| WFC9 | Definition | A non-Markdown root and a path outside the repository fail before a run id is reported | +| WFC10 | `xmd run` | Ordinary `xmd run` is unchanged and still writes into the caller's own filesystem | + +### Tier WFH — Workflow host boundary + +| # | Test | Verify | +|---|------|--------| +| WFH1/WFH3 | Unsupported host | Node and Bun refuse `start` and `resume` with the settled sentence, report no run id or status, and create no run store | +| WFH2 | One grammar | Every runtime renders the same `xmd workflow` help | + +### Tier WFX — A killed run resumes from its frontier + +| # | Test | Verify | +|---|------|--------| +| WFX1 | SIGKILL and resume | A real `SIGKILL` part-way through leaves the run `running` with the effects that committed; the resume replays those exact events by id, performs the rest once each with no duplicate and no gap, advances the current root, and completes | + ### Tier SL — Own-scope context updates | # | Test | Verify | diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index c4b3da41..e1d9daaf 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -205,8 +205,19 @@ eligible history fork. It does not undo completed local or external effects. Only foreground execution that reaches `completed` exits zero. Suspended, failed, interrupted and cancelled executions have distinct nonzero outcomes so -shell automation cannot mistake an incomplete workflow for completion. Their -numeric assignments remain part of the CLI implementation contract. +shell automation cannot mistake an incomplete workflow for completion: + +| Status | Exit | +| --- | --- | +| `completed` | 0 | +| `failed` | 1 | +| `suspended` | 2 | +| `cancelled` | 3 | +| `interrupted` | 130 | + +A request the command refuses rather than runs — bad grammar, a missing run, an +incompatible reuse, damaged storage, an unsupported host — exits 1 and publishes +no status line. Management commands report their own request. `workflow cancel ` exits zero when cancellation succeeds even though the durable run status is `cancelled`. @@ -219,6 +230,49 @@ selector. The contract nevertheless contains no public SQLite access, local process path or other assumption that prevents a later host from owning the same run lifecycle remotely. +### 3.9 What is shipped + +The lifecycle above is the whole design. What a caller can run today is +`start` and `resume`: + +```sh +xmd workflow start [--id=] [--props-*=…] +xmd workflow resume +``` + +Both stream the document's own output to standard output and report two stable +lines on standard error — `workflow run: ` once the run has been created +or found, and `workflow status: ` once the execution settles. + +- `start` takes exactly one Markdown definition path. There is no generic + `--prop`, no `--journal`, no inline `--eval`, no agent option and no host + selector. Without `--id` the host generates an opaque cryptographically random + identifier, so starting the same document twice makes two runs; the local + caller may supply any storage-valid non-empty identifier, and hashing is what + keeps it from becoming a path. +- `resume` takes exactly one run id and nothing else. A document path, a + generated property argument and the aggregate property forms are each refused + rather than ignored. +- The document filesystem (§10.1) is the capability a run has. Repository, + Worktree, Git, Agent, Worker Shell and native services are not: an inherited + `API.Service` provider is refused rather than delegated to, and + `temporaryDirectory` is refused rather than answered with a host directory. +- A function-component root is not supported in this subset and fails before the + run executes. +- The command exists on every runtime and the capability on one: the Deno + entrypoint and the compiled binary own the local run store, and Node and Bun + refuse before creating or executing anything. + +Runs live beneath `~/.xmd/runs` unless `XMD_WORKFLOW_RUNS` names another +absolute directory. Where a run's database is on a host is arrangement, not +identity (§5.2). + +Status, list, history, cancel, fork and delete are designed above and unbuilt. +Concurrent-executor ownership is unbuilt: until it lands, a run left `running` +because its host disappeared is treated as an orphaned interrupted execution by +the next resume, which closes that unfinished execution record as `interrupted` +before beginning its own. + ## 4. Inspection commands ### 4.1 Status and list @@ -868,8 +922,9 @@ delegated without changing the document language. | retained run record and filtered journal | built by #291 | | caller-owned storage transaction | built by #291; Workspace mutations join it in #365 | | provider-backed retained Workspace | document filesystem built by #366; repository, process and attachment capabilities unbuilt (#218) | +| `xmd workflow start` / `resume` | built by #366, Deno entrypoints only | | Repository, Worktree and transactional Git components | defined here; unbuilt | -| lifecycle start/resume/status/history/fork/delete | defined here; unbuilt | +| lifecycle status/history/fork/delete | defined here; unbuilt | | read-only Agent materialization | defined here; proof required | | generated-XMD constrained evaluator | behavior defined; public name/schema open | | Deno-local DOFS persistence | POC proven by #349 / PR #350 |