diff --git a/architecture.md b/architecture.md index 4b029f3f..765d7906 100644 --- a/architecture.md +++ b/architecture.md @@ -134,14 +134,24 @@ or invoking Git. The supplied base must equal the recorded base. Git is not consulted to compare the current value of a moving branch with the pinned commit. +A host that has already created the run's storage record passes +`retainedWorkflowInstallation(run)` instead, with the exact frozen value. +Nothing is left for the execution to decide: it records that value through the +same `workflow_run` durable operation, allocates no identifier and resolves no +base, and every journal state — live, truncated and completed — requires the +recorded run to agree with the supplied one in run ID, base and pinned commit. A +journal that disagrees in any of them is not this run's journal, and the refusal +names the fields rather than their values. + `getWorkflowRun()` returns the frozen `WorkflowRun` for the current document execution. Every call in one live execution returns the same object. It throws -outside a document execution associated with `useWorkflow()`, and it exposes no -journal, Git, workspace or continuation capability. Replay preserves the field -values, not JavaScript object identity. +outside a document execution a workflow installation was passed to, and it +exposes no journal, Git, workspace or continuation capability. Replay preserves +the field values, not JavaScript object identity. -The `@executablemd/workflow` package owns `WorkflowRun`, `useWorkflow()`, -`getWorkflowRun()` and the Git capability. It depends on `@executablemd/core`, +The `@executablemd/workflow` package owns `WorkflowRun`, +`workflowInstallation()`, `retainedWorkflowInstallation()`, `getWorkflowRun()` +and the Git capability. It depends on `@executablemd/core`, `@executablemd/durable-streams` and `@executablemd/runtime`, whose contextual `exec()` and `cwd()` the Git provider invokes; core never imports workflow or Git. The future CLI lifecycle is `xmd workflow start` and `xmd workflow resume`; @@ -434,9 +444,9 @@ one operation does not enlist unrelated durable operations in the same scope. The Deno adapter establishes the canonical module's journal provenance for each WorkflowRun journal and retains that exact witness. The generic pre-persistence guard is policy-neutral and returns an unproven wrapper; the trusted -secret-filter wrapping site preserves provenance explicitly, so a filtered -journal — including one wrapped more than once — still carries the witness its -source carried. Before it opens a transaction, the adapter requires both the +secret-filter and execution-owned admission wrapping sites preserve provenance +explicitly, so a filtered journal — including one wrapped more than once — still +carries the witness its source carried. Before it opens a transaction, the adapter requires both the exact proof executor and the journal provenance from the consumed invocation to belong to the selected WorkflowRun. An in-memory stream, another run's journal, a copied property, an ordinary guard, a custom look-alike, or a wrapper another @@ -464,13 +474,14 @@ rather than invoking an untracked native Git side effect. Successful effect coordination finishes the mutation scope, including child cleanup, before capturing the resulting root. The Deno provider installs that -ordering for its adapter-private Workspace proof operation: the mutation +ordering for its Workspace effect operation: the mutation savepoint, root publication, filtered routed Yield, and caller-owned transaction -commit form one boundary. The proof filesystem uses the pinned synchronous DOFS +commit form one boundary. The Workspace filesystem uses the pinned synchronous DOFS entry points for its string and byte-array surface, so cancellation leaves no eager promise or stream pull able to reach the connection after transaction -authority ends. Public filesystem components and workflow lifecycle commands do -not yet select that operation. +authority ends. The transaction-bound Files provider selects that operation for +every document filesystem read, write and search; workflow lifecycle commands do +not select it yet. An external provider cannot join that transaction. Prompt, Git push and pull request effects derive a stable identity from the run and expansion, ask the @@ -543,11 +554,11 @@ after SQLite has restored the prior frontier. Retained roots, manifests and blobs remain indefinitely. Cloudflare garbage collection is not in the production closure and is never invoked. The provider -exposes no public Workspace mutation effect, history selection or fork -operation at this layer. Its adapter-private coordinator combines one mutation, -immutable-root publication and one filtered journal result atomically for the -provider-level proof; declarative `` and workflow start/resume do not -reach it yet. +exposes no public history selection or fork operation at this layer. Its +coordinator combines one mutation, immutable-root publication and one filtered +journal result atomically, and the transaction-bound Files provider is what +routes a document's `` and `` to it. Workflow start and resume do +not reach it yet. The coordinator treats only errors produced through its private filesystem adapter's documented path and mutation refusals as journalable operation @@ -937,6 +948,25 @@ than implied: Neither claim covers a native command a document runs. +The workflow provider's operations are durable effects. A read, a write and a +search each carry a description derived from the current expansion, the +operation and the resolved logical path, so one authored element is the same +effect across replays and a document edited to name another file is a different +one. `checkFilePath` is not among them: it is lexical admission, it performs no +effect, and it appends nothing. + +A write, the immutable root that results from it, and the filtered journal +result share the one caller-owned transaction. An ordinary refusal rolls its +mutation savepoint back before that result is published, so the retained +outcome describes a Workspace that is exactly what it was, and the reason that +crosses the boundary is selected from the shared vocabulary rather than derived +from anything the filesystem said. Replay restores those recorded outcomes +without performing the mutation, opening a transaction or consulting the current +frontier, which is why a create/delete/create history replays in order. A +temporary directory is refused outright: the provider has no host directory to +hand out, and falling through to the caller's would be the uncontained +filesystem the boundary exists to prevent. + Failure data crosses the boundary as a plain frozen object under a stable tag, carrying a reason from a fixed vocabulary and the phase it came from. No message, errno code, resolved path, temporary name, or symlink target crosses. @@ -1312,8 +1342,13 @@ It qualifies only while every one of these holds: - transfer happens only at a trusted wrapping site — one installed before any code the journal's content could influence, delegating to the exact stream it was handed. A document execution's journal passes through two: the secret - filter and the execution-owned target-admission wrapper. Each transfers only - what its source already had, so an unproven journal stays unproven; and + filter and the execution-owned admission wrapper. That second wrapper holds a + resumed run to its recorded root selection and, in the same read, applies + whatever an installation required of the history through its `admissions` — + workflow-run identity among them. Requirements are contributed to it rather + than wrapped around it, so no wrapping site is added and none of them is + reachable by middleware. Each transfers only what its source already had, so + an unproven journal stays unproven; and - it retains no execution, lifecycle, journal content or provider state. The exception exists because the exact-object, anti-forgery and loaded-copy @@ -1374,7 +1409,8 @@ Status is measured against main. | `xmd targets` | prints one document's catalog as full document references, by inspection alone | built on the #412 stack | | targeted `xmd run` | reads a file argument as a document reference and executes the one exact target its selector resolved to, replacing the selector before execution rereads the file | built on the #412 stack | | targeted workflow definition | the V1 workflow definition optionally carries the exact canonical document target, which takes part in definition identity and in compatible reuse | built on the #412 stack; the workflow CLI does not supply one yet | -| `useWorkflow()` / `getWorkflowRun()` | associates one document execution with a workflow run | built on main | +| `workflowInstallation()` / `getWorkflowRun()` | associates one document execution with a workflow run, through an `ExecutionInstallation` the trusted host passes to `executeInstalled()` | built on the #366 stack | +| `retainedWorkflowInstallation()` | associates one document execution with a run storage already created, requiring exact journal agreement | built on the #366 stack | | `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main | | workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; public workflow execution is unbuilt | | caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in | built on main | @@ -1385,7 +1421,7 @@ Status is measured against main. | `Config` run deadline / exec default / Fetch default | three independently owned contextual timeouts, absent unless configured, each read by exactly one consumer | built on main | | `API.Files` | routes every document filesystem operation to the installed provider, with no host default and structural failure data | built on the #227 stack | | host Files provider / `useHostFiles()` | resolves document paths in the caller's filesystem, containing them while the host namespace is stable; installed by all four CLI entrypoints | built on the #227 stack | -| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | unbuilt; the adapter is #227's second layer, and workflow effect coordination and CLI reachability remain unbuilt | +| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | built on the #366 stack; CLI reachability remains unbuilt | | `service=` | publishes the attachment's endpoint into the live binding overlay for its invocation | built on main | | `ephemeral eval` | reconstructs live middleware and bindings without a journal entry | built on main | | `useWorkflowServiceDenial()` | provides and tests a non-delegating workflow service denial provider; #366 will install it in future start and resume scopes | built on main; no workflow CLI execution branch exists yet | @@ -1394,7 +1430,7 @@ Status is measured against main. | Repository / Worktree / transactional Git effects | compose named checkouts and publish local mutations with their journal result | defined in `specs/workflow-workspace-spec.md`, unbuilt | | workflow inspection and history fork | reads status/history without advancing a run and creates a new run from a checkpoint | defined in `specs/workflow-workspace-spec.md`, unbuilt | | read-only workflow Agent / generated XMD | lets an Agent inspect a derived view and propose constrained executable changes | defined in `specs/workflow-workspace-spec.md`, unbuilt | -| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates an adapter-private mutation proof with its filtered Yield | built on the #365 stack; public mutation and workflow lifecycle reachability are unbuilt | +| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates one Workspace mutation with its filtered Yield | built on the #365 stack; public document filesystem effects route to it on the #366 stack, and workflow lifecycle reachability is unbuilt | | scoped Worker Shell | executes `just-bash` through the Workspace adapter inside a Deno Worker | containment and effect-transaction POCs complete (#351, #357); production integration unbuilt | | `` | retry a region until it completes | defined, unbuilt | | suspension effect | suspend durably | defined, unbuilt | diff --git a/packages/runtime/files.ts b/packages/runtime/files.ts index c3fbe1a0..769f1a50 100644 --- a/packages/runtime/files.ts +++ b/packages/runtime/files.ts @@ -443,6 +443,26 @@ function phaseOf(value: unknown): FilesPhase | undefined { return PHASES.find((phase) => phase === value); } +/** + * The vocabularies, for a provider that reads a failure back out of storage. + * + * A transaction-bound provider retains what it refused rather than a serialized + * error, so restoring one means turning stored text back into the vocabulary. + * Parsing it here is what keeps one list of reasons and phases: a provider that + * declared its own copy would be a second list to keep in agreement with this. + */ +export function parseFilesReason(value: unknown): FilesReason | undefined { + return reasonOf(value); +} + +export function parseFilesPhase(value: unknown): FilesPhase | undefined { + return phaseOf(value); +} + +export function parseFileWritePhase(value: unknown): FileWritePhase | undefined { + return writePhaseOf(value)?.[0]; +} + function invariantCategory(value: unknown): FilesInvariantCategory | undefined { return INVARIANT_CATEGORIES.find((category) => category === value); } diff --git a/packages/runtime/mod.ts b/packages/runtime/mod.ts index 4c150bf4..8e45aae5 100644 --- a/packages/runtime/mod.ts +++ b/packages/runtime/mod.ts @@ -85,7 +85,10 @@ export { fileWriteSuccess, filesFailure, isFilesFatal, + parseFilesPhase, + parseFilesReason, parseFileWriteFailure, + parseFileWritePhase, parseFileWriteSuccess, parseFilesFailure, parseFilesFatal, diff --git a/packages/workflow/deno.ts b/packages/workflow/deno.ts index fac983bd..b09361ae 100644 --- a/packages/workflow/deno.ts +++ b/packages/workflow/deno.ts @@ -28,3 +28,6 @@ export { useWorkflowRunStorage } from "./src/deno/provider.ts"; export type { WorkflowRunStorageOptions } from "./src/deno/provider.ts"; export { hashRunId, workflowRunPath } from "./src/deno/path.ts"; export { APPLICATION_ID, SCHEMA_VERSION } from "./src/deno/schema.ts"; +export { withWorkflowWorkspace } from "./src/deno/workspace/host.ts"; +export { WORKSPACE_FILE } from "./src/deno/workspace/files.ts"; +export { WORKSPACE_ROOT } from "./src/deno/workspace/logical-path.ts"; diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index f505148a..ae32b8e9 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -11,11 +11,13 @@ * data rather than from whoever happened to be holding the journal. * * ```ts - * import { useWorkflow } from "@executablemd/workflow"; - * import { execute } from "@executablemd/core"; + * import { workflowInstallation } from "@executablemd/workflow"; + * import { executeInstalled } from "@executablemd/core/host"; * - * yield* useWorkflow({ base: "main" }); - * const execution = yield* execute({ path: "./workflow.md", stream }); + * const execution = yield* executeInstalled( + * { path: "./workflow.md", stream }, + * [workflowInstallation({ base: "main" })], + * ); * ``` * * A run's durable record lives behind the Workflow Run Storage Api, which @@ -26,7 +28,7 @@ export { Git, GitRevisionError, revParse } from "./src/git.ts"; export type { GitApi } from "./src/git.ts"; -export { getWorkflowRun, useWorkflow } from "./src/run.ts"; +export { getWorkflowRun, retainedWorkflowInstallation, workflowInstallation } from "./src/run.ts"; export type { WorkflowRun } from "./src/run.ts"; export { useWorkflowServiceDenial, WorkflowServiceDeniedError } from "./src/service-denial.ts"; diff --git a/packages/workflow/src/deno/provider.ts b/packages/workflow/src/deno/provider.ts index 5d5379ae..6661c827 100644 --- a/packages/workflow/src/deno/provider.ts +++ b/packages/workflow/src/deno/provider.ts @@ -69,6 +69,7 @@ import { readTransaction } from "./reading.ts"; import { initializeSchema, isUninitialized, translateSqliteError, verifySchema } from "./schema.ts"; import { SavepointObservation } from "./savepoints.ts"; import { usePrivateWorkspace } from "./workspace/private.ts"; +import type { PrivateWorkspaceOptions } from "./workspace/private.ts"; import { useWorkspaceEffects } from "./workspace/effect.ts"; const INSERT_RUN = `INSERT INTO workflow_run @@ -107,14 +108,29 @@ export const WorkflowRunRecognition = createContext * long as the scope that installed the provider and nothing accumulates * between runs. */ -export function* useWorkflowRunStorage(options: WorkflowRunStorageOptions): Operation { +export function useWorkflowRunStorage(options: WorkflowRunStorageOptions): Operation { + return installWorkflowRunStorage(options, {}); +} + +/** + * The same installation, with what only this adapter's own suites supply. + * + * Kept apart from the published entrypoint on purpose: `internal` carries a + * decorator for the authoritative Workspace filesystem, which is provider + * authority rather than host arrangement. It is captured in the provider's + * closure here and never handed to a scope, a context or a descendant. + */ +export function* installWorkflowRunStorage( + options: WorkflowRunStorageOptions, + internal: PrivateWorkspaceOptions, +): Operation { const root = authorizedRoot(options.root); const connections = createWorkflowRunConnections(yield* SavepointObservation.get()); yield* ensure(() => { connections.close(); }); yield* useJournalRouting(connections); - yield* usePrivateWorkspace(connections); + yield* usePrivateWorkspace(connections, internal); yield* useWorkspaceEffects(connections); yield* WorkflowRunStorage.around( diff --git a/packages/workflow/src/deno/workspace/effect.ts b/packages/workflow/src/deno/workspace/effect.ts index 8c695ce2..e84ae526 100644 --- a/packages/workflow/src/deno/workspace/effect.ts +++ b/packages/workflow/src/deno/workspace/effect.ts @@ -218,7 +218,7 @@ export function withWorkspaceEffects( }); } -export function createWorkspaceProofEffect( +export function createWorkspaceEffect( database: WorkflowRunDatabase, description: EffectDescription, mutate: DenoWorkspaceMutation, diff --git a/packages/workflow/src/deno/workspace/errors.ts b/packages/workflow/src/deno/workspace/errors.ts index ebd6c156..723aa1c3 100644 --- a/packages/workflow/src/deno/workspace/errors.ts +++ b/packages/workflow/src/deno/workspace/errors.ts @@ -47,3 +47,16 @@ export function throwWorkspaceFilesystemFailure(error: unknown): never { export function isJournalableWorkspaceFailure(error: unknown): error is Error { return error instanceof JournalableWorkspaceFailure; } + +/** + * The documented filesystem condition this failure is, or `undefined` for one + * that is not documented. + * + * The code is the only part of a DOFS failure anything above this module reads. + * Its message, its cause and the paths either of them names stay here, so a + * consumer selecting a `FilesReason` from this receives a condition rather than + * platform text. + */ +export function journalableWorkspaceCode(error: unknown): string | undefined { + return error instanceof JournalableWorkspaceFailure ? error.code : undefined; +} diff --git a/packages/workflow/src/deno/workspace/files.ts b/packages/workflow/src/deno/workspace/files.ts new file mode 100644 index 00000000..199d8577 --- /dev/null +++ b/packages/workflow/src/deno/workspace/files.ts @@ -0,0 +1,651 @@ +/** + * The transaction-bound `API.Files` provider — a document's filesystem inside a + * workflow run. + * + * This is what `xmd workflow` installs where `xmd run` installs the host + * adapter. A document names the same paths and `` calls the same + * operations; what changes is where those paths land. Here they land in the + * run's own logical Workspace, and every read, write and search is one durable + * effect published by the run's effect transaction — the mutation, the + * resulting immutable Workspace root and the filtered journal result commit + * together or not at all. + * + * ## Why an authored path never reaches a host filesystem call + * + * Resolution is arithmetic on POSIX segments rooted at `/`, and the result is + * handed to the run's DOFS filesystem. No host path appears anywhere in it, so + * the containment claim needs no stable-namespace qualification: nothing + * outside the Workspace can be named, and no other process can replace part of + * a tree that lives inside one database. + * + * `checkFilePath` stays what the Api says it is — pure lexical admission that + * hands back nothing usable. It performs no effect and appends no journal + * entry, so a check that was skipped or answered elsewhere authorizes nothing; + * the write repeats the same admission from the same authored path. + * + * ## What replay does instead + * + * A recorded effect restores its recorded value. A read therefore answers with + * the bytes it read when it ran, even where the current frontier no longer + * holds them, and a write already recorded neither mutates nor captures a root + * again. Nothing here consults current state to decide whether an earlier + * effect happened, which is what lets a create/delete/create history replay in + * order rather than collapsing to whatever the file is now. + * + * ## What crosses the boundary + * + * A documented DOFS refusal selects a `FilesReason` and nothing else travels + * with it: no DOFS message, no errno payload, no SQLite text, no resolved path. + * A refusal is also a *rolled back* refusal — the mutation runs inside a + * savepoint of its own, so partial logical mutation is discarded before the + * sanitized result is durably published, and the run's current root is the one + * it was before. + * + * Everything that is not a documented refusal — connection, authority, + * savepoint, capture, publication, routing, teardown and commit failure — stays + * an infrastructure failure and fails the run. None of them is something a + * document did, and printing one would let the work after this file work run as + * though the file work had happened. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import { globToRegExp } from "@effectionx/fs"; +import { getExpansion } from "@executablemd/core"; +import { + Files, + FilesInvariantError, + FilesOperationDeniedError, + filesFailure, + fileWriteFailure, + fileWriteSuccess, + parseFilesPhase, + parseFilesReason, + parseFileWritePhase, +} from "@executablemd/runtime"; +import type { + FilePathInput, + FilesPhase, + FilesReason, + FileWriteInput, + FileWritePhase, + FileWriteSuccess, + GlobInput, +} from "@executablemd/runtime"; +import type { EffectDescription, Json, Workflow } from "@executablemd/durable-streams"; +import type { WorkflowRunDatabase } from "../../storage/api.ts"; +import { savepoint } from "../transaction.ts"; +import { createWorkspaceEffect } from "./effect.ts"; +import { journalableWorkspaceCode } from "./errors.ts"; +import type { DenoWorkspaceFilesystem, DenoWorkspaceStat } from "./filesystem.ts"; +import { + logicalDirectory, + logicalJoin, + logicalParent, + LogicalPathError, + resolveLogicalPath, + WORKSPACE_ROOT, +} from "./logical-path.ts"; + +/** The effect type every document filesystem operation in a run is recorded under. */ +export const WORKSPACE_FILE = "workspace_file"; + +/** + * The condition each documented DOFS code reports as. + * + * A `Map` rather than an object literal, because a lookup on one answers for + * inherited keys and the code comes from the filesystem rather than from here. + */ +const REASON_BY_CODE: ReadonlyMap = new Map([ + ["ENOENT", "missing"], + ["ENOTDIR", "not-directory"], + ["EISDIR", "directory"], + ["ENOTEMPTY", "directory-not-empty"], + ["EACCES", "permission-denied"], + ["EPERM", "permission-denied"], + ["EROFS", "read-only"], + ["ELOOP", "too-many-symlinks"], +]); + +/** A documented filesystem condition, carrying its reason and nothing else. */ +class WorkspaceRefusal extends Error { + override name = "WorkspaceRefusal"; + readonly reason: FilesReason; + + constructor(reason: FilesReason) { + super("workspace filesystem refused"); + this.reason = reason; + } +} + +/** + * The refusal this failure is, or a rethrow when it is not one. + * + * Rethrowing is what keeps infrastructure failures infrastructure failures: a + * condition DOFS never documented is not something a document did, and turning + * it into a printable reason would let the work after this file work run as + * though the file work had happened. + */ +function asRefusal(error: unknown): WorkspaceRefusal { + const code = journalableWorkspaceCode(error); + if (code === undefined) { + throw error; + } + return new WorkspaceRefusal(REASON_BY_CODE.get(code) ?? "operation-failed"); +} + +function refusalReason(error: Error): FilesReason { + return error instanceof WorkspaceRefusal ? error.reason : "operation-failed"; +} + +function lexicalReason(error: Error): FilesReason { + return error instanceof LogicalPathError ? error.reason : "operation-failed"; +} + +/** + * What one file effect recorded. + * + * A JSON value, because it is what the journal holds and what a replay hands + * back. A refusal is carried as a phase and a reason rather than as a + * serialized error, so nothing a filesystem said is retained and a restored + * refusal is rebuilt from the same vocabulary a live one is. + */ +type FileEffectOutcome = + | { readonly kind: "content"; readonly content: string } + | { readonly kind: "written" } + | { readonly kind: "paths"; readonly paths: string[] } + | { readonly kind: "refused"; readonly phase: Phase; readonly reason: FilesReason }; + +function refused( + phase: Phase, + reason: FilesReason, +): FileEffectOutcome { + return { kind: "refused", phase, reason }; +} + +/** + * The whole of what each outcome carries. + * + * A `Map` rather than an object literal, because the discriminant is read from + * the journal and a lookup on an object answers for keys `Object.prototype` + * happens to hold. A record carrying anything beyond its variant's members — + * `written` with content, `content` with a reason — describes two outcomes at + * once and is therefore no outcome at all. + */ +const OUTCOME_MEMBERS: ReadonlyMap = new Map([ + ["content", ["kind", "content"]], + ["written", ["kind"]], + ["paths", ["kind", "paths"]], + ["refused", ["kind", "phase", "reason"]], +]); + +function carriesExactly(record: Record, kind: string): boolean { + const members = OUTCOME_MEMBERS.get(kind); + if (members === undefined) { + return false; + } + return ( + Object.keys(record).length === members.length && + members.every((member) => Object.hasOwn(record, member)) + ); +} + +function readPaths(value: unknown): string[] | undefined { + if (!Array.isArray(value)) { + return undefined; + } + const paths: string[] = []; + for (const entry of value) { + if (typeof entry !== "string") { + return undefined; + } + paths.push(entry); + } + return paths; +} + +/** + * The outcome a journal record describes, or `undefined` when it describes none. + * + * The journal is parsed, never trusted, and parsing here is total: a record must + * carry its variant's members and no others, each of the declared type, and a + * refusal's phase and reason must both be words this operation's vocabulary + * holds. A record this cannot read has no printable reading, so the caller turns + * it into one fixed provider invariant rather than inventing a filesystem + * condition that was never reported. + */ +function parseOutcome( + value: unknown, + parsePhase: (value: unknown) => Phase | undefined, +): FileEffectOutcome | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + const record: Record = Object.fromEntries(Object.entries(value)); + const kind = record.kind; + if (typeof kind !== "string" || !carriesExactly(record, kind)) { + return undefined; + } + if (kind === "content") { + return typeof record.content === "string" + ? { kind: "content", content: record.content } + : undefined; + } + if (kind === "written") { + return { kind: "written" }; + } + if (kind === "paths") { + const paths = readPaths(record.paths); + return paths === undefined ? undefined : { kind: "paths", paths }; + } + const phase = parsePhase(record.phase); + const reason = parseFilesReason(record.reason); + if (phase === undefined || reason === undefined) { + return undefined; + } + return { kind: "refused", phase, reason }; +} + +/** + * How one file effect is identified, deterministically. + * + * The expansion is what makes two `` elements different effects and one + * element the same effect across replays; the operation separates a read from a + * write performed by the same element; and the resolved logical target is what + * a changed authored path or a changed working directory moves. A document + * edited to name another file therefore diverges rather than quietly replaying + * the previous file's recorded bytes. + */ +function* describeFileEffect( + operation: string, + target: string, + detail: Record, +): Operation { + const expansion = yield* getExpansion(); + return { type: WORKSPACE_FILE, name: `${operation}:${expansion.id}:${target}`, ...detail }; +} + +function* fileEffect( + database: WorkflowRunDatabase, + description: EffectDescription, + perform: (filesystem: DenoWorkspaceFilesystem) => Operation>, +): Workflow { + return yield createWorkspaceEffect(database, description, (filesystem) => perform(filesystem)); +} + +/** + * Run one file effect and read back what it recorded. + * + * The same path serves a live effect and a replayed one: live execution + * publishes the outcome and hands it back, replay hands back the outcome that + * was published. Neither branch is written twice here, which is what makes + * "replay restores the recorded result" a property of the code rather than a + * claim about it. + */ +function* performed( + database: WorkflowRunDatabase, + description: EffectDescription, + parsePhase: (value: unknown) => Phase | undefined, + perform: (filesystem: DenoWorkspaceFilesystem) => Operation>, +): Operation> { + const outcome = parseOutcome(yield* fileEffect(database, description, perform), parsePhase); + if (outcome === undefined) { + throw new FilesInvariantError("protocol"); + } + return outcome; +} + +function* statPath( + filesystem: DenoWorkspaceFilesystem, + path: string, +): Operation> { + try { + return Ok(yield* filesystem.stat(path)); + } catch (error) { + return Err(asRefusal(error)); + } +} + +function* readOutcome( + filesystem: DenoWorkspaceFilesystem, + path: string, +): Operation> { + const info = yield* statPath(filesystem, path); + if (!info.ok) { + return refused("resolution", refusalReason(info.error)); + } + if (info.value.kind !== "file") { + return refused("target", "directory"); + } + try { + return { kind: "content", content: yield* filesystem.readTextFile(path) }; + } catch (error) { + return refused("access", refusalReason(asRefusal(error))); + } +} + +/** + * What the target already is, when that decides the write before it starts. + * + * A directory cannot become a file, and saying so before anything is attempted + * is what keeps the target claim `unchanged` rather than `rolled-back`. A path + * that does not exist yet is the ordinary case and answers `undefined`. + */ +function* classifyWriteTarget( + filesystem: DenoWorkspaceFilesystem, + path: string, +): Operation | undefined> { + const info = yield* statPath(filesystem, path); + if (info.ok) { + return info.value.kind === "file" ? undefined : refused("target", "directory"); + } + const reason = refusalReason(info.error); + return reason === "missing" ? undefined : refused("target", reason); +} + +function* replace( + filesystem: DenoWorkspaceFilesystem, + parent: string, + path: string, + content: string, +): Operation { + if (parent !== WORKSPACE_ROOT) { + yield* filesystem.mkdir(parent, { recursive: true }); + } + yield* filesystem.writeFile(path, content); +} + +/** + * Replace one file, discarding every part of the attempt if any part refuses. + * + * The parents and the replacement share one savepoint, so a write that creates + * two directories and then cannot be written leaves neither behind. The refusal + * that comes back therefore describes a Workspace that is exactly what it was, + * which is the `rolled-back` target claim the write vocabulary already has. + */ +function* writeOutcome( + filesystem: DenoWorkspaceFilesystem, + path: string, + content: string, +): Operation> { + const existing = yield* classifyWriteTarget(filesystem, path); + if (existing !== undefined) { + return existing; + } + + try { + yield* savepoint(replace(filesystem, logicalParent(path), path, content)); + } catch (error) { + return refused("transaction", refusalReason(asRefusal(error))); + } + return { kind: "written" }; +} + +const SUBTREE = "/**"; + +function toRegExp(pattern: string): RegExp { + return globToRegExp(pattern, { extended: true, globstar: true }); +} + +/** + * A matcher for directories whose entire subtree an exclusion covers. + * + * Only a trailing `/**` proves it: matching the directory itself says nothing + * about the files beneath it, so anything else is walked and filtered one file + * at a time. Descending a subtree whose files are all excluded costs reads; + * skipping one that holds a match loses the match. + */ +function pruneMatcher(pattern: string): RegExp | undefined { + if (pattern === "**") { + return toRegExp("**"); + } + if (!pattern.endsWith(SUBTREE)) { + return undefined; + } + return toRegExp(pattern.slice(0, -SUBTREE.length)); +} + +interface Traversal { + readonly include: RegExp[]; + readonly exclude: RegExp[]; + readonly prune: RegExp[]; + readonly matched: string[]; +} + +/** + * Collect matching regular files beneath one logical directory. + * + * A search answers with regular files, which is what `API.Files` says a search + * answers with wherever it runs. A symbolic link is neither reported nor + * descended through, so a link is not a result and no target is reached twice or + * reached at all through a name outside the walk — traversal stays inside the + * directory it started in and cannot cycle. `readdir` classifies a link by what + * the entry is rather than by what it points at, so a link to a directory is + * refused on the same terms as a link to a file. + * + * Exclusion is decided per candidate: a file whose own relative path an + * exclusion matches is not reported. A directory is never a candidate, so its + * own path is not tested against exclusions at all — the only question it raises + * is whether walking it can still produce something. + */ +function* descend( + filesystem: DenoWorkspaceFilesystem, + directory: string, + prefix: string, + walk: Traversal, +): Operation { + for (const entry of yield* filesystem.readdir(directory)) { + const path = prefix === "" ? entry.name : `${prefix}/${entry.name}`; + + if (entry.kind === "directory") { + if (!walk.prune.some((expression) => expression.test(path))) { + yield* descend(filesystem, logicalJoin(directory, entry.name), path, walk); + } + continue; + } + + if (entry.kind !== "file") { + continue; + } + if (walk.exclude.some((expression) => expression.test(path))) { + continue; + } + if (walk.include.some((expression) => expression.test(path))) { + walk.matched.push(path); + } + } +} + +function byCodePoint(left: string, right: string): number { + return left < right ? -1 : left > right ? 1 : 0; +} + +function compile(include: string[], exclude: string[]): Result { + try { + return Ok({ + include: include.map(toRegExp), + exclude: exclude.map(toRegExp), + prune: exclude + .map(pruneMatcher) + .filter((expression): expression is RegExp => expression !== undefined), + matched: [], + }); + } catch (error) { + // The patterns are compiled before the walk, so an unusable one — an + // unterminated character class — arrives as a `SyntaxError` from `RegExp`. + // It is the one failure here a document can fix by editing what it wrote. + if (error instanceof SyntaxError) { + return Err(new WorkspaceRefusal("invalid-pattern")); + } + throw error; + } +} + +function* globOutcome( + filesystem: DenoWorkspaceFilesystem, + directory: string, + include: string[], + exclude: string[], +): Operation> { + const info = yield* statPath(filesystem, directory); + if (!info.ok) { + return refused("target", refusalReason(info.error)); + } + if (info.value.kind !== "directory") { + return refused("target", "not-directory"); + } + + const walk = compile(include, exclude); + if (!walk.ok) { + return refused("pattern", refusalReason(walk.error)); + } + + try { + yield* descend(filesystem, directory, "", walk.value); + } catch (error) { + return refused("traversal", refusalReason(asRefusal(error))); + } + return { kind: "paths", paths: [...new Set(walk.value.matched)].sort(byCodePoint) }; +} + +/** The document filesystem of one workflow run. */ +export interface WorkflowFilesHandler { + checkFilePath(input: FilePathInput): Operation>; + readTextFile(input: FilePathInput): Operation>; + writeTextFile(input: FileWriteInput): Operation>; + globFiles(input: GlobInput): Operation>; + temporaryDirectory(): Operation>; +} + +export function workflowFilesHandler(database: WorkflowRunDatabase): WorkflowFilesHandler { + return { + // deno-lint-ignore require-yield + *checkFilePath(input: FilePathInput): Operation> { + const resolved = resolveLogicalPath(input.cwd, input.path); + if (!resolved.ok) { + return Err( + filesFailure({ + operation: "check-file-path", + phase: "lexical", + reason: lexicalReason(resolved.error), + }), + ); + } + return Ok(undefined); + }, + + *readTextFile(input: FilePathInput): Operation> { + const resolved = resolveLogicalPath(input.cwd, input.path); + if (!resolved.ok) { + return Err( + filesFailure({ + operation: "read", + phase: "lexical", + reason: lexicalReason(resolved.error), + }), + ); + } + const path = resolved.value; + const outcome = yield* performed( + database, + yield* describeFileEffect("read", path, { path: input.path, cwd: input.cwd }), + parseFilesPhase, + (filesystem) => readOutcome(filesystem, path), + ); + if (outcome.kind === "refused") { + return Err( + filesFailure({ operation: "read", phase: outcome.phase, reason: outcome.reason }), + ); + } + if (outcome.kind !== "content") { + throw new FilesInvariantError("protocol"); + } + return Ok(outcome.content); + }, + + *writeTextFile(input: FileWriteInput): Operation> { + const resolved = resolveLogicalPath(input.cwd, input.path); + if (!resolved.ok) { + return Err(fileWriteFailure({ phase: "lexical", reason: lexicalReason(resolved.error) })); + } + const path = resolved.value; + const outcome = yield* performed( + database, + yield* describeFileEffect("write", path, { path: input.path, cwd: input.cwd }), + parseFileWritePhase, + (filesystem) => writeOutcome(filesystem, path, input.content), + ); + if (outcome.kind === "refused") { + return Err(fileWriteFailure({ phase: outcome.phase, reason: outcome.reason })); + } + if (outcome.kind !== "written") { + throw new FilesInvariantError("protocol"); + } + return Ok(fileWriteSuccess("transaction-staged")); + }, + + *globFiles(input: GlobInput): Operation> { + const directory = logicalDirectory(input.cwd); + const include = [...input.include]; + const exclude = [...input.exclude]; + const outcome = yield* performed( + database, + yield* describeFileEffect("glob", directory, { include, exclude }), + parseFilesPhase, + (filesystem) => globOutcome(filesystem, directory, include, exclude), + ); + if (outcome.kind === "refused") { + return Err( + filesFailure({ operation: "glob", phase: outcome.phase, reason: outcome.reason }), + ); + } + if (outcome.kind !== "paths") { + throw new FilesInvariantError("protocol"); + } + return Ok(outcome.paths); + }, + + /** + * A workflow run has no host directory to hand out. + * + * Denied rather than emulated inside the Workspace: `` exists so a + * document can hand a path to a tool the caller already has, and a logical + * path is not one. Falling through to the host would give a run exactly the + * unretained, uncontained filesystem the boundary exists to keep it out of. + */ + // deno-lint-ignore require-yield + *temporaryDirectory(): Operation> { + throw new FilesOperationDeniedError("temporary-directory"); + }, + }; +} + +/** + * Install this run's document filesystem for the current scope and below. + * + * `{ at: "min" }` on the same terms as every other provider: an outer host + * adapter installed by the CLI entrypoint would otherwise answer ahead of this + * one, and the whole point of a workflow run is that it does not. + */ +export function useWorkflowFiles(database: WorkflowRunDatabase): Operation { + const handler = workflowFilesHandler(database); + return Files.around( + { + *checkFilePath([input]) { + return yield* handler.checkFilePath(input); + }, + *readTextFile([input]) { + return yield* handler.readTextFile(input); + }, + *writeTextFile([input]) { + return yield* handler.writeTextFile(input); + }, + *globFiles([input]) { + return yield* handler.globFiles(input); + }, + *temporaryDirectory() { + return yield* handler.temporaryDirectory(); + }, + }, + { at: "min" }, + ); +} diff --git a/packages/workflow/src/deno/workspace/host.ts b/packages/workflow/src/deno/workspace/host.ts new file mode 100644 index 00000000..4857ebf0 --- /dev/null +++ b/packages/workflow/src/deno/workspace/host.ts @@ -0,0 +1,69 @@ +/** + * What a host installs around one workflow document execution. + * + * Three installations, in one place, because they only make sense together: the + * run's effect coordinator decides how a Workspace effect commits, the Files + * provider is what turns a document's `` into one of those effects, and + * the logical working directory is what those paths are relative to. Installing + * any two without the third would leave a document resolving paths one provider + * cannot reach. + * + * They are installed **inside** the execution rather than at the entrypoint, so + * they sit beneath the host adapter `xmd run` installs and answer ahead of it. + * Ordinary `xmd run` keeps its host Files provider untouched; a workflow run's + * document never reaches it. + * + * This is the attachment path, and a completed run does not take it. A root + * result that is already recorded returns without expanding the document, so + * there is nothing to give a filesystem to — and attaching one anyway would + * open a transaction and capture a root for a run that is not going to perform + * an effect. + * + * `withWorkflowWorkspace()` is therefore the whole of what a host may install. + * The three pieces are not published separately: the Files provider alone would + * resolve a document's paths against whatever working directory the host adapter + * answers with, and a host path resolved that way is retained in the durable + * effects a run replays from. + */ + +import { scoped, type Operation } from "effection"; +import { API } from "@executablemd/runtime"; +import type { WorkflowRunDatabase } from "../../storage/api.ts"; +import { withWorkspaceEffects } from "./effect.ts"; +import { useWorkflowFiles } from "./files.ts"; +import { WORKSPACE_ROOT } from "./logical-path.ts"; + +/** + * The working directory a workflow document starts in. + * + * The Workspace root, and a logical path rather than a host one. A document + * that resolves `notes.md` against it names an entry in the run's own + * filesystem, and nothing it can write reaches the directory the caller + * happened to invoke `xmd` from. + */ +function useLogicalWorkspaceCwd(): Operation { + return API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return WORKSPACE_ROOT; + }, + }, + { at: "min" }, + ); +} + +/** Run `operation` with this run's Workspace attached to the document filesystem. */ +export function withWorkflowWorkspace( + database: WorkflowRunDatabase, + operation: Operation, +): Operation { + return withWorkspaceEffects( + database, + scoped(function* () { + yield* useLogicalWorkspaceCwd(); + yield* useWorkflowFiles(database); + return yield* operation; + }), + ); +} diff --git a/packages/workflow/src/deno/workspace/logical-path.ts b/packages/workflow/src/deno/workspace/logical-path.ts new file mode 100644 index 00000000..1b874ddf --- /dev/null +++ b/packages/workflow/src/deno/workspace/logical-path.ts @@ -0,0 +1,127 @@ +/** + * Where a document's path lands in a run's logical Workspace. + * + * Every path a workflow document writes is resolved here, and the result is an + * absolute POSIX path inside the run's own filesystem. Nothing this module + * produces is a host path: there is no drive, no separator to choose, and no + * outside for a resolution to reach — the Workspace root *is* the boundary, so + * containment is decided by arithmetic on segments rather than by observing a + * filesystem. + * + * That is why admission here is purely lexical. The host provider has to defer + * part of its judgement until it can see a symlink, because a host path can + * point anywhere; a logical path resolves inside a tree the run owns entirely, + * and `/..` is `/` the way POSIX says it is. + */ + +import { Err, Ok, type Result } from "effection"; +import type { FilesReason } from "@executablemd/runtime"; + +/** A path the document wrote that names nothing this Workspace can hold. */ +export class LogicalPathError extends Error { + override name = "LogicalPathError"; + readonly reason: FilesReason; + + constructor(reason: FilesReason) { + super("logical path refused"); + this.reason = reason; + } +} + +export const WORKSPACE_ROOT = "/"; + +/** No filesystem holds a name containing one, so no path here may carry one. */ +const NUL = "\u0000"; + +/** + * The segments of a directory this Workspace can be working in. + * + * A caller's working directory is arrangement rather than a document's own + * text, so it is clamped rather than refused: a leading `..` at the root stays + * at the root, exactly as it would in a POSIX filesystem, and a directory that + * is not written as an absolute path is read relative to the root. Neither can + * name anything outside the Workspace, which is the only property this needs. + */ +function directorySegments(cwd: string): string[] { + const segments: string[] = []; + for (const segment of cwd.split("/")) { + if (segment === "" || segment === ".") { + continue; + } + if (segment === "..") { + segments.pop(); + continue; + } + segments.push(segment); + } + return segments; +} + +function posix(segments: readonly string[]): string { + return segments.length === 0 ? WORKSPACE_ROOT : `/${segments.join("/")}`; +} + +/** The absolute logical directory a document is working in. */ +export function logicalDirectory(cwd: string): string { + return posix(directorySegments(cwd)); +} + +/** + * The absolute logical path an authored path names, or why it names none. + * + * The three lexical refusals are the ones a document can act on: it wrote + * nothing, it wrote somewhere absolute, or it wrote its way out of the + * directory it is working in. A NUL is none of those, so it is reported as an + * operation that cannot be carried out rather than described back to the + * document. + */ +export function resolveLogicalPath(cwd: string, path: string): Result { + if (path === "") { + return Err(new LogicalPathError("empty-path")); + } + if (path.startsWith("/")) { + return Err(new LogicalPathError("absolute-path")); + } + if (path.includes(NUL) || cwd.includes(NUL)) { + return Err(new LogicalPathError("operation-failed")); + } + + const base = directorySegments(cwd); + const segments = [...base]; + for (const segment of path.split("/")) { + if (segment === "" || segment === ".") { + continue; + } + if (segment === "..") { + if (segments.length <= base.length) { + return Err(new LogicalPathError("lexical-escape")); + } + segments.pop(); + continue; + } + segments.push(segment); + } + + // `.` and `a/..` normalize back onto the working directory. That is not an + // escape, and saying so would misdescribe it: the path names a directory, and + // target classification is what reports that. + return Ok(posix(segments)); +} + +/** The logical parent directory of an absolute logical path. */ +export function logicalParent(path: string): string { + const segments = path.split("/").filter((segment) => segment !== ""); + segments.pop(); + return posix(segments); +} + +/** The path of `entry` relative to `directory`, POSIX-separated. */ +export function logicalRelative(directory: string, entry: string): string { + const base = directory === WORKSPACE_ROOT ? "" : directory; + return entry.startsWith(`${base}/`) ? entry.slice(base.length + 1) : entry; +} + +/** One more segment beneath an absolute logical directory. */ +export function logicalJoin(directory: string, name: string): string { + return directory === WORKSPACE_ROOT ? `/${name}` : `${directory}/${name}`; +} diff --git a/packages/workflow/src/deno/workspace/private.ts b/packages/workflow/src/deno/workspace/private.ts index 713087ef..44e04e5d 100644 --- a/packages/workflow/src/deno/workspace/private.ts +++ b/packages/workflow/src/deno/workspace/private.ts @@ -42,6 +42,29 @@ function unavailable(): never { ); } +/** + * What decorates the filesystem a Workspace transaction hands its body. + * + * Dependency injection, deliberately, rather than anything a scope can reach. + * It is supplied once when the storage provider is installed — before any + * document exists — and lives in that provider's closure. There is no context + * name to reconstruct, nothing is handed to a descendant, and no module-scoped + * hook accumulates: a component cannot observe the authoritative filesystem, + * and cannot put anything in front of it. + * + * It exists because a mutation discarded part-way through cannot otherwise be + * observed. A write creates the parent directories it needs and then writes the + * file, and DOFS has no condition that stops between the two — a parent chain + * that can be created is a chain the file can then be written into. + */ +export type WorkspaceFilesystemDecorator = ( + filesystem: DenoWorkspaceFilesystem, +) => DenoWorkspaceFilesystem; + +export interface PrivateWorkspaceOptions { + readonly decorateFilesystem?: WorkspaceFilesystemDecorator; +} + const PrivateWorkspace: Api = createApi( "executablemd.workflow.deno.workspace.private", { @@ -68,7 +91,11 @@ const PrivateWorkspace: Api = createApi { +export function usePrivateWorkspace( + connections: WorkflowRunConnections, + options: PrivateWorkspaceOptions = {}, +): Operation { + const decorate = options.decorateFilesystem ?? ((filesystem) => filesystem); return PrivateWorkspace.around( { *transact([database, transaction, body]: [ @@ -85,7 +112,7 @@ export function usePrivateWorkspace(connections: WorkflowRunConnections): Operat connections.authorizeTransaction(database, transaction); }; const workspace: PrivateWorkspaceTransaction = { - filesystem: createDenoWorkspaceFilesystem(connection, authorize), + filesystem: decorate(createDenoWorkspaceFilesystem(connection, authorize)), // deno-lint-ignore require-yield *currentRoot(): Operation { diff --git a/packages/workflow/src/journal.ts b/packages/workflow/src/journal.ts index bc854d28..4dd6781e 100644 --- a/packages/workflow/src/journal.ts +++ b/packages/workflow/src/journal.ts @@ -1,15 +1,23 @@ /** - * The durable `workflow_run` record. + * The durable `workflow_run` record, and what a history has to hold to be a + * run's own. * * One immutable value per workflow run, written before the root document is * imported. The journal is parsed, never trusted: a record that does not * describe a workflow run is refused rather than coerced, and a record made * from a different base is refused rather than quietly standing in for this * run's. + * + * Recognition is deliberately narrow. A record identifies a run only when it is + * the root coroutine's own successfully settled Yield, under the canonical type + * *and* the canonical name, holding a closed value of exactly the three members + * a run has. Anything looser lets a same-typed Yield written under another name, + * or by a child coroutine, stand in for the record that was removed — and a + * recorded terminal result would then be reused on its authority. */ import { StaleInputError } from "@executablemd/durable-streams"; -import type { EffectDescription } from "@executablemd/durable-streams"; +import type { DurableEvent, EffectDescription } from "@executablemd/durable-streams"; /** * One workflow run: an opaque identifier, the base that was asked for, and the @@ -23,23 +31,194 @@ export interface WorkflowRun { export const WORKFLOW_RUN = "workflow_run"; +/** The coroutine a document execution's own history belongs to. */ +const ROOT_COROUTINE = "root"; + +const RUN_MEMBERS: readonly string[] = ["runId", "base", "pinnedCommit"]; + /** How the record identifies itself. `base` is for a reader, never for matching. */ export function describeWorkflowRun(base: string): EffectDescription { return { type: WORKFLOW_RUN, name: WORKFLOW_RUN, base }; } -/** The workflow run a stored value describes, or `undefined` if it describes none. */ +/** + * What a refusal is allowed to carry about the run. + * + * `StaleInputError` retains the description it is given, so handing it the + * recording description would keep the base reachable on the error object even + * though no message prints it. A fresh value each time, holding the two members + * that name the effect and nothing else. + */ +function refusalDescription(): EffectDescription { + return { type: WORKFLOW_RUN, name: WORKFLOW_RUN }; +} + +/** + * Read one value the journal supplied, or answer that reading it refused. + * + * Deliberately narrow: exactly one read of exactly one journal-controlled + * value is inside. A hostile `ownKeys` trap, a `getOwnPropertyDescriptor` trap, + * a throwing getter and a revoked proxy all raise from here, and all of them + * mean the same thing — this value does not describe a run. Widening it would + * start converting programmer and infrastructure errors into "malformed + * journal", which is the opposite of a total parse. + */ +function readingJournalValue(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +/** + * The workflow run a stored value describes, or `undefined` if it describes none. + * + * Closed over its three members: a value carrying a fourth is not a run with + * something extra, it is a value this version cannot account for. + * + * Total over anything the journal can hold. Classification and enumeration are + * both the value's to refuse — `Array.isArray` throws for a revoked proxy, and + * `Object.entries` runs the traps and the getters — so both happen inside one + * guarded read and a refusal is an answer rather than an exception carrying the + * journal's own text out with it. + */ export function readWorkflowRun(value: unknown): WorkflowRun | undefined { - if (typeof value !== "object" || value === null || Array.isArray(value)) { + if (typeof value !== "object" || value === null) { return undefined; } - const { runId, base, pinnedCommit } = Object.fromEntries(Object.entries(value)); + const record = readingJournalValue(() => + Array.isArray(value) ? undefined : Object.fromEntries(Object.entries(value)), + ); + if (record === undefined) { + return undefined; + } + if ( + Object.keys(record).length !== RUN_MEMBERS.length || + !RUN_MEMBERS.every((member) => Object.hasOwn(record, member)) + ) { + return undefined; + } + const { runId, base, pinnedCommit } = record; if (typeof runId !== "string" || typeof base !== "string" || typeof pinnedCommit !== "string") { return undefined; } return Object.freeze({ runId, base, pinnedCommit }); } +/** What one retained event claims about the run, when it claims anything. */ +interface RunClaim { + readonly settled: boolean; + readonly value: unknown; +} + +/** + * The claim this event makes, read totally and fail-closed. + * + * Every member a decision rests on is forced here — the discriminator, the + * coroutine, both halves of the description, the settlement and a successful + * settlement's value — and an event that refuses any of them is a history this + * run cannot describe rather than an unrelated event to step past. Skipping it + * is what would let a record that will not read stand in for one that is + * absent, and a recorded terminal result be reused on the difference. + * + * A Yield that reads cleanly but is not the root coroutine's, or is not under + * both the canonical type and the canonical name, makes no claim at all. + */ +function runClaim(event: DurableEvent): RunClaim | undefined { + try { + if (event.type === "close") { + // Forced, not read for a value: a Close this history cannot classify is + // the same refusal as a Yield that will not read. + void event.coroutineId; + void event.result.status; + return undefined; + } + if (event.type !== "yield" || event.coroutineId !== ROOT_COROUTINE) { + return undefined; + } + if (event.description.type !== WORKFLOW_RUN || event.description.name !== WORKFLOW_RUN) { + return undefined; + } + return event.result.status === "ok" + ? { settled: true, value: event.result.value } + : { settled: false, value: undefined }; + } catch { + throw malformedRecord(); + } +} + +/** + * What one installation requires of a history that is not empty. + * + * The two installations differ in one thing, and it is not strictness. A + * retained run was created by a host before anything executed, so a history of + * its own is something it must have: none, or one that failed, means the + * recorded work is not this run's. A programmatic run allocates itself on first + * execution, and §6 records a base that would not resolve as a *failed* effect — + * so a history whose only record is that failure is this run's history, and + * replaying it reproduces the failure rather than asking Git again. + */ +export interface RunHistoryRules { + /** Whether a non-empty history must carry a successful record. */ + readonly required: boolean; + /** The recorded run, or a refusal naming what it disagrees about. */ + agree(recorded: WorkflowRun): WorkflowRun; +} + +/** + * Hold a retained history to the run it belongs to. + * + * An empty history is the ordinary live start and is held to nothing. Otherwise + * every canonical record it carries must read as a workflow run and agree with + * this installation, and it may carry at most one — the record is written before + * the root document is imported, so two describe two runs. + */ +export function admitWorkflowRunHistory( + retained: readonly DurableEvent[], + rules: RunHistoryRules, +): WorkflowRun | undefined { + if (retained.length === 0) { + return undefined; + } + + const runs: WorkflowRun[] = []; + let claimed = 0; + for (const event of retained) { + const claim = runClaim(event); + if (claim === undefined) { + continue; + } + // Counted whether or not it settled successfully. The record is written + // once, before the root document is imported, so a second entry under the + // canonical identity describes a second run however it ended — and a failed + // one beside a successful one is two runs, not one run with a stumble. + claimed += 1; + if (!claim.settled) { + continue; + } + const run = readWorkflowRun(claim.value); + // A settled record that will not read is damaged rather than absent, and + // saying so is what tells a corrupted journal from somebody else's. + if (run === undefined) { + throw malformedRecord(); + } + runs.push(run); + } + + if (claimed > 1) { + throw duplicateRunRecords(claimed); + } + const only = runs[0]; + if (only === undefined) { + if (rules.required) { + throw missingRunEvidence(); + } + return undefined; + } + return rules.agree(only); +} + /** * The journal holds something that is not a workflow run. * @@ -47,26 +226,73 @@ export function readWorkflowRun(value: unknown): WorkflowRun | undefined { * reporting it would carry whatever it happened to hold into logs and rendered * output. */ -export function malformedRecord(description: EffectDescription): StaleInputError { +export function malformedRecord(): StaleInputError { return new StaleInputError( - `The journal records "${description.name}" holding a value that does not describe a ` + + `The journal records "${WORKFLOW_RUN}" holding a value that does not describe a ` + "workflow run. A record that cannot be read is refused rather than replayed. Re-run " + "the document from the start rather than resuming from this journal.", - { coroutineId: "root", description }, + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, + ); +} + +/** A journal with history offers no record that says whose history it is. */ +export function missingRunEvidence(): StaleInputError { + return new StaleInputError( + "The journal holds recorded history but records no successful workflow run of its " + + "own. A workflow run replays only from history that identifies it. Resume the run " + + "this journal belongs to, or re-run the document from the start.", + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, + ); +} + +/** + * A journal carries more than one entry under the canonical run identity. + * + * The tally is this module's own count rather than anything the journal said, + * so naming it carries nothing across. Failed entries count: the question is + * how many runs the history describes, not how many of them finished. + */ +export function duplicateRunRecords(records: number): StaleInputError { + return new StaleInputError( + `The journal records ${records} workflow run entries where at most one describes a ` + + "run. A history recording more than one run is not one run's history. Resume the " + + "run this journal belongs to, or re-run the document from the start.", + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, + ); +} + +/** + * The recorded run is not the retained run this execution was installed with. + * + * The differing fields are named and their values are not. A run id may be + * caller-selected and a base may be any revision expression, so both are + * external text on the same terms as retained props: naming a field says what + * disagrees without carrying the disagreement into logs and rendered output. + */ +export function retainedRunMismatch(fields: readonly string[]): StaleInputError { + return new StaleInputError( + `The journal records a workflow run whose ${fields.join(", ")} differs from the retained ` + + "run this execution was installed with. A retained run is replayed as itself rather " + + "than onto a different one. Resume the run the journal belongs to.", + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, ); } -/** The recorded run started from a different base than this run supplied. */ -export function baseMismatch( - description: EffectDescription, - recorded: string, - supplied: string, -): StaleInputError { +/** + * The recorded run started from a different base than this run supplied. + * + * Both bases are named, because here they are the two things a caller has to + * compare to understand the refusal, and both came from that caller rather than + * from the journal's opaque content. They are named in the sentence only: the + * description this error retains still holds nothing but the effect's type and + * name. + */ +export function baseMismatch(recorded: string, supplied: string): StaleInputError { return new StaleInputError( `The journal records this workflow run starting from "${recorded}", but this run ` + `supplied "${supplied}". A recorded base cannot be replayed onto a run that asked ` + "for a different one. Re-run the document from the start rather than resuming from " + "this journal.", - { coroutineId: "root", description }, + { coroutineId: ROOT_COROUTINE, description: refusalDescription() }, ); } diff --git a/packages/workflow/src/run.ts b/packages/workflow/src/run.ts index caef5dcb..07483634 100644 --- a/packages/workflow/src/run.ts +++ b/packages/workflow/src/run.ts @@ -1,24 +1,52 @@ /** * Associating one document execution with a workflow run. * - * `useWorkflow({ base })` installs ordinary middleware and nothing else. It + * `workflowInstallation({ base })` is a value, not an installation act. It * creates no workflow run: a run comes into being when a document execution * reaches its first durable operation, which resolves the base once, records * one immutable value, and only then lets the root document be imported. * - * Two middlewares are needed because a journal can be in three states. + * A journal can be in three states, and two of them are held to the run. * - * - **Live** — no record yet. `Execution.document` allocates the run id, - * resolves `${base}^{commit}` through `Git.revParse()`, and records the value - * before `next()` imports the root. - * - **Truncated** — the record is there but the root never closed. Both - * middlewares run: the guard restores the value, and the durable operation - * still runs so the journal cursor advances past its own entry. + * - **Live** — no record yet. The installation's `prepare` hook allocates the + * run id, resolves `${base}^{commit}` through `Git.revParse()`, and records + * the value — inside the durable root, before any public document policy and + * before the root is imported. + * - **Truncated** — the record is there but the root never closed. The durable + * operation replays the stored value, so neither the identifier nor Git is + * reached a second time, and the journal cursor still advances past its own + * entry. * - **Completed** — the root `Close` is recorded, and `durableRun` returns the - * stored result without ever invoking the workflow, so `Execution.document` - * is never reached. The guard's check phase runs before that shortcut, which - * is the only place a completed journal can restore its run — or refuse a - * different base before the recorded result is handed back. + * stored result without entering the durable body at all, so preparation is + * never reached and the admission is what restores the run. + * + * `retainedWorkflowInstallation(run)` is the same three states under a run that already + * exists. A workflow host has created the storage record before anything + * executes, so the live path records exactly the value it was given rather than + * allocating an id and resolving a base, and every state requires the journal to + * agree with that value in full. + * + * ## Where identity is decided + * + * In no middleware at all. `ReplayGuard` is composable policy — a handler + * installed further out may decline to call `next` — and so is `Execution`: a + * handler registered at the same position can rebuild the options a later one + * produced, stream included. A journal held to its run from either place would + * be held to it by registration order. + * + * So this is an *installation* the trusted host hands to `executeInstalled()`, + * and core owns the read. The admission is captured before any installation, + * middleware or document code exists, and applied inside the same trusted + * `readAll` that already holds a resumed run to its recorded root selection. That read happens before any middleware runs, on the retained + * snapshot every later phase consumes — ahead of public guard policy, of any + * retained Yield reaching execution, of a retained `Close` being reused, of + * authored work, and of any append. Public `ReplayGuard` handlers still observe + * and may still reject the history this admits; none of them can widen it. + * + * The two installations differ in what they require, not in how strictly it is + * enforced. See `RunHistoryRules`: a base that would not resolve is recorded as + * a failed effect (§6), so a programmatic run replays that failure rather than + * demanding a successful record it never wrote. * * All of it is operation-scoped. The value is installed in the scope that owns * the document execution, so every descendant of the expansion reads it, the @@ -28,60 +56,174 @@ import { createContext } from "effection"; import type { Context, Operation } from "effection"; -import { createDurableOperation } from "@executablemd/durable-streams"; -import { ReplayGuard } from "@executablemd/durable-streams"; -import type { EffectDescription, Json, Workflow, Yield } from "@executablemd/durable-streams"; -import { Execution } from "@executablemd/core"; +import { createDurableOperation, ephemeral } from "@executablemd/durable-streams"; +import type { + DurableEvent, + EffectDescription, + Json, + Workflow, +} from "@executablemd/durable-streams"; +import type { ExecutionInstallation, JournalAdmission } from "@executablemd/core/host"; import { revParse } from "./git.ts"; import { + admitWorkflowRunHistory, baseMismatch, describeWorkflowRun, malformedRecord, readWorkflowRun, - WORKFLOW_RUN, + retainedRunMismatch, } from "./journal.ts"; -import type { WorkflowRun } from "./journal.ts"; +import type { RunHistoryRules, WorkflowRun } from "./journal.ts"; export type { WorkflowRun } from "./journal.ts"; /** - * The workflow run of the document execution running now. + * Where the workflow run of the document execution running now is kept. * * A stable, namespaced name and a plain value: another loaded copy of this - * package reads the same binding through its own descriptor. By the same - * property a descendant may bind this name for its own descendants, so durable - * enforcement never depends on it. + * package reads the same binding through its own descriptor and finds the same + * frozen run under `run`. By the same property a descendant may bind this name + * for its own descendants, and anyone holding the slot may write to it — which + * is why durable enforcement never depends on it. What holds the run to its + * journal is the admission and the durable record, neither of which this is + * reachable from. */ -const CurrentWorkflowRun: Context = createContext( +const CurrentWorkflowRun: Context = createContext( "executablemd.workflow.run", undefined, ); +/** + * Where the run of one document execution is kept while it is running. + * + * A slot rather than the value itself, because the two places that decide the + * run are on the wrong side of the scope that reads it. Retained-history + * admission runs inside canonical core's own journal read, and preparation runs + * inside the durable root; a context *set* from either would end with the + * operation that set it, while the document that has to read it starts + * afterwards. So the installation puts an empty slot in the execution's scope + * before either of them runs, and they fill it. + * + * The slot is created per installed execution and reclaimed with it. Nothing + * lives at module scope, and one execution cannot see another's. + */ +interface RunSlot { + run?: WorkflowRun; +} + /** The frozen run of the current document execution; throws outside one. */ export function* getWorkflowRun(): Operation { - const run = yield* CurrentWorkflowRun.get(); + const slot = yield* CurrentWorkflowRun.get(); + const run = slot?.run; if (run === undefined) { throw new Error( "getWorkflowRun() is available only inside a document execution associated with a " + - "workflow run. Install useWorkflow({ base }) in the scope that owns the execution.", + "workflow run. Pass workflowInstallation({ base }) to executeInstalled().", ); } return run; } +/** + * How one installation decides what the run is, and what the journal is held to. + * + * Two hosts need different answers to both questions. A programmatic caller + * supplies a base and lets the first live execution allocate an id and resolve + * that base, so the only thing a record can disagree about is the base it was + * made from. A workflow host has already created the storage record, so the run + * is not the execution's to allocate: it arrives whole, and a journal that + * records a different one is not this run's journal. + */ +interface RunPreparation extends RunHistoryRules { + readonly base: string; + /** The run this execution is of, reached only when nothing is recorded yet. */ + allocate(): Operation; +} + /** Append the run to the journal, and answer with what the journal holds. */ -function* record(description: EffectDescription, base: string): Workflow { +function* record(description: EffectDescription, preparation: RunPreparation): Workflow { return yield createDurableOperation(description, function* (): Operation { // Reached only when nothing is recorded yet: a replay hands the stored // value back without running this at all, so neither the identifier nor Git // is reached a second time. - const pinnedCommit = yield* revParse(`${base}^{commit}`); - // Web Crypto rather than `node:crypto`: a run id is allocated in shared - // code, which names no host. - return { runId: crypto.randomUUID(), base, pinnedCommit }; + const { runId, base, pinnedCommit } = yield* preparation.allocate(); + return { runId, base, pinnedCommit }; }); } +function allocating(base: string): RunPreparation { + return { + base, + // A base that would not resolve is recorded as a failed effect (§6), and a + // history whose only record is that failure is this run's own. Requiring a + // successful one would retry Git instead of replaying what happened. + required: false, + *allocate(): Operation { + const pinnedCommit = yield* revParse(`${base}^{commit}`); + // Web Crypto rather than `node:crypto`: a run id is allocated in shared + // code, which names no host. + return { runId: crypto.randomUUID(), base, pinnedCommit }; + }, + /** + * The description carries the base for a reader; divergence detection + * compares only type and name, so the base this run supplied is checked + * against the stored *value* rather than against the entry's identity. + */ + agree(recorded: WorkflowRun): WorkflowRun { + if (recorded.base !== base) { + throw baseMismatch(recorded.base, base); + } + return recorded; + }, + }; +} + +function retaining(run: WorkflowRun): RunPreparation { + return { + base: run.base, + // The host created this run before anything executed, so a history of its + // own is something it must have: none, or one that only failed, means the + // recorded work is not this run's. + required: true, + // deno-lint-ignore require-yield + *allocate(): Operation { + return run; + }, + agree(recorded: WorkflowRun): WorkflowRun { + const differing = (["runId", "base", "pinnedCommit"] as const).filter( + (field) => recorded[field] !== run[field], + ); + if (differing.length > 0) { + throw retainedRunMismatch(differing); + } + return recorded; + }, + }; +} + +/** + * Read one member set off a value a host supplied, or answer that it refused. + * + * One read, nothing else inside — the same narrowness the journal's own reads + * are held to, for the same reason. + */ +function readingRetainedValue(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +/** Read the record this run is held to, refusing anything that is not it. */ +function held(stored: unknown, preparation: RunPreparation): WorkflowRun { + const run = readWorkflowRun(stored); + if (run === undefined) { + throw malformedRecord(); + } + return preparation.agree(run); +} + function same(left: WorkflowRun, right: WorkflowRun): boolean { return ( left.runId === right.runId && @@ -91,64 +233,167 @@ function same(left: WorkflowRun, right: WorkflowRun): boolean { } /** - * Read the record this run is held to, refusing anything that is not it. + * Bring the run into being, inside the durable root. * - * The description carries the base for a reader; divergence detection compares - * only type and name, so the base this run supplied is checked against the - * stored *value* rather than against the entry's identity. + * Canonical core invokes this after retained-history admission and before any + * public `Execution.document` policy, the root import, and every authored + * effect — so the run exists before anything can observe it, and nothing a + * handler does can prevent it, replace it, or reach the journal ahead of it. + * + * A `Workflow` because what it does is journaled. On a live run the durable + * operation allocates and resolves once and records the value; on a partial + * continuation the same operation restores what it already recorded, so + * neither the identifier nor Git is reached a second time; on a completed + * terminal replay core never enters the durable body at all, so this does not + * run and the admission is what installs the recorded run. */ -function held(description: EffectDescription, stored: unknown, base: string): WorkflowRun { - const run = readWorkflowRun(stored); - if (run === undefined) { - throw malformedRecord(description); - } - if (run.base !== base) { - throw baseMismatch(description, run.base, base); +function* prepare(preparation: RunPreparation): Workflow { + const description = describeWorkflowRun(preparation.base); + // Which run this is, and whether the journal agrees, are decided by the + // captured `preparation` and the durable record — never by what the slot + // happens to hold. + const run = held(yield* record(description, preparation), preparation); + const slot = yield* ephemeral(CurrentWorkflowRun.get()); + if (slot === undefined) { + return; } - return run; -} - -function* establish(base: string): Operation { - const description = describeWorkflowRun(base); - const run = held(description, yield* record(description, base), base); - const restored = yield* CurrentWorkflowRun.get(); - // A truncated replay already restored this value in the check phase; keeping - // that object is what makes every read in one execution the same one. + const restored = slot.run; + // A resumed journal already put this value in the slot when it was admitted; + // keeping that object is what makes every read in one execution the same one. if (restored !== undefined && same(restored, run)) { return; } - yield* CurrentWorkflowRun.set(run); + slot.run = run; +} + +/** + * What this installation requires of the history a document execution replays. + * + * Contributed to core rather than wrapped around core. The comparison runs + * inside the execution's own trusted journal read, on the retained snapshot + * every later phase consumes — so no `ReplayGuard` handler that declines to + * delegate, and no `Execution` handler that rebuilds the options a later one + * produced, can suppress, replace or reorder it. By the time any middleware + * runs, the read has already happened. + * + * Publishing the run here is not incidental: a completed journal never enters + * the durable body, so preparation does not run and this is the only place + * inside the execution where the run a recorded result belongs to is known. + */ +function admits(preparation: RunPreparation): JournalAdmission { + return function* (retained: readonly DurableEvent[]): Operation { + // What the history is held to is decided by the captured `preparation`, and + // by nothing that is read here. The slot is reached only afterwards, to + // publish what was already admitted. + const admitted = admitWorkflowRunHistory(retained, preparation); + if (admitted === undefined) { + return; + } + yield* publish(admitted); + }; } -function* restore(event: Yield, base: string): Operation { - if (event.description.type !== WORKFLOW_RUN || event.result.status !== "ok") { +/** + * Put the run where this execution's readers will find it. + * + * The slot is looked up rather than closed over, because the one that matters + * is the one `install()` made for *this* invocation. Nothing is read back out + * of it to decide anything: publication is the whole purpose, and a slot that + * is somehow absent means this run is unreadable, never that it is undecided. + */ +function* publish(run: WorkflowRun): Operation { + const slot = yield* CurrentWorkflowRun.get(); + if (slot === undefined) { return; } - yield* CurrentWorkflowRun.set(held(describeWorkflowRun(base), event.result.value, base)); + slot.run = run; } /** - * Associate the document execution this scope owns with a workflow run. + * What a trusted host attaches to one execution to make it this run's. * - * Installing this creates nothing. Executing a document under it does. + * An installation rather than something installed into an ambient scope: the + * admission is a value the host hands to `executeInstalled()`, so canonical + * core captures it before any middleware or document code exists, and a second + * loaded copy of this package composes by handing over its own closure rather + * than by agreeing on a name. */ -export function* useWorkflow(options: { base: string }): Operation { - const { base } = options; - - yield* ReplayGuard.around({ - *check([event], next) { - // Runs before `durableRun` can short-circuit on a recorded root Close, so - // a completed journal restores its run here — and refuses a different - // base here, before the recorded result is returned. - yield* restore(event, base); - return yield* next(event); +function installation(preparation: RunPreparation): ExecutionInstallation { + return { + admissions: [admits(preparation)], + prepare: () => prepare(preparation), + *install(): Operation { + // A fresh slot per invocation, not per installation value. `install()` + // runs once for each execution this value is passed to, so a host that + // holds one installation and runs two documents with it gets two slots, + // and neither execution can see the other's run. Closing over a slot + // built alongside the value would have shared it between them. + yield* CurrentWorkflowRun.set({}); }, - }); + }; +} - yield* Execution.around({ - *document([props], next) { - yield* establish(base); - return yield* next(props); - }, - }); +/** + * The installation that associates one document execution with a workflow run. + * + * Constructing it creates nothing. Executing a document under it does. + * + * ```ts + * yield* executeInstalled(options, [workflowInstallation({ base: "main" })]); + * ``` + */ +export function workflowInstallation(options: { base: string }): ExecutionInstallation { + return installation(allocating(options.base)); +} + +/** + * Associate the document execution this scope owns with a run that already + * exists. + * + * A workflow host creates the run's storage record before it executes anything, + * so by the time a document runs there is nothing left to allocate or resolve: + * the run id is the one storage answered with, and the pinned commit is the one + * the definition was established from. This installation records exactly that + * value and requires a journal to agree with it in every field, so a resumed + * execution can never continue under a run the storage record does not describe. + * + * Git is not consulted, and no identifier is generated. + */ +export function retainedWorkflowInstallation(run: WorkflowRun): ExecutionInstallation { + return installation(retaining(retainedRun(run))); +} + +/** + * The retained run as a frozen value of its own. + * + * Parsed rather than believed: it arrives from a storage record a host read + * back, so a member that is missing or empty is a value that identifies no run + * rather than one to install and discover later. The three members are named + * here, so a host handing over a wider record installs the run it describes + * rather than being refused for carrying its own bookkeeping. + */ +function retainedRun(run: WorkflowRun): WorkflowRun { + // Named through the same total read as a journal value: a host that hands + // over a record whose members refuse to be read has supplied a value that + // identifies no run, which is the sentence below rather than its exception. + const parsed = readWorkflowRun( + readingRetainedValue(() => ({ + runId: run?.runId, + base: run?.base, + pinnedCommit: run?.pinnedCommit, + })), + ); + if (parsed === undefined || parsed.runId === "" || parsed.base === "") { + throw new Error( + "retainedWorkflowInstallation() needs the retained run's id, base and pinned commit. A run " + + "installed without them identifies no workflow run.", + ); + } + if (parsed.pinnedCommit === "") { + throw new Error( + "retainedWorkflowInstallation() needs the retained run's pinned commit: an empty one pins the " + + "run to no repository state at all.", + ); + } + return parsed; } diff --git a/packages/workflow/tests/retained-run.test.ts b/packages/workflow/tests/retained-run.test.ts new file mode 100644 index 00000000..d8f8ec25 --- /dev/null +++ b/packages/workflow/tests/retained-run.test.ts @@ -0,0 +1,960 @@ +/** + * Tier RR — installing a run that already exists. + * + * `workflowInstallation({ base })` lets the first live execution decide what the run is: + * it allocates an identifier and resolves the base through Git. A workflow host + * has already done both by the time a document runs — storage answered with the + * run id, and the definition was established from a commit it pinned — so + * nothing is left to decide, and a journal that records a different run is not + * this run's journal. + * + * Every test here replaces `Git` with a provider that fails when it is + * consulted, so "the retained installation asks Git nothing" is asserted rather + * than assumed. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { scoped } from "effection"; +import type { Operation } from "effection"; +import { type Api, createApi } from "@effectionx/context-api"; +import { InMemoryStream, ReplayGuard, StaleInputError } from "@executablemd/durable-streams"; +import type { + DurableEvent, + DurableStream, + Json, + Result, + Yield, +} from "@executablemd/durable-streams"; +import { + collect, + DocumentOutput, + execute, + Execution, + inlineSource, + registerComponents, +} from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import type { ExecutionRequest } from "@executablemd/core"; +import { Git } from "../src/git.ts"; +import { getWorkflowRun, retainedWorkflowInstallation } from "../src/run.ts"; +import type { WorkflowRun } from "../src/run.ts"; + +const COMMIT = "9fceb02d0ae598e95dc970b74767f19372d61af8"; +const OTHER_COMMIT = "1111111111111111111111111111111111111111"; + +const RETAINED: WorkflowRun = Object.freeze({ + runId: "release-1.4", + base: "main", + pinnedCommit: COMMIT, +}); + +/** A Git that fails the test if anything consults it. */ +function useForbiddenGit(): Operation { + return Git.around( + { + // deno-lint-ignore require-yield + *revParse([revision]) { + throw new Error(`Git was consulted for "${revision}"`); + }, + }, + { at: "min" }, + ); +} + +/** `` — reports the run it was expanded under. */ +function useProbe(seen: WorkflowRun[]): Operation { + return registerComponents([ + { + name: "Probe", + origin: "tier-rr", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + seen.push(yield* getWorkflowRun()); + return ""; + }, + }, + ]); +} + +interface Attempt { + readonly seen: WorkflowRun[]; + readonly emitted: string[]; + readonly thrown: unknown; +} + +/** + * Run `` under a retained installation, watching everything a refusal + * has to prevent. + * + * `seen` is non-empty only if the document expanded; `emitted` is non-empty only + * if a recorded result reached a consumer. `install` is where a test puts the + * policy it wants to prove cannot widen the boundary — installed *outside* the + * workflow installation, which is the position a suppressing handler wants. + */ +function runRetained( + run: WorkflowRun, + stream: DurableStream, + install: Operation = ok(), + after: Operation = ok(), +): Operation { + return scoped(function* () { + const seen: WorkflowRun[] = []; + const emitted: string[] = []; + yield* useForbiddenGit(); + yield* useProbe(seen); + yield* install; + const workflow = retainedWorkflowInstallation(run); + yield* after; + yield* DocumentOutput.around({ + *output([text], next) { + emitted.push(text); + yield* next(text); + }, + }); + try { + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); + return { seen, emitted, thrown: undefined }; + } catch (error) { + return { seen, emitted, thrown: error }; + } + }); +} + +// deno-lint-ignore require-yield +function* ok(): Operation {} + +/** + * A `ReplayGuard` that answers one phase without delegating. + * + * This is what composable policy is allowed to do, and the whole reason durable + * identity may not live behind it. + */ +function useSuppressingGuard(phase: "check" | "admit" | "decide"): Operation { + return ReplayGuard.around({ + *check([event], next) { + if (phase !== "check") { + yield* next(event); + } + }, + *admit([history], next) { + if (phase !== "admit") { + yield* next(history); + } + }, + decide([event], next) { + return phase === "decide" ? { outcome: "replay" } : next(event); + }, + }); +} + +/** + * The same suppression, through a descriptor this test built for itself. + * + * A contextual Api composes by stable name across loaded copies, so a second + * copy of `durable-streams` is exactly this: the same name, a handler nothing + * here imported. + */ +function useForeignGuard(): Operation { + const foreign: Api<{ + check(event: unknown): Operation; + admit(history: unknown): Operation; + decide(event: unknown): { outcome: "replay" }; + }> = createApi("DurableEffection.ReplayGuard", { + // deno-lint-ignore require-yield + *check(_event: unknown): Operation {}, + // deno-lint-ignore require-yield + *admit(_history: unknown): Operation {}, + decide(_event: unknown): { outcome: "replay" } { + return { outcome: "replay" }; + }, + }); + return foreign.around({ + // deno-lint-ignore require-yield + *check() {}, + // deno-lint-ignore require-yield + *admit() {}, + decide() { + return { outcome: "replay" }; + }, + }); +} + +function workflowEvents(stream: InMemoryStream): DurableEvent[] { + return stream + .snapshot() + .filter((event) => event.type === "yield" && event.description.type === "workflow_run"); +} + +function recordedRun(stream: InMemoryStream): Json | undefined { + const event = workflowEvents(stream)[0]; + if (event === undefined || event.type !== "yield" || event.result.status !== "ok") { + return undefined; + } + return event.result.value; +} + +/** The journal without the root's close, which is what makes the next run replay. */ +function partial(stream: InMemoryStream): InMemoryStream { + return new InMemoryStream( + stream.snapshot().filter((event) => !(event.type === "close" && event.coroutineId === "root")), + ); +} + +/** An `Execution` handler that hands the durable run a journal of its choosing. */ +function useStreamHijack(raw: DurableStream): Operation { + return Execution.around( + { + *execute([request], next) { + yield* next(request.withOptions({ ...request.options, stream: raw })); + }, + }, + { at: "min" }, + ); +} + +/** The same, through a descriptor of the Api's stable name built in this suite. */ +function useForeignStreamHijack(raw: DurableStream): Operation { + const foreign: Api<{ + execute(request: ExecutionRequest): Operation; + }> = createApi("Execution", { + // deno-lint-ignore require-yield + *execute(_request: ExecutionRequest): Operation {}, + }); + return foreign.around( + { + *execute([request], next) { + yield* next(request.withOptions({ ...request.options, stream: raw })); + }, + }, + { at: "min" }, + ); +} + +/** + * A journal that answers differently every time it is read. + * + * The run id shifts on the second read of the recorded value, which is what a + * backend handing out live objects can do. Admission and replay must therefore + * be reading one retained snapshot rather than each taking their own look — + * otherwise a history admitted as one run is replayed as another. + */ +function shiftingJournal(events: readonly DurableEvent[], reads: string[]): DurableStream { + const appended: DurableEvent[] = []; + const shift = (event: DurableEvent): DurableEvent => { + if (event.type !== "yield" || event.description.type !== "workflow_run") { + return event; + } + return { + ...event, + get result(): Result { + const runId = reads.length === 0 ? "release-1.4" : "release-1.5"; + reads.push(runId); + return { status: "ok", value: { runId, base: "main", pinnedCommit: COMMIT } }; + }, + }; + }; + return { + // deno-lint-ignore require-yield + *readAll(): Operation { + return [...events.map(shift), ...appended]; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }; +} + +/** + * A journal handing back exactly these events, accessors and all. + * + * `InMemoryStream` clones what it is given, which is the right thing for it and + * the wrong thing here: a value that refuses to be read cannot survive being + * copied. These events reach the execution as written. + */ +interface WatchedJournal { + readonly journal: DurableStream; + /** What a refused run managed to append, which must stay empty. */ + readonly appended: DurableEvent[]; +} + +function unreadableJournal(events: readonly DurableEvent[]): WatchedJournal { + const appended: DurableEvent[] = []; + return { + appended, + journal: { + // deno-lint-ignore require-yield + *readAll(): Operation { + return [...events, ...appended]; + }, + // deno-lint-ignore require-yield + *append(event: DurableEvent): Operation { + appended.push(event); + }, + }, + }; +} + +/** + * The whole error, not only its sentence. + * + * `StaleInputError` retains what it is handed, so a refusal is inspected as an + * object: anything reachable on it is something a log or a rendered document + * could carry. + */ +function wholeError(thrown: unknown): string { + return JSON.stringify(thrown, (_key, value) => + value instanceof Error ? { ...value, message: value.message, name: value.name } : value, + ); +} + +/** + * The completed journal, with what it records about the run replaced. + * + * A completed journal is where the run record matters most and is read least: it + * answers with the recorded root result without expanding anything, so whatever + * these cases leave behind is the whole of the evidence that the result is this + * run's. + */ +function completedWith( + stream: InMemoryStream, + change: (event: Yield) => DurableEvent[], +): InMemoryStream { + return new InMemoryStream( + stream + .snapshot() + .flatMap((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? change(event) + : [event], + ), + ); +} + +describe("Tier RR — retained workflow runs", () => { + it("RR1: records exactly the retained run, without allocating or resolving", function* () { + const stream = new InMemoryStream(); + const attempt = yield* runRetained(RETAINED, stream); + + expect(attempt.thrown).toBeUndefined(); + expect(attempt.seen).toEqual([RETAINED]); + expect(recordedRun(stream)).toEqual({ + runId: "release-1.4", + base: "main", + pinnedCommit: COMMIT, + }); + expect(workflowEvents(stream)).toHaveLength(1); + }); + + it("RR2: restores the retained run from a truncated journal without recording again", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const resumed = partial(first); + const attempt = yield* runRetained(RETAINED, resumed); + + expect(attempt.thrown).toBeUndefined(); + expect(attempt.seen).toEqual([RETAINED]); + expect(workflowEvents(resumed)).toHaveLength(1); + }); + + it("RR3: restores the retained run from a completed journal", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const replayed = new InMemoryStream(first.snapshot()); + const attempt = yield* runRetained(RETAINED, replayed); + + expect(attempt.thrown).toBeUndefined(); + expect(workflowEvents(replayed)).toHaveLength(1); + }); + + it("RR4: refuses a journal recording a different run id", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const attempt = yield* runRetained({ ...RETAINED, runId: "release-1.5" }, partial(first)); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + const message = attempt.thrown instanceof Error ? attempt.thrown.message : ""; + expect(message).toContain("runId"); + // The differing values are named nowhere: a run id is caller-selected text. + expect(message).not.toContain("release-1.4"); + expect(message).not.toContain("release-1.5"); + expect(attempt.seen).toEqual([]); + }); + + it("RR5: refuses a journal recording a different base or pinned commit", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const base = yield* runRetained({ ...RETAINED, base: "release/1.4" }, partial(first)); + expect(base.thrown).toBeInstanceOf(StaleInputError); + expect(base.thrown instanceof Error ? base.thrown.message : "").toContain("base"); + + const pinned = yield* runRetained({ ...RETAINED, pinnedCommit: OTHER_COMMIT }, partial(first)); + expect(pinned.thrown).toBeInstanceOf(StaleInputError); + expect(pinned.thrown instanceof Error ? pinned.thrown.message : "").toContain("pinnedCommit"); + }); + + it("RR6: refuses a record that does not describe a workflow run at all", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const damaged = new InMemoryStream( + partial(first) + .snapshot() + .map((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? { ...event, result: { status: "ok", value: { runId: 7 } } } + : event, + ), + ); + + const attempt = yield* runRetained(RETAINED, damaged); + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + }); + + // RR8: what a completed journal has to hold. `check` can only object to an + // event a journal contains, so a journal recording a terminal result and no + // readable run for it leaves nothing to object to — and the recorded result + // would answer for history that never identified this run. + it("RR8: refuses a completed journal that does not record exactly one run", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const cases: Array<{ says: string; stream: InMemoryStream }> = [ + { says: "no successful workflow run", stream: completedWith(first, () => []) }, + { + says: "no successful workflow run", + stream: completedWith(first, (event) => [ + { ...event, result: { status: "err", error: { message: "recorded failure" } } }, + ]), + }, + { + says: "2 workflow run entries", + stream: completedWith(first, (event) => [event, event]), + }, + ]; + + for (const refused of cases) { + const before = refused.stream.snapshot().length; + const attempt = yield* runRetained(RETAINED, refused.stream); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain(refused.says); + // Refused before the recorded root result was handed back, and before + // anything expanded. + expect(attempt.seen).toEqual([]); + expect(refused.stream.snapshot().length).toEqual(before); + } + }); + + it("RR9: refuses a completed journal recording another run, or an unreadable one", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + const completed = () => new InMemoryStream(first.snapshot()); + + const foreign = yield* runRetained({ ...RETAINED, runId: "release-1.5" }, completed()); + expect(foreign.thrown).toBeInstanceOf(StaleInputError); + expect(foreign.thrown instanceof Error ? foreign.thrown.message : "").toContain("runId"); + + const damaged = yield* runRetained( + RETAINED, + completedWith(first, (event) => [ + { ...event, result: { status: "ok", value: { runId: 7 } } }, + ]), + ); + expect(damaged.thrown).toBeInstanceOf(StaleInputError); + expect(damaged.seen).toEqual([]); + }); + + // RR10: the reproduction the architecture review ran. Under a guard that + // answers without delegating, run-a's completed journal used to hand its + // recorded root result back to run-b. Identity is not policy, so it does not. + it("RR10: refuses another run's completed journal beneath a suppressing guard", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + for (const phase of ["check", "admit", "decide"] as const) { + const completed = new InMemoryStream(first.snapshot()); + const before = completed.snapshot().length; + + const attempt = yield* runRetained( + { ...RETAINED, runId: "release-1.5" }, + completed, + useSuppressingGuard(phase), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain("runId"); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(completed.snapshot().length).toEqual(before); + } + }); + + it("RR11: refuses it beneath a same-named guard this suite built itself", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const completed = new InMemoryStream(first.snapshot()); + const before = completed.snapshot().length; + + const attempt = yield* runRetained( + { ...RETAINED, runId: "release-1.5" }, + completed, + useForeignGuard(), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(completed.snapshot().length).toEqual(before); + }); + + it("RR12: a valid completed journal still replays beneath a suppressing guard", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + for (const phase of ["check", "admit", "decide"] as const) { + const completed = new InMemoryStream(first.snapshot()); + const attempt = yield* runRetained(RETAINED, completed, useSuppressingGuard(phase)); + + expect(attempt.thrown).toBeUndefined(); + // Zero live execution: the recorded result answered. + expect(attempt.seen).toEqual([]); + expect(workflowEvents(completed)).toHaveLength(1); + } + }); + + // RR13: the guard surface still works as policy. It observes what admission + // let through, and it may still refuse — it simply cannot widen. + it("RR13: a public guard still observes admitted history and may reject it", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const observed: string[] = []; + const watched = yield* runRetained( + RETAINED, + new InMemoryStream(first.snapshot()), + ReplayGuard.around({ + *check([event], next) { + observed.push(event.description.type); + yield* next(event); + }, + }), + ); + expect(watched.thrown).toBeUndefined(); + expect(observed).toContain("workflow_run"); + + const rejected = yield* runRetained( + RETAINED, + new InMemoryStream(first.snapshot()), + ReplayGuard.around({ + // deno-lint-ignore require-yield + *check([event]) { + if (event.description.type === "workflow_run") { + throw new Error("this guard says no"); + } + }, + }), + ); + expect(rejected.thrown).toBeInstanceOf(Error); + expect(rejected.emitted).toEqual([]); + }); + + // RR14: the record has to be the canonical one. A same-typed Yield under + // another name, or under a child coroutine, establishes nothing — otherwise + // removing the genuine record and adding one of these would authorize reuse. + it("RR14: refuses a record that is not the root coroutine's canonical one", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const counterfeits: Array<{ says: string; change: (event: Yield) => DurableEvent[] }> = [ + { + says: "wrong name", + change: (event) => [ + { ...event, description: { ...event.description, name: "workflow_run_v2" } }, + ], + }, + { + says: "child coroutine", + change: (event) => [{ ...event, coroutineId: "root.0" }], + }, + { + says: "extra member", + change: (event) => [ + { + ...event, + result: { + status: "ok", + value: { ...RETAINED, executor: "someone-else" }, + }, + }, + ], + }, + ]; + + for (const counterfeit of counterfeits) { + const damaged = completedWith(first, counterfeit.change); + const before = damaged.snapshot().length; + const attempt = yield* runRetained(RETAINED, damaged); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(damaged.snapshot().length).toEqual(before); + // Nothing the journal held is quoted back. + expect(String(attempt.thrown)).not.toContain("someone-else"); + expect(String(attempt.thrown)).not.toContain("workflow_run_v2"); + } + }); + + // RR15: a truncated journal is held to the same rule. A recorded + // establishment failure is not a licence to establish the run again. + it("RR15: refuses a truncated journal whose run record failed", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const failed = new InMemoryStream( + partial(first) + .snapshot() + .map((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? { ...event, result: { status: "err", error: { message: "planted-establishment" } } } + : event, + ), + ); + const before = failed.snapshot().length; + + const attempt = yield* runRetained(RETAINED, failed); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(failed.snapshot().length).toEqual(before); + // The recorded failure text is not replayed as this run's diagnostic. + expect(String(attempt.thrown)).not.toContain("planted-establishment"); + }); + + // RR16: nothing about the run reaches the error object, only the message. + it("RR16: a refusal retains no run id, base, pinned commit or planted member", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const attempt = yield* runRetained( + { ...RETAINED, runId: "release-1.5" }, + new InMemoryStream(first.snapshot()), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + // The whole error, not only its sentence: `StaleInputError` keeps what it + // is handed, so the description it retains is inspected too. + const whole = wholeError(attempt.thrown); + for (const secret of ["release-1.4", "release-1.5", COMMIT, "main"]) { + expect(whole).not.toContain(secret); + } + expect(whole).toContain("runId"); + }); + + // RR17: the second bypass the architecture review found. `Execution` is + // composable too — a handler registered at the same position can rebuild the + // options a later one produced, stream included — so an installation that + // wrapped the stream itself was still held to registration order. Nothing is + // wrapped now: core reads the requirement and applies it to the journal it + // built, after every handler has had its turn. + it("RR17: refuses another run's journal however Execution middleware is ordered", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + const other = { ...RETAINED, runId: "release-1.5" }; + + const hijacks: Array<{ says: string; install: (raw: DurableStream) => Operation }> = [ + { says: "the package's own descriptor", install: useStreamHijack }, + { says: "a descriptor built here", install: useForeignStreamHijack }, + ]; + + for (const hijack of hijacks) { + // Registered before the workflow installation, and registered after it. + // One of these was the order that used to succeed. + for (const order of ["before", "after"] as const) { + const raw = new InMemoryStream(first.snapshot()); + const handler = hijack.install(raw); + const attempt = yield* runRetained( + other, + new InMemoryStream(first.snapshot()), + order === "before" ? handler : ok(), + order === "after" ? handler : ok(), + ); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain("runId"); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(raw.snapshot().length).toEqual(first.snapshot().length); + } + } + }); + + // RR18: admission and replay read one history, not two. The journal below + // answers differently on a second read; what makes that harmless is that the + // history is retained once, before anything is decided, and every phase is + // handed those same objects. + it("RR18: identity, guard observation and replay consume one retained snapshot", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const reads: string[] = []; + const observed: Json[] = []; + const attempt = yield* runRetained( + RETAINED, + shiftingJournal(first.snapshot(), reads), + ReplayGuard.around({ + *check([event], next) { + if (event.description.type === "workflow_run" && event.result.status === "ok") { + observed.push(event.result.value ?? null); + } + yield* next(event); + }, + }), + ); + + // The backend was asked once and never again, which is what makes the + // shift unreachable: the history is retained before anything is decided. + expect(reads).toEqual(["release-1.4"]); + // Admission agreed with that settlement, so nothing was refused... + expect(attempt.thrown).toBeUndefined(); + // ...and the guard was handed the same settled value, not a second look. + expect(observed).toEqual([{ runId: "release-1.4", base: "main", pinnedCommit: COMMIT }]); + expect(attempt.seen).toEqual([]); + }); + + // RR19: an event that refuses to be read is a history this run cannot + // describe, not one to step past on the way to a record that does read. + it("RR19: refuses a history holding an event that will not read", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const refusing: DurableEvent = { + get type(): never { + throw new Error("planted-unreadable-discriminator"); + }, + } as unknown as DurableEvent; + + const watched = unreadableJournal([refusing, ...first.snapshot()]); + const attempt = yield* runRetained(RETAINED, watched.journal); + + expect(attempt.thrown).toBeInstanceOf(Error); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(String(attempt.thrown)).not.toContain("planted-unreadable-discriminator"); + }); + + // RR20: nothing a journal holds gets to raise its own exception. Each value + // below refuses a different way — enumeration, descriptors, a getter, and + // classification of a revoked proxy — and every one of them has to arrive as + // the same fixed refusal, carrying none of its own text. + it("RR20: a hostile recorded value becomes the fixed refusal and leaks nothing", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const hostile: Array<{ says: string; value: () => unknown }> = [ + { + says: "ownKeys refuses", + value: () => + new Proxy( + { ...RETAINED }, + { + ownKeys() { + throw new Error("PLANTED-OWNKEYS"); + }, + }, + ), + }, + { + says: "getOwnPropertyDescriptor refuses", + value: () => + new Proxy( + { ...RETAINED }, + { + getOwnPropertyDescriptor() { + throw new Error("PLANTED-DESCRIPTOR"); + }, + }, + ), + }, + { + says: "a getter refuses", + value: () => ({ + base: "main", + pinnedCommit: COMMIT, + get runId(): never { + throw new Error("PLANTED-GETTER"); + }, + }), + }, + { + says: "classification refuses", + value: () => { + const revoked = Proxy.revocable({ ...RETAINED }, {}); + revoked.revoke(); + return revoked.proxy; + }, + }, + ]; + + for (const planted of hostile) { + const watched = unreadableJournal( + first + .snapshot() + .map((event) => + event.type === "yield" && event.description.type === "workflow_run" + ? { ...event, result: { status: "ok", value: planted.value() as Json } } + : event, + ), + ); + + const attempt = yield* runRetained(RETAINED, watched.journal); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + // No terminal result reused, no document code, no output, no append. + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(watched.appended).toEqual([]); + // The whole error object, not only its sentence. + const whole = wholeError(attempt.thrown); + for (const secret of [ + "PLANTED-OWNKEYS", + "PLANTED-DESCRIPTOR", + "PLANTED-GETTER", + "release-1.4", + COMMIT, + ]) { + expect(whole).not.toContain(secret); + } + } + }); + + // RR21: the record is written once, before the root import, so a second entry + // under the canonical identity describes a second run — however either of them + // ended. A failure beside a success is two runs, not one run with a stumble. + it("RR21: refuses a duplicate canonical record in every settlement combination", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + const failure = (event: Yield): DurableEvent => ({ + ...event, + result: { status: "err", error: { message: "planted-second-establishment" } }, + }); + + const combinations: Array<{ + says: string; + change: (event: Yield) => DurableEvent[]; + }> = [ + { says: "successful + successful", change: (event) => [event, event] }, + { says: "successful + failed", change: (event) => [event, failure(event)] }, + { says: "failed + successful", change: (event) => [failure(event), event] }, + { says: "failed + failed", change: (event) => [failure(event), failure(event)] }, + ]; + + for (const combination of combinations) { + const journal = completedWith(first, combination.change); + const before = journal.snapshot().length; + const attempt = yield* runRetained(RETAINED, journal); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.thrown instanceof Error ? attempt.thrown.message : "").toContain("2"); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(journal.snapshot().length).toEqual(before); + expect(wholeError(attempt.thrown)).not.toContain("planted-second-establishment"); + } + }); + + // RR22: the same totality on the other side of the boundary. A host hands the + // retained run over directly, so its members are read here rather than out of + // a journal — and a member that refuses is a value identifying no run, which + // is a sentence rather than the host's own exception. + it("RR22: refuses a retained run whose members refuse to be read", function* () { + const hostile = { + base: "main", + pinnedCommit: COMMIT, + get runId(): never { + throw new Error("PLANTED-INSTALL-GETTER"); + }, + }; + + const thrown = yield* scoped(function* () { + try { + const workflow = retainedWorkflowInstallation(hostile as unknown as WorkflowRun); + return undefined; + } catch (error) { + return error; + } + }); + + expect(thrown).toBeInstanceOf(Error); + expect(String(thrown)).toContain("identifies no workflow run"); + expect(wholeError(thrown)).not.toContain("PLANTED-INSTALL-GETTER"); + }); + + // RR23: a workflow definition may name an exact document target (#431). That + // is definition data — what the run is a run *of* — and never part of what + // identifies the run. A journal recording it inside the run value describes + // something this version cannot account for. + it("RR23: a definition target is not part of run identity", function* () { + const first = new InMemoryStream(); + yield* runRetained(RETAINED, first); + + const withTarget = completedWith(first, (event) => [ + { + ...event, + result: { + status: "ok", + value: { ...RETAINED, targetPath: "Release/Publish" }, + }, + }, + ]); + const before = withTarget.snapshot().length; + const attempt = yield* runRetained(RETAINED, withTarget); + + expect(attempt.thrown).toBeInstanceOf(StaleInputError); + expect(attempt.seen).toEqual([]); + expect(attempt.emitted).toEqual([]); + expect(withTarget.snapshot().length).toEqual(before); + // The target is not quoted back either. + expect(wholeError(attempt.thrown)).not.toContain("Release/Publish"); + }); + + it("RR7: refuses to install a run that identifies nothing", function* () { + const empty = yield* scoped(function* () { + try { + const workflow = retainedWorkflowInstallation({ + runId: "", + base: "main", + pinnedCommit: COMMIT, + }); + return undefined; + } catch (error) { + return error; + } + }); + expect(empty).toBeInstanceOf(Error); + + const unpinned = yield* scoped(function* () { + try { + const workflow = retainedWorkflowInstallation({ + runId: "r", + base: "main", + pinnedCommit: "", + }); + return undefined; + } catch (error) { + return error; + } + }); + expect(unpinned).toBeInstanceOf(Error); + }); +}); diff --git a/packages/workflow/tests/support/storage.ts b/packages/workflow/tests/support/storage.ts index 94b09b23..0914488d 100644 --- a/packages/workflow/tests/support/storage.ts +++ b/packages/workflow/tests/support/storage.ts @@ -23,7 +23,9 @@ import { type WorkflowRunDatabase, WorkflowRunStorage, } from "../../mod.ts"; -import { useWorkflowRunStorage, workflowRunPath } from "../../deno.ts"; +import { workflowRunPath } from "../../deno.ts"; +import { installWorkflowRunStorage } from "../../src/deno/provider.ts"; +import type { PrivateWorkspaceOptions } from "../../src/deno/workspace/private.ts"; export const SHA1 = "9fceb02d0ae598e95dc970b74767f19372d61af8"; @@ -67,10 +69,21 @@ export function request( }; } -/** Run `body` with this host's storage installed for its scope only. */ -export function withStorage(root: string, body: () => Operation): Operation { +/** + * Run `body` with this host's storage installed for its scope only. + * + * `internal` is the provider's own installation option, supplied here and + * nowhere a document could reach: the decorator it may carry replaces the + * authoritative Workspace filesystem, and that decision belongs to whoever + * installs the provider. + */ +export function withStorage( + root: string, + body: () => Operation, + internal: PrivateWorkspaceOptions = {}, +): Operation { return scoped(function* () { - yield* useWorkflowRunStorage({ root }); + yield* installWorkflowRunStorage({ root }, internal); return yield* body(); }); } diff --git a/packages/workflow/tests/support/workspace-crash-child.ts b/packages/workflow/tests/support/workspace-crash-child.ts index aa3d125b..fab3aa59 100644 --- a/packages/workflow/tests/support/workspace-crash-child.ts +++ b/packages/workflow/tests/support/workspace-crash-child.ts @@ -43,7 +43,7 @@ import { useJournalRouting } from "../../src/deno/journal-route.ts"; import { readTransaction } from "../../src/deno/reading.ts"; import { verifySchema } from "../../src/deno/schema.ts"; import { - createWorkspaceProofEffect, + createWorkspaceEffect, useWorkspaceEffects, withWorkspaceEffects, } from "../../src/deno/workspace/effect.ts"; @@ -146,7 +146,7 @@ function* crash(root: string, runId: string): Operation { // The run this process resumes already holds this effect's result, so it // replays. Executing it would mean the crash effect below is not the // first live work of the process, and the count says which happened. - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: BASELINE_EFFECT }, // deno-lint-ignore require-yield @@ -155,7 +155,7 @@ function* crash(root: string, runId: string): Operation { return null; }, ); - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: CRASH_EFFECT }, function* (selected) { diff --git a/packages/workflow/tests/support/workspace-restart-child.ts b/packages/workflow/tests/support/workspace-restart-child.ts index c3a94e04..4613c488 100644 --- a/packages/workflow/tests/support/workspace-restart-child.ts +++ b/packages/workflow/tests/support/workspace-restart-child.ts @@ -25,10 +25,7 @@ import { durableRun, type Workflow } from "@executablemd/durable-streams"; import { main, type Operation } from "effection"; import { WorkflowRunStorage, type WorkflowRunDatabase } from "../../mod.ts"; import { useWorkflowRunStorage } from "../../deno.ts"; -import { - createWorkspaceProofEffect, - withWorkspaceEffects, -} from "../../src/deno/workspace/effect.ts"; +import { createWorkspaceEffect, withWorkspaceEffects } from "../../src/deno/workspace/effect.ts"; import { setPrivateWorkspaceClock, transactWorkspaceRoots, @@ -62,7 +59,7 @@ const DEFINITION = { */ function workflow(database: WorkflowRunDatabase, marker: string, clock: { now: number }) { return function* (): Workflow { - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: "seed" }, function* (filesystem) { @@ -74,7 +71,7 @@ function workflow(database: WorkflowRunDatabase, marker: string, clock: { now: n return null; }, ); - yield createWorkspaceProofEffect( + yield createWorkspaceEffect( database, { type: "workspace-proof", name: "revise" }, function* (filesystem) { diff --git a/packages/workflow/tests/workflow-run.test.ts b/packages/workflow/tests/workflow-run.test.ts index b3c90bc6..c1fe8445 100644 --- a/packages/workflow/tests/workflow-run.test.ts +++ b/packages/workflow/tests/workflow-run.test.ts @@ -22,12 +22,17 @@ import { collect, DocumentOutput, execute, + Execution, getExpansion, inlineSource, registerComponents, } from "@executablemd/core"; +import { createApi } from "@effectionx/context-api"; +import type { Api } from "@effectionx/context-api"; +import { executeInstalled } from "@executablemd/core/host"; +import type { ExecutionInstallation } from "@executablemd/core/host"; import { Git } from "../src/git.ts"; -import { getWorkflowRun, useWorkflow } from "../src/run.ts"; +import { getWorkflowRun, workflowInstallation } from "../src/run.ts"; import type { WorkflowRun } from "../src/run.ts"; const COMMIT = "9fceb02d0ae598e95dc970b74767f19372d61af8"; @@ -99,7 +104,7 @@ describe("Tier WR — workflow runs", () => { yield* scoped(function* () { yield* useGit(COMMIT, []); yield* useProbe(seen); - yield* useWorkflow({ base: "main" }); + const workflow = workflowInstallation({ base: "main" }); // Installing the middleware creates no workflow run. try { @@ -109,7 +114,9 @@ describe("Tier WR — workflow runs", () => { } yield* collect( - yield* execute({ ...inlineSource("\n"), stream: new InMemoryStream() }), + yield* executeInstalled({ ...inlineSource("\n"), stream: new InMemoryStream() }, [ + workflow, + ]), ); // ...and the run does not outlive the execution that owns it, even though @@ -123,8 +130,8 @@ describe("Tier WR — workflow runs", () => { expect(seen).toHaveLength(1); expect(seen[0]).toEqual({ runId: expect.any(String), base: "main", pinnedCommit: COMMIT }); - expect(before[0]).toContain("useWorkflow"); - expect(after[0]).toContain("useWorkflow"); + expect(before[0]).toContain("workflowInstallation"); + expect(after[0]).toContain("workflowInstallation"); }); it("WR2: every read inside one execution answers with the same frozen value", function* () { @@ -133,12 +140,15 @@ describe("Tier WR — workflow runs", () => { yield* scoped(function* () { yield* useGit(COMMIT, []); yield* useProbe(seen); - yield* useWorkflow({ base: "main" }); + const workflow = workflowInstallation({ base: "main" }); yield* collect( - yield* execute({ - ...inlineSource("\n\n\n"), - stream: new InMemoryStream(), - }), + yield* executeInstalled( + { + ...inlineSource("\n\n\n"), + stream: new InMemoryStream(), + }, + [workflow], + ), ); }); @@ -155,9 +165,12 @@ describe("Tier WR — workflow runs", () => { return scoped(function* () { yield* useGit(commit, []); yield* useProbe(into); - yield* useWorkflow({ base }); + const workflow = workflowInstallation({ base }); yield* collect( - yield* execute({ ...inlineSource("\n"), stream: new InMemoryStream() }), + yield* executeInstalled( + { ...inlineSource("\n"), stream: new InMemoryStream() }, + [workflow], + ), ); }); } @@ -176,6 +189,105 @@ describe("Tier WR — workflow runs", () => { expect(first[0]?.runId).not.toBe(second[0]?.runId); }); + it("WR20: one installation value reused by two executions gives each its own run", function* () { + // A host may hold one installation value and run more than one document + // with it. Each of those is a separate execution, and the run one of them + // records is not the other's — including while both are live. + const shared = workflowInstallation({ base: "main" }); + const prepared = withResolvers(); + const release = withResolvers(); + const streamA = new InMemoryStream(); + const streamB = new InMemoryStream(); + const seenA: WorkflowRun[] = []; + const seenB: WorkflowRun[] = []; + + const first = yield* spawn(() => + scoped(function* () { + yield* useGit(COMMIT, []); + yield* registerComponents([ + { + name: "Hold", + origin: "tier-wr", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + // Prepared, and holding here while the sibling runs end to end. + prepared.resolve(); + yield* release.operation; + seenA.push(yield* getWorkflowRun()); + seenA.push(yield* getWorkflowRun()); + return ""; + }, + }, + ]); + return yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream: streamA }, [shared]), + ); + }), + ); + + yield* prepared.operation; + + yield* scoped(function* () { + yield* useGit(COMMIT, []); + yield* useProbe(seenB); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream: streamB }, [shared]), + ); + }); + + release.resolve(); + yield* first; + + const recordedA = recordedRun(streamA); + const recordedB = recordedRun(streamB); + expect(recordedA).toBeDefined(); + expect(recordedB).toBeDefined(); + const idA = + recordedA !== null && typeof recordedA === "object" && "runId" in recordedA + ? recordedA.runId + : undefined; + const idB = + recordedB !== null && typeof recordedB === "object" && "runId" in recordedB + ? recordedB.runId + : undefined; + // Two executions, two runs. + expect(idA).toBeDefined(); + expect(idB).toBeDefined(); + expect(idA).not.toEqual(idB); + + // Each document read the run its own journal records — the sibling's + // preparation and completion changed neither. + expect(seenA[0]?.runId).toEqual(idA); + expect(seenB[0]?.runId).toEqual(idB); + // And every read inside one execution is the same frozen object. + expect(seenA[0]).toBe(seenA[1]); + + // Nothing leaked into a later ordinary execution. + let leaked: unknown; + yield* scoped(function* () { + yield* useForbiddenGit(); + yield* registerComponents([ + { + name: "Probe", + origin: "tier-wr", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + try { + leaked = yield* getWorkflowRun(); + } catch { + leaked = "refused"; + } + return ""; + }, + }, + ]); + yield* collect( + yield* execute({ ...inlineSource("\n"), stream: new InMemoryStream() }), + ); + }); + expect(leaked).toEqual("refused"); + }); + it("WR4: a completed journal restores the run without consulting Git", function* () { const stream = new InMemoryStream(); const live: WorkflowRun[] = []; @@ -184,8 +296,10 @@ describe("Tier WR — workflow runs", () => { yield* scoped(function* () { yield* useGit(COMMIT, asked); yield* useProbe(live); - yield* useWorkflow({ base: "main" }); - yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + const workflow = workflowInstallation({ base: "main" }); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); }); expect(asked).toEqual(["main^{commit}"]); @@ -194,14 +308,16 @@ describe("Tier WR — workflow runs", () => { const replayed: WorkflowRun[] = []; yield* scoped(function* () { yield* useForbiddenGit(); - yield* useWorkflow({ base: "main" }); + const workflow = workflowInstallation({ base: "main" }); yield* DocumentOutput.around({ *output([text], next) { replayed.push(yield* getWorkflowRun()); yield* next(text); }, }); - yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); }); expect(replayed).not.toHaveLength(0); @@ -215,21 +331,23 @@ describe("Tier WR — workflow runs", () => { yield* scoped(function* () { yield* useGit(COMMIT, []); yield* useProbe([]); - yield* useWorkflow({ base: "main" }); - yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + const workflow = workflowInstallation({ base: "main" }); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); }); const emitted: string[] = []; const result = yield* scoped(function* () { yield* useForbiddenGit(); - yield* useWorkflow({ base: "release" }); + const workflow = workflowInstallation({ base: "release" }); yield* DocumentOutput.around({ *output([text], next) { emitted.push(text); yield* next(text); }, }); - return yield* yield* execute({ ...inlineSource("\n"), stream }); + return yield* yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]); }); expect(result.ok).toBe(false); @@ -246,8 +364,10 @@ describe("Tier WR — workflow runs", () => { yield* scoped(function* () { yield* useGit(COMMIT, []); yield* useProbe(live); - yield* useWorkflow({ base: "main" }); - yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + const workflow = workflowInstallation({ base: "main" }); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); }); // The branch moved between runs. The recorded commit is what stands. @@ -255,14 +375,16 @@ describe("Tier WR — workflow runs", () => { const asked: string[] = []; yield* scoped(function* () { yield* useGit(OTHER_COMMIT, asked); - yield* useWorkflow({ base: "main" }); + const workflow = workflowInstallation({ base: "main" }); yield* DocumentOutput.around({ *output([text], next) { restored.push(yield* getWorkflowRun()); yield* next(text); }, }); - yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); }); expect(asked).toHaveLength(0); @@ -285,8 +407,8 @@ describe("Tier WR — workflow runs", () => { { at: "min" }, ); yield* useProbe(expanded); - yield* useWorkflow({ base: "main" }); - return yield* yield* execute({ ...inlineSource("\n"), stream }); + const workflow = workflowInstallation({ base: "main" }); + return yield* yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]); }); expect(result.ok).toBe(false); @@ -294,17 +416,133 @@ describe("Tier WR — workflow runs", () => { expect(recordedRun(stream)).toBeUndefined(); }); + // WR18: a base that would not resolve is journaled as a failed effect, and + // that history is this run's own — so a programmatic installation contributes + // no refusal about it. Requiring a *successful* record here would refuse a + // journal this run wrote, and retry Git on the way to doing so. + // + // What this does not assert is that the recorded Git failure is what the + // caller sees. It is not, and it was not before this PR either: the journal + // holds a root Close and no root import, which core's target admission refuses + // on its own terms (verified against `main` at b324b97). That contradiction + // between core's rule and workflow-spec §6 is recorded in §6 and is not this + // PR's to settle. + it("WR18: a recorded base-resolution failure is not refused as missing evidence", function* () { + const stream = new InMemoryStream(); + + const first = yield* scoped(function* () { + yield* Git.around( + { + // deno-lint-ignore require-yield + *revParse() { + throw new Error("fatal: not a git repository"); + }, + }, + { at: "min" }, + ); + const workflow = workflowInstallation({ base: "main" }); + return yield* yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]); + }); + expect(first.ok).toBe(false); + expect(recordedRun(stream)).toBeUndefined(); + + const before = stream.snapshot().length; + const expanded: WorkflowRun[] = []; + const replayed = yield* scoped(function* () { + yield* useForbiddenGit(); + yield* useProbe(expanded); + const workflow = workflowInstallation({ base: "main" }); + return yield* yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]); + }); + + expect(replayed.ok).toBe(false); + const message = replayed.ok ? "" : replayed.error.message; + // Nothing about workflow-run evidence: this installation had no objection. + expect(message).not.toContain("workflow run"); + expect(message).not.toContain("identifies"); + // Git was never asked, the root never expanded, and nothing was appended. + expect(expanded).toEqual([]); + expect(stream.snapshot().length).toEqual(before); + }); + + it("WR19: a preparation that failed replays as the failure it was", function* () { + const stream = new InMemoryStream(); + const expanded: WorkflowRun[] = []; + + const live = yield* scoped(function* () { + yield* Git.around( + { + // deno-lint-ignore require-yield + *revParse() { + throw new Error("fatal: not a git repository"); + }, + }, + { at: "min" }, + ); + yield* useProbe(expanded); + return yield* yield* executeInstalled({ ...inlineSource("\n"), stream }, [ + workflowInstallation({ base: "main" }), + ]); + }); + + expect(live.ok).toBe(false); + expect(expanded).toHaveLength(0); + // Preparation failed before the root import, so the durable root records + // the bound terminal core writes at that position (#433) — a history the + // identical execution can read back rather than one it must refuse. + const recorded = stream.snapshot(); + expect(recorded.some((event) => event.type === "close")).toBe(true); + expect( + recorded.some( + (event) => event.type === "yield" && event.description.type === "import_component", + ), + ).toBe(false); + + const replay = new InMemoryStream(recorded); + const asked: string[] = []; + const order: string[] = []; + const replayedExpansions: WorkflowRun[] = []; + + const replayed = yield* scoped(function* () { + yield* useGit(COMMIT, asked); + yield* useProbe(replayedExpansions); + yield* Execution.around({ + *document([request], next) { + order.push("policy"); + yield* next(request); + }, + }); + return yield* yield* executeInstalled({ ...inlineSource("\n"), stream: replay }, [ + workflowInstallation({ base: "main" }), + ]); + }); + + // The same failure, read back rather than re-decided. + expect(replayed.ok).toBe(false); + expect(String(replayed.ok ? "" : replayed.error)).not.toContain( + "cannot be read by this version", + ); + // Nothing ran: no Git, no preparation, no policy, no import, no append. + expect(asked).toEqual([]); + expect(order).toEqual([]); + expect(replayedExpansions).toHaveLength(0); + expect(replay.snapshot().length).toEqual(recorded.length); + }); + it("WR8: a run survives a document that fails after it was recorded", function* () { const stream = new InMemoryStream(); const result = yield* scoped(function* () { yield* useGit(COMMIT, []); - yield* useWorkflow({ base: "main" }); - return yield* yield* execute({ - // A value root that produces no fails after expansion begins. - ...inlineSource("---\nreturns:\n type: object\n---\n\nbody\n"), - stream, - }); + const workflow = workflowInstallation({ base: "main" }); + return yield* yield* executeInstalled( + { + // A value root that produces no fails after expansion begins. + ...inlineSource("---\nreturns:\n type: object\n---\n\nbody\n"), + stream, + }, + [workflow], + ); }); expect(result.ok).toBe(false); @@ -317,7 +555,7 @@ describe("Tier WR — workflow runs", () => { // Ordinary execution stays Git-independent: with no workflow middleware // installed, nothing reaches the process boundary looking for `git`. - it("WR9: an execution without useWorkflow never invokes Git", function* () { + it("WR9: an execution without a workflow installation never invokes Git", function* () { const commands: string[][] = []; const output = yield* scoped(function* () { @@ -365,7 +603,7 @@ describe("Tier WR — workflow runs", () => { }); expect(failures).toHaveLength(1); - expect(failures[0]).toContain("useWorkflow"); + expect(failures[0]).toContain("workflowInstallation"); }); it("WR11: a journal holding something else under the workflow name is refused", function* () { @@ -379,8 +617,8 @@ describe("Tier WR — workflow runs", () => { const result = yield* scoped(function* () { yield* useForbiddenGit(); - yield* useWorkflow({ base: "main" }); - return yield* yield* execute({ ...inlineSource("# Hello\n"), stream }); + const workflow = workflowInstallation({ base: "main" }); + return yield* yield* executeInstalled({ ...inlineSource("# Hello\n"), stream }, [workflow]); }); expect(result.ok).toBe(false); @@ -422,14 +660,14 @@ describe("Tier WR — workflow runs", () => { const restored: WorkflowRun[] = []; yield* scoped(function* () { yield* useForbiddenGit(); - yield* useWorkflow({ base: "main" }); + const workflow = workflowInstallation({ base: "main" }); yield* DocumentOutput.around({ *output([text], next) { restored.push(yield* getWorkflowRun()); yield* next(text); }, }); - yield* collect(yield* execute({ ...inlineSource("# Hello\n"), stream })); + yield* collect(yield* executeInstalled({ ...inlineSource("# Hello\n"), stream }, [workflow])); }); expect(restored[0]).toEqual({ runId: "seeded-run", base: "main", pinnedCommit: COMMIT }); @@ -448,8 +686,10 @@ describe("Tier WR — workflow runs", () => { yield* scoped(function* () { yield* useForbiddenGit(); yield* useProbe(seen); - yield* useWorkflow({ base: "main" }); - yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + const workflow = workflowInstallation({ base: "main" }); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); }); expect(seen).toHaveLength(1); @@ -476,11 +716,12 @@ describe("Tier WR — workflow runs", () => { }, }, ]); - if (base !== undefined) { - yield* useWorkflow({ base }); - } + const workflow = base === undefined ? undefined : workflowInstallation({ base }); yield* collect( - yield* execute({ ...inlineSource("\n"), stream: new InMemoryStream() }), + yield* executeInstalled( + { ...inlineSource("\n"), stream: new InMemoryStream() }, + workflow === undefined ? [] : [workflow], + ), ); }); return seen; @@ -518,10 +759,12 @@ describe("Tier WR — workflow runs", () => { }, }, ]); - yield* useWorkflow({ base: "main" }); + const workflow = workflowInstallation({ base: "main" }); const execution = yield* spawn(function* () { - yield* collect(yield* execute({ ...inlineSource("\n"), stream })); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [workflow]), + ); }); yield* observed.operation; yield* execution.halt(); @@ -537,6 +780,11 @@ describe("Tier WR — workflow runs", () => { // `run.ts` makes the same loaded-copy claim core does: a second copy of this // package reads the run through its own descriptor of the same context name. it("WR17: a descriptor of the same name built elsewhere reads the run", function* () { + // The binding is the execution's own slot, and what it holds is the exact + // frozen run. Retained-history admission and durable preparation both + // decide the run from outside the scope that reads it, so a context set + // from either would end with the operation that set it; the slot is what + // gives them somewhere to put it that the document can still see. let own: WorkflowRun | undefined; let observed: unknown; const elsewhere = createContext("executablemd.workflow.run", undefined); @@ -555,14 +803,190 @@ describe("Tier WR — workflow runs", () => { }, }, ]); - yield* useWorkflow({ base: "main" }); + const workflow = workflowInstallation({ base: "main" }); yield* collect( - yield* execute({ ...inlineSource("\n"), stream: new InMemoryStream() }), + yield* executeInstalled({ ...inlineSource("\n"), stream: new InMemoryStream() }, [ + workflow, + ]), ); }); expect(own).toBeDefined(); - expect(observed).toBe(own); + expect(observed).toBeDefined(); + const carried = + observed !== null && typeof observed === "object" && "run" in observed + ? observed.run + : undefined; + expect(carried).toBe(own); + }); + + /** + * A descriptor of the Api's name whose `document` may return anything. + * + * The canonical surface types `Operation`, so a handler cannot offer a + * document result through it. Middleware from elsewhere is under no such + * obligation — its types are its own — and this is what that looks like. + */ + function loose(): Api<{ document(request: unknown): Operation }> { + return createApi("Execution", { + // deno-lint-ignore require-yield + *document(_request: unknown): Operation { + return undefined; + }, + }); + } + + /** + * Tier WM — what public middleware cannot do to a workflow run. + * + * The run is prepared by canonical core inside the durable root, after + * retained-history admission and before any public `Execution.document` + * policy. These are the ways a handler might try to get in front of that, and + * what happens instead. + */ + + it("WM1: a non-delegating document handler cannot fabricate a successful run", function* () { + const journal = new InMemoryStream(); + const seen: WorkflowRun[] = []; + + const result = yield* scoped(function* () { + yield* useGit(COMMIT, []); + yield* useProbe(seen); + yield* loose().around({ + // deno-lint-ignore require-yield + *document() { + return { status: "ok", output: "FABRICATED", value: "FABRICATED" }; + }, + }); + return yield* yield* executeInstalled({ ...inlineSource("\n"), stream: journal }, [ + workflowInstallation({ base: "main" }), + ]); + }); + + // The handler decided nothing. Core refused it. + expect(result.ok).toBe(false); + expect(seen).toEqual([]); + // The run was prepared before that handler ever ran, so the journal holds + // the canonical record — and then the failed terminal, never a success + // carrying "FABRICATED". + expect(recordedRun(journal)).toBeDefined(); + expect(JSON.stringify(journal.snapshot())).not.toContain("FABRICATED"); + const closes = journal.snapshot().filter((event) => event.type === "close"); + expect(closes.length).toEqual(1); + }); + + it("WM2: a substitute result after delegating replaces nothing", function* () { + const journal = new InMemoryStream(); + const seen: WorkflowRun[] = []; + + const output = yield* scoped(function* () { + yield* useGit(COMMIT, []); + yield* useProbe(seen); + yield* loose().around({ + *document([request], next) { + yield* next(request); + return { status: "ok", output: "SUBSTITUTE", value: "SUBSTITUTE" }; + }, + }); + return yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream: journal }, [ + workflowInstallation({ base: "main" }), + ]), + ); + }); + + expect(String(output)).not.toContain("SUBSTITUTE"); + expect(seen.length).toEqual(1); + expect(recordedRun(journal)).toBeDefined(); + }); + + it("WM3: the run is prepared before any document policy observes it", function* () { + const order: string[] = []; + const journal = new InMemoryStream(); + + yield* scoped(function* () { + yield* useGit(COMMIT, []); + yield* registerComponents([ + { + name: "Probe", + origin: "tier-wr", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + order.push(`expanded:${(yield* getWorkflowRun()).pinnedCommit === COMMIT}`); + return ""; + }, + }, + ]); + yield* Execution.around({ + *document([request], next) { + // The record is already in the journal by the time policy runs. + order.push(`policy:${workflowEvents(journal).length}`); + yield* next(request); + }, + }); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream: journal }, [ + workflowInstallation({ base: "main" }), + ]), + ); + }); + + expect(order).toEqual(["policy:1", "expanded:true"]); + }); + + it("WM4: an installation another loaded copy built composes when the host passes it", function* () { + const seen: WorkflowRun[] = []; + const journal = new InMemoryStream(); + + // Built through this package's own constructor, then relayed to canonical + // core as an opaque record of closures — which is exactly how a separately + // loaded copy reaches core: it hands over functions, and nobody agrees on + // a name or a type. + const built = workflowInstallation({ base: "main" }); + const relayed: ExecutionInstallation = { + admissions: built.admissions, + prepare: built.prepare, + install: built.install, + }; + + yield* scoped(function* () { + yield* useGit(COMMIT, []); + yield* useProbe(seen); + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream: journal }, [relayed]), + ); + }); + + expect(seen.length).toEqual(1); + expect(recordedRun(journal)).toBeDefined(); + }); + + it("WM5: valid public observation still composes", function* () { + const observed: string[] = []; + const seen: WorkflowRun[] = []; + const journal = new InMemoryStream(); + + const output = yield* scoped(function* () { + yield* useGit(COMMIT, []); + yield* useProbe(seen); + yield* Execution.around({ + *document([request], next) { + observed.push("before"); + yield* next(request); + observed.push("after"); + }, + }); + return yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream: journal }, [ + workflowInstallation({ base: "main" }), + ]), + ); + }); + + expect(observed).toEqual(["before", "after"]); + expect(seen.length).toEqual(1); + expect(String(output).trim()).toEqual(""); + expect(recordedRun(journal)).toBeDefined(); }); it("WR12: a slow Git does not stall a sibling execution", function* () { @@ -581,17 +1005,23 @@ describe("Tier WR — workflow runs", () => { }, { at: "min" }, ); - yield* useWorkflow({ base: "slow" }); + const workflow = workflowInstallation({ base: "slow" }); yield* collect( - yield* execute({ ...inlineSource("\n"), stream: new InMemoryStream() }), + yield* executeInstalled( + { ...inlineSource("\n"), stream: new InMemoryStream() }, + [workflow], + ), ); }), ); yield* scoped(function* () { yield* useGit(OTHER_COMMIT, []); - yield* useWorkflow({ base: "fast" }); + const workflow = workflowInstallation({ base: "fast" }); yield* collect( - yield* execute({ ...inlineSource("\n"), stream: new InMemoryStream() }), + yield* executeInstalled( + { ...inlineSource("\n"), stream: new InMemoryStream() }, + [workflow], + ), ); }); yield* slow; diff --git a/packages/workflow/tests/workspace-effect-transaction.test.ts b/packages/workflow/tests/workspace-effect-transaction.test.ts index 7b947ff7..1fe5ca9b 100644 --- a/packages/workflow/tests/workspace-effect-transaction.test.ts +++ b/packages/workflow/tests/workspace-effect-transaction.test.ts @@ -37,7 +37,7 @@ import { useJournalRouting } from "../src/deno/journal-route.ts"; import { SavepointObservation, type SavepointObserver } from "../src/deno/savepoints.ts"; import { initializeSchema } from "../src/deno/schema.ts"; import { - createWorkspaceProofEffect, + createWorkspaceEffect, useWorkspaceEffects, withWorkspaceEffects, } from "../src/deno/workspace/effect.ts"; @@ -132,7 +132,7 @@ function* workspaceStep( name: string, mutate: (filesystem: DenoWorkspaceFilesystem) => Operation, ): Workflow { - yield createWorkspaceProofEffect(database, { type: "workspace-proof", name }, mutate); + yield createWorkspaceEffect(database, { type: "workspace-proof", name }, mutate); } function* inspectWorkspace( diff --git a/packages/workflow/tests/workspace-files.test.ts b/packages/workflow/tests/workspace-files.test.ts new file mode 100644 index 00000000..5dea6744 --- /dev/null +++ b/packages/workflow/tests/workspace-files.test.ts @@ -0,0 +1,936 @@ +/** + * Tier WF — the document filesystem of a workflow run. + * + * These drive the real `` and `` definitions through `execute()` + * against a real run database, because what is under test is where a document's + * paths land and what survives in the journal — neither of which a stand-in for + * DOFS or for SQLite could show. + * + * Two observations do most of the work. A second connection counts committed + * journal rows, which says whether a transaction has already published rather + * than whether a row is there now; and a host `API.Files` spy is installed + * *outside* the workflow provider, so any call that fell through to the caller's + * filesystem would be recorded rather than merely suspected. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { createContext, scoped, type Operation } from "effection"; +import { type Api, createApi } from "@effectionx/context-api"; +import { collect, execute, inlineSource, registerComponents } from "@executablemd/core"; +import type { Json } from "@executablemd/durable-streams"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { API, FILES_FATAL, parseFilesFatal, useHostFiles } from "@executablemd/runtime"; +import type { HostFilesEvent } from "@executablemd/runtime"; +import type { WorkflowRunDatabase } from "../mod.ts"; +import { withWorkflowWorkspace } from "../src/deno/workspace/host.ts"; +import { WORKSPACE_FILE } from "../src/deno/workspace/files.ts"; +import { throwWorkspaceFilesystemFailure } from "../src/deno/workspace/errors.ts"; +import type { DenoWorkspaceFilesystem } from "../src/deno/workspace/filesystem.ts"; +import { transactWorkspaceRoots } from "../src/deno/workspace/private.ts"; +import type { PrivateWorkspaceTransaction } from "../src/deno/workspace/private.ts"; +import { + committedEventCount, + createRun, + runPath, + tamper, + useStorageRoot, + withStorage, +} from "./support/storage.ts"; + +/** What an operation threw, so a suite can assert on it rather than fail. */ +function* raised(operation: Operation): Operation { + try { + yield* operation; + return undefined; + } catch (error) { + return error; + } +} + +/** + * The infrastructure failure somewhere in this failure's causes. + * + * A denied operation is raised where `` acquired it and reaches the + * caller wrapped in whatever the document execution reported, so the assertion + * follows the chain the engine builds rather than the top of it. + */ +function fatalOf(error: unknown): unknown { + let current = error; + for (let depth = 0; depth < 16 && current instanceof Error; depth += 1) { + if (parseFilesFatal(current) !== undefined) { + return current; + } + current = current.cause; + } + return error; +} + +/** The current root pointer, as a second connection sees it. */ +function committedRoot(path: string): unknown { + let found: unknown; + tamper(path, (database) => { + found = database.prepare("SELECT current_root_id AS root FROM workspace_state").get()?.root; + }); + return found; +} + +/** The Workspace root the newest committed journal row is associated with. */ +function rootOfLastEvent(path: string): unknown { + let found: unknown; + tamper(path, (database) => { + found = database + .prepare( + "SELECT workspace_root_id AS root FROM journal_events ORDER BY sequence DESC LIMIT 1", + ) + .get()?.root; + }); + return found; +} + +/** Every host document-filesystem step this run performed. Must stay empty. */ +interface HostSpy { + readonly seen: HostFilesEvent[]; +} + +function* useHostSpy(): Operation { + const seen: HostFilesEvent[] = []; + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return "/nowhere-the-workflow-may-reach"; + }, + }, + { at: "min" }, + ); + yield* useHostFiles({ observe: (event) => seen.push(event) }); + return { seen }; +} + +interface Run { + readonly output: Json; + readonly host: HostSpy; +} + +/** + * Execute `source` as this run's root document, with the run's Workspace + * attached and a host provider installed outside it. + */ +function runDocument(database: WorkflowRunDatabase, source: string): Operation { + return scoped(function* () { + const host = yield* useHostSpy(); + const output = yield* withWorkflowWorkspace( + database, + scoped(function* () { + return yield* collect( + yield* execute({ ...inlineSource(source), stream: database.journal }), + ); + }), + ); + return { output, host }; + }); +} + +/** The same document again, replaying the journal the first execution wrote. */ +function replayDocument(database: WorkflowRunDatabase, source: string): Operation { + return runDocument(database, source); +} + +function* workspaceEvents(database: WorkflowRunDatabase): Operation { + const events = yield* database.journal.readAll(); + return events.filter( + (event) => event.type === "yield" && event.description.type === WORKSPACE_FILE, + ); +} + +/** + * The file effects this run recorded, as operation and outcome. + * + * What a document rendered is not evidence that a file effect happened: an + * element's own expansion is journaled too, so a provider that never recorded + * anything can still replay the text it produced. These rows are the provider's + * own history, which is what the durability claims are about. + */ +function* recordedFileEffects( + database: WorkflowRunDatabase, +): Operation> { + const events = yield* workspaceEvents(database); + return events.flatMap((event) => + event.type === "yield" ? [{ name: event.description.name, result: event.result }] : [], + ); +} + +function* workspaceText(database: WorkflowRunDatabase, path: string): Operation { + const read = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.readTextFile(path); + }); + if (!read.ok) { + throw read.error; + } + return read.value; +} + +/** + * A Workspace filesystem that refuses one write the way DOFS refuses one. + * + * The failure is raised through the adapter's own wrapping, so what reaches the + * provider is indistinguishable from a real `EACCES`: it selects a reason and + * carries nothing else. It exists because no ordinary DOFS condition stops a + * write between creating its parents and writing the file — a parent chain that + * can be created is a chain the file can then be written into — so the state a + * savepoint is there to discard cannot otherwise be produced. + */ +function refusingWrite( + target: string, +): (filesystem: DenoWorkspaceFilesystem) => DenoWorkspaceFilesystem { + return (filesystem) => ({ + ...filesystem, + *writeFile(path, content, mode) { + if (path === target) { + throwWorkspaceFilesystemFailure( + Object.assign(new Error("planted"), { name: "WorkspaceFsError", code: "EACCES" }), + ); + } + yield* filesystem.writeFile(path, content, mode); + }, + }); +} + +/** + * Every name a Workspace filesystem decorator has answered to, rebuilt here. + * + * A contextual Api composes by stable name across loaded copies, so an + * independently constructed descriptor of the same name *is* the second copy. + * Each handler records that it was consulted and then delegates, so a seam that + * still existed would show up as a name in `reached` rather than as a broken + * run. + */ +const SEAM_NAMES: readonly string[] = [ + "executablemd.workflow.deno.workspace.private.filesystem", + "executablemd.workflow.deno.workspace.private", + "executablemd.workflow.deno.workspace.effect.mutation", +]; + +interface SeamShape { + interpose(value: unknown): Operation; +} + +function* useImpostorSeams(reached: string[]): Operation { + for (const name of SEAM_NAMES) { + const impostor: Api = createApi(name, { + // deno-lint-ignore require-yield + *interpose(value: unknown): Operation { + return value; + }, + }); + yield* impostor.around({ + *interpose([value], next) { + reached.push(name); + return yield* next(value); + }, + }); + yield* createContext(name, undefined).set({ seized: true }); + } +} + +/** `` — the same impostors, installed from inside the document. */ +function useTamper(reached: string[]): Operation { + return registerComponents([ + { + name: "Tamper", + origin: "tier-wf", + props: { type: "object", properties: {}, additionalProperties: false }, + *fn() { + yield* useImpostorSeams(reached); + return ""; + }, + }, + ]); +} + +/** Whether one journal row is the recorded `operation` on `target`. */ +function namesEffect(record: unknown, operation: string, target: string): boolean { + if (typeof record !== "string") { + return false; + } + const parsed: unknown = JSON.parse(record); + if (typeof parsed !== "object" || parsed === null) { + return false; + } + const description = Reflect.get(parsed, "description"); + const name = + typeof description === "object" && description !== null + ? Reflect.get(description, "name") + : undefined; + return ( + Reflect.get(parsed, "type") === "yield" && + typeof name === "string" && + name.startsWith(`${operation}:`) && + name.endsWith(`:${target}`) + ); +} + +/** + * Replace what one recorded file effect settled to. + * + * Written through SQL rather than through the provider, because the point of + * these cases is a journal holding something the provider would never write. + */ +function plantOutcome(path: string, operation: string, target: string, value: Json): void { + tamper(path, (database) => { + let planted = 0; + for (const row of database.prepare("SELECT sequence, record FROM journal_events").all()) { + if (!namesEffect(row["record"], operation, target)) { + continue; + } + const record = JSON.parse(String(row["record"])); + record.result = { status: "ok", value }; + database + .prepare("UPDATE journal_events SET record = ? WHERE sequence = ?") + .run(`${JSON.stringify(record)}\n`, row["sequence"]); + planted += 1; + } + if (planted !== 1) { + throw new Error(`the journal records ${planted} ${operation} effects on ${target}`); + } + }); +} + +/** + * Take away the root's Close. + * + * A completed journal answers with its recorded root result without replaying + * anything, so a record planted in one of its effects is never read. Removing + * the Close is what makes the effects replay. + */ +function dropRootClose(path: string): void { + tamper(path, (database) => { + let dropped = 0; + for (const row of database.prepare("SELECT sequence, record FROM journal_events").all()) { + const parsed: unknown = JSON.parse(String(row["record"])); + if (typeof parsed !== "object" || parsed === null) { + continue; + } + if ( + Reflect.get(parsed, "type") !== "close" || + Reflect.get(parsed, "coroutineId") !== "root" + ) { + continue; + } + database.prepare("DELETE FROM journal_events WHERE sequence = ?").run(row["sequence"]); + dropped += 1; + } + if (dropped !== 1) { + throw new Error(`the journal records ${dropped} root closes`); + } + }); +} + +/** + * Drop the journal from the recorded effect on `target` onward. + * + * What is left replays up to that point and runs live after it, which is how a + * test observes what a document does *after* a replayed effect rather than only + * what that effect answers. + */ +function truncateFromEffect(path: string, operation: string, target: string): void { + tamper(path, (database) => { + const rows = database.prepare("SELECT sequence, record FROM journal_events").all(); + const found = rows.find((row) => namesEffect(row["record"], operation, target)); + if (found === undefined) { + throw new Error(`the journal records no ${operation} of ${target}`); + } + database.prepare("DELETE FROM journal_events WHERE sequence >= ?").run(found["sequence"]); + }); +} + +function* mutateWorkspace( + database: WorkflowRunDatabase, + body: (workspace: PrivateWorkspaceTransaction) => Operation, +): Operation { + const changed = yield* transactWorkspaceRoots(database, function* (workspace) { + yield* body(workspace); + const root = yield* workspace.capture(); + yield* workspace.publish(root.rootId); + }); + if (!changed.ok) { + throw changed.error; + } +} + +describe("WF workflow document filesystem", () => { + it("writes a file into the run's own Workspace and records one effect", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument( + database, + ["# Release", "", 'Prepared', ""].join("\n"), + ); + + expect(run.host.seen).toEqual([]); + expect(yield* workspaceText(database, "/notes/release.md")).toEqual("Prepared"); + + const events = yield* workspaceEvents(database); + expect(events).toHaveLength(1); + expect(events[0]?.type === "yield" && events[0].description.name).toContain("write:"); + }); + }); + + it("reads a file back through the same logical Workspace", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument( + database, + [ + '{"channel":"stable"}', + "", + '', + "", + "Read: {config}", + ].join("\n"), + ); + + expect(run.host.seen).toEqual([]); + expect(String(run.output)).toContain('Read: {"channel":"stable"}'); + }); + }); + + it("restores a read's recorded content when the frontier no longer holds it", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const source = [ + 'first', + "", + '', + "", + "Seen: {seen}", + ].join("\n"); + const first = yield* runDocument(database, source); + expect(String(first.output)).toContain("Seen: first"); + + // The provider recorded the read itself, not merely the element that + // asked for it: a read that answered from the frontier would leave one + // effect here instead of two. + const recorded = yield* recordedFileEffects(database); + expect(recorded.map((effect) => effect.name.split(":")[0])).toEqual(["write", "read"]); + expect(recorded[1]?.result).toEqual({ + status: "ok", + value: { kind: "content", content: "first" }, + }); + + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/seed.txt", "replaced"); + }); + expect(yield* workspaceText(database, "/seed.txt")).toEqual("replaced"); + + const replayed = yield* replayDocument(database, source); + expect(String(replayed.output)).toContain("Seen: first"); + expect(replayed.host.seen).toEqual([]); + expect(yield* recordedFileEffects(database)).toEqual(recorded); + }); + }); + + // WF6: the three refusals a document can act on, each decided before any + // effect exists. Nothing is recorded and nothing outside the run is asked, + // which is what "lexical" means here. + it("refuses an empty, absolute or escaping path without an effect or a host call", function* () { + const cases: Array<{ path: string; says: string }> = [ + { path: "", says: "path is empty" }, + { path: "/etc/passwd", says: "an absolute path is not accepted" }, + { path: "../escape.txt", says: "resolves outside the working directory" }, + ]; + + for (const refused of cases) { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + const before = committedRoot(path); + const run = yield* runDocument(database, `no`); + + expect(String(run.output)).toContain(refused.says); + expect(run.host.seen).toEqual([]); + expect(yield* workspaceEvents(database)).toEqual([]); + expect(committedRoot(path)).toEqual(before); + }); + } + }); + + // WF11: the search's document-facing shape, on the same contract the host + // provider answers on (HF3). A link is not a file, so it is neither a result + // nor a way into the tree it names. + it("searches regular files only, reporting no symbolic link and following none", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.mkdir("/docs", { recursive: true }); + yield* workspace.filesystem.writeFile("/docs/a.md", "a"); + yield* workspace.filesystem.mkdir("/hidden", { recursive: true }); + yield* workspace.filesystem.writeFile("/hidden/b.md", "b"); + yield* workspace.filesystem.symlink("/docs/a.md", "/link.md"); + // Named so that walking *through* it would produce a second, matching + // path for a file the walk already reaches by its own name. + yield* workspace.filesystem.symlink("/hidden", "/mirror.md"); + }); + + const run = yield* runDocument( + database, + ['', "", "Found: {found}"].join("\n"), + ); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "paths", paths: ["docs/a.md", "hidden/b.md"] }, + }); + // The file link is not a result, and the directory link is neither a + // result nor a second route to `b.md`. + expect(String(run.output)).not.toContain("link.md"); + expect(String(run.output)).not.toContain("mirror.md"); + }); + }); + + it("reports a missing file as missing rather than reaching the host for it", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument(database, ''); + + expect(String(run.output)).toContain("absent.txt"); + expect(run.host.seen).toEqual([]); + const events = yield* workspaceEvents(database); + expect(events).toHaveLength(1); + }); + }); + + it("searches the logical Workspace with ", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const run = yield* runDocument( + database, + [ + 'a', + 'b', + 'c', + "", + '', + "", + "Found: {found}", + ].join("\n"), + ); + + expect(run.host.seen).toEqual([]); + expect(String(run.output)).toContain("docs/a.md"); + expect(String(run.output)).toContain("docs/b.md"); + expect(String(run.output)).not.toContain("skip.txt"); + }); + }); + + it("commits the bytes, the current root and the filtered result together", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + yield* runDocument(database, 'committed'); + + // Counted through a second connection, so what it reports is what the + // transaction published rather than what this handle is holding. + expect(committedEventCount(path)).toBeGreaterThan(0); + expect(committedRoot(path)).toEqual(rootOfLastEvent(path)); + expect(yield* workspaceText(database, "/atomic.txt")).toEqual("committed"); + + const events = yield* workspaceEvents(database); + const written = events[0]; + expect(written?.type === "yield" && written.result).toEqual({ + status: "ok", + value: { kind: "written" }, + }); + }); + }); + + it("replays a create/delete/create history without consulting the current file", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const source = [ + 'one', + "", + '', + "", + "Seen: {seen}", + ].join("\n"); + + const first = yield* runDocument(database, source); + expect(String(first.output)).toContain("Seen: one"); + + // The history the replay has to survive, built through the seam a + // provider owns rather than through a public delete component: the file + // the document created is removed, and then a different file is created + // at the same path. + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.remove("/x.txt"); + }); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/x.txt", "three"); + }); + + const before = yield* database.journal.readAll(); + const recorded = yield* recordedFileEffects(database); + expect(recorded).toEqual([ + { name: recorded[0]?.name ?? "", result: { status: "ok", value: { kind: "written" } } }, + { + name: recorded[1]?.name ?? "", + result: { status: "ok", value: { kind: "content", content: "one" } }, + }, + ]); + const frontier = yield* workspaceText(database, "/x.txt"); + expect(frontier).toEqual("three"); + + const replayed = yield* replayDocument(database, source); + + expect(String(replayed.output)).toContain("Seen: one"); + expect(replayed.host.seen).toEqual([]); + // The write did not run again, so the frontier is still what the private + // history left there rather than the document's own content. + expect(yield* workspaceText(database, "/x.txt")).toEqual(frontier); + expect((yield* database.journal.readAll()).length).toEqual(before.length); + expect(yield* recordedFileEffects(database)).toEqual(recorded); + }); + }); + + it("refuses to publish a workflow file effect into a stream that is not the run's", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const before = yield* database.journal.readAll(); + + const failure = yield* raised( + scoped(function* () { + yield* useHostSpy(); + return yield* withWorkflowWorkspace( + database, + scoped(function* () { + return yield* collect( + yield* execute({ + ...inlineSource('no'), + stream: new InMemoryStream(), + }), + ); + }), + ); + }), + ); + + expect(failure).toBeInstanceOf(Error); + expect((yield* database.journal.readAll()).length).toEqual(before.length); + const present = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.readTextFile("/smuggled.txt"); + }); + expect(present.ok).toEqual(false); + }); + }); + + it("publishes a refusal as rolled back, leaving the Workspace as it was", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/blocked", "a file, not a directory"); + }); + const before = committedRoot(path); + + const run = yield* runDocument(database, 'no'); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded).toHaveLength(1); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "refused", phase: "transaction", reason: "not-directory" }, + }); + // Nothing the attempt created survives, so the root the effect published + // is the one it started from. + expect(committedRoot(path)).toEqual(before); + + const created = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.stat("/blocked/deep"); + }); + expect(created.ok).toEqual(false); + }); + }); + + // WF12: the savepoint around parent creation and the write together. The + // write fails after two directories exist, so what the assertion below sees + // is the rollback rather than an attempt that never started. + it("discards the parent directories a refused write already created", function* () { + const root = yield* useStorageRoot(); + yield* withStorage( + root, + function* () { + const database = yield* createRun(); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.writeFile("/kept.txt", "kept"); + }); + + const run = yield* runDocument( + database, + ['no', "", 'yes'].join( + "\n", + ), + ); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "refused", phase: "transaction", reason: "permission-denied" }, + }); + + // Both directories the attempt created are gone. + for (const created of ["/made", "/made/deep"]) { + const stat = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.stat(created); + }); + expect(stat.ok).toEqual(false); + } + // What the Workspace already held is what it still holds. + expect(yield* workspaceText(database, "/kept.txt")).toEqual("kept"); + // The savepoint took back the mutation rather than the transaction, so + // the next effect still commits. + expect(recorded[1]?.result).toEqual({ status: "ok", value: { kind: "written" } }); + expect(yield* workspaceText(database, "/after.txt")).toEqual("yes"); + }, + { decorateFilesystem: refusingWrite("/made/deep/x.txt") }, + ); + }); + + // WF14: the transaction filesystem is the provider's, decided where the + // provider was installed. The adversary here sits in the strongest position + // any composed code can occupy — a scope that encloses the whole document and + // was installed *after* the provider — and rebuilds, by name, every seam a + // filesystem decorator has answered to. A stable name is composition; this is + // what it means for authority not to travel through one. + it("keeps composed middleware away from the transaction filesystem", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const reached: string[] = []; + + const run = yield* scoped(function* () { + yield* useImpostorSeams(reached); + yield* useTamper(reached); + return yield* runDocument( + database, + ["", "", 'written'].join("\n"), + ); + }); + + expect(run.host.seen).toEqual([]); + // The write went to the run's own Workspace, untouched. + expect(yield* workspaceText(database, "/guarded.txt")).toEqual("written"); + const recorded = yield* recordedFileEffects(database); + expect(recorded.at(-1)?.result).toEqual({ status: "ok", value: { kind: "written" } }); + // Neither position reached the filesystem: not the enclosing scope, and + // not the component that installed the same names from inside the + // document. + expect(reached).toEqual([]); + }); + }); + + // WF13: durable history is parsed, not believed. A record carrying more than + // its variant carries, or a word the vocabulary does not hold, describes no + // outcome — and nothing it happens to hold is repeated back. + it("refuses a recorded outcome carrying extra or contradictory members", function* () { + const source = [ + 'first', + "", + '', + "", + '', + "", + "Seen: {seen}", + ].join("\n"); + + const cases: Array<{ operation: string; target: string; value: Json }> = [ + // content, carrying a refusal's members as well as its own + { + operation: "read", + target: "/seed.txt", + value: { kind: "content", content: "first", reason: "missing" }, + }, + // written, which carries nothing but its kind + { operation: "write", target: "/seed.txt", value: { kind: "written", content: "first" } }, + // paths, carrying content + { + operation: "glob", + target: "/", + value: { kind: "paths", paths: ["seed.txt"], content: "x" }, + }, + // refused, missing the reason it is refused for + { operation: "read", target: "/seed.txt", value: { kind: "refused", phase: "target" } }, + // refused, in a vocabulary this provider does not speak + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "target", reason: "unspeakable" }, + }, + // refused, with planted text riding along beside the vocabulary + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "target", reason: "missing", detail: "PLANTED-SECRET" }, + }, + // Every member, holding the wrong kind of value. + { operation: "read", target: "/seed.txt", value: { kind: "content", content: 7 } }, + { operation: "glob", target: "/", value: { kind: "paths", paths: "seed.txt" } }, + { operation: "glob", target: "/", value: { kind: "paths", paths: ["seed.txt", 7] } }, + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: 7, reason: "missing" }, + }, + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "target", reason: 7 }, + }, + // A word from the other operation's vocabulary is not this one's. + { + operation: "read", + target: "/seed.txt", + value: { kind: "refused", phase: "commit", reason: "missing" }, + }, + ]; + + for (const planted of cases) { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + yield* runDocument(database, source); + + dropRootClose(path); + plantOutcome(path, planted.operation, planted.target, planted.value); + const before = (yield* workspaceEvents(database)).length; + + const failure = yield* raised(replayDocument(database, source)); + + expect(failure).toBeInstanceOf(Error); + // Exactly the fixed provider invariant — not merely "something failed". + const fatal = fatalOf(failure); + expect(parseFilesFatal(fatal)).toEqual({ + type: FILES_FATAL, + kind: "invariant", + category: "protocol", + }); + // Cause-free: nothing the journal held is carried along underneath it. + expect(fatal instanceof Error ? fatal.cause : "not an error").toBeUndefined(); + expect(String(failure)).not.toContain("PLANTED-SECRET"); + // The failed run performed no file effect of its own — the history it + // could not read is the whole of what it has. + expect((yield* workspaceEvents(database)).length).toEqual(before); + }); + } + }); + + it("performs no later file effect once the history it replays is malformed", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const path = runPath(root, database.record.runId); + const source = [ + 'one', + "", + 'two', + ].join("\n"); + yield* runDocument(database, source); + + // The journal now replays the first write and runs everything from the + // second one live, and the first write's record describes no outcome. + truncateFromEffect(path, "write", "/second.txt"); + plantOutcome(path, "write", "/first.txt", { kind: "written", content: "one" }); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.remove("/second.txt"); + }); + const before = (yield* workspaceEvents(database)).length; + + const failure = yield* raised(replayDocument(database, source)); + + expect(parseFilesFatal(fatalOf(failure))).toEqual({ + type: FILES_FATAL, + kind: "invariant", + category: "protocol", + }); + expect((yield* workspaceEvents(database)).length).toEqual(before); + const second = yield* transactWorkspaceRoots(database, function* (workspace) { + return yield* workspace.filesystem.stat("/second.txt"); + }); + expect(second.ok).toEqual(false); + }); + }); + + it("refuses to replace a directory before it changes anything", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + yield* mutateWorkspace(database, function* (workspace) { + yield* workspace.filesystem.mkdir("/held", { recursive: true }); + yield* workspace.filesystem.writeFile("/held/inner.txt", "kept"); + }); + + const run = yield* runDocument(database, 'replacement'); + + expect(run.host.seen).toEqual([]); + const recorded = yield* recordedFileEffects(database); + expect(recorded[0]?.result).toEqual({ + status: "ok", + value: { kind: "refused", phase: "target", reason: "directory" }, + }); + expect(yield* workspaceText(database, "/held/inner.txt")).toEqual("kept"); + }); + }); + + it("denies a temporary directory instead of handing out a host one", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const failure = yield* raised( + runDocument( + database, + ["", 'x', ""].join("\n"), + ), + ); + + expect(failure).toBeInstanceOf(Error); + expect(parseFilesFatal(fatalOf(failure))).toEqual({ + type: FILES_FATAL, + kind: "operation-denied", + operation: "temporary-directory", + }); + }); + }); + + it("keeps an unrelated in-memory journal out of the run's storage", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const before = (yield* database.journal.readAll()).length; + yield* scoped(function* () { + yield* useHostSpy(); + yield* collect( + yield* execute({ ...inlineSource("# plain"), stream: new InMemoryStream() }), + ); + }); + expect((yield* database.journal.readAll()).length).toEqual(before); + }); + }); +}); diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index a1f35802..de99659f 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -138,6 +138,12 @@ const DENO_ONLY_TOOLING: RuntimeExclusion[] = [ "kills real Deno child processes with SIGKILL and reads the recovered node:sqlite WorkflowRun database they leave behind; the children run under the Deno executable and node:sqlite remains behind --experimental-sqlite on Node 22", issue: "https://github.com/taras/executable.md/issues/365", }, + { + path: "packages/workflow/tests/workspace-files.test.ts", + reason: + "drives and against a real node:sqlite WorkflowRun database through the Deno DOFS Workspace adapter; node:sqlite remains behind --experimental-sqlite on Node 22", + issue: "https://github.com/taras/executable.md/issues/366", + }, ]; /** diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 601e8c79..08c65e49 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -1601,6 +1601,10 @@ run but are absent from the diagnostic trace. | `packages/cli/src/{deno,node,bun,compiled}-service.ts` | runtime-named service adapters for token, environment and stdio behavior | | `packages/cli/src/{deno,node,bun,compiled}.ts` | Entrypoints — each installs matching `API.Env` and `API.Service` adapters, then calls `runXmd` | | `packages/workflow/src/service-denial.ts` | `useWorkflowServiceDenial()`, the tested non-delegating provider for future workflow start and resume scopes (#366) | +| `packages/workflow/src/deno/workspace/files.ts` | the transaction-bound `API.Files` provider — one durable Workspace effect per document read, write and search | +| `packages/workflow/src/deno/workspace/host.ts` | `withWorkflowWorkspace()` — the run's effect coordinator, logical cwd `/`, and Files provider installed together inside one execution | +| `packages/workflow/src/journal.ts` | the `workflow_run` record, canonical-record recognition, and the refusals that name differing fields without their values | +| `packages/workflow/src/run.ts` | `workflowInstallation()` / `retainedWorkflowInstallation()` — the `ExecutionInstallation` values a trusted host passes to `executeInstalled()`, each contributing a mandatory run-identity admission and the `prepare` hook that creates or restores the run inside the durable root | | `packages/cli/src/file-stream.ts` | `FileStream` — JSONL-backed `DurableStream` implementation | Dependencies: `@effectionx/scope-eval`, `@effectionx/timebox`, @@ -7750,11 +7754,19 @@ Defined in [Workflow runs](./workflow-spec.md). | WR6 | A moving base | A branch that moved cannot change the recorded pinned commit | | WR7 | Git fails | No run recorded, and the root document never expands | | WR8 | A later failure | A document that fails after the run was recorded does not erase it | -| WR9 | No workflow | An execution without `useWorkflow()` invokes no process at all | +| WR9 | No workflow | An execution without a workflow installation invokes no process at all | +| WR20 | One installation, two executions | A host that passes the same installation value to two concurrent executions gets two runs: each journal records its own, each document reads the run its own journal records, a sibling preparing or completing changes neither, repeated reads in one execution answer with the same frozen object, and a later ordinary execution sees no workflow run at all | | WR10 | No workflow, inside | `getWorkflowRun()` throws inside an execution that installed none | | WR11 | A malformed record | Refused, and the refusal never quotes what the journal held | | WR12 | A slow base | Resolving one run's base does not stall a sibling execution | | WR13/WR14 | Seeded journals | A completed and a truncated journal written by hand restore without any live run having happened | +| WR18 | Recorded base-resolution failure | A programmatic installation raises no evidence objection to the journal §6 describes: Git is not consulted, the root does not expand, and nothing is appended | +| WR19 | Replaying a failed preparation | A preparation that failed before the root import left the bound pre-root terminal, and the identical execution reads that failure back: no Git, no preparation, no document policy, no root import, no expansion and no append | +| WM1 | A non-delegating handler | A document handler that answers instead of delegating cannot fabricate a successful run: core refuses it, nothing expands, and the journal holds the canonical `workflow_run` record and one failed terminal rather than the handler's value | +| WM2 | A substitute result | A result returned after delegating replaces neither the document nor the recorded run | +| WM3 | Preparation precedes policy | The `workflow_run` record is already in the journal when the first public document handler runs | +| WM4 | A relayed installation | An installation handed to canonical core as an opaque record of closures composes exactly as the constructor's own value does | +| WM5 | Valid observation | A handler that observes and delegates still composes, and the run is unaffected | ### Tier WD — Workflow definitions and storage contracts @@ -7997,6 +8009,52 @@ Defined in [Workflow runs](./workflow-spec.md) §9.4 and §9.6–§9.7. | TX43 | One read across phases | Two valid recorded selections behind one accessor — Alpha then Beta — resume as Alpha: Alpha's section executes, Beta's never does, the source is read once, and the appended Close describes the Alpha execution | | TX38–TX41 | Totality, on the envelope | A result that refuses to be read, a value that refuses to be read, a settlement that refuses to be read, and a successful result with no value are each malformed rather than unrelated — the fixed cause-free diagnostic, no recorded terminal result reused, no planted text anywhere, nothing expanded and nothing appended, for the original failing selector and for a different selector that would otherwise succeed | | TX42 | Ordinary failed settlement | A root import recorded as failed for non-selection reasons is left alone by this protocol | +### Tier RR — Retained workflow-run installation + +Defined in [Workflow runs](./workflow-spec.md) §3.1. + +| # | Test | Verify | +|---|------|--------| +| RR1 | Exact record | The retained run is recorded verbatim; no identifier is allocated and Git is never consulted | +| RR2/RR3 | Restoration | A truncated and a completed journal each restore the retained run without recording it again | +| RR4 | A different run id | Refused as `StaleInputError` naming the field, never either value, and the root document does not expand | +| RR5 | A different base or pinned commit | Refused on the same terms, naming the fields that differ | +| RR6 | A malformed record | Refused rather than coerced, without quoting what the journal held | +| RR7 | An unusable installation | A missing run id, base or pinned commit is refused before any document executes | +| RR8 | A completed journal recording no run, or more than one | A terminal result whose journal holds no successful `workflow_run` record, holds one that failed, or holds two is refused before the recorded root result is handed back, and without appending anything | +| RR9 | A completed journal recording another run | A completed journal whose record names a different run, or holds a value that does not read as a run at all, is refused on the same terms as a truncated one | +| RR10 | Suppressed guard policy | A completed run-a journal resumed as run-b is refused beneath a `ReplayGuard` that answers check, admit or decide without delegating — nothing expands, nothing is emitted, nothing is appended | +| RR11 | A guard from another loaded copy | The same, beneath a suppressing handler installed through an independently constructed descriptor of the guard's stable name | +| RR12 | Valid replay under the same suppression | A completed journal that does agree still returns its recorded result with zero live execution | +| RR13 | Policy still composes | A public guard observes the admitted history, and one that rejects it still refuses | +| RR14 | Non-canonical records | A same-typed Yield under another name, under a child coroutine, or holding a value with an extra member establishes nothing, and none of what it held is quoted back | +| RR15 | Failed establishment in a truncated history | A recorded `workflow_run` failure fails closed rather than replaying its planted text for a run storage already describes | +| RR16 | What a refusal retains | Inspecting the whole error object — not only its message — finds no run id, base, pinned commit or planted description member | +| RR17 | Suppressed or reordered `Execution` policy | A handler that hands the durable run a different stream — the package's own descriptor or one built elsewhere, registered before or after the workflow installation — cannot make another run's journal replay | +| RR18 | One retained snapshot | A journal whose recorded value shifts between reads is settled once: identity admission, guard observation and replay are handed the same objects, and the second answer is never reached | +| RR20 | A hostile recorded value | A value whose `ownKeys`, `getOwnPropertyDescriptor` or getter refuses, and one whose classification refuses, each become the same fixed refusal — nothing reused, executed, emitted or appended, and the whole error object carries none of the planted text | +| RR21 | A duplicate canonical record | Two entries under the canonical run identity are refused in every settlement combination — successful+successful, successful+failed, failed+successful, failed+failed — because the record is written once and a second entry describes a second run however it ended | + +### Tier WF — The workflow document filesystem + +Defined in [Workflow runs](./workflow-spec.md) §10. + +| # | Test | Verify | +|---|------|--------| +| WF1 | Public routing | `` and `` reach the run's logical Workspace, and a host `API.Files` spy installed outside the run observes nothing for any read, write, refusal or search | +| WF2 | Atomic write | File bytes, the current-root pointer and one filtered Yield are all visible to a second connection together, and the newest journal row names the published root | +| WF3 | Recorded read | A read is its own durable effect whose recorded value is the content it read | +| WF4 | Historical read | A read restores its recorded content where the current frontier holds something else | +| WF5 | Create/delete/create | A history built through the provider-private mutation seam replays recorded results in order, performing no mutation and consulting no current file | +| WF6 | Lexical refusal | An empty, absolute or escaping path is refused without an effect being recorded and without a host call | +| WF7 | Rolled-back refusal | A documented refusal publishes a `rolled-back` outcome, leaves the current root unchanged and creates none of the parents the attempt would have needed | +| WF8 | Target refusal | Replacing a directory is refused as `unchanged` before anything is attempted | +| WF9 | Foreign journal | A file effect published into a stream that is not the run's is refused before mutation, leaving the journal and the Workspace as they were | +| WF10 | Denied temporary directory | `` receives the operation-denied infrastructure failure and no host directory | +| WF11 | The search's shape | Sorted, deduplicated, POSIX-relative regular files, on HF3's contract: neither a file symlink nor a directory symlink is a result, and a file reachable through a directory symlink is reported once, by its own path | +| WF12 | Discarded partial mutation | A write that refuses after creating two parent directories leaves neither behind, leaves what the Workspace already held untouched, records the sanitized refusal, and does not stop the next effect from committing | +| WF13 | Unreadable history | A recorded outcome carrying a member its variant does not have, a member of the wrong type, or a phase or reason the operation's vocabulary does not hold, is refused as exactly the cause-free `protocol` provider invariant — nothing the record held is repeated back, and no later file effect is performed | +| WF14 | The transaction filesystem is the provider's | Contextual middleware installed from inside the document — including descriptors rebuilt for every name a filesystem seam has used — neither observes nor replaces the filesystem a Workspace transaction hands its body | ### Tier EP — The execution protocol diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 50f6f9b3..bd4d8ad6 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -2,8 +2,9 @@ * **Status:** Current * **Scope:** `@executablemd/workflow` — associating a document execution with a - workflow run whose starting repository state is pinned once, and retaining - that run so another process can find it. + workflow run whose starting repository state is pinned once, retaining that + run so another process can find it, and giving that run's document its own + transactional filesystem. --- @@ -19,14 +20,16 @@ commit** the first time it is created. A branch that moves afterwards does not change what the run started from. ```ts -import { execute } from "@executablemd/core"; -import { useWorkflow, getWorkflowRun } from "@executablemd/workflow"; +import { executeInstalled } from "@executablemd/core/host"; +import { workflowInstallation, getWorkflowRun } from "@executablemd/workflow"; -yield* useWorkflow({ base: "main" }); -const execution = yield* execute({ path: "./workflow.md", stream }); +const execution = yield* executeInstalled( + { path: "./workflow.md", stream }, + [workflowInstallation({ base: "main" })], +); ``` -The package owns `WorkflowRun`, `useWorkflow()`, `getWorkflowRun()` and the Git +The package owns `WorkflowRun`, `workflowInstallation()`, `getWorkflowRun()` and the Git capability. It depends on `@executablemd/core`, `@executablemd/durable-streams` and `@executablemd/runtime`, whose contextual `exec()` and `cwd()` the Git provider invokes. Core never imports workflow or @@ -56,22 +59,162 @@ replay preserves the field values, never JavaScript object identity. It throws outside a document execution associated with a run, and exposes no journal, Git, workspace or continuation capability. -## 3. Where the run is installed +## 3. Where the run is prepared -`useWorkflow({ base })` installs ordinary middleware in the scope that owns one -document execution, and does nothing else. **Installing it creates no workflow -run**; executing a document under it does. +`workflowInstallation({ base })` returns an `ExecutionInstallation` — a value, +not an installation act. **Constructing it creates no workflow run**; executing +a document under it does. A trusted host passes it to `executeInstalled()`, and +the installation contributes exactly two things: -The value is installed in the scope that owns the document execution, so every -descendant of the expansion reads it, output emitted after the durable run still -sees it, and ordinary teardown takes it away. It is not readable before -`execute()` and not readable after that execution completes, even while the -installing scope is still alive. +- **an admission**, which canonical core captures before any installation, + middleware or document code exists and applies inside its own trusted journal + read, on the retained snapshot every later phase consumes; and +- **a `prepare` hook** — a `DurablePreparation`, the trusted durable + preparation canonical core invokes *inside the durable root*, after + retained-history admission and before any public `Execution.document` policy, + the root import and every authored effect. -Concurrent runs are isolated by scope ownership: each document execution and its -middleware installation share one child scope. A later document execution — -including one continuing the same workflow run — gets a new child scope and a -new installation. +No public middleware carries workflow-run authority. `Execution.execute`, +`Execution.document` and `ReplayGuard` handlers may observe, transform or refuse +what they are given; none of them can suppress the preparation, complete it, or +substitute a run for it. + +The run is readable through `getWorkflowRun()` for the lifetime of the document +execution: every descendant of the expansion reads it, output emitted after the +durable run still sees it, and ordinary teardown takes it away. It is not +readable before the execution and not readable after it completes, even while +the host scope is still alive. + +Concurrent runs are isolated by execution ownership: each execution gets its own +slot, so a host that passes the same installation value twice runs two +executions and neither sees the other's run. + +### 3.1 Installing a run that already exists + +A host that keeps runs in retained storage (§9) has decided what the run is +before anything executes: `create()` answered with the run id, and the +definition was established from a commit the host pinned. There is nothing left +for the execution to allocate or resolve, and a run id an execution invented +could not agree with the record storage already holds. + +```ts +yield* executeInstalled(options, [retainedWorkflowInstallation({ runId, base, pinnedCommit })]); +``` + +This installs the same middleware in the same place and records through the same +`workflow_run` durable operation. What differs is both ends of it. The live path +writes exactly the value it was given: no identifier is generated and +`Git.revParse()` is never called. And every journal state holds the record to +that value in full — run id, base and pinned commit — rather than to the base +alone. + +A journal recording a different run is refused as `StaleInputError`, naming the +fields that differ and never their values: a run id may be caller-selected and a +base is any revision expression, so both are external text on the same terms as +retained props. A value installed without a run id, a base or a pinned commit +identifies no run and is refused before any document executes. + +### 3.2 Where workflow-run identity is decided + +**Workflow-run identity is execution-owned, and it is not middleware of any +kind.** `ReplayGuard` is composable policy by design: a handler installed +further out may answer without delegating. So is `Execution`: a handler +registered at the same position may rebuild the options a later one produced, +stream included. Identity decided from either place is identity decided by +registration order — a completed journal reached under a suppressing guard, or +under a handler that swapped the stream back, would hand its recorded root +result to whichever run asked. + +An installation therefore contributes the *requirement* rather than a wrapper. +Its `admissions` say what a history has to satisfy; canonical core captures them +by value before any installation, middleware or document code exists, and +applies them inside the same trusted `readAll` that already holds a resumed run +to its recorded root selection. No wrapping site is added, nothing is reachable +through a context a document can rebind, and by the time any middleware runs the +read has already happened. It runs: + +- before any public `ReplayGuard` check, admit or decide; +- before a recorded root `Close` can be reused; +- before live execution, Workspace mutation, or any append. + +Public `ReplayGuard` handlers still observe the history this admits and may +still reject it. None of them can widen it. + +Creating the run is separated from admitting the history for the same reason. +The installation's `prepare` hook is a `DurablePreparation` canonical core +captures by value at the same moment, and invokes inside the durable root — +after admission and before any public `Execution.document` policy, the root +import and every authored effect. A handler that answers without delegating +cannot get in front of it: by the time any document policy runs, the +`workflow_run` record is already in the journal, and a fabricated result is +refused by core rather than published. + +The wrapper is a trusted wrapping site: it is installed before any document code +exists, delegates every append to the exact stream it was handed, and carries +that stream's journal-provenance witness onto itself without establishing one. + +What a history is held to depends on the installation, and they differ in one +thing beyond which fields must agree. + +`retainedWorkflowInstallation(run)` requires `runId`, `base` and `pinnedCommit` to match +exactly, and requires the record to be *there*: a host created the run before +anything executed, so a non-empty history carrying no successful record — none +at all, or only one that failed — is not this run's history. + +`workflowInstallation({ base })` requires a recorded run's base to match, and requires +nothing to be present. It allocates its run on first execution, and §6 records a +base that would not resolve as a failed effect; a history whose only record is +that failure is this run's own, and refusing it would refuse a journal this run +wrote. Both installations refuse a history carrying more than one successful +record, and both refuse one that cannot be read. + +A workflow definition may name an exact document target. That is part of what +the run is a run *of*, not part of what identifies it: run identity stays the +three members below, and a recorded value carrying a target as a fourth member +is refused as a value this version cannot account for. + +A record identifies a run only when it is all of these at once: + +- a Yield owned by the root coroutine; +- under the canonical effect type **and** the canonical effect name; +- successfully settled; +- holding a closed value of exactly `runId`, `base` and `pinnedCommit`, each a + string; and +- in agreement with the identity the installation supplied. + +An empty journal is the ordinary live start and is held to nothing. Otherwise a +history may carry **at most one** entry under the canonical run identity — the +record is written before the root document is imported, so a second entry +describes a second run, *however either of them settled*. Two successful +records, a successful one beside a failed one in either order, and two failed +ones are all refused. Under a retained installation the history must carry +exactly one, and it must have succeeded. + +Malformed, carrying an extra member, written under another name, written by a +child coroutine, and naming another run are refused under either installation, +whether the history is truncated or completed. A single missing or failed record +is refused under a retained installation and permitted under a programmatic one, +for the reason above. + +Reading a recorded value is total. A value whose enumeration, property +descriptors, getters or classification refuse describes no run, and becomes the +same fixed refusal every other unreadable record becomes — it never escapes +carrying its own text. + +The history admission reads is the retained history: every discriminator settled +once, before anything is decided, and the same objects every later phase +consumes. An event that refuses any member a decision rests on — its type, its +coroutine, either half of its description, its settlement, or a successful +settlement's value — is a history this run cannot describe and is refused, never +stepped past as unrelated. + +A refusal is a `StaleInputError` that names the fields that differ and never +their values, and the description it retains carries only the effect's type and +name — so nothing about the run, and nothing the journal held, is reachable on +the error object. + +`getWorkflowRun()`, the three journal states and the lifetime rules above are +otherwise identical under either installation. ## 4. The three journal states @@ -79,12 +222,12 @@ The journal decides which middleware does the work. | State | What runs | What happens | | --- | --- | --- | -| **live** — no record | `Execution.document` | allocates the run id, resolves the base through `Git.revParse()`, records one immutable value, and only then imports the root | -| **truncated** — record present, root not closed | the replay guard, then `Execution.document` | the guard restores the value; the durable operation still runs, so the journal cursor advances past its own entry | -| **completed** — root `Close` recorded | the replay guard only | the durable run returns the stored result without invoking the workflow, so the guard's check phase is the only place the run can be restored — or a different base refused | +| **live** — no record | the admission, then `prepare` | the admission finds nothing to hold the run to; preparation allocates the run id, resolves the base through `Git.revParse()`, records one immutable value, and only then is the root imported | +| **truncated** — record present, root not closed | the admission, then `prepare` | the admission restores the recorded value; preparation re-enters and its durable operation restores what it already recorded, so neither the identifier nor Git is reached again and the journal cursor still advances past its own entry | +| **completed** — root `Close` recorded | the admission only | canonical core returns the recorded result without entering the durable body, so preparation never runs and the admission is the only place the run is restored — or a disagreeing one refused | -The check phase runs before the recorded root result is returned. That ordering -is what lets a completed journal refuse a supplied base that disagrees with the +The admission runs before the recorded root result is returned. That ordering is +what lets a completed journal refuse a supplied base that disagrees with the recorded one, rather than handing back a result the caller did not ask for. Replay invokes neither run-id allocation nor `Git.revParse()`. The current value @@ -112,9 +255,16 @@ cannot be invoked, the working directory is not a Git repository, or the base does not resolve to a commit. Such a failure is journaled the way every durable effect's failure is — as a -recorded failed effect. Resuming the same journal therefore reproduces the -failure rather than retrying Git. No `WorkflowRun` value exists in either case, -which is what "records no workflow run" means. +recorded failed effect, and no `WorkflowRun` value exists, which is what +"records no workflow run" means. + +Preparation fails before the root import, so the durable root records the +**bound pre-root terminal** canonical core writes at that position: a terminal +carrying core's own binding to the exact root source and target it was about. +Resuming that journal reports the recorded failure — it does not retry Git, does +not re-enter preparation, does not run document policy, imports no root, expands +nothing, and appends nothing. The workflow installation raises no objection to +it (§3.2). ## 7. The Git capability @@ -578,9 +728,9 @@ retained root from that state without the process that wrote it. The provider-neutral coordinator receives the failure-activation continuation needed for this boundary. The default live coordinator ignores it and preserves ordinary success/failure publication. Replay bypasses coordination, and only an -explicit Workspace operation selects the Workspace coordinator. The Deno proof -operation is adapter-private: public filesystem effects and workflow -start/resume do not reach it. +explicit Workspace operation selects the Workspace coordinator. The +transaction-bound Files provider of §10 selects it for every document +filesystem effect; workflow start and resume do not reach it. The private restoration materializer loads a fully validated retained root and rebuilds directories, files, chunks, modes, mtimes, symbolic links and hardlink @@ -656,10 +806,73 @@ Version 1 reads and writes version 1. Unsupported versions are refused without the file being touched; partial version-1 initialization is corruption and is also left unchanged. -## 10. Intentionally excluded +## 10. The document filesystem of a run + +A host attaches one run's Workspace to a document execution with +`withWorkflowWorkspace(database, operation)` from +`@executablemd/workflow/deno`. It installs three things together, inside the +execution rather than at an entrypoint, so they answer ahead of the host adapter +`xmd run` installs: the run's Workspace effect coordinator, the logical working +directory `/`, and the transaction-bound `API.Files` provider. + +That composed helper is the whole of what the entrypoint publishes. The three +pieces are not installable separately, because the Files provider alone would +resolve a document's paths against whatever working directory the surrounding +host adapter answers with, and a host path resolved that way is what the run +then retains in the durable effects it replays from. + +The filesystem a Workspace transaction hands its body is the provider's, decided +where the provider is installed and held in its closure. It is reached through no +context and no contextual Api, so a document cannot observe it and cannot put +anything in front of it — a stable name is composition, and composition is not +where authority belongs. + +Paths are absolute POSIX paths inside the run's own filesystem. An authored path +is resolved by arithmetic on segments and handed to the run's DOFS filesystem; +no host path exists anywhere in it, so containment needs no stable-namespace +qualification. An empty path, an absolute path and a lexical escape are refused +with the vocabulary `API.Files` already has. + +`readTextFile`, `writeTextFile` and `globFiles` are durable effects. +`checkFilePath` is not: it is lexical admission, it performs no effect, and it +appends nothing, so the write repeats the same admission from the same authored +path. Each effect's description is derived from the current expansion, the +operation and the resolved logical path, so one authored element is the same +effect across replays while an element edited to name another file is a +different one. + +A search answers with sorted, deduplicated, POSIX-relative regular files, which +is the contract `API.Files` holds wherever it runs. A symbolic link is neither a +result nor a way into the tree it names, so a file reachable through a directory +link is reported once, under its own path. + +One effect is one effect transaction. The mutation, the resulting immutable root +and the filtered journal result commit together. An ordinary filesystem refusal +rolls its mutation savepoint back before its result is published, so a write +that created two parent directories and was then refused leaves neither behind, +the retained outcome describes a Workspace that is exactly what it was, the +write reports its target as rolled back, and the next effect still commits. What +crosses the boundary is a `FilesReason` selected from the shared vocabulary — no +DOFS message, errno payload, SQLite text or resolved path. Everything that is +not a documented refusal stays an infrastructure failure and fails the run. + +Replay restores the recorded outcome: it performs no mutation, opens no +transaction and consults no current state, which is what lets a read answer with +the bytes it read at the time and a create/delete/create history replay in +order. What it restores is parsed rather than believed. A record must carry its +variant's members and no others, each of the declared type, and a refusal's +phase and reason must both be words the operation's vocabulary holds. Anything +else describes no outcome, and becomes the one fixed cause-free provider +invariant — carrying nothing the record happened to hold, and performing no +further file effect. + +`temporaryDirectory` is refused with the existing operation-denied failure. A +run has no host directory to hand out, and reaching the caller's would be the +uncontained filesystem this boundary exists to prevent. + +## 11. Intentionally excluded Public `xmd workflow` lifecycle commands; lifecycle transition policy, executor -leases and stale-owner recovery; public Workspace mutation and filesystem -effects; public root selection, history checkpoints and forks; `` integration; -workflow-owned worktrees; and deterministic Git and GitHub effects. Retained -roots and private restoration do not expose any of those behaviors. +leases and stale-owner recovery; public root selection, history checkpoints and +forks; workflow-owned worktrees; and deterministic Git and GitHub effects. +Retained roots and private restoration do not expose any of those behaviors. diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 6b40503e..98280f95 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -852,9 +852,9 @@ and cannot return until the exact Result has been appended and recorded. Its supported filesystem calls use synchronous pinned DOFS primitives, leaving no asynchronous continuation after mutation teardown. It distinguishes a documented filesystem refusal from infrastructure failure and cancellation, and activates the durable -fail-stop fence for infrastructure failures. This foundation is not a public -filesystem effect: ``, `API.Files`, workflow start/resume and history -commands do not route to it in this slice. +fail-stop fence for infrastructure failures. ``, `` and every other +`API.Files` operation route to it through the transaction-bound Files provider. +Workflow start, resume and the history commands do not reach it in this slice. That boundary holds across processes as well as within one. A host killed between the mutation and the commit publishes nothing, and a process that @@ -875,7 +875,7 @@ delegated without changing the document language. | workflow-run and expansion identity | built by #289 / PR #341 | | retained run record and filtered journal | built by #291 | | caller-owned storage transaction | built by #291; Workspace mutations join it in #365 | -| provider-backed retained Workspace | defined here; unbuilt (#218) | +| provider-backed retained Workspace | document filesystem built by #366; repository, process and attachment capabilities unbuilt (#218) | | Repository, Worktree and transactional Git components | defined here; unbuilt | | lifecycle start/resume/status/history/fork/delete | defined here; unbuilt | | read-only Agent materialization | defined here; proof required |