diff --git a/.github/workflows/publish-packages.yml b/.github/workflows/publish-packages.yml index 8afc866c..c5297191 100644 --- a/.github/workflows/publish-packages.yml +++ b/.github/workflows/publish-packages.yml @@ -30,7 +30,7 @@ jobs: - name: Validate the manifests declare this version run: | VERSION="${{ steps.resolve.outputs.value }}" - for f in packages/durable-streams/deno.json packages/runtime/deno.json packages/core/deno.json packages/acp/deno.json packages/testing/deno.json packages/test-agent/deno.json packages/web/deno.json packages/cli/deno.json packages/code-review-agent/deno.json packages/workflow/deno.json; do + for f in packages/durable-streams/deno.json packages/runtime/deno.json packages/core/deno.json packages/acp/deno.json packages/testing/deno.json packages/test-agent/deno.json packages/web/deno.json packages/workflow/deno.json packages/cli/deno.json packages/code-review-agent/deno.json; do declared="$(jq -r .version "$f")" if [ "$declared" != "$VERSION" ]; then echo "::error::$f declares $declared, not $VERSION — the tag does not match the manifests" @@ -110,8 +110,15 @@ jobs: package: packages/web version: ${{ needs.version.outputs.value }} + workflow: + needs: [version, core, durable-streams, runtime] + uses: ./.github/workflows/publish-one.yml + with: + package: packages/workflow + version: ${{ needs.version.outputs.value }} + cli: - needs: [version, acp, core, durable-streams, runtime, test-agent, testing, web] + needs: [version, acp, core, durable-streams, runtime, test-agent, testing, web, workflow] uses: ./.github/workflows/publish-one.yml with: package: packages/cli @@ -124,13 +131,6 @@ jobs: package: packages/code-review-agent version: ${{ needs.version.outputs.value }} - workflow: - needs: [version, core, durable-streams, runtime] - uses: ./.github/workflows/publish-one.yml - with: - package: packages/workflow - version: ${{ needs.version.outputs.value }} - jsr: needs: [version] runs-on: ubuntu-latest diff --git a/README.md b/README.md index 6bea43a2..c8d70dfd 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,58 @@ Useful flags: - `--verbose`, `-V` - print durable journal entries to stderr while running. - `--component-dir` - add component search directories. Defaults to `components` and `.`. +## Run a workflow + +`xmd run` executes against the directory you are in and promises nothing +afterwards. `xmd workflow` executes against a **run**: one retained Workspace and +one filtered journal, in a database that outlives the process, so an interrupted +procedure resumes from where it stopped instead of starting again. + +```bash +xmd workflow start flows/prepare-release.md +xmd workflow start --id=release-1.4 --props-channel=stable flows/prepare-release.md +xmd workflow resume release-1.4 +``` + +`start` names a document; `resume` names a run. A document path locates a +definition and never selects a previous run, so starting the same document twice +without `--id` creates two runs. Reusing an `--id` addresses the same run when +the definition, base and normalized properties all agree, and is refused when +any of them differ. `resume` takes no document and no properties: it uses the +ones its run retained. + +What a run is, is a Git object: the repository containing the document, the +commit `HEAD` resolves to, and the document's path inside it. **The committed +document runs**, so uncommitted edits in your working tree do not change what a +run is a run of, and a resume months later loads the same object rather than +whatever the file says now. + +Inside a run, `` and `` name entries in the run's own logical +filesystem rather than yours. Each read, write and search is one durable effect: +the mutation, the Workspace root it produces and the journal result commit +together, so a crash leaves all three or none, and a resume restores what was +recorded instead of doing it again. Operations a run does not have — a temporary +directory, a native service — fail explicitly rather than reaching your machine. + +Identity and outcome go to standard error, so piping stdout still gives you the +document: + +```text +workflow run: release-1.4 +workflow status: completed +``` + +Only a completed run exits `0`. Failed exits `1`, suspended `2`, cancelled `3` +and interrupted `130`, so a script cannot mistake an incomplete workflow for a +finished one. + +Runs live under `~/.xmd/runs`; set `XMD_WORKFLOW_RUNS` to an absolute directory +to keep them somewhere else. `xmd workflow` is available through the Deno +entrypoint and the compiled binary; under Node and Bun the command exists and +refuses before creating anything. + +Status, list, history, cancel and fork are designed but not yet shipped. + ## Coding agents Run ACP-compatible coding agents directly from a document with ``, diff --git a/architecture.md b/architecture.md index 765d7906..3fb20ed7 100644 --- a/architecture.md +++ b/architecture.md @@ -154,9 +154,8 @@ The `@executablemd/workflow` package owns `WorkflowRun`, and the Git capability. It depends on `@executablemd/core`, `@executablemd/durable-streams` and `@executablemd/runtime`, whose contextual `exec()` and `cwd()` the Git provider invokes; core never imports workflow or -Git. The future CLI lifecycle is `xmd workflow start` and `xmd workflow resume`; -there is no workflow CLI execution branch yet. The durable lookup that resume -will require is the run storage below. Ordinary `xmd run` remains unchanged. +Git. `xmd workflow start` and `xmd workflow resume` are the CLI lifecycle, and +they resume through the run storage below. Ordinary `xmd run` remains unchanged. ## Workflow run storage @@ -352,6 +351,68 @@ describe what failed without repeating retained props or journal payloads — including their member *names*, which can carry a credential as readily as a member value can. +## The workflow lifecycle + +`xmd workflow start [--id=] [--props-*=…] ` and +`xmd workflow resume ` are the two commands. Both run in the foreground, +stream the document's own output to standard output, and report identity and +outcome on standard error as two stable lines — `workflow run: ` once +the run has been created or found, and `workflow status: ` once the +execution settles. Only a completed run exits zero: failed exits 1, suspended 2, +cancelled 3 and interrupted 130, so shell automation cannot mistake an +incomplete workflow for a finished one. A request the command refuses — bad +grammar, a missing run, an incompatible reuse, damaged storage, an unsupported +host — exits 1. + +`start` establishes an immutable definition from Git rather than identifying a +working-tree file. It locates the repository containing the supplied path, +resolves `HEAD^{commit}` once because the command has no base option, reads the +repository's object format, and stores version 1 of the descriptor with that +format, the lowercase commit ID and the normalized repository-relative POSIX +path. **The bytes that execute are the ones that commit holds**, so a working +tree with uncommitted edits runs the committed document. Where the repository is +checked out is retrieval metadata: replaceable, credential-free, excluded from +identity, and reauthorized before it is used again. `resume` loads exactly the +retained object through that locator and never substitutes the current `HEAD` or +a same-named working-tree file. A missing object, missing or unreadable +retrieval metadata, a path outside the repository, or a root that is not +Markdown fails explicitly; none of them creates a replacement run or an empty +definition. A workflow definition is one immutable object, so the component +search path is empty and a repository component fails to resolve rather than +resolving to content beside the definition in a mutable checkout. + +Every actual execution opens or creates the run's database, begins a +document-execution record, installs the exact retained WorkflowRun, installs the +service denial before the root is imported, and — for a live or partial +execution — installs the logical working directory `/`, the transaction-bound +Files provider and that database's Workspace effect coordinator. It executes +against the database's own journal with the retained props and secret detection, +then finishes the execution record and publishes the run's status. A completed +run still replays, so its retained output and result are emitted, but it +attaches no Workspace provider or coordinator and performs no filesystem +mutation. + +A failure retains `failed` and uses a journal stop reason when a retained event +identifies it, and a categorical host code otherwise; no exception text is +retained beside the journal that filtered it. Graceful foreground interruption +finishes the execution `interrupted` and exits 130. `suspended` is resumable; +`failed` and `cancelled` are refused by `resume`, and `completed` replays under +either command. + +The initial host is Deno-local. The Deno entrypoints — source and the compiled +binary — install the local run store, beneath `~/.xmd/runs` unless +`XMD_WORKFLOW_RUNS` names another absolute directory. Node and Bun expose the +same grammar and refuse before creating or executing anything. The shared CLI +module imports no SQLite, no DOFS and no runtime detection: it asks a host +adapter to open storage and to attach a run's Workspace, and the entrypoints +decide which adapter exists. + +Durable ownership and concurrent-executor enforcement belong to #367. Until it +lands, a run left `running` because its host disappeared is treated as an +orphaned interrupted execution by the next resume, which closes that unfinished +execution record as `interrupted` before beginning its own. Nothing here claims +concurrent resume is safe. + ## Workflow Workspace The command selects the environment; the document describes the procedure. @@ -480,8 +541,8 @@ commit form one boundary. The Workspace filesystem uses the pinned synchronous D entry points for its string and byte-array surface, so cancellation leaves no eager promise or stream pull able to reach the connection after transaction authority ends. The transaction-bound Files provider selects that operation for -every document filesystem read, write and search; workflow lifecycle commands do -not select it yet. +every document filesystem read, write and search, and the workflow lifecycle +commands install that provider. An external provider cannot join that transaction. Prompt, Git push and pull request effects derive a stable identity from the run and expansion, ask the @@ -557,8 +618,8 @@ collection is not in the production closure and is never invoked. The provider exposes no public history selection or fork operation at this layer. Its coordinator combines one mutation, immutable-root publication and one filtered journal result atomically, and the transaction-bound Files provider is what -routes a document's `` and `` to it. Workflow start and resume do -not reach it yet. +routes a document's `` and `` to it. `xmd workflow start` and +`xmd workflow resume` install that provider around each execution. The coordinator treats only errors produced through its private filesystem adapter's documented path and mutation refusals as journalable operation @@ -1034,12 +1095,12 @@ consumed. If collision handling terminates immediately, receives no terminal event; restoring the compatible definition can still replay it. -#390 provides and tests the non-delegating `useWorkflowServiceDenial()` provider. -#366 will install it in the future `xmd workflow start` and `xmd workflow resume` -scopes. No workflow CLI execution branch exists yet. The provider prevents a -workflow from reaching an inherited host adapter, because a run-owned durable -service requires stable identity and reconciliation rather than an -execution-owned live process. +`xmd workflow start` and `xmd workflow resume` install the non-delegating +`useWorkflowServiceDenial()` provider inside each execution scope, before the +root document is imported — in the same place `xmd run` installs its host +service adapter. The provider prevents a workflow from reaching an inherited +host adapter, because a run-owned durable service requires stable identity and +reconciliation rather than an execution-owned live process. ## State ownership @@ -1412,7 +1473,7 @@ Status is measured against main. | `workflowInstallation()` / `getWorkflowRun()` | associates one document execution with a workflow run, through an `ExecutionInstallation` the trusted host passes to `executeInstalled()` | built on the #366 stack | | `retainedWorkflowInstallation()` | associates one document execution with a run storage already created, requiring exact journal agreement | built on the #366 stack | | `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main | -| workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; public workflow execution is unbuilt | +| workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; the CLI lifecycle reaches it on the #366 stack | | caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in | built on main | | live durable-operation coordinator | explicitly coordinates structured live execution with existing Yield publication while leaving replay and callback effects unchanged | built on the #365 stack | | Workspace coordination API | fails closed by default; replaceable context routes only a one-use provider selection, while the selected provider directly invokes an execution-owned credentialed capability for execution, publication and failure activation | built on the #365 stack; the Deno provider installs an adapter-private atomic handler | @@ -1421,16 +1482,16 @@ Status is measured against main. | `Config` run deadline / exec default / Fetch default | three independently owned contextual timeouts, absent unless configured, each read by exactly one consumer | built on main | | `API.Files` | routes every document filesystem operation to the installed provider, with no host default and structural failure data | built on the #227 stack | | host Files provider / `useHostFiles()` | resolves document paths in the caller's filesystem, containing them while the host namespace is stable; installed by all four CLI entrypoints | built on the #227 stack | -| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | built on the #366 stack; CLI reachability remains unbuilt | +| transaction-bound Files provider | resolves document paths in the run-owned logical Workspace inside the caller-owned transaction | built on the #366 stack | | `service=` | publishes the attachment's endpoint into the live binding overlay for its invocation | built on main | | `ephemeral eval` | reconstructs live middleware and bindings without a journal entry | built on main | -| `useWorkflowServiceDenial()` | provides and tests a non-delegating workflow service denial provider; #366 will install it in future start and resume scopes | built on main; no workflow CLI execution branch exists yet | -| `xmd workflow start` / `xmd workflow resume` | starts or resumes a workflow run from the CLI | defined in `specs/workflow-workspace-spec.md`, unbuilt; the lookup it resumes through is built | -| implicit workflow Workspace | retains provider-neutral filesystem, repository and attachment state by run ID | defined in `specs/workflow-workspace-spec.md`, unbuilt (#218) | +| `useWorkflowServiceDenial()` | provides a non-delegating workflow service denial provider, installed inside every start and resume execution scope | built on the #366 stack | +| `xmd workflow start` / `xmd workflow resume` | starts or resumes a workflow run from the CLI, under the Deno entrypoints only | built on the #366 stack; status, list, history, cancel, fork and delete are unbuilt | +| implicit workflow Workspace | retains provider-neutral filesystem, repository and attachment state by run ID | document filesystem built on the #366 stack; repository, process and attachment capabilities unbuilt (#218) | | Repository / Worktree / transactional Git effects | compose named checkouts and publish local mutations with their journal result | defined in `specs/workflow-workspace-spec.md`, unbuilt | | workflow inspection and history fork | reads status/history without advancing a run and creates a new run from a checkpoint | defined in `specs/workflow-workspace-spec.md`, unbuilt | | read-only workflow Agent / generated XMD | lets an Agent inspect a derived view and propose constrained executable changes | defined in `specs/workflow-workspace-spec.md`, unbuilt | -| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates one Workspace mutation with its filtered Yield | built on the #365 stack; public document filesystem effects route to it on the #366 stack, and workflow lifecycle reachability is unbuilt | +| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates one Workspace mutation with its filtered Yield | built on the #365 stack; public document filesystem effects and the CLI lifecycle route to it on the #366 stack | | scoped Worker Shell | executes `just-bash` through the Workspace adapter inside a Deno Worker | containment and effect-transaction POCs complete (#351, #357); production integration unbuilt | | `` | retry a region until it completes | defined, unbuilt | | suspension effect | suspend durably | defined, unbuilt | diff --git a/bun.lock b/bun.lock index 69f14894..794de04a 100644 --- a/bun.lock +++ b/bun.lock @@ -74,6 +74,7 @@ "@executablemd/test-agent": "workspace:*", "@executablemd/testing": "workspace:*", "@executablemd/web": "workspace:*", + "@executablemd/workflow": "workspace:*", "@standard-schema/spec": "^1.0.0", "configliere": "^0.4.0", "effection": "4.1.0", @@ -211,6 +212,7 @@ "dependencies": { "@effectionx/context-api": "0.6.0", "@effectionx/fs": "0.3.0", + "@effectionx/process": "0.8.1", "@executablemd/core": "workspace:*", "@executablemd/durable-streams": "workspace:*", "@executablemd/runtime": "workspace:*", diff --git a/packages/cli/package.json b/packages/cli/package.json index af34a288..fe1954f5 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -19,6 +19,7 @@ "@executablemd/test-agent": "workspace:*", "@executablemd/testing": "workspace:*", "@executablemd/web": "workspace:*", + "@executablemd/workflow": "workspace:*", "@standard-schema/spec": "^1.0.0", "configliere": "^0.4.0", "effection": "4.1.0", diff --git a/packages/cli/src/bun.ts b/packages/cli/src/bun.ts index 6f8ba15f..3a76f161 100644 --- a/packages/cli/src/bun.ts +++ b/packages/cli/src/bun.ts @@ -11,6 +11,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileDataUri } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { unsupportedWorkflowHost } from "./workflow.ts"; import { useBunService } from "./bun-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -35,5 +36,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useBunService); + yield* runXmd(args, useBunService, unsupportedWorkflowHost); }); diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 809d8f28..f8d2b232 100755 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -5,9 +5,12 @@ * xmd run [options] * xmd [options] (run is the default command) * xmd targets + * xmd workflow start [options] + * xmd workflow resume * * A document reference is a path, optionally followed by `#` and one target - * selector naming a section of the document (spec §5.4). + * selector naming a section of the document (spec §5.4). `workflow start` takes + * a plain path: a definition descriptor cannot record a target yet. * * Examples: * xmd run packages/core/examples/hello-world.md @@ -15,6 +18,8 @@ * xmd run packages/core/examples/hello-world.md --journal events.jsonl * xmd targets README.md * xmd run README.md#Release/Publish + * xmd workflow start --id=release-1.4 flows/prepare-release.md + * xmd workflow resume release-1.4 */ import { @@ -55,10 +60,13 @@ import { installAgentComponents, installPermissionMode, registerAgentProvider, + retainedSource, rootSourcePath, useNormalizedOutput, useTerminalOutput, } from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import type { ExecutionInstallation } from "@executablemd/core/host"; import type { DocumentInfo, FileRootDocument, RootDocumentSource } from "@executablemd/core"; import { env as readEnv } from "@executablemd/runtime"; import { createAcpxProvider, DEFAULT_AGENT_NAME } from "@executablemd/acp"; @@ -86,6 +94,17 @@ import type { Binding, Extraction } from "./props.ts"; import { componentSearchPath, resolveTestTarget } from "./test-target.ts"; import { EVAL_ALIAS, EVAL_OPTION, evalGrammarError, readEvalFlags } from "./eval-source.ts"; import type { EvalFlags } from "./eval-source.ts"; +import { + parseWorkflowRequest, + runWorkflow, + UNSUPPORTED_WORKFLOW_HOST, + unsupportedWorkflowHost, + workflowConfig, +} from "./workflow.ts"; +import type { HostWorkflowInstaller, WorkflowHost, WorkflowStart } from "./workflow.ts"; +import { establishDefinition } from "./workflow-definition.ts"; +import type { EstablishedDefinition } from "./workflow-definition.ts"; +import { useWorkflowServiceDenial } from "@executablemd/workflow"; import denoJson from "../deno.json" with { type: "json" }; const SECRET_DETECTION_OPTION = "--secret-detection"; @@ -240,6 +259,7 @@ const xmd = program({ test: testConfig, targets: targetsConfig, "test-agent": testAgentConfig, + workflow: workflowConfig, }, { default: "run" }, ), @@ -475,12 +495,29 @@ interface DocumentConfig { raw: boolean; /** Whether this document's durable events are scanned before they persist. */ secretDetection: boolean; + /** + * The journal this execution reads and appends, when the caller owns one. + * + * A workflow run does: its journal is the run's retained history, and + * replacing it with a fresh stream would make every execution a first one. + * `xmd run` supplies none and gets the empty stream below. + */ + stream?: DurableStream; } interface DocumentMode { testing: boolean; agent?: AgentFlags; props?: Record; + /** + * What a trusted host attaches to this one execution. + * + * Values passed straight to `executeInstalled()`, so canonical core captures + * their admissions and preparations before any installation, middleware or + * document code exists. `xmd run` and `xmd test` attach none, and an empty + * list is exactly what `execute()` itself does. + */ + installations?: readonly ExecutionInstallation[]; } export type HostServiceInstaller = () => Operation; @@ -500,11 +537,14 @@ function* runDocument( ): Operation> { const { root, componentDir, verbose, journal, raw, secretDetection } = config; - // Every CLI invocation starts from an empty stream. --journal writes - // current-run diagnostics only; existing traces are never loaded. + // Every CLI invocation starts from an empty stream unless the caller owns + // one. --journal writes current-run diagnostics only; existing traces are + // never loaded. let stream: DurableStream; - if (journal) { + if (config.stream) { + stream = config.stream; + } else if (journal) { yield* createJournalFile(journal); stream = new FileStream(journal); } else { @@ -587,13 +627,20 @@ function* runDocument( // the provider for a service. yield* installService(); - const execution = yield* execute({ - ...root, - stream, - props: mode.props, - componentDirs: componentDir, - secretDetection, - }); + // One authoritative execution, and only one. What a host attaches travels as + // values canonical core captures before anything else exists — never as a + // second call, and never as middleware that could be reordered around this + // one. + const execution = yield* executeInstalled( + { + ...root, + stream, + props: mode.props, + componentDirs: componentDir, + secretDetection, + }, + mode.installations ?? [], + ); // Consume the output stream with forEach. // A value root reserves stdout for its result: its rendered body is @@ -1003,12 +1050,29 @@ function readPatternFlags(args: string[]): PatternFlags { interface PropsPhase { /** argv with document-derived tokens removed. */ args: string[]; + /** + * The subcommand and target `xmd workflow` resolved, when it is the command. + * + * Carried rather than re-parsed downstream: `args` is the head, so a + * positional written after `--` is not in it, and asking the parser again + * would lose exactly the token the separator was there to protect. + */ + workflow?: { action?: string; target?: string }; root?: RootDocumentSource; bindings: Binding[]; extraction?: Extraction; propsSchema?: unknown; declared?: string[]; error?: string; + /** + * The immutable definition a `workflow start` established, when it did. + * + * Established here rather than later because the props a run is created with + * are the ones the *pinned* document declares: reading the working tree to + * build the bindings and then executing the commit would let help and parsing + * describe a document that is not the one running. + */ + established?: EstablishedDefinition; } /** @@ -1019,7 +1083,15 @@ interface PropsPhase { * of argv, so it needs no parse at all. */ function* preparePropsPhase(args: string[], evalFlags: EvalFlags): Operation { - const provisional = xmd.parse({ args }); + // `xmd workflow` reads its options from the head and its remaining positionals + // from the tail. The parser only ever sees the head, so a dash-leading token + // after `--` is never offered to it as an option; the grammar check below + // still sees the argv that had the separator, because where options stopped + // is what decides whether a later token is a third positional. + const workflow = namesWorkflow(args); + const separated = separateArgs(args); + const parsed = workflow ? separated.head : args; + const provisional = xmd.parse({ args: parsed }); // `program` short-circuits on `--version` and leaves no configuration // behind, so there is nothing to inspect. const selected = provisional.ok ? provisional.value.config : undefined; @@ -1028,6 +1100,15 @@ function* preparePropsPhase(args: string[], evalFlags: EvalFlags): Operation { + const workflow = { action: config.action, target: config.target }; + if (inlineDocument !== undefined) { + return { + args: rawArgs, + bindings: [], + workflow, + error: `unrecognized option for xmd workflow: ${EVAL_OPTION} — inline documents are exclusive to xmd run`, + }; + } + + // Read before anything is stripped: `--` ends option parsing, so a token + // after it is positional however it is spelled, and the count has to be taken + // while the separator is still there to say where options stopped. + const extra = extraWorkflowArgument(rawArgs); + if (extra !== undefined) { + return { + args: rawArgs, + bindings: [], + workflow, + error: + `unrecognized argument for xmd workflow: ${extra} — start names one definition and ` + + "resume names one run", + }; + } + + const stray = findPropsFlag(args); + if (config.action !== "start") { + if (stray) { + return { + args, + bindings: [], + workflow, + error: + `unrecognized option for xmd workflow ${config.action ?? "resume"}: ${stray} — a resume ` + + "runs the props its run retained", + }; + } + return { args, bindings: [], workflow }; + } + + if (config.target === undefined || config.target === "") { + return { args, bindings: [], workflow }; + } + + const established = yield* establishDefinition(config.target); + if (!established.ok) { + return { args, bindings: [], workflow, error: established.error.message }; + } + + const root = retainedSource( + established.value.definition.rootDocumentPath, + established.value.source, + ); + try { + const document = yield* inspectDocument(root); + const bindings = buildBindings(document.props); + const extraction = extractPropsArgs(args, bindings); + return { + args: extraction.rest, + workflow, + root, + bindings, + extraction, + propsSchema: document.props, + declared: declaredProperties(document.props), + established: established.value, + }; + } catch (error) { + return { args, bindings: [], workflow, root, error: describeError(error) }; + } +} + +/** + * Whether these arguments select the `workflow` command. + * + * Read from argv rather than from a parse, because the answer is needed before + * the props phase and the props phase is what makes a parse meaningful. + * Whatever the command turns out to be, only `workflow` names it first. + */ +function namesWorkflow(args: string[]): boolean { + return args[0] === "workflow"; +} + +/** + * A third positional argument to `xmd workflow`, when one was written. + * + * The parser stops at the first token it does not define rather than rejecting + * it, so `xmd workflow resume ` would otherwise run the resume + * and silently ignore the document — exactly the confusion the lifecycle rule + * exists to prevent, since a document never selects a run. + * + * Read from the argv the props phase already stripped, so a generated property + * value is not mistaken for an argument. `--id` is the one option that takes a + * separated value, and `--` ends option parsing: every token after it is + * positional, including one that begins with `-`. + */ +function extraWorkflowArgument(args: string[]): string | undefined { + const start = args.indexOf("workflow"); + if (start === -1) { + return undefined; + } + let positionals = 0; + let skip = false; + let parsingOptions = true; + for (const arg of args.slice(start + 1)) { + if (parsingOptions && !skip && arg === "--") { + // The end of *option* parsing, and nothing more. What follows is + // positional however it is spelled, so a third argument is still a third + // argument — writing it after `--` used to end the check instead of the + // options, which let it through to storage. + parsingOptions = false; + continue; + } + if (skip) { + skip = false; + continue; + } + if (parsingOptions && arg.startsWith("-")) { + skip = arg === "--id"; + continue; + } + positionals += 1; + if (positionals > 2) { + return arg; + } + } + return undefined; +} + +/** + * An argv split at its end-of-options separator. + * + * The tail is carried rather than folded back in. Dropping the separator and + * rejoining would hand a dash-leading positional — `-run-id`, `-definition.md` + * — back to a parser that reads a leading dash as an option, which is exactly + * what `--` was written to prevent. + */ +interface SeparatedArgs { + /** Everything before `--`: the options, and any positionals written early. */ + head: string[]; + /** Everything after it, each token positional however it is spelled. */ + tail: string[]; +} + +function separateArgs(args: string[]): SeparatedArgs { + const at = args.indexOf("--"); + return at === -1 + ? { head: args, tail: [] } + : { head: args.slice(0, at), tail: args.slice(at + 1) }; +} + +/** + * The subcommand and target one `xmd workflow` invocation names. + * + * The parser classifies what it can — everything before `--` — and the tail + * supplies the rest in order. A token's spelling decides nothing here: after the + * separator it is positional because of where it is. + */ +function workflowPositionals( + config: { action?: string; target?: string }, + tail: string[], +): { action?: string; target?: string } { + const named = [config.action, config.target].filter((value) => value !== undefined); + const [action, target] = [...named, ...tail]; + return { action, target }; +} + +const COMMAND_NAMES = ["run", "test", "targets", "test-agent", "workflow"]; /** * What a caller has to know to write a filename that contains reference @@ -1230,11 +1492,16 @@ function* resolveRunProps( * deadline — document inspection, target and props preparation, provider * installation, execution and output consumption all included. Nothing here * reads an option the caller has not already had validated. + * + * `workflowHost` is present only for `xmd workflow`, and only on a host that + * supports it: every other invocation is handed nothing, which is what keeps a + * run store from being inherited by omission. */ function* dispatch( evalFlags: EvalFlags, helpRequest: { requested: boolean; args: string[] }, installService: HostServiceInstaller, + workflowHost: WorkflowHost | undefined, ): Operation { const propsPhase = yield* preparePropsPhase(helpRequest.args, evalFlags); @@ -1386,10 +1653,80 @@ function* dispatch( case "test-agent": yield* runTestAgentWorker({ connect: command.config.connect }); break; + case "workflow": { + // The parser saw only the head, so the positionals the separator carried + // come from the props phase rather than from a second parse of an argv + // they are not in. + const config = { ...command.config, ...propsPhase.workflow }; + const agentFlag = findAgentOnlyFlag(evalFlags.rest); + if (agentFlag) { + console.error( + `unrecognized option for xmd workflow: ${agentFlag} — agent options are exclusive to xmd run`, + ); + yield* exit(1); + break; + } + if (workflowHost === undefined) { + console.error(UNSUPPORTED_WORKFLOW_HOST); + yield* exit(1); + break; + } + const request = parseWorkflowRequest(config); + if (!request.ok) { + console.error(request.error.message); + yield* exit(1); + break; + } + const props = yield* resolveRunProps(propsPhase); + if (props.error) { + console.error(props.error); + yield* exit(1); + break; + } + const start: WorkflowStart | undefined = + propsPhase.established === undefined + ? undefined + : { established: propsPhase.established, props: props.value ?? {} }; + announceSecretDetection(config.secretDetection); + const outcome = yield* runWorkflow(request.value, start, workflowHost, (execution) => + execution.around( + runScopedDocument( + { + root: execution.root, + // A workflow definition is one immutable object. A component + // search path would read the mutable checkout beside it, so a + // repository component fails to resolve rather than resolving + // to content the definition does not describe. + componentDir: [], + verbose: request.value.verbose, + journal: undefined, + raw: request.value.raw, + secretDetection: request.value.secretDetection, + stream: execution.stream, + }, + { testing: false, props: execution.props, installations: execution.installations }, + // The workflow authority boundary sits exactly where a host + // service adapter would: installed inside the execution scope, + // before the root document is imported. + useWorkflowServiceDenial, + ), + ), + ); + yield* exit(outcome.exitCode); + break; + } } } -export function* runXmd(args: string[], installService: HostServiceInstaller): Operation { +export function* runXmd( + args: string[], + installService: HostServiceInstaller, + // Defaults to the host that refuses. A caller driving this without naming a + // workflow host has no run store, and inheriting one by omission is the + // failure mode the whole boundary exists to prevent — so the default is the + // one that creates and executes nothing. + installWorkflowHost: HostWorkflowInstaller = unsupportedWorkflowHost, +): Operation { // First, so that no later scanner — help, properties, agent flags — can // mistake the inline document's own text for an option. const evalFlags = readEvalFlags(args); @@ -1401,6 +1738,23 @@ export function* runXmd(args: string[], installService: HostServiceInstaller): O } const helpRequest = takeHelpFlag(evalFlags.rest); + + // Before the props phase, because that phase establishes a workflow start's + // definition from Git in order to read what the *pinned* document declares. + // On a host without workflow support the first thing a caller would otherwise + // see is whatever Git said about their directory, which is not the reason the + // command is not going to run. Help is exempt: the grammar is the same + // everywhere, and describing it costs nothing. + let workflowHost: WorkflowHost | undefined; + if (!helpRequest.requested && namesWorkflow(helpRequest.args)) { + try { + workflowHost = yield* installWorkflowHost(); + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + yield* exit(1); + return; + } + } // Recognized before anything reads a document: a malformed duration is a // grammar failure, and a grammar failure never depends on what is on disk. // Help, `--version`, and the other commands stay outside a run lifecycle, @@ -1411,7 +1765,7 @@ export function* runXmd(args: string[], installService: HostServiceInstaller): O !helpRequest.requested && selected !== undefined && !selected.help && selected.name === "run"; if (!isRun) { - return yield* dispatch(evalFlags, helpRequest, installService); + return yield* dispatch(evalFlags, helpRequest, installService, workflowHost); } const timeouts = resolveRunTimeouts(evalFlags.rest); @@ -1421,5 +1775,7 @@ export function* runXmd(args: string[], installService: HostServiceInstaller): O return; } - yield* underRunDeadline(timeouts, () => dispatch(evalFlags, helpRequest, installService)); + yield* underRunDeadline(timeouts, () => + dispatch(evalFlags, helpRequest, installService, workflowHost), + ); } diff --git a/packages/cli/src/compiled.ts b/packages/cli/src/compiled.ts index e1abf66a..7c1f2377 100644 --- a/packages/cli/src/compiled.ts +++ b/packages/cli/src/compiled.ts @@ -10,6 +10,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileDataUri } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { useDenoWorkflowHost } from "./deno-workflow.ts"; import { useCompiledService } from "./compiled-service.ts"; await main(function* (args) { @@ -32,5 +33,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useCompiledService); + yield* runXmd(args, useCompiledService, useDenoWorkflowHost); }); diff --git a/packages/cli/src/deno-workflow.ts b/packages/cli/src/deno-workflow.ts new file mode 100644 index 00000000..c86e8373 --- /dev/null +++ b/packages/cli/src/deno-workflow.ts @@ -0,0 +1,43 @@ +/** + * The Deno workflow host — where `xmd workflow` keeps runs, and what it attaches. + * + * This is the only module in the CLI that names the local run store. SQLite, + * DOFS and the path a run lives at are all behind `@executablemd/workflow/deno`, + * and the shared command module reaches none of them: it asks a `WorkflowHost` + * to open storage and to attach a run's Workspace, and this is what Deno and the + * compiled binary supply. The binary is Deno too, so both entrypoints install + * this one rather than each carrying a copy. + * + * Runs live beneath `~/.xmd/runs` by default. `XMD_WORKFLOW_RUNS` names a + * different absolute directory, which is how a test — or a caller keeping one + * project's runs apart from another's — works without the real user state + * directory being involved at all. + */ + +import { homedir } from "node:os"; +import { join } from "node:path"; +import type { Operation } from "effection"; +import { env as readEnv } from "@executablemd/runtime"; +import { useWorkflowRunStorage, withWorkflowWorkspace } from "@executablemd/workflow/deno"; +import type { WorkflowRunDatabase } from "@executablemd/workflow"; +import type { WorkflowHost } from "./workflow.ts"; + +/** Where a run lives when nothing says otherwise. */ +export const DEFAULT_RUN_STORAGE_ROOT: string = join(homedir(), ".xmd", "runs"); + +/** The variable that names a different run store. Absolute, or it is refused. */ +export const RUN_STORAGE_ROOT_ENV = "XMD_WORKFLOW_RUNS"; + +export function* useDenoWorkflowHost(): Operation { + const configured = yield* readEnv(RUN_STORAGE_ROOT_ENV); + const root = + configured === undefined || configured === "" ? DEFAULT_RUN_STORAGE_ROOT : configured; + return { + useStorage(): Operation { + return useWorkflowRunStorage({ root }); + }, + attach(database: WorkflowRunDatabase, operation: Operation): Operation { + return withWorkflowWorkspace(database, operation); + }, + }; +} diff --git a/packages/cli/src/deno.ts b/packages/cli/src/deno.ts index 4415aebc..b0ed4e73 100644 --- a/packages/cli/src/deno.ts +++ b/packages/cli/src/deno.ts @@ -13,6 +13,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileDataUri } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { useDenoWorkflowHost } from "./deno-workflow.ts"; import { useDenoService } from "./deno-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -37,5 +38,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useDenoService); + yield* runXmd(args, useDenoService, useDenoWorkflowHost); }); diff --git a/packages/cli/src/node.ts b/packages/cli/src/node.ts index 878ab6fc..6a34daa7 100755 --- a/packages/cli/src/node.ts +++ b/packages/cli/src/node.ts @@ -17,6 +17,7 @@ import process from "node:process"; import { API, useHostFiles } from "@executablemd/runtime"; import { compileTempFile } from "@executablemd/core"; import { runXmd } from "./cli.ts"; +import { unsupportedWorkflowHost } from "./workflow.ts"; import { useNodeService } from "./node-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -42,5 +43,5 @@ await main(function* (args) { // no host default: a run with no provider must fail rather than reach the // host by accident. yield* useHostFiles(); - yield* runXmd(args, useNodeService); + yield* runXmd(args, useNodeService, unsupportedWorkflowHost); }); diff --git a/packages/cli/src/workflow-definition.ts b/packages/cli/src/workflow-definition.ts new file mode 100644 index 00000000..2e8159b0 --- /dev/null +++ b/packages/cli/src/workflow-definition.ts @@ -0,0 +1,253 @@ +/** + * What a workflow run is a run of, established from Git. + * + * `xmd workflow start notes.md` names a file in a working tree. A working tree + * changes, so it cannot be a run's identity — a resume months later has to mean + * the same document. What becomes identity is the object: the repository's + * object format, the full commit id `HEAD` resolved to once, and the document's + * repository-relative path inside it. + * + * The consequence is the part worth stating plainly. **The bytes that execute + * come from that commit, not from the file the caller pointed at.** A working + * tree with uncommitted edits runs the committed document, because running the + * edited one while recording the commit as identity would make the record a + * claim about something that never ran. + * + * Where the repository is *checked out* is not identity. It is retrieval + * metadata: replaceable, credential-free, excluded from the comparison that + * decides whether a reused run id addresses the same run, and reauthorized + * before it is used again. A run that moves between machines is the same run. + * + * Everything here goes through the contextual `Git` capability, so nothing + * below runs a command of its own or names a host. + */ + +import { isAbsolute, relative, resolve, sep } from "node:path"; +import { Err, Ok, scoped } from "effection"; +import type { Operation, Result } from "effection"; +import type { Json } from "@executablemd/durable-streams"; +import { API } from "@executablemd/runtime"; +import { + gitObjectFormat, + parseWorkflowDefinition, + readGitObject, + repositoryRoot, + revParse, +} from "@executablemd/workflow"; +import type { GitWorkflowDefinitionV1, WorkflowDefinition } from "@executablemd/workflow"; + +/** The base a `start` records. The command has no base option, so it is this. */ +export const DEFINITION_BASE = "HEAD"; + +/** How this host will find the definition again. Replaceable, never a credential. */ +export const RETRIEVAL_KIND = "local-checkout"; + +/** Everything one `start` establishes before a run can exist. */ +export interface EstablishedDefinition { + readonly definition: WorkflowDefinition; + readonly base: string; + readonly pinnedCommit: string; + readonly retrieval: Json; + /** The document as the pinned commit holds it. */ + readonly source: string; +} + +/** A definition that cannot be established, or a retained one that cannot be loaded. */ +export class WorkflowDefinitionUnavailableError extends Error { + override name = "WorkflowDefinitionUnavailableError"; +} + +function unavailable(message: string, cause?: unknown): WorkflowDefinitionUnavailableError { + return new WorkflowDefinitionUnavailableError(message, cause === undefined ? {} : { cause }); +} + +/** + * Run `body` with the repository as the contextual working directory. + * + * Git answers about the directory it is asked in, so every question about one + * repository is asked from the same place rather than from wherever the process + * started. Keeping Git's own output out of the caller's is the capability's + * own business and is done there. + */ +function inRepository(directory: string, body: () => Operation): Operation { + return scoped(function* () { + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return directory; + }, + }, + { at: "min" }, + ); + return yield* body(); + }); +} + +/** + * The document's path inside its repository, as a definition may hold it. + * + * Repository-relative, POSIX-separated, and refused rather than repaired when + * it leaves the working tree: a path outside the repository names no object in + * the commit, and normalizing one would silently run a different document. + */ +function repositoryRelativePath(root: string, documentPath: string): Result { + const absolute = resolve(documentPath); + const within = relative(resolve(root), absolute); + if (within === "" || within.startsWith("..") || isAbsolute(within)) { + return Err( + unavailable( + "the document is not inside the repository this command resolved, so no commit in it " + + "holds the document. Run the command from the repository the document belongs to.", + ), + ); + } + return Ok(within.split(sep).join("/")); +} + +/** + * Establish the immutable definition of a run that is starting. + * + * The order matters: the repository is located from the document's own + * directory, `HEAD` is resolved once, and the object format is read from the + * same repository — so a definition never mixes one repository's commit with + * another's format. + */ +export function* establishDefinition( + documentPath: string, +): Operation> { + const absolute = resolve(documentPath); + try { + const root = yield* inRepository(absolute.slice(0, absolute.lastIndexOf(sep)) || sep, () => + repositoryRoot(), + ); + + return yield* inRepository(root, function* (): Operation> { + const rootDocumentPath = repositoryRelativePath(root, absolute); + if (!rootDocumentPath.ok) { + return rootDocumentPath; + } + + const pinnedCommit = yield* revParse(`${DEFINITION_BASE}^{commit}`); + const objectFormat = yield* gitObjectFormat(); + + const definition = parseWorkflowDefinition({ + version: 1, + kind: "git", + objectFormat, + objectId: pinnedCommit.toLowerCase(), + rootDocumentPath: rootDocumentPath.value, + }); + if (!definition.ok) { + return definition; + } + + const source = yield* readGitObject(pinnedCommit, rootDocumentPath.value); + return Ok({ + definition: definition.value, + base: DEFINITION_BASE, + pinnedCommit, + retrieval: { version: 1, kind: RETRIEVAL_KIND, checkout: root }, + source, + }); + }); + } catch (error) { + return Err( + unavailable( + `the workflow definition could not be established from ${documentPath}: ` + + (error instanceof Error ? error.message : String(error)), + error, + ), + ); + } +} + +/** + * The checkout a retained locator names, reauthorized before it is used. + * + * A retained path is replaceable metadata rather than permission a host already + * has, so it is checked against the repository it claims to be: a directory + * that is no longer a working tree, or is now a different one, fails rather + * than quietly resolving the definition somewhere else. + */ +function parseRetrieval(metadata: Json | undefined): Result { + if (typeof metadata !== "object" || metadata === null || Array.isArray(metadata)) { + return Err( + unavailable( + "this run retains no usable retrieval metadata, so its definition cannot be located. " + + "The run is left exactly as it is.", + ), + ); + } + const record = Object.fromEntries(Object.entries(metadata)); + if (record.kind !== RETRIEVAL_KIND || typeof record.checkout !== "string") { + return Err( + unavailable( + "this run's retrieval metadata does not describe a local checkout this host can " + + "reach. The run is left exactly as it is.", + ), + ); + } + return Ok(record.checkout); +} + +/** + * Load the exact object a retained run's definition names. + * + * It never substitutes the current `HEAD` or a same-named file in the working + * tree. A resume that could do either would silently continue a different + * document under the same run id. + */ +export function* loadRetainedDefinition( + definition: WorkflowDefinition, + metadata: Json | undefined, +): Operation> { + const checkout = parseRetrieval(metadata); + if (!checkout.ok) { + return checkout; + } + + try { + return yield* inRepository(checkout.value, function* (): Operation> { + const root = yield* repositoryRoot(); + if (resolve(root) !== resolve(checkout.value)) { + return Err( + unavailable( + "the checkout this run retains is no longer the root of the repository it names. " + + "The run is left exactly as it is.", + ), + ); + } + const format = yield* gitObjectFormat(); + if (format !== definition.objectFormat) { + return Err( + unavailable( + "the retained checkout names its objects with a different format than this run's " + + "definition. The run is left exactly as it is.", + ), + ); + } + return Ok(yield* readGitObject(definition.objectId, definition.rootDocumentPath)); + }); + } catch (error) { + return Err( + unavailable( + "this run's retained definition could not be loaded: " + + (error instanceof Error ? error.message : String(error)), + error, + ), + ); + } +} + +/** Whether this definition names a document this slice can execute. */ +export function supportedRootDocument(definition: GitWorkflowDefinitionV1): Result { + if (definition.rootDocumentPath.endsWith(".md")) { + return Ok(undefined); + } + return Err( + unavailable( + "xmd workflow runs Markdown definitions. A function-component root is not supported yet.", + ), + ); +} diff --git a/packages/cli/src/workflow.ts b/packages/cli/src/workflow.ts new file mode 100644 index 00000000..d207dfab --- /dev/null +++ b/packages/cli/src/workflow.ts @@ -0,0 +1,591 @@ +/** + * `xmd workflow start` and `xmd workflow resume` — running a document as a + * retained workflow run. + * + * The command selects the environment; the document describes the procedure. + * `xmd run` executes against the caller's own filesystem and promises nothing + * afterwards. These two execute against a run: one implicit logical Workspace, + * one filtered journal, both in one database that outlives the process, so an + * interrupted procedure continues from its journal frontier rather than from + * the beginning. + * + * ```sh + * xmd workflow start [--id=] [--props-*=…] + * xmd workflow resume + * ``` + * + * `start` names a document; `resume` names a run. That asymmetry is the whole + * lifecycle rule: a document path locates a definition and never selects a + * previous run, and a run id addresses a run whose definition and props are + * already retained. Starting the same document twice without `--id` therefore + * makes two runs, and `resume` accepts no definition, no props and no generated + * prop arguments at all — supplying them would ask this run to be a different + * one. + * + * ## Where the boundary is + * + * This module is runtime-neutral: it names no SQLite, no DOFS and no host. What + * it cannot do itself — open a run's storage, and attach that run's Workspace — + * it asks a `WorkflowHost` for, and the entrypoints decide which one exists. + * Deno and the compiled binary supply the local one; Node and Bun supply a host + * that refuses, so the grammar is the same everywhere and the refusal arrives + * before anything is created or executed. + * + * ## Exit status + * + * Only a completed run exits zero. The rest are distinguishable, because shell + * automation that could not tell a suspended run from a finished one would + * treat every incomplete workflow as success. + */ + +import { Err, Ok, ensure, scoped } from "effection"; +import type { Operation, Result } from "effection"; +import { field, object, cli } from "configliere"; +import { z } from "zod"; +import type { DurableStream, Json } from "@executablemd/durable-streams"; +import { retainedSource } from "@executablemd/core"; +import type { RootDocumentSource } from "@executablemd/core"; +import { retainedWorkflowInstallation, WorkflowRunStorage } from "@executablemd/workflow"; +import type { ExecutionInstallation } from "@executablemd/core/host"; +import type { + WorkflowRunDatabase, + WorkflowRunStatus, + WorkflowStopReason, +} from "@executablemd/workflow"; +import { loadRetainedDefinition, supportedRootDocument } from "./workflow-definition.ts"; +import type { EstablishedDefinition } from "./workflow-definition.ts"; + +/** + * What this module cannot do without knowing the host. + * + * Two operations, both of which reach storage. Everything else about the + * lifecycle — the grammar, the definition, the props, the statuses, the exit + * codes — is the same wherever a run lives. + */ +export interface WorkflowHost { + /** Install this host's run storage for the current scope and its descendants. */ + useStorage(): Operation; + /** + * Attach one run's Workspace around a live or partial document execution. + * + * A completed run does not take this path: its root result is already + * recorded, so there is nothing to give a filesystem to. + */ + attach(database: WorkflowRunDatabase, operation: Operation): Operation; +} + +export type HostWorkflowInstaller = () => Operation; + +/** The one sentence a host without workflow support says. */ +export const UNSUPPORTED_WORKFLOW_HOST = + "xmd workflow is available only through the Deno entrypoint or compiled xmd binary"; + +export class WorkflowHostUnsupportedError extends Error { + override name = "WorkflowHostUnsupportedError"; + + constructor() { + super(UNSUPPORTED_WORKFLOW_HOST); + } +} + +/** The installer Node and Bun supply: the grammar exists, the capability does not. */ +// deno-lint-ignore require-yield +export function* unsupportedWorkflowHost(): Operation { + throw new WorkflowHostUnsupportedError(); +} + +/** Only a completed run exits zero. */ +const EXIT_BY_STATUS: Readonly> = Object.freeze({ + completed: 0, + failed: 1, + suspended: 2, + cancelled: 3, + interrupted: 130, + // A run this process is still holding has not reported an outcome. Reaching + // here with one is a defect, and zero would call it success. + running: 1, +}); + +/** A failure the host classified, rather than an exception message it retained. */ +const HOST_FAILURE_CODE = "document-execution-failed"; +const HOST_INTERRUPTED_CODE = "executor-interrupted"; +const HOST_ORPHANED_CODE = "executor-disappeared"; + +export const WORKFLOW_ACTIONS = ["start", "resume"]; + +export const workflowConfig = object({ + action: { + description: "start or resume", + ...field(z.string().optional(), cli.argument()), + }, + target: { + description: "markdown definition to start, or the run id to resume", + ...field(z.string().optional(), cli.argument()), + }, + id: { + description: "run id to create or address (start only; generated when absent)", + ...field(z.string().optional()), + }, + verbose: { + description: "log journal entries to stderr", + aliases: ["-V"], + ...field(z.boolean(), field.default(false)), + }, + raw: { + description: "output raw markdown without normalization or terminal formatting", + ...field(z.boolean(), field.default(false)), + }, + secretDetection: { + description: + "scan durable events for credentials before they persist; " + + "disable with --no-secret-detection", + ...field(z.boolean(), field.default(true)), + }, +}); + +/** What one invocation asks for, after the grammar has been read. */ +export interface WorkflowRequest { + readonly action: "start" | "resume"; + readonly target: string; + readonly id: string | undefined; + readonly verbose: boolean; + readonly raw: boolean; + readonly secretDetection: boolean; +} + +/** What the shared CLI needs in order to execute this run's document. */ +export interface WorkflowExecution { + readonly root: RootDocumentSource; + readonly props: Record; + readonly stream: DurableStream; + /** + * What the trusted host attaches to this run's one execution. + * + * An `ExecutionInstallation` rather than something installed into a scope: + * its retained-run admission is applied inside canonical core's own journal + * read, before any middleware or document code exists, and its `prepare` + * hook records the run inside the durable root before the root import. + */ + readonly installations: readonly ExecutionInstallation[]; + /** Wraps the whole document execution, or passes it through on completed replay. */ + around(operation: Operation): Operation; +} + +/** How one `start` or `resume` ended. */ +export interface WorkflowOutcome { + readonly exitCode: number; +} + +/** + * The run id `start` uses when the caller named none. + * + * Opaque and cryptographically random, so two starts of one document are two + * runs and neither id says anything about what it runs. A caller who wants a + * stable id supplies one; the local caller is authorized to use any + * storage-valid id, and hashing is what keeps that id from becoming a path. + */ +function generatedRunId(): string { + return crypto.randomUUID(); +} + +function report(message: string): void { + console.error(message); +} + +function reportRun(runId: string): void { + report(`workflow run: ${runId}`); +} + +function reportStatus(status: WorkflowRunStatus): void { + report(`workflow status: ${status}`); +} + +/** Whether this journal already holds the root's terminal event. */ +function* isCompleted(stream: DurableStream): Operation { + const events = yield* stream.readAll(); + return events.some((event) => event.type === "close" && event.coroutineId === "root"); +} + +/** + * The stop reason a failure gets. + * + * A retained event that already crossed the secret filter is preferable to a + * code, because it says which effect failed. Anything else becomes one + * categorical host code: the alternative is retaining an exception message + * beside the journal that filtered it, which is history nothing has filtered. + */ +function* failureReason(database: WorkflowRunDatabase): Operation { + const entries = yield* database.readJournalEntries(); + if (entries.ok) { + for (let index = entries.value.length - 1; index >= 0; index -= 1) { + const entry = entries.value[index]; + if (entry !== undefined && entry.event.result.status === "err") { + return { kind: "journal", eventId: entry.eventId }; + } + } + } + return { kind: "host", code: HOST_FAILURE_CODE }; +} + +/** + * Close an execution record this process did not start and cannot finish. + * + * A run left `running` by a host that disappeared has an execution record with + * no end. Closing it as interrupted before a new one begins is what keeps the + * records a history of executions rather than a history with a hole in it. + * Concurrent ownership is #367's; nothing here claims a second executor is safe. + */ +function* closeOrphanedExecutions(database: WorkflowRunDatabase): Operation> { + const executions = yield* database.readDocumentExecutions(); + if (!executions.ok) { + return executions; + } + for (const execution of executions.value) { + if (execution.stoppedAt !== undefined) { + continue; + } + const finished = yield* database.finishDocumentExecution({ + executionId: execution.executionId, + status: "interrupted", + reason: { kind: "host", code: HOST_ORPHANED_CODE }, + }); + if (!finished.ok) { + return finished; + } + } + return Ok(undefined); +} + +/** The grammar this command accepts, refusing what it does not. */ +export function parseWorkflowRequest(config: { + action?: string; + target?: string; + id?: string; + verbose: boolean; + raw: boolean; + secretDetection: boolean; +}): Result { + const { action, target } = config; + if (action === undefined) { + return Err( + new Error( + "xmd workflow requires a subcommand — `xmd workflow start ` or " + + "`xmd workflow resume `", + ), + ); + } + if (action !== "start" && action !== "resume") { + return Err( + new Error( + `unrecognized subcommand for xmd workflow: ${action} — ` + + `expected ${WORKFLOW_ACTIONS.join(" or ")}`, + ), + ); + } + if (target === undefined || target === "") { + return Err( + new Error( + action === "start" + ? "xmd workflow start requires a markdown definition — `xmd workflow start `" + : "xmd workflow resume requires a run id — `xmd workflow resume `", + ), + ); + } + if (action === "resume" && config.id !== undefined) { + return Err( + new Error( + "unrecognized option for xmd workflow resume: --id — a resume names its run as its " + + "only argument", + ), + ); + } + return Ok({ + action, + target, + id: config.id, + verbose: config.verbose, + raw: config.raw, + secretDetection: config.secretDetection, + }); +} + +/** What the props phase already established for a `start`. */ +export interface WorkflowStart { + readonly established: EstablishedDefinition; + readonly props: Record; +} + +/** The statuses a resume may continue from, and what the rest mean. */ +function admitResume(status: WorkflowRunStatus): Result { + switch (status) { + case "failed": + case "cancelled": + // Terminal, and terminal in a direction a resume cannot move. Replaying + // what a failed run recorded is what a compatible `start --id` is for; + // asking to *continue* one is asking for something that is over. + return Err( + new Error( + `workflow run ${status}: a run that ${status === "failed" ? "failed" : "was cancelled"} ` + + "is not resumed. The run is left exactly as it is.", + ), + ); + // A completed run replays what it recorded, and attaches no Workspace to do + // it. `running` keeps its documented temporary treatment until #367 settles + // durable ownership. + case "completed": + case "interrupted": + case "suspended": + case "running": + return Ok(undefined); + } +} + +/** + * Open the run this request addresses, creating it when `start` describes a new + * one. + * + * `create()` is also how a run is found: a request describing the stored run + * answers with it, and one differing in any immutable field is refused with the + * conflict diagnostics storage already has. Nothing here repairs, replaces or + * initializes anything it did not create, and a lookup that finds nothing + * creates nothing. + */ +function* openRun( + request: WorkflowRequest, + start: WorkflowStart | undefined, +): Operation> { + if (request.action === "resume") { + const found = yield* WorkflowRunStorage.operations.lookup(request.target); + if (!found.ok) { + return found; + } + const database = found.value; + // Before the definition is fetched, before an orphaned execution is closed, + // and before anything is begun: a run that ended is not a run to continue, + // and finding that out after Git has been consulted and a record opened + // would be finding it out too late. + const admitted = admitResume(database.record.status); + if (!admitted.ok) { + return admitted; + } + const source = yield* loadRetainedDefinition( + database.record.definition, + database.retrieval?.metadata, + ); + if (!source.ok) { + return source; + } + return Ok({ database, source: source.value }); + } + + if (start === undefined) { + return Err(new Error("xmd workflow start has no definition to run")); + } + + const supported = supportedRootDocument(start.established.definition); + if (!supported.ok) { + return supported; + } + + const created = yield* WorkflowRunStorage.operations.create({ + runId: request.id ?? generatedRunId(), + definition: start.established.definition, + base: start.established.base, + props: start.props, + }); + if (!created.ok) { + return created; + } + + const replaced = yield* created.value.replaceRetrievalMetadata(start.established.retrieval); + if (!replaced.ok) { + return replaced; + } + return Ok({ database: created.value, source: start.established.source }); +} + +/** + * Run one `start` or `resume` to completion, and answer with its exit status. + * + * `execute` is the shared CLI's own document machinery, handed everything this + * run decided: the pinned source, the retained props, the run's journal, the + * installations that belong inside the execution scope, and the attachment that + * wraps it. + */ +export function runWorkflow( + request: WorkflowRequest, + start: WorkflowStart | undefined, + host: WorkflowHost, + execute: (execution: WorkflowExecution) => Operation>, +): Operation { + return scoped(function* () { + yield* host.useStorage(); + + const opened = yield* openRun(request, start); + if (!opened.ok) { + report(opened.error.message); + return { exitCode: 1 }; + } + + const { database, source } = opened.value; + const { record } = database; + reportRun(record.runId); + + const orphaned = yield* closeOrphanedExecutions(database); + if (!orphaned.ok) { + report(orphaned.error.message); + return { exitCode: 1 }; + } + + const begun = yield* database.beginDocumentExecution(); + if (!begun.ok) { + report(begun.error.message); + return { exitCode: 1 }; + } + const executionId = begun.value.executionId; + + // Interruption is the outcome nothing else publishes. Registered before the + // execution starts, so a scope torn down by Ctrl-C settles the run rather + // than leaving a record with no end and a status of `running`. + // + // The phase, rather than a boolean: "the document produced an outcome" and + // "this invocation is durably settled" are different facts, and collapsing + // them is how a post-execution storage refusal would be republished as an + // interruption. Teardown speaks only while the phase is still `running`. + const phase: LifecyclePhase = { state: "running" }; + yield* ensure(function* () { + if (phase.state !== "running") { + return; + } + const reason: WorkflowStopReason = { kind: "host", code: HOST_INTERRUPTED_CODE }; + const retained = yield* retain(database, executionId, "interrupted", reason); + if (!retained.ok) { + // Never claim a status storage refused. What went wrong is reported, + // and no `workflow status:` line is published. + // + // The process still leaves on the signal's terms. `main()` resolves its + // exit continuation with 130 when SIGINT arrives, before teardown + // begins, and that continuation is first-settlement-wins — so an + // interrupted run exits 130 whether or not its last write landed. What + // this run owes the caller is an accurate account, not a different exit + // code: the refusal is reported, and no status is claimed that storage + // did not accept. An ordinary settlement refusal, which happens while + // there is still an outcome to return, does exit 1. + report(retained.error.message); + return; + } + reportStatus("interrupted"); + }); + + const completed = yield* isCompleted(database.journal); + const execution: WorkflowExecution = { + root: retainedSource(record.definition.rootDocumentPath, source), + props: record.props, + stream: database.journal, + // The run already exists: storage created it before anything executed, + // so this installation records exactly that value, allocates nothing and + // never consults Git. Service denial is installed beside it, through the + // same host-service slot `xmd run` fills with a real adapter. + installations: [ + retainedWorkflowInstallation({ + runId: record.runId, + base: record.base, + pinnedCommit: record.definition.objectId, + }), + ], + around(operation: Operation): Operation { + // A completed run replays its retained output and result. Attaching a + // Workspace for it would open a transaction and capture a root for + // work that is not going to happen. + return completed ? operation : host.attach(database, operation); + }, + }; + + const result = yield* attempt(execution, execute); + // The document is over, whatever storage does next — so teardown must not + // relabel this run interrupted, even if what follows refuses. + phase.state = "executed"; + + const status: WorkflowRunStatus = result.ok ? "completed" : "failed"; + const reason = result.ok ? undefined : yield* failureReason(database); + const retained = yield* retain(database, executionId, status, reason); + + // A document failure is still the failure worth reading, so it is reported + // whether or not the lifecycle writes landed. + if (!result.ok) { + report(result.error.message); + } + if (!retained.ok) { + // The status was not retained, so it is not published and its exit code + // is not this invocation's. Storage refusing is its own failure. + report(retained.error.message); + return { exitCode: 1 }; + } + phase.state = "settled"; + reportStatus(status); + return { exitCode: EXIT_BY_STATUS[status] }; + }); +} + +/** + * How far this invocation has durably got. + * + * `running` — the document may still be running, and a torn-down scope is an + * interruption to retain. `executed` — the document produced an outcome, so + * teardown has nothing left to say about it whatever storage does next. + * `settled` — both lifecycle writes persisted and the status was published. + */ +interface LifecyclePhase { + state: "running" | "executed" | "settled"; +} + +/** + * Retain one outcome: the execution record first, then the run state. + * + * Ordered, and the first refusal is the answer. The run state describes a + * document execution that ended, so publishing it after the record that says so + * was refused would state a conclusion whose premise storage rejected. + * + * Only what the caller already decided crosses into storage — a status and a + * filtered reason. A storage diagnostic is reported to the caller and never + * written back into what the run retains. + */ +function* retain( + database: WorkflowRunDatabase, + executionId: string, + status: WorkflowRunStatus, + reason: WorkflowStopReason | undefined, +): Operation> { + const finished = yield* database.finishDocumentExecution({ executionId, status, reason }); + if (!finished.ok) { + return finished; + } + const updated = yield* database.updateRunState({ status, reason }); + if (!updated.ok) { + return updated; + } + return Ok(undefined); +} + +/** + * Run the document, converting a failure the shared machinery did not catch. + * + * Whatever escapes is still this execution's outcome rather than an + * interruption, so it is published as a failure and the interruption finalizer + * stays out of the way. + */ +function* attempt( + execution: WorkflowExecution, + execute: (execution: WorkflowExecution) => Operation>, +): Operation> { + try { + return yield* execute(execution); + } catch (error) { + return Err(error instanceof Error ? error : new Error(String(error))); + } +} + +/** The exit code a status reports, for a caller composing its own outcome. */ +export function workflowExitCode(status: WorkflowRunStatus): number { + return EXIT_BY_STATUS[status]; +} diff --git a/packages/cli/tests/workflow-cli.test.ts b/packages/cli/tests/workflow-cli.test.ts new file mode 100644 index 00000000..1a75e61b --- /dev/null +++ b/packages/cli/tests/workflow-cli.test.ts @@ -0,0 +1,466 @@ +/** + * Tier WFC — `xmd workflow start` and `xmd workflow resume`. + * + * Every run here shells out, so exit status, the two stderr metadata lines and + * the document's own stdout are observed exactly as a caller sees them. Each + * fixture is a real Git repository in a temporary directory with a real commit, + * because what the definition *is* comes from Git, and each run store is an + * isolated absolute directory named by `XMD_WORKFLOW_RUNS` — nothing here goes + * near `~/.xmd`. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped } from "effection"; +import type { Operation } from "effection"; +import { ensureDir, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { exec } from "@effectionx/process"; +import { randomUUID } from "node:crypto"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { runCli } from "@executablemd/test-support/launch"; +import { workflowRunPath } from "@executablemd/workflow/deno"; + +interface Fixture { + /** The repository the definition lives in. */ + readonly repository: string; + /** The isolated run store. */ + readonly runs: string; + /** An isolated HOME, so nothing reaches the developer's own configuration. */ + readonly home: string; +} + +const RELEASE = [ + "---", + "props:", + " channel:", + " type: string", + " default: stable", + "---", + "", + "# Release", + "", + 'channel={props.channel}', + "", + '', + "", + "Wrote: {notes}", + "", +].join("\n"); + +/** + * A document that fails rather than printing. + * + * A `` refusal is a *printed* error — data, and the run still completes — + * so a failed run needs both an error nothing prints and a region that decides + * one. `` is that region: it makes an undecided error the document + * execution's own outcome. The unresolvable component doubles as the evidence for the other claim this + * fixture carries: a workflow definition is one immutable object, so the + * component search path is empty and a repository component fails to resolve + * rather than resolving to content beside the definition in a mutable checkout. + */ +const REFUSING = [ + "# Refusing", + "", + "", + "", + "", + "the run's result", + "", + "", +].join("\n"); + +function* git(repository: string, args: string[]): Operation { + const result = yield* exec("git", { arguments: args, cwd: repository }).expect(); + if (result.code !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr}`); + } +} + +/** A committed repository, an empty run store, and an isolated HOME. */ +function useFixture( + files: Record, + body: (fixture: Fixture) => Operation, +): Operation { + return scoped(function* () { + const root = join(tmpdir(), `xmd-wfc-${randomUUID()}`); + const fixture: Fixture = { + repository: join(root, "repository"), + runs: join(root, "runs"), + home: join(root, "home"), + }; + yield* ensure(() => rm(root, { recursive: true, force: true })); + yield* ensureDir(fixture.repository); + yield* ensureDir(fixture.home); + + for (const [name, content] of Object.entries(files)) { + const path = join(fixture.repository, name); + yield* ensureDir(join(path, "..")); + yield* writeTextFile(path, content); + } + + yield* git(fixture.repository, ["init", "-q", "--initial-branch=main", "."]); + yield* git(fixture.repository, ["config", "user.email", "tier-wfc@example.test"]); + yield* git(fixture.repository, ["config", "user.name", "Tier WFC"]); + yield* git(fixture.repository, ["add", "-A"]); + yield* git(fixture.repository, ["commit", "-q", "-m", "definition"]); + + return yield* body(fixture); + }); +} + +function xmd(fixture: Fixture, args: string[]) { + return runCli(args, { + cwd: fixture.repository, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }); +} + +/** The run id the `workflow run:` line reported. */ +function reportedRunId(stderr: string): string | undefined { + const line = stderr.split("\n").find((entry) => entry.startsWith("workflow run: ")); + return line?.slice("workflow run: ".length).trim(); +} + +/** The status the `workflow status:` line reported. */ +function reportedStatus(stderr: string): string | undefined { + const line = stderr.split("\n").find((entry) => entry.startsWith("workflow status: ")); + return line?.slice("workflow status: ".length).trim(); +} + +describe("Tier WFC — xmd workflow start and resume", () => { + it("WFC1: two starts without an id make two runs", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const first = yield* xmd(fixture, ["workflow", "start", "flows/release.md"]).join(); + const second = yield* xmd(fixture, ["workflow", "start", "flows/release.md"]).join(); + + expect(first.code).toBe(0); + expect(second.code).toBe(0); + expect(reportedStatus(first.stderr)).toBe("completed"); + + const one = reportedRunId(first.stderr); + const other = reportedRunId(second.stderr); + expect(one).toBeDefined(); + expect(other).toBeDefined(); + expect(one).not.toBe(other); + expect(first.stdout).toContain("Wrote: channel=stable"); + }); + }); + + it("WFC2: reusing an id compatibly finds the run, and incompatibly is refused", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const created = yield* xmd(fixture, [ + "workflow", + "start", + "--id=release-1", + "flows/release.md", + ]).join(); + expect(created.code).toBe(0); + expect(reportedRunId(created.stderr)).toBe("release-1"); + + const reused = yield* xmd(fixture, [ + "workflow", + "start", + "--id=release-1", + "flows/release.md", + ]).join(); + expect(reused.code).toBe(0); + expect(reportedRunId(reused.stderr)).toBe("release-1"); + + const conflicting = yield* xmd(fixture, [ + "workflow", + "start", + "--id=release-1", + "flows/release.md", + "--props-channel=beta", + ]).join(); + expect(conflicting.code).toBe(1); + expect(conflicting.stderr).toContain("release-1"); + expect(conflicting.stderr).toContain("props"); + // A refused reuse creates nothing: the run that is there is the one that + // was there, still reporting the props it was created with. + expect(reportedStatus(conflicting.stderr)).toBeUndefined(); + + const unchanged = yield* xmd(fixture, ["workflow", "resume", "release-1"]).join(); + expect(unchanged.stdout).toContain("Wrote: channel=stable"); + }); + }); + + it("WFC3: generated prop arguments belong to start alone", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const started = yield* xmd(fixture, [ + "workflow", + "start", + "--id=props-1", + "flows/release.md", + "--props-channel=beta", + ]).join(); + expect(started.code).toBe(0); + expect(started.stdout).toContain("Wrote: channel=beta"); + + const help = yield* xmd(fixture, ["workflow", "start", "flows/release.md", "--help"]).join(); + expect(help.code).toBe(0); + expect(help.stdout).toContain("--props-channel"); + + const refused = yield* xmd(fixture, [ + "workflow", + "resume", + "props-1", + "--props-channel=stable", + ]).join(); + expect(refused.code).toBe(1); + expect(refused.stderr).toContain("--props-channel"); + expect(reportedStatus(refused.stderr)).toBeUndefined(); + + const aggregate = yield* xmd(fixture, [ + "workflow", + "resume", + "props-1", + "--props", + '{"channel":"stable"}', + ]).join(); + expect(aggregate.code).toBe(1); + }); + }); + + it("WFC4: the definition is the committed object, not the working tree", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + yield* writeTextFile( + join(fixture.repository, "flows/release.md"), + `${RELEASE}\nUNCOMMITTED\n`, + ); + + const started = yield* xmd(fixture, [ + "workflow", + "start", + "--id=pinned-1", + "flows/release.md", + ]).join(); + + expect(started.code).toBe(0); + expect(started.stdout).toContain("Wrote: channel=stable"); + expect(started.stdout).not.toContain("UNCOMMITTED"); + }); + }); + + it("WFC5: resume names only a run, and finds its definition again", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + yield* xmd(fixture, ["workflow", "start", "--id=resume-1", "flows/release.md"]).expect(); + + const resumed = yield* xmd(fixture, ["workflow", "resume", "resume-1"]).join(); + expect(resumed.code).toBe(0); + expect(reportedRunId(resumed.stderr)).toBe("resume-1"); + expect(resumed.stdout).toContain("Wrote: channel=stable"); + + const withDocument = yield* xmd(fixture, [ + "workflow", + "resume", + "resume-1", + "flows/release.md", + ]).join(); + expect(withDocument.code).toBe(1); + }); + }); + + it("WFC6: a failing document reports failed and exits 1, and replays as failed", function* () { + yield* useFixture( + { + "flows/refusing.md": REFUSING, + "components/ComponentBesideTheDefinition.md": "resolved from the checkout\n", + }, + function* (fixture) { + const failed = yield* xmd(fixture, [ + "workflow", + "start", + "--id=failing-1", + "flows/refusing.md", + ]).join(); + expect(failed.code).toBe(1); + expect(reportedStatus(failed.stderr)).toBe("failed"); + // Nothing beside the definition was searched, so nothing beside it could + // have been read. + expect(failed.stderr).toContain("ComponentBesideTheDefinition"); + expect(failed.stderr).toContain("searched: "); + + // A compatible reuse of retained failed history replays that failure + // rather than retrying the work. + const replayed = yield* xmd(fixture, [ + "workflow", + "start", + "--id=failing-1", + "flows/refusing.md", + ]).join(); + expect(replayed.code).toBe(1); + expect(reportedStatus(replayed.stderr)).toBe("failed"); + expect(replayed.stdout).not.toContain("resolved from the checkout"); + }, + ); + }); + + it("WFC7: absent, foreign and unreadable storage is reported and left alone", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const missing = yield* xmd(fixture, ["workflow", "resume", "never-started"]).join(); + expect(missing.code).toBe(1); + expect(missing.stderr).toContain("never-started"); + expect(reportedRunId(missing.stderr)).toBeUndefined(); + + yield* xmd(fixture, ["workflow", "start", "--id=corrupt-1", "flows/release.md"]).expect(); + const runPath = workflowRunPath(fixture.runs, "corrupt-1"); + const before = yield* readTextFile(runPath); + yield* writeTextFile(runPath, "this is not a workflow run database"); + + const corrupt = yield* xmd(fixture, ["workflow", "resume", "corrupt-1"]).join(); + expect(corrupt.code).toBe(1); + expect(reportedStatus(corrupt.stderr)).toBeUndefined(); + // Described, never replaced: the bytes are exactly what the test wrote. + expect(yield* readTextFile(runPath)).toBe("this is not a workflow run database"); + expect(before).not.toBe("this is not a workflow run database"); + }); + }); + + it("WFC8: the grammar refuses what the command does not have", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const noAction = yield* xmd(fixture, ["workflow"]).join(); + expect(noAction.code).toBe(1); + expect(noAction.stderr).toContain("start"); + + const unknown = yield* xmd(fixture, ["workflow", "cancel", "release-1"]).join(); + expect(unknown.code).toBe(1); + expect(unknown.stderr).toContain("cancel"); + + const noTarget = yield* xmd(fixture, ["workflow", "start"]).join(); + expect(noTarget.code).toBe(1); + + const resumeId = yield* xmd(fixture, [ + "workflow", + "resume", + "release-1", + "--id=other", + ]).join(); + expect(resumeId.code).toBe(1); + expect(resumeId.stderr).toContain("--id"); + + const agent = yield* xmd(fixture, [ + "workflow", + "start", + "flows/release.md", + "--approve-all", + ]).join(); + expect(agent.code).toBe(1); + expect(agent.stderr).toContain("--approve-all"); + + const inline = yield* xmd(fixture, ["workflow", "start", "-e", "# hi"]).join(); + expect(inline.code).toBe(1); + }); + }); + + it("WFC11: `--` ends the options, not the grammar", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + // A third argument is a third argument however it is written. Before this, + // `--` ended the check rather than the options, so these reached storage. + const extraResume = yield* xmd(fixture, [ + "workflow", + "resume", + "release-1", + "--", + "unexpected.md", + ]).join(); + expect(extraResume.code).toBe(1); + // The grammar refused it — not storage, not Git, and not a missing run. + expect(extraResume.stderr).toContain("unexpected.md"); + expect(extraResume.stderr).not.toContain("workflow run:"); + expect(extraResume.stderr).not.toContain("workflow status:"); + + const extraStart = yield* xmd(fixture, [ + "workflow", + "start", + "flows/release.md", + "--", + "second.md", + ]).join(); + expect(extraStart.code).toBe(1); + expect(extraStart.stderr).toContain("second.md"); + expect(extraStart.stderr).not.toContain("workflow run:"); + + // And one valid target after `--` is still one valid target. + const started = yield* xmd(fixture, ["workflow", "start", "--", "flows/release.md"]).join(); + expect(started.code).toBe(0); + expect(started.stderr).toContain("workflow status: completed"); + const runId = /workflow run: (\S+)/.exec(started.stderr)?.[1]; + expect(runId).toBeDefined(); + + const resumed = yield* xmd(fixture, ["workflow", "resume", "--", runId ?? ""]).join(); + expect(resumed.code).toBe(0); + expect(resumed.stderr).toContain("workflow status: completed"); + }); + }); + + it("WFC12: a dash-leading definition and run id are what `--` is for", function* () { + yield* useFixture({ "-release.md": RELEASE }, function* (fixture) { + // The whole point of the separator: a name that begins with `-` is a name, + // not an option, and the parser never gets the chance to read it as one. + const started = yield* xmd(fixture, [ + "workflow", + "start", + "--id=-architect-probe-run", + "--", + "-release.md", + ]).join(); + expect(started.code).toBe(0); + expect(started.stderr).toContain("workflow run: -architect-probe-run"); + expect(started.stderr).toContain("workflow status: completed"); + + // A dash-leading run id resumes on the same terms. + const resumed = yield* xmd(fixture, [ + "workflow", + "resume", + "--", + "-architect-probe-run", + ]).join(); + expect(resumed.code).toBe(0); + expect(resumed.stderr).toContain("workflow run: -architect-probe-run"); + expect(resumed.stderr).toContain("workflow status: completed"); + + // And a third positional after it is still a third positional. + const extra = yield* xmd(fixture, [ + "workflow", + "resume", + "--", + "-architect-probe-run", + "unexpected.md", + ]).join(); + expect(extra.code).toBe(1); + expect(extra.stderr).toContain("unexpected.md"); + expect(extra.stderr).not.toContain("workflow run:"); + expect(extra.stderr).not.toContain("workflow status:"); + }); + }); + + it("WFC9: a definition outside a repository, and one that is not Markdown", function* () { + yield* useFixture( + { "flows/release.md": RELEASE, "flows/root.ts": "export default 1;\n" }, + function* (fixture) { + const notMarkdown = yield* xmd(fixture, ["workflow", "start", "flows/root.ts"]).join(); + expect(notMarkdown.code).toBe(1); + expect(notMarkdown.stderr).toMatch(/markdown/i); + expect(reportedRunId(notMarkdown.stderr)).toBeUndefined(); + + const outside = yield* xmd(fixture, ["workflow", "start", "../elsewhere.md"]).join(); + expect(outside.code).toBe(1); + expect(reportedRunId(outside.stderr)).toBeUndefined(); + }, + ); + }); + + it("WFC10: ordinary xmd run is unchanged by any of this", function* () { + yield* useFixture({ "flows/release.md": RELEASE }, function* (fixture) { + const run = yield* xmd(fixture, ["run", "flows/release.md", "--props-channel=beta"]).join(); + expect(run.code).toBe(0); + expect(run.stdout).toContain("Wrote: channel=beta"); + expect(run.stderr).not.toContain("workflow run:"); + // `xmd run` writes into the caller's own filesystem, which is exactly + // what a workflow run does not do. + expect(yield* readTextFile(join(fixture.repository, "notes.md"))).toBe("channel=beta"); + }); + }); +}); diff --git a/packages/cli/tests/workflow-crash.test.ts b/packages/cli/tests/workflow-crash.test.ts new file mode 100644 index 00000000..541f931c --- /dev/null +++ b/packages/cli/tests/workflow-crash.test.ts @@ -0,0 +1,253 @@ +/** + * Tier WFX — what a killed `xmd workflow start` leaves, and what a resume does + * with it. + * + * The point of a retained workflow is that losing the host is survivable, so + * this is made of real processes and a real `SIGKILL`. Nothing runs after the + * signal — no cleanup, no commit, no rollback — and what the next process finds + * is whatever the last committed transaction left. + * + * The document writes many files, one durable effect each, and the parent kills + * the child once a second connection can see that some of them have committed. + * Where the kill lands is deliberately not controlled: it may fall between two + * effects, or inside one whose transaction has not committed. Both are real, and + * the invariant that has to hold either way is the one asserted — every effect + * appears exactly once, in order, with no duplicate and no gap, and the effects + * that committed before the kill are not performed again. + * + * The stricter in-transaction kill, where the child is stopped at a point it has + * announced from inside an open transaction, is Tier WAC's + * (`packages/workflow/tests/workspace-crash-recovery.test.ts`). This suite is + * about the CLI lifecycle built on top of it. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped, spawn } from "effection"; +import type { Operation } from "effection"; +import { ensureDir, rm, writeTextFile } from "@effectionx/fs"; +import { exec } from "@effectionx/process"; +import { when } from "@effectionx/converge"; +import { randomUUID } from "node:crypto"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import process from "node:process"; +import { DatabaseSync } from "node:sqlite"; +import { cliCommand, runCli } from "@executablemd/test-support/launch"; +import { workflowRunPath } from "@executablemd/workflow/deno"; + +/** Enough effects that a kill lands part-way through rather than after. */ +const EFFECTS = 60; + +const RUN_ID = "crashed-run"; + +function definition(): string { + const lines = ["# Many effects", ""]; + for (let index = 0; index < EFFECTS; index += 1) { + lines.push(`effect ${index}`, ""); + } + return lines.join("\n"); +} + +interface Fixture { + readonly repository: string; + readonly runs: string; + readonly home: string; +} + +function* git(repository: string, args: string[]): Operation { + const result = yield* exec("git", { arguments: args, cwd: repository }).expect(); + if (result.code !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr}`); + } +} + +function useFixture(body: (fixture: Fixture) => Operation): Operation { + return scoped(function* () { + const root = join(tmpdir(), `xmd-wfx-${randomUUID()}`); + const fixture: Fixture = { + repository: join(root, "repository"), + runs: join(root, "runs"), + home: join(root, "home"), + }; + yield* ensure(() => rm(root, { recursive: true, force: true })); + yield* ensureDir(join(fixture.repository, "flows")); + yield* ensureDir(fixture.home); + yield* writeTextFile(join(fixture.repository, "flows/many.md"), definition()); + + yield* git(fixture.repository, ["init", "-q", "--initial-branch=main", "."]); + yield* git(fixture.repository, ["config", "user.email", "tier-wfx@example.test"]); + yield* git(fixture.repository, ["config", "user.name", "Tier WFX"]); + yield* git(fixture.repository, ["add", "-A"]); + yield* git(fixture.repository, ["commit", "-q", "-m", "definition"]); + + return yield* body(fixture); + }); +} + +interface FileEffect { + readonly eventId: string; + readonly name: string; +} + +/** + * The file effects a second connection can see, in append order. + * + * Read from outside the running process on purpose: rows inside an open + * transaction are invisible here until that transaction commits, so this + * reports what has been *published* rather than what some handle is holding. + */ +function committedEffects(path: string): FileEffect[] { + const database = new DatabaseSync(path, { readOnly: true }); + try { + const rows = database + .prepare("SELECT event_id AS id, record FROM journal_events ORDER BY sequence") + .all(); + const effects: FileEffect[] = []; + for (const row of rows) { + const record = typeof row["record"] === "string" ? row["record"] : ""; + const parsed = JSON.parse(record); + const description = parsed?.description; + if (description?.type === "workspace_file" && typeof description.name === "string") { + effects.push({ eventId: String(row["id"]), name: description.name }); + } + } + return effects; + } finally { + database.close(); + } +} + +/** The run's retained status, as a second connection sees it. */ +function committedStatus(path: string): string { + const database = new DatabaseSync(path, { readOnly: true }); + try { + const row = database.prepare("SELECT status FROM workflow_run WHERE id = 1").get(); + return String(row?.["status"]); + } finally { + database.close(); + } +} + +/** The current Workspace root, as a second connection sees it. */ +function committedRoot(path: string): string { + const database = new DatabaseSync(path, { readOnly: true }); + try { + const row = database.prepare("SELECT current_root_id AS root FROM workspace_state").get(); + return String(row?.["root"]); + } finally { + database.close(); + } +} + +function exists(path: string): boolean { + try { + new DatabaseSync(path, { readOnly: true }).close(); + return true; + } catch { + return false; + } +} + +describe("Tier WFX — a killed workflow run resumes from its frontier", () => { + it("WFX1: SIGKILL mid-run, then resume: every effect exactly once", function* () { + yield* useFixture(function* (fixture) { + const path = workflowRunPath(fixture.runs, RUN_ID); + + const killed = yield* scoped(function* () { + const cli = cliCommand(["workflow", "start", `--id=${RUN_ID}`, "flows/many.md"]); + const child = yield* exec(cli.command, { + arguments: cli.arguments, + cwd: fixture.repository, + env: { + ...inherited(), + HOME: fixture.home, + XMD_WORKFLOW_RUNS: fixture.runs, + }, + }); + // Read the child's streams so a full pipe cannot stall it before it has + // committed anything. + yield* spawn(function* () { + const subscription = yield* child.stdout; + let next = yield* subscription.next(); + while (!next.done) { + next = yield* subscription.next(); + } + }); + yield* spawn(function* () { + const subscription = yield* child.stderr; + let next = yield* subscription.next(); + while (!next.done) { + next = yield* subscription.next(); + } + }); + + // The condition is what the database has published, never a sleep. + yield* when( + function* () { + expect(exists(path)).toBe(true); + expect(committedEffects(path).length).toBeGreaterThanOrEqual(3); + }, + { timeout: 60_000 }, + ); + const before = committedEffects(path); + const rootBefore = committedRoot(path); + process.kill(child.pid, "SIGKILL"); + const status = yield* child.join(); + return { before, rootBefore, status }; + }); + + expect(killed.status.signal).toBe("SIGKILL"); + expect(killed.before.length).toBeGreaterThanOrEqual(3); + expect(killed.before.length).toBeLessThan(EFFECTS); + + // Nothing ran after the signal, so the run is still `running`: no status + // was published and no execution record was closed. + expect(committedStatus(path)).toBe("running"); + + const resumed = yield* runCli(["workflow", "resume", RUN_ID], { + cwd: fixture.repository, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + timeout: 180_000, + }).join(); + + expect(resumed.code).toBe(0); + expect(resumed.stderr).toContain(`workflow run: ${RUN_ID}`); + expect(resumed.stderr).toContain("workflow status: completed"); + expect(committedStatus(path)).toBe("completed"); + + const after = committedEffects(path); + + // Every effect the kill left committed is still there, by its own event + // id and in the same order: the resume replayed them rather than + // performing them again. + expect(after.slice(0, killed.before.length)).toEqual(killed.before); + + // And the whole history is exactly one effect per authored element — no + // duplicate from an interrupted transaction that had already published, + // and no gap from one that had not. + expect(after).toHaveLength(EFFECTS); + const targets = after.map((effect) => effect.name.split(":").at(-1)); + expect(new Set(targets).size).toBe(EFFECTS); + expect(targets[0]).toBe("/out/f0.txt"); + expect(targets.at(-1)).toBe(`/out/f${EFFECTS - 1}.txt`); + + // The resume continued from the root the last committed effect left, so + // the frontier moved on rather than starting again. + expect(committedRoot(path)).not.toBe(killed.rootBefore); + }); + }); +}); + +/** What a child needs from this process, without a developer's own HOME. */ +function inherited(): Record { + const names = ["PATH", "DENO_DIR", "DENO_INSTALL_ROOT", "XDG_CACHE_HOME", "TMPDIR"]; + const env: Record = {}; + for (const name of names) { + const value = process.env[name]; + if (typeof value === "string") { + env[name] = value; + } + } + return env; +} diff --git a/packages/cli/tests/workflow-host.test.ts b/packages/cli/tests/workflow-host.test.ts new file mode 100644 index 00000000..2b12748a --- /dev/null +++ b/packages/cli/tests/workflow-host.test.ts @@ -0,0 +1,107 @@ +/** + * Tier WFH — which hosts run a workflow. + * + * `xmd workflow` has the same grammar everywhere and the capability in one + * place. Deno and the compiled binary own the local run store; Node and Bun + * expose the command and refuse it before anything is created or executed, so + * a caller learns the boundary from one sentence rather than from a run that + * half-happened. + * + * Which entrypoint is under test is asked of `@executablemd/test-support`, + * which is the one place runtime detection belongs. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped } from "effection"; +import type { Operation } from "effection"; +import { ensureDir, exists, rm, writeTextFile } from "@effectionx/fs"; +import { randomUUID } from "node:crypto"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { cliRuntime, runCli } from "@executablemd/test-support/launch"; + +/** The one sentence a host without workflow support says. */ +const UNSUPPORTED = + "xmd workflow is available only through the Deno entrypoint or compiled xmd binary"; + +const DOCUMENT = ["# Nothing", "", "no effects at all", ""].join("\n"); + +interface Fixture { + readonly dir: string; + readonly runs: string; + readonly home: string; +} + +function useFixture(body: (fixture: Fixture) => Operation): Operation { + return scoped(function* () { + const root = join(tmpdir(), `xmd-wfh-${randomUUID()}`); + const fixture: Fixture = { + dir: join(root, "work"), + runs: join(root, "runs"), + home: join(root, "home"), + }; + yield* ensure(() => rm(root, { recursive: true, force: true })); + yield* ensureDir(fixture.dir); + yield* ensureDir(fixture.home); + yield* writeTextFile(join(fixture.dir, "flow.md"), DOCUMENT); + return yield* body(fixture); + }); +} + +describe("Tier WFH — workflow host boundary", () => { + it("WFH1: an unsupported host refuses before creating or executing anything", function* () { + yield* useFixture(function* (fixture) { + const result = yield* runCli(["workflow", "start", "flow.md"], { + cwd: fixture.dir, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }).join(); + + if (cliRuntime() === "deno") { + // The Deno entrypoint has the capability. What it refuses here is the + // definition — the fixture is not in a repository — which is a + // different sentence and a different reason. + expect(result.stderr).not.toContain(UNSUPPORTED); + return; + } + + expect(result.code).toBe(1); + expect(result.stderr).toContain(UNSUPPORTED); + // Nothing was created: no run store, and no `workflow run:` line, so no + // run id was ever allocated. + expect(result.stderr).not.toContain("workflow run:"); + expect(result.stderr).not.toContain("workflow status:"); + expect(yield* exists(fixture.runs)).toBe(false); + }); + }); + + it("WFH2: every host reads the same grammar", function* () { + yield* useFixture(function* (fixture) { + const help = yield* runCli(["workflow", "--help"], { + cwd: fixture.dir, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }).join(); + + expect(help.code).toBe(0); + expect(help.stdout).toContain("xmd workflow"); + expect(help.stdout).toContain("--id"); + }); + }); + + it("WFH3: an unsupported host refuses a resume too, and reads no store", function* () { + yield* useFixture(function* (fixture) { + const result = yield* runCli(["workflow", "resume", "any-run"], { + cwd: fixture.dir, + env: { HOME: fixture.home, XMD_WORKFLOW_RUNS: fixture.runs }, + }).join(); + + expect(result.code).toBe(1); + if (cliRuntime() === "deno") { + expect(result.stderr).toContain("any-run"); + return; + } + expect(result.stderr).toContain(UNSUPPORTED); + expect(yield* exists(fixture.runs)).toBe(false); + }); + }); +}); diff --git a/packages/cli/tests/workflow-installation.test.ts b/packages/cli/tests/workflow-installation.test.ts new file mode 100644 index 00000000..cb4a9b22 --- /dev/null +++ b/packages/cli/tests/workflow-installation.test.ts @@ -0,0 +1,612 @@ +/** + * Tier WFI — what `start` and `resume` hand to canonical core. + * + * The end-to-end tier (WFC) proves the lifecycle a caller sees. This one proves + * its *shape*: that a run reaches core as an `ExecutionInstallation` the trusted + * host passes to `executeInstalled()`, through exactly one execution, and that a + * completed replay is given no Workspace to mutate. + * + * `runWorkflow()` takes its document machinery as a parameter, so none of this + * needs a subprocess: the executor here is the observation point the shared CLI + * fills with `runScopedDocument`. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, Err, Ok, resource, scoped, spawn, suspend, withResolvers } from "effection"; +import type { Operation, Result } from "effection"; +import { rm, writeTextFile } from "@effectionx/fs"; +import { exec } from "@effectionx/process"; +import { mkdtemp } from "node:fs/promises"; +import { until } from "effection"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { useWorkflowRunStorage } from "@executablemd/workflow/deno"; +import { Git, WorkflowRunStorage } from "@executablemd/workflow"; +import type { WorkflowRunDatabase, WorkflowRunStatus } from "@executablemd/workflow"; +import type { Json } from "@executablemd/core"; +import { runWorkflow } from "../src/workflow.ts"; +import type { WorkflowExecution, WorkflowHost, WorkflowRequest } from "../src/workflow.ts"; + +/** + * The fixture repository, answered through the Git Api itself. + * + * The boundary a definition is read across, substituted at that boundary and + * nowhere else: the object id and the bytes are the ones `git` reported for a + * real commit, so nothing about what a definition *is* is faked here. + */ +function useGit( + repository: string, + objectId: string, + contents: string, + asked: string[] = [], +): Operation { + return Git.around( + { + // deno-lint-ignore require-yield + *repositoryRoot(): Operation { + asked.push("repositoryRoot"); + return repository; + }, + // deno-lint-ignore require-yield + *objectFormat(): Operation<"sha1" | "sha256"> { + asked.push("objectFormat"); + return "sha1"; + }, + // deno-lint-ignore require-yield + *readObject([commit, path]): Operation { + asked.push(`readObject:${commit}:${path}`); + if (commit !== objectId) { + throw new Error(`unexpected commit ${commit}`); + } + if (path !== "workflow.md") { + throw new Error(`unexpected path ${path}`); + } + return contents; + }, + // deno-lint-ignore require-yield + *revParse([revision]): Operation { + asked.push(`revParse:${revision}`); + return objectId; + }, + }, + { at: "min" }, + ); +} + +/** + * Capture what the CLI reports, and put `console.error` back afterwards. + * + * A status line is an observable of this lifecycle, so a test that only reads + * exit codes cannot tell "no status was published" from "a status was published + * beside an exit 1". + */ +function useReported(lines: string[]): Operation { + return resource(function* (provide) { + const written = console.error; + yield* ensure(() => { + console.error = written; + }); + console.error = (...parts: unknown[]) => { + lines.push(parts.map((part) => String(part)).join(" ")); + }; + yield* provide(); + }); +} + +/** An isolated run store for one test. */ +function useRunStore(): Operation { + return resource(function* (provide) { + const root = yield* until(mkdtemp(join(tmpdir(), "xmd-wfi-"))); + yield* ensure(function* () { + yield* rm(root, { recursive: true, force: true }); + }); + yield* provide(root); + }); +} + +/** A host that records every attachment rather than opening a Workspace. */ +function recordingHost(root: string, attached: string[]): WorkflowHost { + return { + useStorage(): Operation { + return useWorkflowRunStorage({ root }); + }, + attach(database: WorkflowRunDatabase, operation: Operation): Operation { + attached.push(database.record.runId); + return operation; + }, + }; +} + +/** + * A host whose storage refuses one lifecycle write. + * + * Substituted at the database boundary the CLI already depends on, so what is + * being tested is how the lifecycle reacts to a refusal rather than how storage + * produces one. + */ +function refusingHost( + root: string, + refuse: "finish" | "update" | "none", + attempted: string[], +): WorkflowHost { + const host = recordingHost(root, []); + return { + *useStorage(): Operation { + yield* host.useStorage(); + yield* WorkflowRunStorage.around({ + *lookup([runId], next) { + const found = yield* next(runId); + return found.ok ? Ok(refusingDatabase(found.value, refuse, attempted)) : found; + }, + }); + }, + attach: host.attach, + }; +} + +function refusingDatabase( + database: WorkflowRunDatabase, + refuse: "finish" | "update" | "none", + attempted: string[], +): WorkflowRunDatabase { + return { + ...database, + *finishDocumentExecution(request) { + attempted.push(`finish:${request.status}`); + if (refuse === "finish") { + return Err(new Error("PLANTED-STORAGE-REFUSAL")); + } + return yield* database.finishDocumentExecution(request); + }, + *updateRunState(state) { + attempted.push(`update:${state.status}`); + if (refuse === "update") { + return Err(new Error("PLANTED-STORAGE-REFUSAL")); + } + return yield* database.updateRunState(state); + }, + }; +} + +/** Record a root terminal, so the next pass over this journal is a replay. */ +function* closeRoot(root: string, runId: string): Operation { + yield* scoped(function* () { + yield* useWorkflowRunStorage({ root }); + const found = yield* WorkflowRunStorage.operations.lookup(runId); + if (!found.ok) { + throw found.error; + } + yield* found.value.journal.append({ + type: "close", + coroutineId: "root", + result: { status: "ok", value: { status: "ok", output: "", value: "" } }, + }); + }); +} + +/** Put a run into the state a previous invocation would have left it in. */ +function* endRun(root: string, runId: string, status: WorkflowRunStatus): Operation { + yield* scoped(function* () { + yield* useWorkflowRunStorage({ root }); + const found = yield* WorkflowRunStorage.operations.lookup(runId); + if (!found.ok) { + throw found.error; + } + const updated = yield* found.value.updateRunState({ status }); + if (!updated.ok) { + throw updated.error; + } + }); +} + +/** + * Everything about a stored run that a refused resume must leave alone. + * + * Not a summary, and not the database's bytes either: the complete document + * execution records and the complete journal *entries* — in order, each with the + * opaque id storage gave it. A length would not notice a record opened and + * closed again, and reading events rather than entries would not notice one + * appended and rewritten under a new id. + */ +function* runSnapshot(root: string, runId: string): Operation { + return yield* scoped(function* () { + yield* useWorkflowRunStorage({ root }); + const found = yield* WorkflowRunStorage.operations.lookup(runId); + if (!found.ok) { + throw found.error; + } + const database = found.value; + const executions = yield* database.readDocumentExecutions(); + if (!executions.ok) { + throw executions.error; + } + const journal = yield* database.readJournalEntries(); + if (!journal.ok) { + throw journal.error; + } + return JSON.stringify({ + status: database.record.status, + stopReason: database.record.stopReason ?? null, + updatedAt: database.record.updatedAt, + props: database.record.props, + definition: database.record.definition, + retrieval: database.retrieval ?? null, + executions: executions.value, + journal: journal.value, + }); + }); +} + +const REQUEST: WorkflowRequest = { + action: "start", + target: "workflow.md", + id: undefined, + verbose: false, + raw: false, + secretDetection: false, +}; + +describe("Tier WFI — what a run hands to canonical core", () => { + it("WFI1: a run reaches core as one installation carrying its admission and preparation", function* () { + const attached: string[] = []; + const seen: WorkflowExecution[] = []; + let executions = 0; + + yield* scoped(function* () { + const root = yield* useRunStore(); + const created = yield* startedRun(root); + yield* useGit(created.repository, created.objectId, created.contents); + yield* runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + recordingHost(root, attached), + function* (execution): Operation> { + executions += 1; + seen.push(execution); + return Ok(undefined); + }, + ); + }); + + // Exactly one execution — not an `executeInstalled()` followed by an + // `execute()`, and not one per phase. + expect(executions).toEqual(1); + const execution = seen[0]; + expect(execution).toBeDefined(); + // Exactly one installation, and it carries both halves of the contract: + // the mandatory retained-history admission core applies inside its own + // journal read, and the durable preparation core invokes inside the + // durable root. + expect(execution?.installations.length).toEqual(1); + const installation = execution?.installations[0]; + expect(installation?.admissions?.length).toEqual(1); + expect(typeof installation?.prepare).toEqual("function"); + }); + + it("WFI2: a completed run is given no Workspace to attach", function* () { + const attached: string[] = []; + let executions = 0; + + yield* scoped(function* () { + const root = yield* useRunStore(); + const created = yield* startedRun(root); + yield* useGit(created.repository, created.objectId, created.contents); + // First pass: live, so the Workspace is attached. + yield* runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + recordingHost(root, attached), + function* (execution): Operation> { + executions += 1; + // Close the root, so the next pass is a completed replay. + yield* execution.stream.append({ + type: "close", + coroutineId: "root", + result: { status: "ok", value: { status: "ok", output: "", value: "" } }, + }); + return yield* execution.around( + (function* (): Operation> { + return Ok(undefined); + })(), + ); + }, + ); + + const live = attached.length; + expect(live).toEqual(1); + + // Second pass over the same journal: nothing left to give a filesystem to. + yield* runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + recordingHost(root, attached), + function* (execution): Operation> { + executions += 1; + return yield* execution.around( + (function* (): Operation> { + return Ok(undefined); + })(), + ); + }, + ); + }); + + expect(executions).toEqual(2); + // The completed pass attached nothing. + expect(attached.length).toEqual(1); + }); + + it("WFI3: a failed or cancelled run is refused before anything is opened", function* () { + for (const terminal of ["failed", "cancelled"] as const) { + const attached: string[] = []; + const asked: string[] = []; + const reported: string[] = []; + let executions = 0; + + const outcome = yield* scoped(function* () { + const root = yield* useRunStore(); + const created = yield* startedRun(root); + yield* useGit(created.repository, created.objectId, created.contents, asked); + yield* endRun(root, created.runId, terminal); + yield* useReported(reported); + const before = yield* runSnapshot(root, created.runId); + // Everything the fixture itself asked of Git is behind us. + asked.length = 0; + + const result = yield* runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + recordingHost(root, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + executions += 1; + return Ok(undefined); + }, + ); + const after = yield* runSnapshot(root, created.runId); + return { result, before, after }; + }); + + expect(outcome.result.exitCode).toEqual(1); + // Nothing was fetched, attached or run — the definition in particular was + // never read out of Git. + expect(asked).toEqual([]); + expect(attached).toEqual([]); + expect(executions).toEqual(0); + // No status was published for a run whose status did not change. + expect(reported.some((line) => line.includes("workflow status:"))).toBe(false); + expect(reported.some((line) => line.includes(terminal))).toBe(true); + // Structurally unchanged: the same status, stop reason, definition, + // props and retrieval metadata, the same execution records, and the same + // journal entries under the same ids in the same order. + expect(outcome.after).toEqual(outcome.before); + } + }); + + it("WFI4: completed and interrupted runs are still admitted", function* () { + const outcomes: Array<{ status: string; executions: number; attached: number }> = []; + + for (const admitted of ["completed", "interrupted"] as const) { + const attached: string[] = []; + let executions = 0; + + yield* scoped(function* () { + const root = yield* useRunStore(); + const created = yield* startedRun(root); + yield* useGit(created.repository, created.objectId, created.contents); + if (admitted === "completed") { + yield* closeRoot(root, created.runId); + } + yield* endRun(root, created.runId, admitted); + yield* runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + recordingHost(root, attached), + function* (execution): Operation> { + executions += 1; + // The shared CLI wraps its document work in this; a completed run + // is what decides there is nothing to wrap it with. + return yield* execution.around( + (function* (): Operation> { + return Ok(undefined); + })(), + ); + }, + ); + }); + + outcomes.push({ status: admitted, executions, attached: attached.length }); + } + + // Both reached the executor; only the completed one was spared a Workspace. + expect(outcomes).toEqual([ + { status: "completed", executions: 1, attached: 0 }, + { status: "interrupted", executions: 1, attached: 1 }, + ]); + }); + + it("WFI5: a lifecycle write storage refused is never published as a status", function* () { + const faults: Array<{ refuse: "finish" | "update" | "none"; attempts: string[] }> = [ + { refuse: "finish", attempts: ["finish:completed"] }, + { refuse: "update", attempts: ["finish:completed", "update:completed"] }, + ]; + + for (const fault of faults) { + const attempted: string[] = []; + const reported: string[] = []; + + const outcome = yield* scoped(function* () { + const root = yield* useRunStore(); + const created = yield* startedRun(root); + yield* useGit(created.repository, created.objectId, created.contents); + yield* useReported(reported); + return yield* runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + refusingHost(root, fault.refuse, attempted), + // deno-lint-ignore require-yield + function* (): Operation> { + return Ok(undefined); + }, + ); + }); + + // A refusal is this invocation's failure, not a status. + expect(outcome.exitCode).toEqual(1); + expect(reported.some((line) => line.includes("workflow status:"))).toBe(false); + // The planted refusal is what the caller is told about. + expect(reported.some((line) => line.includes("PLANTED-STORAGE-REFUSAL"))).toBe(true); + // A refused prerequisite is not followed by its dependent write, and + // nothing relabelled the run interrupted on the way out. + expect(attempted).toEqual(fault.attempts); + } + }); + + it("WFI6: an interruption storage refusal is never published as a status", function* () { + const faults: Array<{ refuse: "finish" | "update" | "none"; attempts: string[] }> = [ + { refuse: "finish", attempts: ["finish:interrupted"] }, + { refuse: "update", attempts: ["finish:interrupted", "update:interrupted"] }, + ]; + + for (const fault of faults) { + const attempted: string[] = []; + const reported: string[] = []; + const running = withResolvers(); + + yield* scoped(function* () { + const root = yield* useRunStore(); + const created = yield* startedRun(root); + yield* useGit(created.repository, created.objectId, created.contents); + yield* useReported(reported); + + const invocation = yield* spawn(() => + runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + refusingHost(root, fault.refuse, attempted), + function* (): Operation> { + // The document is live and the interruption finalizer is + // registered: halting now is a real interruption rather than a + // race against a delay. + running.resolve(); + yield* suspend(); + return Ok(undefined); + }, + ), + ); + + yield* running.operation; + // Teardown runs to completion before this returns. + yield* invocation.halt(); + }); + + // The finalizer attempted exactly what it was allowed to, in order. + expect(attempted).toEqual(fault.attempts); + // And claimed nothing storage refused. + expect(reported.some((line) => line.includes("workflow status: interrupted"))).toBe(false); + expect(reported.some((line) => line.includes("PLANTED-STORAGE-REFUSAL"))).toBe(true); + } + }); + + it("WFI7: an interruption that was retained keeps the ordinary outcome", function* () { + const attempted: string[] = []; + const reported: string[] = []; + const running = withResolvers(); + + yield* scoped(function* () { + const root = yield* useRunStore(); + const created = yield* startedRun(root); + yield* useGit(created.repository, created.objectId, created.contents); + yield* useReported(reported); + + const invocation = yield* spawn(() => + runWorkflow( + { ...REQUEST, action: "resume", target: created.runId }, + undefined, + // Storage refuses nothing here. + refusingHost(root, "none", attempted), + function* (): Operation> { + running.resolve(); + yield* suspend(); + return Ok(undefined); + }, + ), + ); + + yield* running.operation; + yield* invocation.halt(); + }); + + // Both writes landed, so the status this run publishes is one storage + // accepted. + expect(attempted).toEqual(["finish:interrupted", "update:interrupted"]); + expect(reported.some((line) => line.includes("workflow status: interrupted"))).toBe(true); + }); +}); + +/** + * A real repository with one committed document, and a run that retains it. + * + * The definition a resume loads is committed Git bytes, so nothing here can be + * faked: the object id is the one `git` reports, and the retrieval metadata + * names the checkout the resume reads it back through. + */ +function* startedRun(root: string): Operation { + const repository = yield* until(mkdtemp(join(tmpdir(), "xmd-wfi-repo-"))); + yield* ensure(function* () { + yield* rm(repository, { recursive: true, force: true }); + }); + + yield* git(repository, ["init", "--quiet"]); + yield* git(repository, ["config", "user.email", "wfi@example.test"]); + yield* git(repository, ["config", "user.name", "WFI"]); + yield* writeTextFile(join(repository, "workflow.md"), "recorded\n"); + yield* git(repository, ["add", "workflow.md"]); + yield* git(repository, ["commit", "--quiet", "-m", "definition"]); + const contents = "recorded\n"; + const objectId = (yield* git(repository, ["rev-parse", "HEAD:workflow.md"])).trim(); + const objectFormat = (yield* git(repository, ["rev-parse", "--show-object-format"])).trim(); + + return yield* scoped(function* () { + yield* useWorkflowRunStorage({ root }); + const runId = crypto.randomUUID(); + const created = yield* WorkflowRunStorage.operations.create({ + runId, + base: "main", + definition: { + version: 1, + kind: "git", + objectFormat: objectFormat === "sha256" ? "sha256" : "sha1", + objectId, + rootDocumentPath: "workflow.md", + }, + props: {}, + }); + if (!created.ok) { + throw created.error; + } + yield* created.value.replaceRetrievalMetadata({ + kind: "local-checkout", + checkout: repository, + }); + return { runId, repository, objectId, contents }; + }); +} + +/** A created run, and everything the Git boundary must answer for it. */ +interface Started { + readonly runId: string; + readonly repository: string; + readonly objectId: string; + readonly contents: string; +} + +/** One `git` invocation in `repository`, answering with its stdout. */ +function* git(repository: string, args: string[]): Operation { + const result = yield* exec("git", { arguments: args, cwd: repository }).expect(); + if (result.code !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr}`); + } + return result.stdout; +} diff --git a/packages/core/mod.ts b/packages/core/mod.ts index 290d64c1..cffb1ec5 100644 --- a/packages/core/mod.ts +++ b/packages/core/mod.ts @@ -144,11 +144,13 @@ export { formatDocumentReference, INLINE_SOURCE_PATH, inlineSource, + retainedSource, rootSourcePath, } from "./src/root-source.ts"; export type { FileRootDocument, InlineRootDocument, + RetainedRootDocument, RootDocumentSource, } from "./src/root-source.ts"; export { diff --git a/packages/core/src/root-source.ts b/packages/core/src/root-source.ts index b651f448..139434b1 100644 --- a/packages/core/src/root-source.ts +++ b/packages/core/src/root-source.ts @@ -28,8 +28,41 @@ export interface InlineRootDocument { readonly target?: string; } +/** + * A root document whose text came from somewhere the engine cannot read again. + * + * A workflow definition is the case this exists for: the bytes are the ones a + * pinned Git object held, and the identity they report is the document's path + * inside that object. Text and identity travel together, as they do for an + * inline document, so nothing can execute one document while reporting another. + * + * `retained` is what keeps supplied text from travelling under an identity + * nobody vouched for. An inline document is branded by its path — there is only + * one `` — but a retained document's path is an ordinary one, so without + * this member `{ path, source }` would type-check for any path at all and the + * engine would report a document as having come from a file it never read. + * Constructing one is therefore a deliberate act, the way `inlineSource()` is. + */ +export interface RetainedRootDocument { + readonly path: string; + readonly source: string; + readonly retained: true; + /** The requested target selector, still encoded and possibly a glob. */ + readonly target?: string; +} + /** Where a root document's text comes from: a path, or supplied text. */ -export type RootDocumentSource = FileRootDocument | InlineRootDocument; +export type RootDocumentSource = FileRootDocument | InlineRootDocument | RetainedRootDocument; + +/** Supplied text as a root document reported by the path it came from. */ +export function retainedSource( + path: string, + source: string, + options?: { readonly target?: string }, +): RetainedRootDocument { + const target = options?.target; + return { path, source, retained: true, ...(target === undefined ? {} : { target }) }; +} /** Supplied text as a root document carrying the `` identity. */ export function inlineSource( diff --git a/packages/runtime/apis.ts b/packages/runtime/apis.ts index fd32e91e..1c1aa47a 100644 --- a/packages/runtime/apis.ts +++ b/packages/runtime/apis.ts @@ -85,7 +85,7 @@ import { stat as fsStat, writeTextFile as fsWriteTextFile, } from "@effectionx/fs"; -import { exec as processExec } from "@effectionx/process"; +import { exec as processExec, Stdio } from "@effectionx/process"; import { race, sleep, until } from "effection"; import type { Operation } from "effection"; import { timeoutFetch as contextualFetchTimeout } from "./config.ts"; @@ -587,3 +587,20 @@ export const platform: typeof API.Env.operations.platform = API.Env.operations.p export const command: typeof API.Env.operations.command = API.Env.operations.command; export const compile: typeof API.Env.operations.compile = API.Env.operations.compile; + +/** + * Discard the standard output of subprocesses started in this scope. + * + * For a caller whose subprocess output is an *answer* rather than something to + * show: a command whose stdout is parsed and returned would otherwise also + * print itself into whatever the process was rendering. `stderr` is left alone, + * because that is where a failing command explains itself and a diagnostic is + * worth seeing. + * + * It lives here because reaching the process Api's stdio directly is host + * behavior, and modules held to the runtime-neutral boundary may not import a + * host process module of their own. + */ +export function useQuietProcessOutput(): Operation { + return Stdio.around({ *stdout() {} }); +} diff --git a/packages/runtime/mod.ts b/packages/runtime/mod.ts index 8e45aae5..21f0a94e 100644 --- a/packages/runtime/mod.ts +++ b/packages/runtime/mod.ts @@ -39,6 +39,7 @@ export { platform, command, compile, + useQuietProcessOutput, } from "./apis.ts"; export type { EvalBlock, ResponseHeaders, RuntimeFetchResponse, StatResult } from "./apis.ts"; export { diff --git a/packages/test-support/launch.ts b/packages/test-support/launch.ts index 61689e0a..c5b48f0b 100644 --- a/packages/test-support/launch.ts +++ b/packages/test-support/launch.ts @@ -13,6 +13,20 @@ function entry(runtime: string): string { return join(ROOT, "packages", "cli", "src", `${runtime}.ts`); } +/** + * Which entrypoint `runCli` launches. + * + * Runtime detection belongs to this package and nowhere else (Code Rule 12), so + * a suite whose subject differs by host — `xmd workflow`, which only the Deno + * entrypoints support — asks here rather than reading a global of its own. + */ +export function cliRuntime(): "deno" | "bun" | "node" { + if (Reflect.has(globalThis, "Deno")) { + return "deno"; + } + return Reflect.has(globalThis, "Bun") ? "bun" : "node"; +} + export function cliBase(): string[] { if (Reflect.has(globalThis, "Deno")) { return [process.execPath, "run", "--allow-all", entry("deno")]; diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index ae32b8e9..060e75d9 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -26,8 +26,17 @@ * imports SQLite, Deno or any other host. */ -export { Git, GitRevisionError, revParse } from "./src/git.ts"; -export type { GitApi } from "./src/git.ts"; +export { + Git, + gitObjectFormat, + GitObjectError, + GitRepositoryError, + GitRevisionError, + readGitObject, + repositoryRoot, + revParse, +} from "./src/git.ts"; +export type { GitApi, GitObjectFormat } from "./src/git.ts"; export { getWorkflowRun, retainedWorkflowInstallation, workflowInstallation } from "./src/run.ts"; export type { WorkflowRun } from "./src/run.ts"; export { useWorkflowServiceDenial, WorkflowServiceDeniedError } from "./src/service-denial.ts"; diff --git a/packages/workflow/src/git.ts b/packages/workflow/src/git.ts index 06aab2a0..e33db796 100644 --- a/packages/workflow/src/git.ts +++ b/packages/workflow/src/git.ts @@ -1,16 +1,46 @@ /** * The Git capability. * - * Workflow infrastructure asks one question of the repository — resolve this - * revision expression to a commit — and asks it through a contextual Api, so a - * host or a test replaces the answer lexically rather than by arranging a - * repository on disk. Core never reaches Git at all: ordinary `execute()` and - * `xmd run` stay Git-independent. + * Workflow infrastructure asks a small, fixed set of questions of the + * repository, and asks them through a contextual Api, so a host or a test + * replaces the answers lexically rather than by arranging a repository on disk. + * Core never reaches Git at all: ordinary `execute()` and `xmd run` stay + * Git-independent. + * + * All four questions are about *one* repository — the one containing the + * contextual working directory — and none of them mutates it. Together they are + * what an immutable workflow definition is made of: which repository, which + * commit, in which object format, and what the root document held in it. */ import { type Api, createApi, type Operations } from "@effectionx/context-api"; +import { scoped } from "effection"; import type { Operation } from "effection"; -import { cwd, exec } from "@executablemd/runtime"; +import { cwd, exec, useQuietProcessOutput } from "@executablemd/runtime"; + +/** + * What Git said, without Git having said it to the terminal. + * + * Every answer here is a payload this module reads and returns, so echoing it + * would put a commit id and a document's own bytes into whatever the caller was + * rendering. `stderr` is left alone: it is where a failing Git explains itself, + * and that is a diagnostic rather than a payload. + */ +function asked(command: string[], directory: string): Operation { + return scoped(function* () { + yield* useQuietProcessOutput(); + return yield* exec({ command, cwd: directory }); + }); +} + +interface ExecResult { + readonly exitCode: number; + readonly stdout: string; + readonly stderr: string; +} + +/** The hash algorithm a repository names its objects with. */ +export type GitObjectFormat = "sha1" | "sha256"; export interface GitApi { /** @@ -20,6 +50,33 @@ export interface GitApi { * contextual working directory. */ revParse(revision: string): Operation; + + /** + * The absolute path of the working tree containing the contextual working + * directory. + * + * The semantics of `git rev-parse --show-toplevel`. A directory that is not + * inside a working tree is an error rather than an empty answer. + */ + repositoryRoot(): Operation; + + /** + * The repository's object format. + * + * The semantics of `git rev-parse --show-object-format`. It is part of a + * definition's identity: two hosts that agree about a commit only agree about + * the run if they agree about which algorithm named it. + */ + objectFormat(): Operation; + + /** + * The bytes one path held in one commit, as text. + * + * The semantics of `git cat-file blob :`. What comes back is the + * pinned object rather than whatever the working tree holds now, which is what + * lets a run claim a commit as its identity and mean it. + */ + readObject(commit: string, path: string): Operation; } /** Git could not answer for this revision. Carries what Git reported, not a guess. */ @@ -35,13 +92,43 @@ export class GitRevisionError extends Error { } } +/** Git could not answer a question about the repository itself. */ +export class GitRepositoryError extends Error { + override name = "GitRepositoryError"; + + constructor(question: string, result: { exitCode: number; stderr: string }) { + const reported = result.stderr.trim(); + super( + `git could not answer ${question}: exited ${result.exitCode}` + + (reported === "" ? " with no output" : ` — ${reported}`), + ); + } +} + +/** The object a definition names is not in the repository, or is not a file. */ +export class GitObjectError extends Error { + override name = "GitObjectError"; + + constructor(commit: string, path: string, result: { exitCode: number; stderr: string }) { + const reported = result.stderr.trim(); + super( + `git could not read "${path}" from commit ${commit}: exited ${result.exitCode}` + + (reported === "" ? " with no output" : ` — ${reported}`), + ); + } +} + +function objectFormat(value: string): GitObjectFormat | undefined { + return value === "sha1" || value === "sha256" ? value : undefined; +} + export const Git: Api = createApi("Git", { *revParse(revision: string): Operation { // `--verify` makes an unresolvable revision an error rather than an echo, // and `--end-of-options` stops a revision that looks like a flag from being // read as one. The command is an array, so nothing is ever parsed by a shell. const command = ["git", "rev-parse", "--verify", "--end-of-options", revision]; - const result = yield* exec({ command, cwd: yield* cwd() }); + const result = yield* asked(command, yield* cwd()); if (result.exitCode !== 0) { throw new GitRevisionError(revision, result); } @@ -53,6 +140,38 @@ export const Git: Api = createApi("Git", { } return objectId; }, + + *repositoryRoot(): Operation { + const result = yield* asked(["git", "rev-parse", "--show-toplevel"], yield* cwd()); + const root = result.stdout.trim(); + if (result.exitCode !== 0 || root === "") { + throw new GitRepositoryError("which working tree this directory is in", result); + } + return root; + }, + + *objectFormat(): Operation { + const result = yield* asked(["git", "rev-parse", "--show-object-format"], yield* cwd()); + const format = objectFormat(result.stdout.trim()); + if (result.exitCode !== 0 || format === undefined) { + throw new GitRepositoryError("which object format it uses", result); + } + return format; + }, + + *readObject(commit: string, path: string): Operation { + // `cat-file blob` rather than `show`: it refuses a tree or a commit instead + // of rendering one, so a root document path that names a directory fails + // here rather than executing as whatever `show` chose to print. + const result = yield* asked(["git", "cat-file", "blob", `${commit}:${path}`], yield* cwd()); + if (result.exitCode !== 0) { + throw new GitObjectError(commit, path, result); + } + return result.stdout; + }, }); export const revParse: Operations["revParse"] = Git.operations.revParse; +export const repositoryRoot: Operations["repositoryRoot"] = Git.operations.repositoryRoot; +export const gitObjectFormat: Operations["objectFormat"] = Git.operations.objectFormat; +export const readGitObject: Operations["readObject"] = Git.operations.readObject; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a18cf6a8..f5982842 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -174,6 +174,9 @@ importers: '@executablemd/web': specifier: workspace:* version: link:../web + '@executablemd/workflow': + specifier: workspace:* + version: link:../workflow '@standard-schema/spec': specifier: ^1.0.0 version: 1.1.0 diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index de99659f..9e36302d 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -144,6 +144,24 @@ const DENO_ONLY_TOOLING: RuntimeExclusion[] = [ "drives and against a real node:sqlite WorkflowRun database through the Deno DOFS Workspace adapter; node:sqlite remains behind --experimental-sqlite on Node 22", issue: "https://github.com/taras/executable.md/issues/366", }, + { + path: "packages/cli/tests/workflow-cli.test.ts", + reason: + "drives `xmd workflow start` and `resume` against a real node:sqlite run store, which only the Deno entrypoints open; under Node and Bun the command refuses, and workflow-host.test.ts asserts that refusal on every runtime", + issue: "https://github.com/taras/executable.md/issues/366", + }, + { + path: "packages/cli/tests/workflow-crash.test.ts", + reason: + "kills a real `xmd workflow start` child with SIGKILL and reads the recovered node:sqlite run database it leaves behind; the command only exists on the Deno entrypoints", + issue: "https://github.com/taras/executable.md/issues/366", + }, + { + path: "packages/cli/tests/workflow-installation.test.ts", + reason: + "opens a real node:sqlite run store through @executablemd/workflow/deno to drive runWorkflow() directly; Bun has no node:sqlite at all and Node 22 keeps it behind --experimental-sqlite", + issue: "https://github.com/taras/executable.md/issues/366", + }, ]; /** diff --git a/site/routes/docs/index.tsx b/site/routes/docs/index.tsx index 71a26eec..65bcfcf0 100644 --- a/site/routes/docs/index.tsx +++ b/site/routes/docs/index.tsx @@ -57,13 +57,15 @@ export default define.page(function GettingStarted({ url }) { xmd workflow - Not yet shipped + start · resume - Will run a document in a retained Workspace so an interrupted - workflow can reattach and continue. Unsupported imperative - operations fail explicitly instead of falling back to the host. + Runs a document in a retained Workspace, so an interrupted workflow + resumes from its journal frontier instead of starting again. + Unsupported operations fail explicitly instead of falling back to + the host. Available through the Deno entrypoint and the compiled + binary. diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 08c65e49..be3a4b4b 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -1600,7 +1600,10 @@ run but are absent from the diagnostic trace. | `packages/cli/src/service-host.ts` | shared XMD service handshake observer and supervised host-process adapter | | `packages/cli/src/{deno,node,bun,compiled}-service.ts` | runtime-named service adapters for token, environment and stdio behavior | | `packages/cli/src/{deno,node,bun,compiled}.ts` | Entrypoints — each installs matching `API.Env` and `API.Service` adapters, then calls `runXmd` | -| `packages/workflow/src/service-denial.ts` | `useWorkflowServiceDenial()`, the tested non-delegating provider for future workflow start and resume scopes (#366) | +| `packages/workflow/src/service-denial.ts` | `useWorkflowServiceDenial()`, the non-delegating provider `xmd workflow` installs in every start and resume scope | +| `packages/cli/src/workflow.ts` | the runtime-neutral `xmd workflow` lifecycle: grammar, run opening, execution records, statuses and exit codes | +| `packages/cli/src/workflow-definition.ts` | establishing an immutable Git definition from a working-tree path, and loading a retained one again | +| `packages/cli/src/deno-workflow.ts` | the Deno run store and Workspace attachment; Node and Bun install the refusing host instead | | `packages/workflow/src/deno/workspace/files.ts` | the transaction-bound `API.Files` provider — one durable Workspace effect per document read, write and search | | `packages/workflow/src/deno/workspace/host.ts` | `withWorkflowWorkspace()` — the run's effect coordinator, logical cwd `/`, and Files provider installed together inside one execution | | `packages/workflow/src/journal.ts` | the `workflow_run` record, canonical-record recognition, and the refusals that name differing fields without their values | @@ -8127,6 +8130,46 @@ Defined in §8.1. | DP28 | The complete canonical terminal | A recorded success carrying an otherwise-valid binding, inner failure data of the wrong shape or carrying an extra member, and rendered output a pre-root terminal never has are each the fixed unreadable-root diagnostic, quoting nothing the record held | | DP29 | Hostile after inspection | At both pre-root boundaries, a failure that answers safely while canonical core inspects it and throws afterwards is not the object the completion reports: reading the reported error repeatedly, every way, never throws; no planted text is in the result or the journal; and the bound terminal still replays without running preparation, policy, import or authored work | | DP30 | A mutated canonical refusal | Middleware that catches the refusal the expansion raised, installs throwing `name`, `stack` and `cause` accessors and a planted `message`, and rethrows the same object does not decide what the completion reports: the result is a fresh `DocumentProtocolError` carrying core's original reason, repeated inspection never throws, no planted text is in the result or the journal, nothing authored runs, and the bound terminal replays without policy, preparation, import or append | +### Tier WFC — `xmd workflow start` and `xmd workflow resume` + +Defined in [Workflow workspaces](./workflow-workspace-spec.md) §3. + +| # | Test | Verify | +|---|------|--------| +| WFC1 | Two starts | Two starts without `--id` report two different run ids and both complete | +| WFC2 | Reuse | A compatible reuse of an explicit id addresses the same run; an incompatible one is refused, publishes no status and leaves the stored run as it was | +| WFC3 | Properties | Generated `--props-*` arguments and their help come from the pinned definition; every property form is refused on `resume` | +| WFC4 | Pinned bytes | An uncommitted working-tree edit does not reach the run: the committed document executes | +| WFC5 | Resume | A resume names only a run and finds its definition through retained metadata; a document argument is refused | +| WFC6 | Failure | A failing document exits 1 and retains `failed`; a compatible reuse replays that failure. The component search path is empty, so a component beside the definition never resolves | +| WFC7 | Storage | A missing run and an unreadable database are reported without a status line, and the database is left byte-identical | +| WFC8 | Grammar | A missing or unknown subcommand, a missing target, a third argument, `--id` on resume, an agent option and an inline document are each refused | +| WFC9 | Definition | A non-Markdown root and a path outside the repository fail before a run id is reported | +| WFC10 | `xmd run` | Ordinary `xmd run` is unchanged and still writes into the caller's own filesystem | + +### Tier WFI — What a run hands to canonical core + +| # | Test | Verify | +|---|------|--------| +| WFI1 | One installation, one execution | A run reaches core as exactly one `ExecutionInstallation` carrying its retained-run admission and its `prepare` hook, through exactly one execution — not an `executeInstalled()` followed by an `execute()` | +| WFI2 | Completed replay attaches nothing | A live pass attaches the run's Workspace; a completed replay of the same journal attaches none | +| WFI3 | Terminal resume refusal | A `failed` or `cancelled` run refuses `resume` with exit 1, and the substituted Git Api — which records every operation — is never invoked: no definition is retrieved, no Workspace attached, no executor run, no status line published, and status, stop reason, definition, props, retrieval metadata, every document-execution record and every journal entry under its own id remain structurally unchanged | +| WFI4 | Admission controls | `completed` and `interrupted` runs are both admitted and both reach the executor; only the completed one is spared a Workspace | +| WFI5 | Normal lifecycle-storage refusal | A refused completion record stops the run-state write from being attempted; a refused run-state write after a successful record keeps that failure; captured stderr carries no `workflow status:` line and does carry the refusal; both exit 1, and neither is relabelled `interrupted` | +| WFI6 | Interruption lifecycle-storage refusal | Halting a live document at a barrier reaches the real interruption finalizer: a refused completion record attempts only `finish:interrupted`, a refused run state attempts both in order, neither publishes `workflow status: interrupted`, and the first refusal is reported | + +### Tier WFH — Workflow host boundary + +| # | Test | Verify | +|---|------|--------| +| WFH1/WFH3 | Unsupported host | Node and Bun refuse `start` and `resume` with the settled sentence, report no run id or status, and create no run store | +| WFH2 | One grammar | Every runtime renders the same `xmd workflow` help | + +### Tier WFX — A killed run resumes from its frontier + +| # | Test | Verify | +|---|------|--------| +| WFX1 | SIGKILL and resume | A real `SIGKILL` part-way through leaves the run `running` with the effects that committed; the resume replays those exact events by id, performs the rest once each with no duplicate and no gap, advances the current root, and completes | ### Tier SL — Own-scope context updates diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 98280f95..9e01db07 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -213,8 +213,37 @@ eligible history fork. It does not undo completed local or external effects. Only foreground execution that reaches `completed` exits zero. Suspended, failed, interrupted and cancelled executions have distinct nonzero outcomes so -shell automation cannot mistake an incomplete workflow for completion. Their -numeric assignments remain part of the CLI implementation contract. +shell automation cannot mistake an incomplete workflow for completion: + +| Status | Exit | +| --- | --- | +| `completed` | 0 | +| `failed` | 1 | +| `suspended` | 2 | +| `cancelled` | 3 | +| `interrupted` | 130 | + +A request the command refuses rather than runs — bad grammar, a missing run, an +incompatible reuse, damaged storage, an unsupported host — exits 1 and publishes +no status line. + +**A status line says what was retained**, so a lifecycle write storage refused +publishes none. What that costs the exit code depends on when it happened: + +- **Ordinary settlement.** The document finished and there is still an outcome + to return, so a refused completion record or run state exits **1**. The + refusal is reported; a document failure that also occurred is reported too. +- **Signal-driven interruption.** SIGINT begins orderly teardown and the process + exits **130**, and a refused interruption write does not change that: by the + time a finalizer discovers it, the outcome the signal chose is already the + process's. The refusal is still reported, and no status storage rejected is + ever claimed — an interrupted run whose last write failed says nothing about + `interrupted` rather than saying something untrue about it. + +In both cases the first refusal is authoritative and stops what depends on it: +a refused completion record is not followed by a run-state write. Only the +status and a categorical host reason are retained; a storage diagnostic reaches +the caller and is never written into the run. Management commands report their own request. `workflow cancel ` exits zero when cancellation succeeds even though the durable run status is `cancelled`. @@ -227,6 +256,70 @@ selector. The contract nevertheless contains no public SQLite access, local process path or other assumption that prevents a later host from owning the same run lifecycle remotely. +### 3.9 What is shipped + +The lifecycle above is the whole design, including §3.7's rule that a status +line is published only once both of its lifecycle writes have persisted. What a +caller can run today is `start` and `resume`: + +```sh +xmd workflow start [--id=] [--props-*=…] +xmd workflow resume +``` + +Both stream the document's own output to standard output and report two stable +lines on standard error — `workflow run: ` once the run has been created +or found, and `workflow status: ` once the execution settles. + +A status line is published only after both of its lifecycle writes have +persisted — the document execution's completion record first, then the run +state — and interruption teardown attempts the same two writes in the same +order. What a refusal costs the exit code is §3.7's; what it costs the *record* +is the same everywhere: the first refusal stops the write that depends on it, +the unpersisted status is neither published nor claimed, and a document failure +that also occurred is still reported. + +A refusal after the document finished is that refusal — never a host +interruption, and a completed or failed document is never relabelled +`interrupted` on the way out. + +**A run that ended is not a run to continue.** `resume` admits `interrupted`, +`suspended` and (as a full replay) `completed`; `failed` and `cancelled` are +refused with exit 1 — before the definition is fetched from Git, before an +orphaned execution is closed, before a document-execution record is begun, +before a Workspace is attached, and before anything is appended. Reusing a +compatible id through `start` is a separate rule: it replays a failed run's +retained failure, and that does not make the run eligible for `resume`. + +- `start` takes exactly one Markdown definition path. There is no generic + `--prop`, no `--journal`, no inline `--eval`, no agent option and no host + selector. Without `--id` the host generates an opaque cryptographically random + identifier, so starting the same document twice makes two runs; the local + caller may supply any storage-valid non-empty identifier, and hashing is what + keeps it from becoming a path. +- `resume` takes exactly one run id and nothing else. A document path, a + generated property argument and the aggregate property forms are each refused + rather than ignored. +- The document filesystem (§10.1) is the capability a run has. Repository, + Worktree, Git, Agent, Worker Shell and native services are not: an inherited + `API.Service` provider is refused rather than delegated to, and + `temporaryDirectory` is refused rather than answered with a host directory. +- A function-component root is not supported in this subset and fails before the + run executes. +- The command exists on every runtime and the capability on one: the Deno + entrypoint and the compiled binary own the local run store, and Node and Bun + refuse before creating or executing anything. + +Runs live beneath `~/.xmd/runs` unless `XMD_WORKFLOW_RUNS` names another +absolute directory. Where a run's database is on a host is arrangement, not +identity (§5.2). + +Status, list, history, cancel, fork and delete are designed above and unbuilt. +Concurrent-executor ownership is unbuilt: until it lands, a run left `running` +because its host disappeared is treated as an orphaned interrupted execution by +the next resume, which closes that unfinished execution record as `interrupted` +before beginning its own. + ## 4. Inspection commands ### 4.1 Status and list @@ -876,8 +969,9 @@ delegated without changing the document language. | retained run record and filtered journal | built by #291 | | caller-owned storage transaction | built by #291; Workspace mutations join it in #365 | | provider-backed retained Workspace | document filesystem built by #366; repository, process and attachment capabilities unbuilt (#218) | +| `xmd workflow start` / `resume` | built by #366, Deno entrypoints only | | Repository, Worktree and transactional Git components | defined here; unbuilt | -| lifecycle start/resume/status/history/fork/delete | defined here; unbuilt | +| lifecycle status/history/fork/delete | defined here; unbuilt | | read-only Agent materialization | defined here; proof required | | generated-XMD constrained evaluator | behavior defined; public name/schema open | | Deno-local DOFS persistence | POC proven by #349 / PR #350 |