Skip to content

Latest commit

 

History

History
390 lines (315 loc) · 19.1 KB

File metadata and controls

390 lines (315 loc) · 19.1 KB

Maestro workflow spec — v1

A workflow is a directed graph of nodes described in one YAML file. It is executed by the lead agent (the user's interactive session running /maestro <slug> or $maestro <slug>), which never interprets the graph itself: the deterministic resolver (engine/maestroctl.py next) reads the workflow + the per-feature state ledger and serves exactly one next action as JSON.

The machine-readable contract is engine/schemas/workflow.schema.json. The builder UI (ui/builder.html) embeds the same schema and reads/writes this format losslessly.

YAML subset

Workflow files are restricted to a strict YAML subset (parsed by engine/wf.py, no external dependencies): block mappings and lists, flow lists [a, b] and flow maps {a: b} on one line, plain / single- / double-quoted scalars, | and |- block literals, comments. Anchors, aliases, tags, multi-document files and folded scalars (>) are not supported. The builder UI emits only this subset.

Minimal authoring

Only nodes: is required. Everything else has a sensible default:

  • version: defaults to 1 · name: optional · start: defaults to the first node
  • a node with no type: is an agent
  • a node with no next:/routes: implicitly ends the workflow (next: end)
  • slug is always available as an input; other inputs only need declaring when used

The smallest valid workflow:

nodes:
  - id: review
    instruction: Review the changes and summarize what you find.

Top level

version: 1                  # spec version; anything else is rejected
name: design                # kebab-case identifier
description: Design phase.  # optional prose
inputs:                     # declared inputs (all optional keys except the map itself)
  slug:    {type: string, required: true, description: kebab-case feature id}
  feature: {type: string, required: false, default: "${inputs.slug}"}
defaults:                   # optional per-workflow fallbacks
  model: haiku              # model for agent nodes that don't set one
  agent: general            # subagent type fallback (agents/general.md)
  max_visits: 10            # global revisit cap fallback
start: first_node_id        # entry node
nodes: [ ... ]              # the graph (see node types)
outputs:                    # optional; values surfaced to a calling workflow / final report
  hld_path: ".maestro/${inputs.slug}/hld.md"
ui: { ... }                 # optional, free-form editor metadata — engine ignores, round-trips

inputs.<name>.type is one of string | number | boolean | list. Inputs with required: true must be provided at init; others fall back to default (which may itself contain placeholders over other inputs).

Placeholders

Pure string substitution — no templating engine, no filters, no expressions. Five namespaces:

Placeholder Meaning
${inputs.<name>} workflow input value
${steps.<id>.outputs.<field>} recorded output of a completed step
${steps.<id>.branches.<key>.outputs.<field>} a parallel-branch result
${config.<dot.path>} value from an optional maestro.config.yaml at the repo root (advanced; nothing ships or requires one)
${memory.knowledge.<domain>} a lesson file from the memory store, frozen at init (see memory.md) — resolves leniently to empty when absent

Placeholders may nest: ${memory.knowledge.${inputs.stack}-review} resolves inner-first (bounded to a few passes). ${memory.knowledge.*} is read once at init from a per-run snapshot and never re-read mid-run, so a run stays reproducible even as the shared memory store changes between runs.

Resolution strictness differs by where the placeholder sits. The validator flags a statically unresolvable placeholder (undeclared input, unknown step) at lint time. At runtime, the substitutions that feed an LLM or a shell — agent instructions, gate prompts, agent inputs:, and script argv — resolve leniently: an unresolved reference becomes the empty string rather than aborting the run (an early step may legitimately reference a field a later step has not produced yet). The strict-at-runtime cases are the ones that must be correct to make progress: artifact: paths and skill: names. Inside a parallel branch, ${steps.<id>…} resolves branch-local step ids first, then workflow-level ids.

Conditions

when: on routes uses a tiny grammar — exactly these forms, nothing else:

${...} == <literal>        ${...} != <literal>
${...}                     # truthy: false for "", "false", "0", "null", missing
${...} in [a, b, c]

Literals: bare words, numbers, true/false, single- or double-quoted strings. Comparison is string-wise after normalising numbers and booleans ("3" == 3, "true" == true).

Routing

Every node routes with exactly one of:

next: <node-id>             # unconditional
routes:                     # first matching wins
  - {when: "${steps.review.outputs.blocking} == true", to: fix}
  - {to: contract_gate}     # last entry MUST be the default (no `when`)

Gate nodes route through their options instead (below). Reserved targets:

  • end — workflow completes successfully.
  • abort — workflow terminates as failed.

Back-edges (cycles)

A route may target any node, including earlier ones — that is how loops are expressed. Engine semantics when a route lands on an already-done node:

  • Re-entry reset: the target and every done node reachable from it are reset to pending (cascade, graph-aware), so the flow genuinely re-runs from that point. Gate decision history and visit counters are preserved.
  • Visit caps: the engine counts how many times each node is entered. A node exceeding its max_visits (node value → defaults.max_visits → 10) routes to its on_exhausted: (<node-id> | abort | ask; default ask = a synthesized "loop limit reached — continue anyway / abort" gate).

The validator allows cycles but warns about cycles containing no gate and no script node (pure agent↔agent cycles burn tokens with no human or deterministic brake).

Failure

on_fail: <node-id> | abort | ask (default ask) applies when a node fails after exhausting its retries. ask synthesizes a gate: Retry / Skip this step / Abort, with the failure reason. Skipping marks the node skipped and takes its default route.

Node types

agent

The workhorse. execution: worker (default) spawns a subagent; execution: lead runs the served prompt in the current session without spawning one.

- id: author_hld
  type: agent
  instruction: |            # REQUIRED — what this step must do, in plain language.
    Write the high-level design from the approved PRD and accepted architecture decisions.
    Emit deferred choices separately and propose follow-up questions only for genuinely new gaps.
  skill: hld-writing        # optional pin: agent loads the installed skill by name
                            # (BY NAME, not a path — could be yours or a 3rd-party pack).
                            # Omit for "auto": harness skill-discovery picks the best match.
  execution: lead           # keep the clarified architecture context in the main session
  inputs:                   # optional map, passed verbatim into the subagent prompt
    feature: "${inputs.feature}"
    slug: "${inputs.slug}"
  outputs: [hld_summary]    # fields the agent must return as last-line JSON (small scalars)
  artifact:                # string or list; engine refuses to mark the step done unless every
    - ".maestro/${inputs.slug}/hld.md"
    - ".maestro/${inputs.slug}/hld-post-questions.json"
                            # artifact exists non-empty ("proof, not promises"). The engine
                            # injects owned paths into the prompt.
  retries: 1                # re-dispatches on failure before on_fail applies (default 1)
  next: validate_hld_post_questions

The node owns what/where/wheninstruction, inputs, artifact, outputs, ordering — and the engine renders all of them into the subagent prompt. A skill supplies only how. That split is what makes skills swappable: pin one of ours, one of yours, or a third-party skill (Obra, Superpowers, …), or omit skill: and let the harness auto-pick — the graph is unchanged either way.

Use execution: lead only for small, context-continuous work such as confirming and writing a PRD. The action prompt remains the authority: it should name a narrow read/write boundary. Pinned skills are resolved only from the active repository's .agents/skills/, .claude/skills/, or .cursor/skills/ trees; a lead must never search the wider filesystem.

interview

An ordered, durable clarification step. next serves one unresolved question as ask_interview; interview-record accepts its proposal or stores a human correction. No AI is called by the node. After every question is confirmed, the engine atomically writes the structured context artifact and advances. Use either inline sections or a validated dynamic questions_artifact, never both.

Set batch_size above 1 to serve up to that many unresolved questions in one ask_interview_batch action. The lead records all clear answers atomically with interview-record-batch; omitted answers remain pending and are served again. No question is discarded.

Set presentation: popup to require the lead to collect the served round through the harness's native question UI. If the UI caps questions per popup, the lead opens consecutive popups but does not process or record between them. chat is the backward-compatible default.

- id: prd_interview
  type: interview
  skill: prd-interview
  context: "Confirmed feature: ${steps.feature_goal.outputs.feature_goal}"
  sections:
    - {id: users, title: Users and jobs, proposal: "${steps.proposals.outputs.users}"}
    - {id: scope, title: Functional scope, prompt: "What behavior is in scope?"}
  artifact: ".maestro/runs/${inputs.slug}/prd-context.json"
  next: author_prd

For feature-specific follow-ups, a preceding agent writes a JSON queue containing schema_version: 2, questions, and audit; every question has id, title, question, why, proposal, and one to four short must_resolve facts. audit.unresolved and audit.contradictions explain why another round exists. Both arrays must be empty before an empty queue may declare the interview clear. The engine appends each completed queue, including its resolution facts, to the cumulative decision artifact. Schema version 1 remains readable so an installed run can be upgraded in place.

- id: clarify_edges
  type: interview
  questions_artifact: ".maestro/runs/${inputs.slug}/prd-questions.json"
  batch_size: 12
  presentation: popup
  artifact: ".maestro/runs/${inputs.slug}/prd-context.json"
  next: find_more_gaps

A proposal may be empty. In that case the human must provide an answer; the engine rejects a blank answer, acceptance without a proposal, and answers for any question other than the one currently served.

The shipped design workflow checks an existing PRD with validate_prd.py --compatible, which accepts common equivalent headings for its fast path. Maestro-authored PRDs use the exact 11-heading contract and are validated in strict mode before approval. Traceability IDs appear only as sequential AC-01, AC-02, … bullets under Acceptance criteria. Generated PRDs begin with one concise level-1 feature title plus Feature slug and Status: Ready for review metadata. The author validates inside its existing call; the workflow then runs --fix-mechanical to join wrapped acceptance criteria and restore sequential AC numbers without another model. Only a remaining semantic or structural defect reaches the fast fallback repair agent.

gate

A human decision. Options ARE the outgoing edges. Gates are never skipped on resume.

- id: hld_approval
  type: gate
  prompt: "HLD ready: ${steps.author_hld.outputs.hld_summary}. Approve?"
  options:
    - {id: approve, label: "Approve — create repo LLD workstreams", to: lld_scope_serve}
    - {id: revise,  label: "Request revisions", to: prepare_hld_questions, input: feedback}
    - {id: reject,  label: "Reject — abort", to: abort}

input: <name> collects free text into ${steps.hld_approval.outputs.<name>}. A revise option is simply an option whose to: is a back-edge — re-entry reset cascades automatically.

script

A deterministic command. Exit 0 → next/routes; non-zero → on_fail. If stdout is a single JSON object, its fields become the step's outputs and are routable. Validators commonly expose valid, state, or another small routing field.

- id: validate_hld
  type: script
  run: ["python3", "engine/validate_hld.py", ".maestro/${inputs.slug}/hld.md",
        "--open-questions", ".maestro/${inputs.slug}/open-questions.json"]
  timeout: 60               # seconds, optional (default 300)
  routes:
    - {when: "${steps.validate_hld.outputs.valid} == false", to: repair_hld}
    - {to: hld_approval}

parallel

Static fork with inline branch subgraphs. The node itself joins; branch results land at ${steps.<id>.branches.<branch-id>.outputs.<field>}.

- id: author_llds
  type: parallel
  join: all                 # all | any (default all)
  on_branch_fail: fail_all  # fail_all | continue | ask (default fail_all)
  branches:
    - id: backend
      start: backend_design
      steps:
        - id: backend_design
          type: agent
          instruction: Write the backend low-level design from the HLD.
          skill: backend-design
          artifact: ".maestro/${inputs.slug}/lld/backend.md"
          next: end         # `end` inside a branch = branch complete
    - id: frontend
      start: frontend_design
      steps: [ ... ]
  next: contract

The shipped design workflow does not use one shared parallel node for team-authored LLDs. It creates a separate repo-lld.yaml run per selected repository. Each child has its own state.yaml, decision context, LLD revision gate, and approval history. Its lead agent inspects a bounded repository seam, serves material questions as a grouped native interview, writes once, and runs validate_lld.py; post-write questions and narrow repair are conditional. Its publish step writes a hash-bound receipt to the parent; the parent joins only after every selected child is complete and still matches the current HLD. Generic parallel remains useful when one owner controls every branch in one run.

Branch bodies may contain agent, gate, script and subworkflow nodes (no nested parallel in v1) — a branch wrapping a subworkflow is how sdlc-main runs one impl.yaml per stack. In harnesses with parallel subagents, ready agent steps across branches are dispatched as one wave; elsewhere (next --serial) branches run one at a time.

subworkflow

Runs another workflow file inline. The child's outputs: map becomes the step's outputs; child steps are namespaced in state (design/author_hld). Maximum nesting depth: 4.

- id: design
  type: subworkflow
  workflow: workflows/design.yaml
  inputs: {slug: "${inputs.slug}", feature: "${inputs.feature}"}
  next: arch_review

Trust & execution model

Workflow files are trusted code, on the same footing as a Makefile or a CI config in the repo: a script node runs an arbitrary argv, and values from earlier steps are interpolated into later agent prompts and script argv. Treat authoring or editing a workflow as a change that gets code review, and only run workflows you trust.

Two properties keep interpolation from becoming injection, and both must be preserved:

  • argv is a list, never a shell string. The engine emits run: as a JSON array and the lead agent MUST execute it as an argument vector (e.g. subprocess-style), never by joining it into a single string handed to a shell. A value like "; rm -rf /" is then just one inert argument. (The example pack's stubs use bash -c "…" deliberately, with no interpolation inside the command — do not add ${…} inside a bash -c string.)
  • Conditions are parsed before substitution. when: expressions are parsed into a fixed grammar first, so an interpolated value can only be a comparison operand — it can never introduce a new operator or clause.

slug is validated to a single safe path segment ([a-z0-9][a-z0-9._-]*, no / or ..) so it cannot redirect writes outside .maestro/runs/<slug>/.

State — .maestro/runs/<slug>/state.yaml

Written only by engine/maestroctl.py (fcntl-locked, atomic tmp+rename). Records workflow file + sha256 (edits mid-run halt with instructions to rebase), inputs, run status + cursor (active frontier), per-step status / attempts / visits / timing / outputs / artifacts, interview answers, append-only gate decision history, and parallel-branch bookkeeping.

version is the serialization contract; run_format is the shipped workflow-layout marker. A missing/older run-format marker triggers the explicit one-time upgrade-run preview instead of guessing how removed step identifiers map. Compatible ledgers are backed up, rebased, and stamped; legacy ledgers are rebuilt from preserved artifacts and require current validation plus human approval before their documents can advance.

Resume: done steps are skipped only while their artifacts still exist non-empty on disk; interrupted (running) steps are re-served; gates always re-ask.

Lead-agent protocol (summary)

maestroctl validate <wf>                 # refuse to start on errors
maestroctl init --slug S --workflow <wf> [--input k=v ...]
maestroctl upgrade-run --slug S --workflow <wf> [--apply]  # preview, then one-time upgrade
loop:
  maestroctl next --slug S [--serial]    # → ONE action JSON
    run_agent  → spawn subagent with the pre-rendered prompt → complete --outputs '<json>'
    run_agents → spawn all listed subagents in one parallel wave → complete each
    run_lead   → execute the bounded prompt in the current session → complete --outputs '<json>'
    run_script → execute argv → complete --exit-code N --stdout '...'
    ask_interview → ask one decision → interview-record --section X (--accept | --answer TEXT)
    ask_interview_batch → ask the served round once → interview-record-batch --responses JSON
    ask_gate   → ask the human for a choice → gate-record --option X
    ask_input  → ask for required free text → gate-input-record --input '...'
    done | failed → report and stop
  on step failure: maestroctl fail --step P --reason '...'

The action payload is fully resolved — placeholders substituted, prompts pre-rendered. The lead agent performs zero graph interpretation, never edits state, and never skips a gate. It reads artifacts only when a run_lead prompt explicitly grants bounded access.

An option with input: <field> is a durable two-stage interaction. Recording the choice without text leaves the gate active and makes next return ask_input; only a non-blank gate-input-record completes it and exposes <field> in the gate outputs. Atomic gate-record --input remains accepted for API compatibility, while interactive harnesses use the two-stage path so a choice click cannot silently become empty feedback.