From 58f581f2a08fec9b93f943d912df4667def47d45 Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Sat, 15 Aug 2026 01:02:39 -0600 Subject: [PATCH 1/5] docs(readme): lead with the front doors consumers actually use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A measured DX audit read the README cold and counted the entry points that 183 importing files across 10 consumer repositories really use. The README led with the one API family that has zero real call sites, and never named the most imported symbol in the fleet. README: 591 lines and 3,433 words become 127 lines and 1,176 words. - The quickstart is now `examples/quickstart/minimal.ts` in full. It compiles and runs as pasted. The old excerpt used five identifiers it never defined. - One decision table replaces the three routers that disagreed. Rows run in the order real products use them, and each row links to a runnable example. - A five-word table defines worker, driver, decision, verdict, and harness under the quickstart. - The improvement reference moves to docs/improve.md, and the PrimeIntellect adapter to docs/primeintellect.md. The pinned pip revisions move with them. - Retained-run depth moves to examples/retained-run, which also carries the table that maps what died to the mechanism that continues the work. Examples: three new front doors, each with a README in three sections — when to use it against its siblings, how to use it, and why it exists. - examples/stream-a-turn: `runAgentTaskStream` plus the `RuntimeStreamEvent` union, the most imported symbol in the fleet and previously unmentioned. - examples/tool-loop: `runToolLoop` with your model turn and your executors. - examples/retained-run: start, persist the claim ticket, and reattach. examples/quickstart absorbs the minimal call and the refine loop in one place. Two API honesty fixes: - Export `TERMINAL_DECISIONS`, `TerminalDecision`, and `isTerminalDecision`. The four values that stop the loop were a private string comparison, so a caller could not tell kernel keywords from its own vocabulary. - `Driver.name` documents that it is a trace label with no behavioral effect, and the quickstart stops naming a driver after a decision value. The 767-export kernel barrel stays as it is. Tiering it is out of scope here. --- README.md | 612 +++--------------------- docs/api/index.md | 11 +- docs/api/primitive-catalog.md | 5 +- docs/api/runtime.md | 43 ++ docs/improve.md | 288 +++++++++++ docs/primeintellect.md | 76 +++ examples/README.md | 36 +- examples/chat-handler/README.md | 91 ++-- examples/improve/README.md | 68 ++- examples/quickstart/README.md | 65 ++- examples/quickstart/minimal.ts | 42 ++ examples/quickstart/quickstart.ts | 50 +- examples/retained-run/README.md | 72 +++ examples/retained-run/retained-run.ts | 112 +++++ examples/runtime-run/README.md | 90 ++-- examples/stream-a-turn/README.md | 60 +++ examples/stream-a-turn/stream-a-turn.ts | 106 ++++ examples/stream-backends/README.md | 62 +-- examples/supervise/README.md | 74 ++- examples/tool-loop/README.md | 55 +++ examples/tool-loop/tool-loop.ts | 83 ++++ src/runtime/index.ts | 9 +- src/runtime/run-loop.ts | 20 +- src/runtime/types.ts | 11 +- tsconfig.examples.json | 1 + 25 files changed, 1325 insertions(+), 817 deletions(-) create mode 100644 docs/improve.md create mode 100644 docs/primeintellect.md create mode 100644 examples/quickstart/minimal.ts create mode 100644 examples/retained-run/README.md create mode 100644 examples/retained-run/retained-run.ts create mode 100644 examples/stream-a-turn/README.md create mode 100644 examples/stream-a-turn/stream-a-turn.ts create mode 100644 examples/tool-loop/README.md create mode 100644 examples/tool-loop/tool-loop.ts diff --git a/README.md b/README.md index 5b0abe3f..13ef6b0f 100644 --- a/README.md +++ b/README.md @@ -9,526 +9,103 @@ Domain behavior (models, tools, knowledge) plugs in as adapters; the scoring sta pnpm add @tangle-network/agent-runtime @tangle-network/agent-eval @tangle-network/sandbox ``` -## Contents - -- [Quickstart](#quickstart-offline-no-api-keys) -- [What you do with it](#what-you-do-with-it) -- [Run a chat turn](#run-a-chat-turn) -- [Retain and reconnect a run](#retain-and-reconnect-a-run) -- [Supervise a team of agents](#supervise-a-team-of-agents) -- [Improve an agent](#improve-an-agent) -- [Improve a knowledge base](#improve-a-knowledge-base) -- [Run on PrimeIntellect](#run-on-primeintellect) -- [How it works](#how-it-works-the-short-version) -- [Primitives](#primitives) -- [Examples](#examples) -- [Where to go next](#where-to-go-next) +New here? Read [`docs/concepts.md`](./docs/concepts.md) for the mental model in plain terms, then pick a front door below. ## Quickstart (offline, no API keys) -A driver runs a worker, reads its output, and writes the next prompt until a check passes. -This excerpt shows the driver from the runnable [`examples/quickstart/quickstart.ts`](./examples/quickstart/quickstart.ts). -That file defines the scripted `worker`, `output`, and `validator` used below so it runs without credentials. -Replace the scripted worker with a sandbox, CLI bridge, or router backend without changing the driver. +One agent attempt, run by a loop you control. +This is [`examples/quickstart/minimal.ts`](./examples/quickstart/minimal.ts) in full: it compiles and runs as pasted, with no credentials. ```ts import type { AgentProfile } from '@tangle-network/agent-interface' -import { inProcessSandboxClient, runAgentRounds } from '@tangle-network/agent-runtime/kernel' +import { + inProcessSandboxClient, + runAgentRounds, + type TerminalDecision, +} from '@tangle-network/agent-runtime/kernel' +import type { SandboxEvent } from '@tangle-network/sandbox' -const noteWriterProfile = { +const profile = { name: 'note-writer', harness: 'cli-base', model: { provider: 'scripted', default: 'scripted/note-writer' }, } satisfies AgentProfile -const result = await runAgentRounds({ - task: { prompt: 'Write a one-line release note for one-click restore.' }, +// A scripted worker. Swap in a sandbox, CLI-harness, or router backend later. +const worker = inProcessSandboxClient({ + onPrompt: (): SandboxEvent[] => [ + { type: 'result', data: { result: { note: 'Shipped one-click restore.' } } }, + ], +}) + +const result = await runAgentRounds({ + task: 'Write a one-line release note for one-click restore.', driver: { - name: 'refine', - plan: async (task, history) => { - const last = history[history.length - 1] - if (!last) return [task] // shot 0: run the task as written - if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots - // The core move: read the last worker's real output, write the next prompt FROM it. - return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] - }, - decide: (history) => - history.some((shot) => shot.verdict?.valid) ? 'pick-winner' : history.length < 3 ? 'refine' : 'fail', + // plan returns the tasks to run this iteration; [] means no more work. + plan: async (task, history) => (history.length === 0 ? [task] : []), + // 'done' is one of the four kernel keywords in TERMINAL_DECISIONS. + decide: (): TerminalDecision => 'done', }, - agentRun: { profile: noteWriterProfile, taskToPrompt: (t) => t.prompt }, - output, // parses the worker's event stream into { note } - validator, // pass/fail check: does the note mention "rollback"? + agentRun: { profile, taskToPrompt: (t) => t }, + output: { parse: (events) => events }, ctx: { sandboxClient: worker }, - maxIterations: 3, }) + +console.log(`decision: ${result.decision} — ${result.iterations.length} iteration(s)`) ``` Run it from a clone of this repo and you get exactly this: ```bash $ pnpm i && pnpm build -$ pnpm tsx examples/quickstart/quickstart.ts -shot 0: reject — "Shipped one-click restore." -shot 1: PASS — "Shipped one-click restore with an instant rollback path." -decision: pick-winner — winner: shot 1 +$ pnpm tsx examples/quickstart/minimal.ts +decision: done — 1 iteration(s) ``` -The annotated version is [`examples/driver-loop`](./examples/driver-loop). +[`examples/quickstart`](./examples/quickstart) grows the same call into a loop that reads each output and writes the next prompt from it. -## What you do with it +Five words appear everywhere: -| You want to… | Call | +| Word | What it means | |---|---| -| Run a **chat turn** for a production product agent | `handleChatTurn(...)` | -| Have one agent **supervise a team of agents** toward a goal | `supervise(profile, task, opts)` | -| **Improve** an agent and prove the gain on fresh tasks | `improve(profile, opts)` | -| Produce a measured knowledge-base candidate with agents and checks | `runKnowledgeImprovementJob(...)` | -| Evaluate or train the same agent on **PrimeIntellect** | `createPrimeIntellectPackage(...)` | - -### Run a chat turn - -A product agent is one `handleChatTurn` call inside a route. You give it how to produce the response and how to persist it; it streams, traces, and persists. - -```ts -import { deriveExecutionId, handleChatTurn } from '@tangle-network/agent-runtime/durable' - -const turnIndex = 0 -const executionId = deriveExecutionId({ projectId, sessionId: threadId, turnIndex }) -const result = handleChatTurn({ - identity: { tenantId, sessionId: threadId, userId, turnIndex }, - hooks: { - produce: () => ({ - stream: box.streamPrompt(userMessage, { - sessionId: threadId, - executionId, - turnId: executionId, - detach: true, - }), - finalText: () => box.lastResponse(), - }), - persistAssistantMessage: async ({ identity, finalText }) => db.insertMessage(identity, finalText), - }, - waitUntil, -}) -return new Response(result.body, { headers: { 'content-type': result.contentType } }) -``` - -For a stream reconnect, call `streamPrompt` with the same `executionId` and the last event id the client received. -For a repeated initial dispatch, reuse both `sessionId` and `turnId`; `executionId` alone is not an idempotency key. - -### Retain and reconnect a run - -Use the retained-run API when the provider owns a job that must outlive one HTTP reader or application process. -The provider must advertise exact run identity, replay, result identity, and idempotent cancellation. - -```ts -import { - reconnectRetainedRun, - recoverRetainedRun, - startRetainedRun, -} from '@tangle-network/agent-runtime/kernel' - -const run = await startRetainedRun({ - provider, - environment: { idempotencyKey: 'workspace-42', profile }, - turn: { turnId: 'turn-7', prompt: 'Finish the migration and run its tests.' }, - identity: { sessionId: 'thread-42', executionId: 'execution-7' }, - onAdmission: async (admission) => { - await journal.write(admission) - }, -}) - -for await (const event of run.events()) { - await journal.write(event) -} - -const recovered = await reconnectRetainedRun({ - provider: freshProvider, - controlRef: (await journal.readDispatchedAdmission()).controlRef, -}) -if (!recovered) throw new Error('the provider no longer retains this environment') - -const snapshot = await recovered.status({ waitMs: 30_000 }) -const result = await recovered.result() -``` - -The runtime awaits `onAdmission` after environment creation and again after dispatch. -The start promise resolves only after the dispatched admission, so the exact reference is durable before any caller observes success. -Persist each admission record inside the hook before it returns. -A hook rejection fails the start with `RetainedRunAdmissionError` and keeps the environment for recovery. -When you omit `identity`, the runtime mints deterministic coordinates from the two keys, so every process derives the same values. -After dispatch, the runtime verifies the provider honored the requested identity and fails with `RetainedRunDispatchBindingError` when it did not. -Persist each event cursor and sequence before advancing the visible transcript. -`reconnectRetainedRun` reconstructs a client from a dispatched admission's `controlRef` and rejects any provider, environment, session, execution, run, or digest mismatch. -After a crash that left only the `environment` admission, call `recoverRetainedRun` with its coordinates. -It reports `recovered` with a handle, `not_found` when the environment is gone, or `unverifiable` when the provider cannot self-identify the session. -Never destroy an environment on `unverifiable`; keep it, retry `reconnectRetainedRun` later, or inspect it with provider-native tools. -An unknown provider result remains unknown; the runtime never converts it into success or confirmed cancellation. - -### Supervise a team of agents - -One supervisor spawns and steers workers toward a goal. Where the workers run (an in-process loop, or a sandboxed coding harness) is one data value; the budget, journaling, and stopping are handled for you. - -```ts -import { supervise } from '@tangle-network/agent-runtime/kernel' - -const result = await supervise( - { - name: 'supervisor', - harness: 'cli-base', - model: { provider: 'tangle-router', default: process.env.TANGLE_MODEL! }, - prompt: { - systemPrompt: 'Delegate to workers; do not solve the task yourself.', - }, - }, - 'Implement the feature and make the tests pass.', - { budget, router, backend }, // backend = where workers run: router-tools | sandbox+harness | bridge -) -``` - -### Improve an agent - -`improve` runs one complete `OptimizationMethod` against one profile field. -The method owns candidate generation and selection. -Runtime keeps the final test set out of the method, scores the baseline and selected candidate on it, and returns `ship` only when the paired confidence interval clears `minimumLift`. -The profile is never changed. - -```ts -import { improve, officialGepa } from '@tangle-network/agent-runtime' -import { profileOptimizerModelCall } from '@tangle-network/agent-runtime/kernel' -import { - type AgentProfile, - canonicalAgentProfileDigest, - canonicalCandidateDigest, -} from '@tangle-network/agent-interface' - -const executionRef = canonicalCandidateDigest({ - deployment: process.env.AGENT_DEPLOYMENT_SHA!, - model: process.env.AGENT_MODEL!, - tools: process.env.AGENT_TOOLSET_SHA!, -}) - -const optimizerProfile = { - name: 'support-prompt-optimizer', - harness: 'cli-base', - model: { - provider: 'tangle-router', - default: process.env.OPTIMIZER_MODEL!, - metadata: { maxTokens: 16_384 }, - }, -} satisfies AgentProfile -const optimizerPricing = { - inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION), - outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION), -} -const optimizer = { - model: optimizerProfile.model.default, - call: profileOptimizerModelCall({ - profile: optimizerProfile, - context: 'support-prompt optimizer', - executor: { - backend: 'router', - routerBaseUrl: process.env.OPTIMIZER_BASE_URL!, - routerKey: process.env.OPTIMIZER_API_KEY!, - }, - pricing: optimizerPricing, - }), - callRef: canonicalCandidateDigest({ - profile: canonicalAgentProfileDigest(optimizerProfile), - deployment: process.env.OPTIMIZER_DEPLOYMENT_SHA!, - }), - budget: { - maxCostUsd: 10, - maxRequests: 50, - maxRequestBytes: 2_000_000, - maxResponseBytes: 2_000_000, - maxOutputTokensPerRequest: 16_384, - pricing: optimizerPricing, - }, -} - -const result = await improve(baseProfile, { - surface: 'prompt', - executionRef, - method: officialGepa({ - objective: 'Improve the complete support-agent prompt.', - recipe: { - kind: 'engine', - run: { - engine: 'gepa', - maxEvaluations: 40, - maxProposerCostUsd: 10, - }, - }, - optimizer, - resume: 'if-compatible', - trustResumeState: true, - describeScenario: ({ input }) => ({ input }), - }), - findings, - trainScenarios, - selectionScenarios, - testScenarios, - judges: [judge], - agent: (candidateProfile, scenario, ctx) => - runProfile(candidateProfile, scenario, ctx), - runDir: '.runs/support-prompt', - costCeiling: 25, -}) - -if (result.decision === 'ship') { - console.log(result.candidate.profile, result.liftInterval) -} -``` - -`officialGepa(...)` delegates the complete search to GEPA's upstream Optimize Anything API through agent-eval. -Pass one explicit `engine`, `sequential`, `adaptive-sequential`, `best-of`, `vote`, or `omni` recipe. -Runtime derives the upstream resume identity from `executionRef`, the complete baseline profile, and the selected surface. -With `resume: 'if-compatible'`, agent-eval resumes only when the saved run identity matches the candidate, recipe, data, optimizer settings, runner, and derived execution identity. -Set `trustResumeState: true` only when that run directory is private to the current operator. -Use `resume: 'required'` to fail when no matching run exists. -`result.provenance` reports the upstream package, run ID, resume status, evaluation count, and artifact directory. -`result.candidatePopulation` verifies and joins callback observations with an optimizer's official candidate graph. -It returns every unique candidate as a complete profile with ordered Interface diffs, or as an explicit materialization refusal. -GEPA candidates retain exact parent indices and selection scores; callback-only proposals report lineage as unavailable. -Methods without either artifact return `status: 'unavailable'` instead of treating the winner as the full population. -There is no local fallback. -Install its optional Python process before using it: - -```bash -python -m pip install "agent-eval-rpc==0.145.0" -python -m pip install "gepa[full]==0.1.4" -``` - -The published GEPA 0.1.4 wheel supports the direct `gepa` engine. -Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N require the tested official source revision: - -```bash -python -m pip install "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f" -``` - -Use `officialSkillOpt(...)` for Microsoft's SkillOpt: - -```bash -python -m pip install "agent-eval-rpc==0.145.0" -python -m pip install "skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90" -``` - -SkillOpt 0.2.0's published wheel omits prompt files required by `ReflACTTrainer`, so the tested SkillOpt source revision remains necessary. -SkillOpt and GEPA's standard reflection engine require `optimizer: { model, call, callRef, budget }`. -Agent-based GEPA engines may own their model connection instead. -Runtime owns those model calls through one exact `AgentProfile`; Agent Eval enforces the nested budget and records their measured cost and execution evidence without receiving provider credentials. -`costCeiling` is the total limit for optimizer calls, candidate runs, judges, and final scoring. -Runtime returns `hold` when any part of that cost is unknown. -Runtime rejects a reported total above the limit. - -SkillOpt accepts one text surface. -GEPA accepts text or named components. -Any complete method from `@tangle-network/agent-eval` uses the same call. -The `agent` callback receives the complete immutable candidate profile, not a raw prompt or component fragment. -Runtime uses that exact profile for every candidate run and returns the same measured profile in `result.candidate.profile`. -`executionRef` is a content digest of the agent callback, profile component mapping, model, tools, and closure settings. -Runtime combines it with the complete baseline profile and selected surface for saved work. -Changing any of them runs the affected work again. -For a skill, set `surface: 'skills'` and `skills.resourceName`. -For the complete profile, set `surface: 'agent-profile'`. -To optimize several named profile fields together, also provide `profileComponents.read` and `profileComponents.apply`. -Tools, MCP, hooks, subagents, curated instructions, and rollout policy are also exact profile coordinates. -Runtime does not choose an optimizer for them. - -Without `describeScenario`, the external optimizer receives only each development case ID. -Without `describeArtifact`, evaluation feedback contains no artifact body. -When either descriptor is present, its result passes through `redact` together with findings, background text, profile name, and judge notes. -The built-in redactor removes common credentials and email addresses. -Supply a domain redactor for customer names, account IDs, or other private data the built-in rules cannot identify. -Runtime applies that hook first and then still applies its built-in scrubber. -Set `redact: false` only when every outbound value is public and already reviewed. - -The selected profile surface is the optimizer's candidate and cannot be redacted without changing the measured candidate. -Runtime always rejects recognized credentials in those bytes. -It also rejects structurally sensitive fields such as MCP env, headers, URLs, metadata, and extensions. -For `tools`, `mcp`, `hooks`, `subagents`, and `agent-profile`, Runtime treats the entire selected coordinate as execution-capable. -Use `authorizeSensitiveCandidate` to inspect and accept each exact immutable profile containing public values or safe references. -The callback runs for the baseline and every distinct candidate before either reaches your agent. -Its `sensitivePaths` includes `$` when the whole coordinate requires review. - -Code is the exception. -It uses Runtime's isolated git worktrees and coding-agent candidate execution: - -```ts -const result = await improve({ - surface: 'code', - code: { repoRoot, baseRef, profile, generator }, - scenarios, - judge, - agent, - budget, -}) -``` - -`improve` is the search call. -For production, `proposeAgentImprovement` adds trace analysis and reruns the exact frozen baseline and winner before creating a reviewable proposal. -Runtime rejects a candidate bundle that differs from the search winner. - -```ts -import { - createAgentImprovementActivation, - executeAgentImprovementActivation, - proposeAgentImprovement, - reviewAgentImprovementProposal, -} from '@tangle-network/agent-runtime/intelligence' - -const baseline = freezeBaseline(liveProfile) -const result = await proposeAgentImprovement({ - runId, - profile: liveProfile, - analysis, - improvement: { - surface: 'prompt', - executionRef, - method, - trainScenarios, - selectionScenarios, - testScenarios, - judges: [judge], - agent, - }, - buildExperiment: ({ improvement }) => - buildExperimentMaterial({ - baseline, - candidate: compileCandidateBundle({ baseline, improvement: improvement.candidate }), - benchmark: heldOutBenchmark, - policy: comparisonPolicy, - }), - placeCell, -}) - -const review = reviewAgentImprovementProposal(result.proposal, { - decision: 'approve', - reviewedBy: user.id, - reason: 'The measured gain is worth the cost.', -}) -const activation = createAgentImprovementActivation(result.proposal, review, { - intent: 'activate-candidate', - targets: [{ surface: 'prompt', identity: profileId }], - fundingOwner: tenantId, - authorizedBy: user.id, - expiresAt, -}) -const outcome = await executeAgentImprovementActivation( - { proposal: result.proposal, review, activation }, - { transition: commitProfileTransaction, reconcile: readCommittedResult }, -) -``` - -`buildExperimentMaterial`, `placeCell`, and the transaction functions are application ports because storage and compute differ by product. -The builder returns only baseline, candidate, tasks, and policy; Runtime adds the search ancestry and seals the final experiment. -Runtime owns candidate identity, measurement, review binding, expiry, retry identity, and result validation; the application owns its atomic write. -Official optimizer proposals carry the observed package versions, optimizer model, evaluation and token usage, separate optimization and final-test costs, and resumed-run identity. -`createOptimizationActivationReceipt(result)` exposes the same detached record for callers that need to inspect an `improve()` result before building a proposal. - -### Improve a knowledge base - -`runKnowledgeImprovementJob` runs KB, wiki, memory-backed, and RAG improvement jobs. -It creates a candidate copy, runs agents against it, checks it through `@tangle-network/agent-knowledge`, and returns frozen baseline and candidate snapshots with spend and timing. -It never changes the live knowledge base. -Use `improve(profile, { surface: 'memory', ... })` for the agent's curated lesson document. -Use this job for source, retrieval, and knowledge-store changes. - -```ts -import { runKnowledgeImprovementJob } from '@tangle-network/agent-runtime/knowledge' - -const result = await runKnowledgeImprovementJob({ - root: './kb', - goal: 'Improve support refund-policy knowledge', - implementationRef: 'git:0123456789abcdef0123456789abcdef01234567', - readinessSpecs, - budget: { maxIterations: 8, maxTokens: 120_000, maxUsd: 10 }, - backend, -}) - -console.log(result.knowledge?.reference.candidateHash, result.measurement.supervisedSpent) -``` - -Use it when the product needs one knob for "make this knowledge base better" instead of wiring `improveKnowledgeBase`, a runtime supervisor, candidate workspaces, and readiness checks by hand. -Set `implementationRef` to the deployed `git:<40 hex>` revision or a `sha256:<64 hex>` digest covering every callback, model, index, and external setting that can change the result. -The same run ID resumes only when this identity still matches. -Measure the returned bundle pair, record the review, then activate through `executeAgentImprovementActivation`; activation is the only write path. - -### Run on PrimeIntellect - -`@tangle-network/agent-runtime/primeintellect` packages typed train and eval tasks as a PrimeIntellect Verifiers environment. -Prime launches your actual runtime program against an intercepted model endpoint, so `runPersonified`, `runAgentic`, product agents, tool calls, and multiple rounds stay intact. -Reference answers remain in Prime's task process and never enter the agent workspace. -The runner file must be one executable bundle containing the app and its runtime dependencies. - -```ts -import { readFile } from 'node:fs/promises' -import { - createPrimeIntellectPackage, - writePrimeIntellectPackage, -} from '@tangle-network/agent-runtime/primeintellect' - -const bundledRunner = await readFile('./dist/prime-runner.mjs', 'utf8') -const bundle = createPrimeIntellectPackage({ - name: 'support-agent', - version: '1.0.0', - tasks: [ - { - id: 'train-refund-policy', - split: 'train', - prompt: 'Can a subscription renewal be refunded?', - answer: 'No', - }, - { - id: 'eval-final-sale', - split: 'eval', - prompt: 'Can a final-sale order be refunded?', - answer: 'No', - }, - ], - scoring: { kind: 'exact', normalization: 'trim-casefold' }, - runner: { - image: 'node:22-bookworm-slim', - files: { 'runner.mjs': bundledRunner }, - command: ['node', 'runner.mjs'], - }, -}) - -await writePrimeIntellectPackage(bundle, './prime/support-agent') -``` - -The runner reads the episode and uses the normal runtime APIs: -Here, `runProductAgent` is the application's existing entry point, not another loop supplied by this adapter. - -```ts -import { - primeIntellectExecutorConfig, - runPrimeIntellectProgram, -} from '@tangle-network/agent-runtime/primeintellect' -import { - collectAgentTurn, - createExecutor, - streamAgentTurn, -} from '@tangle-network/agent-runtime/kernel' - -await runPrimeIntellectProgram(async (episode) => { - const profile = makeProductProfile({ model: episode.model.name }) - return collectAgentTurn( - streamAgentTurn( - { - kind: 'executor', - profile, - factory: createExecutor(primeIntellectExecutorConfig(episode)), - }, - episode.task.prompt, - ), - ) -}) -``` - -Prime writes complete `traces.jsonl` rows. -Use `importPrimeIntellectTraces(...)` to convert them to agent-eval `RunRecord`s for the existing reports and release checks. +| **worker** | An agent that produces an answer. Here it is a `SandboxClient`. | +| **driver** | Your code. It runs a worker, reads the output, and writes the next prompt. | +| **decision** | What `decide` returns. The four keywords in `TERMINAL_DECISIONS` (`stop`, `pick-winner`, `fail`, `done`) end the loop; every other value is your own vocabulary and continues it. | +| **verdict** | What a validator returns: valid or not, with a score. | +| **harness** | What drives an agent: an in-process model loop (`cli-base`), or a coding CLI such as `claude-code`, `codex`, or `opencode`. | + +## Which front door + +One row per entry point, ordered by how often real products use it. +Each row links to a runnable example. + +| Front door | When to call it | What you give it | What you get back | +|---|---|---|---| +| **`runAgentTaskStream`** · [example](./examples/stream-a-turn) | You run one agent turn and read its events yourself. | a task, a backend, a message | an async stream of `RuntimeStreamEvent` | +| **`handleChatTurn`** (`/durable`) · [example](./examples/chat-handler) | A web route must stream one turn to a browser and save the reply. | how to produce tokens, how to persist | an HTTP body plus a persist call after the last token | +| **`AgentExecutionBackend`** · [example](./examples/stream-backends) | You choose where the tokens come from: your loop, a sandbox, or an exact profile. | `kind` plus a `stream()` generator | the same event union from any source | +| **`runToolLoop`** (`/tool-loop`) · [example](./examples/tool-loop) | The model must call your tools and answer in the same turn. | one model turn, your executors | final text, every tool outcome, a stop reason | +| **`startRuntimeRun`** · [example](./examples/runtime-run) | You must record what a run cost and whether it succeeded. | run identity, a store adapter | a live cost tally and one persisted row | +| **`runAgentRounds`** (`/kernel`) · [example](./examples/quickstart) | One prompt is not enough, and your code owns the stop rule. | `plan`, `decide`, an output adapter, a sandbox client | every attempt, the verdicts, and a winner | +| **`supervise`** (`/kernel`) · [example](./examples/supervise) | A model must decide the plan and drive other agents. | a supervisor profile, a goal, a budget | the delivered result, or a typed reason and the spend | +| **`startRetainedRun`** (`/kernel`) · [example](./examples/retained-run) | The job must outlive the process that started it. | a provider, keys, a durable admission hook | a claim ticket any process can reattach to | +| **`improve`** · [example](./examples/improve) | You must change one part of an agent and prove the gain. | a profile field, three case sets, a judge | a detached candidate, a lift interval, ship or hold | + +Five mechanisms continue interrupted work. +Pick by what died: the HTTP connection (`streamPrompt` with the same `executionId`), nothing but you want the same box (`openSandboxRun`), the coordinator process (`supervise({ runDir })`), the user's chat session (`/conversation` stores), or everything except the provider ([retained runs](./examples/retained-run)). + +## Also in the box + +- **Benchmarks and leaderboards** — compare strategies with significance stats (`runBenchmark`), or stand up a harness×model board (`defineLeaderboard`): [`examples/coding-benchmark`](./examples/coding-benchmark), [`examples/webcode-matrix`](./examples/webcode-matrix). +- **Agent graphs** — fixed topologies authored as data and run through `runGraph`: [`examples/graphs`](./examples/graphs). +- **Improve a knowledge base** — a measured candidate copy of a KB, wiki, or RAG corpus: [`docs/improve.md`](./docs/improve.md). +- **PrimeIntellect** — package the same runtime program as a Verifiers environment: [`docs/primeintellect.md`](./docs/primeintellect.md). +- **Conversations** (`/conversation`) — multi-turn two-agent sessions with SQL-backed resume. +- **MCP servers** (`/mcp`) — give any agent a `delegate` tool plus live coordination tools. +- **Live run view** (`/tui`) — `agent-runtime-top` shows every supervisor run in a workspace, with steer and cancel. +- **Telemetry** — every loop emits `loop.*` trace events, exported as OpenTelemetry GenAI spans when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. + +All 33 examples live in [`examples/`](./examples). ## How it works (the short version) @@ -537,55 +114,14 @@ Use `importPrimeIntellectTraces(...)` to convert them to agent-eval `RunRecord`s - **Candidates face fresh tasks.** The optimizer uses train and selection tasks. Promotion uses a separate final set. - **Scores come from executed attempts.** Runtime recomputes results from the recorded cells and rejects incomplete cost or source evidence. -## Primitives - -The general-purpose pieces, by import path. Every export with its one-line summary lives in the generated [`docs/api/primitive-catalog.md`](./docs/api/primitive-catalog.md): check it before building anything new. - -| Primitive | What it does | Import | -|---|---|---| -| Chat-turn runtime | Stream and persist one production chat turn (`handleChatTurn`); derive its stable execution and turn identity (`deriveExecutionId`); normalize any backend's stream into one event shape (`streamAgentTurn`) | `/durable` · `/kernel` | -| Retained provider runs | Start one detached provider job with a durable admission hook, replay exact events, reconnect after restart, rebuild from pre-dispatch coordinates, continue its native context, and cancel idempotently (`startRetainedRun`, `reconnectRetainedRun`, `recoverRetainedRun`) | `/kernel` | -| Tool-call loop | Run one model turn, execute requested tools, feed results back, and stop on completion, repetition, time, or cost limits (`runToolLoop`, `streamToolLoop`) | `/tool-loop` | -| Supervision | One agent spawns, budgets, and steers workers toward a goal (`supervise`, `delegate`), on an in-process loop or a sandboxed coding harness | `/kernel` · `/mcp` | -| Loop kernel + combinators | Write a driver (`plan`/`decide`) and run it (`runAgentRounds`), or compose fixed shapes: refine (`loopUntil`), best-of-N (`fanout`), chain (`pipeline`), multi-judge (`panel`) | `/kernel` | -| Improvement driver | Optimize one part of an agent and ship only if it wins on tasks it never practiced on (`improve`); production proposal/review/activation flow | root · `/intelligence` | -| Benchmarks + leaderboards | Compare strategies with significance stats (`runBenchmark`), stand up a harness×model leaderboard (`defineLeaderboard`, `leaderboard`) | `/kernel` | -| Knowledge improvement | Produce a measured candidate copy of a KB/wiki/RAG corpus without touching the live one (`runKnowledgeImprovementJob`) | `/knowledge` | -| MCP tool servers | Give an agent a `delegate` tool or live worker-coordination tools over MCP | `/mcp` | -| Conversations + durability | Multi-turn two-agent sessions with SQL-backed resume (D1/pg/sqlite/libSQL adapters) | `/conversation` | -| Training/eval adapter | Package the same runtime program as a PrimeIntellect Verifiers environment; import its traces back | `/primeintellect` | - -| Watching live runs | A terminal view of every supervisor run in a workspace — workers, spend, tokens, latency, plus steer and cancel (`agent-runtime-top`) | `/tui` | - -Remaining subpaths: `/agent`, `/profiles`, `/platform`, `/analyst-loop`, `/environment-provider`, `/testing` (validated fixture records for consumer tests). - -## Examples - -Runnable, grouped by what they show. Copy the one nearest your task: - -| Do this | Example | -|---|---| -| The smallest complete loop (start here) | [`quickstart`](./examples/quickstart) · [`driver-loop`](./examples/driver-loop) | -| Run a product chat turn | [`chat-handler`](./examples/chat-handler) | -| Drive a team of agents to a goal | [`supervise`](./examples/supervise) · [`recursive-supervisor`](./examples/recursive-supervisor) | -| Benchmark strategies on your own domain | [`coding-benchmark`](./examples/coding-benchmark) | -| Benchmark **harnesses × models** over a real task suite (the real WebCode dataset) | [`webcode-matrix`](./examples/webcode-matrix) | -| Render a **multi-profile leaderboard** with ranked board, score matrix, and SVG/HTML charts | `leaderboard(records)` → `renderLeaderboardMarkdown` / `Svg` / `Html` | -| Trace + bill + effort-gate the WebCode benchmark (the Intelligence SDK) | [`intelligence-webcode`](./examples/intelligence-webcode) | -| Self-improve an agent, gated on a held-out set | [`improve`](./examples/improve) · [`self-improving-coder`](./examples/self-improving-coder) | -| Improve a KB, wiki, or RAG corpus with runtime agents | [`docs/canonical-api.md`](./docs/canonical-api.md) | -| Evaluate or train a runtime program on PrimeIntellect | `@tangle-network/agent-runtime/primeintellect` | -| Study coordination vs raw compute | [`ablation-suite`](./examples/ablation-suite) | - -All 29 live in [`examples/`](./examples). - ## Where to go next -- New here? [`docs/concepts.md`](./docs/concepts.md), the mental model in plain terms. +- [`docs/concepts.md`](./docs/concepts.md), the mental model in plain terms. - [`docs/canonical-api.md`](./docs/canonical-api.md), find the primitive: "I want to ___ → use ___". - [`docs/api/primitive-catalog.md`](./docs/api/primitive-catalog.md), every export in one generated, never-stale list with its import path. Check it before building anything new. -- [`docs/STABILITY.md`](./docs/STABILITY.md), what `@stable` / `@experimental` promise you, and how a symbol graduates. -- [`docs/design.md`](./docs/design.md), the design philosophy and the internal research docs behind it: background reading, not required to use the package. +- [`docs/improve.md`](./docs/improve.md), the improvement reference: optimizers, surfaces, redaction, proposal, review, activation. +- [`docs/STABILITY.md`](./docs/STABILITY.md), what `@stable` and `@experimental` promise you, and how a symbol graduates. +- [`docs/design.md`](./docs/design.md), the design philosophy and the research behind it: background reading, not required to use the package. - [`bench/HARNESS.md`](./bench/HARNESS.md), the experiment harness and how to run a benchmark. **Contributing:** `pnpm i && pnpm build && pnpm test` gets you running; the full local gate is the [`package.json`](./package.json) scripts (`lint`, `typecheck`, `docs:check`). diff --git a/docs/api/index.md b/docs/api/index.md index bc27fc5b..94ce2014 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -8903,7 +8903,8 @@ The spawn label, when the node's `spawned` event is in this journal tree. > `readonly` `optional` **name?**: `string` -Stable identifier surfaced in trace events. Default `'driver'`. +Trace label surfaced in trace events. No behavioral effect: it never +selects a strategy or a decision path. Default `'driver'`. #### Methods @@ -8933,9 +8934,11 @@ readonly [`Iteration`](runtime.md#iteration-1)\<`Task`, `Output`\>[] > **decide**(`history`): `Decision` \| `Promise`\<`Decision`\> Inspect history and return the next state. The kernel terminates the -loop when `decide` returns a value listed in `isTerminalDecision` -(`'stop' | 'pick-winner' | 'fail' | 'done'`), when `maxIterations` -is hit, or when the abort signal fires. +loop when `decide` returns a `TerminalDecision` +(`'stop' | 'pick-winner' | 'fail' | 'done'`, exported as +`TERMINAL_DECISIONS` with the `isTerminalDecision` guard), when +`maxIterations` is hit, or when the abort signal fires. Every other +value is caller vocabulary and continues the loop. ###### Parameters diff --git a/docs/api/primitive-catalog.md b/docs/api/primitive-catalog.md index 33533c57..ed750ae7 100644 --- a/docs/api/primitive-catalog.md +++ b/docs/api/primitive-catalog.md @@ -525,7 +525,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. ### Execution kernel — recursive atom, supervision, executors, round-synchronous loop -Import from `@tangle-network/agent-runtime/kernel` — 767 exports. +Import from `@tangle-network/agent-runtime/kernel` — 770 exports. | Symbol | Kind | Summary | |---|---|---| @@ -624,6 +624,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 767 exports. | `harvestSurfaceDiffs` | function | Re-read every mounted (and watched) surface and report the ones whose settled state differs from | | `inlineSandboxClient` | function | Adapt an `ExecutorFactory` into a `SandboxClient` for `runAgentRounds`. The factory is | | `inProcessSandboxClient` | function | Adapt a single `onPrompt(prompt, ctx)` callback into a `SandboxClient` for | +| `isTerminalDecision` | function | True when the kernel stops the loop for this decision value. | | `isWaitOutcome` | function | Narrow a settlement's `out` to a wait outcome — a wait settles on the SAME cursor as workers, | | `jjWorkspace` | function | A jj-backed `Workspace` (Jujutsu, colocated with git for the durable remote). | | `kernelPromptRegistry` | function | The kernel's seeded registry: every surface the runtime's own builders derive from. A caller | @@ -782,6 +783,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 767 exports. | `strategyAuthorContract` | const | The compressed consumable a skill carries: everything an author needs to emit a loop. | | `strategyAuthorSystemPrompt` | const | Standing behavior callers put in the strategy-author AgentProfile. | | `supervisorPolicyPrompt` | const | THE supervisor policy — one stance, both front doors. The work-vs-delegate rule is conditional | +| `TERMINAL_DECISIONS` | const | Decision values the kernel treats as terminal. Every other value returned by | | `VERIFY_TAIL_CHARS` | const | Tail of the verify output — the failing assertion lives at the END of a test log. | | `WORKER_TOOL_TRACE_SCHEMA_VERSION` | const | Schema version for content-addressed worker tool-trace artifacts. | | `workerTraceSeamKey` | const | Seam key the `Scope` seeds a {@link TraceContext} under on each child's `ExecutorContext.seams`. | @@ -1130,6 +1132,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 767 exports. | `SupervisorSpanAttributes` | type | OTLP span attribute values. Exported because `SupervisorSpanOptions.attributes` is public and | | `SurfaceReader` | type | The read seam: fetch the current bytes at a mounted path. Implemented by a sandbox box's | | `SurfaceReadOutcome` | type | Outcome of reading one surface back at settle. `missing: true` means the path no longer exists | +| `TerminalDecision` | type | One of the kernel's terminal decision values. | | `ToolLoopCompactionOptions` | type | Public supervisor-facing compaction config: same knobs as the primitive, but `distill` is optional | | `ToolLoopMessageRecord` | type | Provider-neutral conversation record accepted by a tool-loop brain. | | `TrajectoryReportFn` | type | `trajectoryReport(...)` — the tree+cost reconstructor. Async (reads journal + optionally blobs). | diff --git a/docs/api/runtime.md b/docs/api/runtime.md index 09b69807..f76b3af9 100644 --- a/docs/api/runtime.md +++ b/docs/api/runtime.md @@ -19635,6 +19635,16 @@ A checkable task domain — implement these 5 hooks and the suite does the rest. *** +### TerminalDecision + +> **TerminalDecision** = *typeof* [`TERMINAL_DECISIONS`](#terminal_decisions)\[`number`\] + +**`Stable`** + +One of the kernel's terminal decision values. + +*** + ### Deliverable > **Deliverable**\<`Out`\> = \{ `kind`: `"events"`; `fromEvents`: (`events`) => `Out`; \} \| \{ `kind`: `"artifact"`; `path`: `string`; `fromArtifact`: (`raw`, `events`) => `Out`; \} @@ -21132,6 +21142,19 @@ The default registry `runPersonified` resolves a shape name against. Empty by co *** +### TERMINAL\_DECISIONS + +> `const` **TERMINAL\_DECISIONS**: readonly \[`"stop"`, `"pick-winner"`, `"fail"`, `"done"`\] + +**`Stable`** + +Decision values the kernel treats as terminal. Every other value returned by +`decide` continues the loop. Type a driver's `decide` return as +`'your-word' | TerminalDecision` so caller vocabulary and kernel keywords +stay visibly distinct. + +*** + ### strategyAuthorContract > `const` **strategyAuthorContract**: "\nYou author an OPTIMIZATION STRATEGY for an agentic loop system. A strategy decides how to\nspend a compute budget to beat a task's deployable check. You compose exactly two steps:\n\n shot(spec?: \{ handle?, messages?, steer?, persona?, tools? \}): Promise\\n Runs ONE worker attempt (a bounded tool loop) over an artifact.\n - omit handle =\> the shot opens its OWN fresh artifact and closes it after (a sample).\n - pass handle =\> the shot CONTINUES that artifact (state accumulates across shots).\n - messages =\> the carried conversation (pass the previous ShotResult.messages to continue).\n - steer =\> a corrective instruction injected before the shot.\n - persona =\> \{ systemPrompt?, model? \} — give THIS shot its own role and/or model\n (multi-agent strategies: a researcher shot then an engineer shot, a panel of k\n personas over one budget). On a fresh shot the systemPrompt replaces the task's; on\n a carried conversation it arrives as a hand-off message. Same conserved budget.\n - tools =\> string\[\] — restrict THIS shot to a subset of the task's tools by\n name (focus an explore shot on read-only tools, an execute shot on write tools).\n Restriction-only; unknown names make the shot fail. ALWAYS select from\n await listTools(handle) — never hardcode. Omitted =\> the shot sees every tool.\n ShotResult = \{ messages, score (0..1 on the task's check), passes, total, completions, toolErrors \}\n Returns null if the attempt failed infra-wise.\n\n critique(messages): Promise\\n A firewalled trace-analyst reads the attempt's trajectory and returns ONE corrective\n instruction (or null when it judges the work complete). Costs ~1 completion.\n\n consult(messages, instruction): Promise\\n The RAW analyst channel: the same firewalled critic answers YOUR instruction over the\n trajectory verbatim (no reformatting) — use it when you need a specific reply format\n (a decision, a prediction). Costs ~1 completion.\n\n surface.open(task) / surface.close(handle)\n Open a persistent artifact you manage yourself (remember to close in a finally).\n close is idempotent — closing an already-closed handle is a safe no-op.\n\n listTools(handle): Promise\\>\n The tools THIS task actually offers. TOOL SETS VARY PER TASK — if you restrict a\n shot with \`tools\`, you MUST pick names from await listTools(handle); hardcoding\n names from an example kills your shots on every task whose tools differ.\n\nRules:\n- ALWAYS await every shot/critique/surface call — a floating promise that rejects\n crashes the whole benchmark run.\n- Stay within ~budget total shots; every shot/critique spends from a conserved pool.\n- For a FRESH attempt OMIT \`messages\` entirely (never pass \`\[\]\` — an empty array is a\n fresh conversation too, but be explicit). To CONTINUE, pass the previous\n ShotResult.messages unchanged.\n- Return \{ score, resolved, completions, progression, shots \} — score = the BEST checkpoint\n you reached (keep-best, never final-state), progression = score after each shot.\n- The module must be EXACTLY this shape (no other imports, no commentary outside code):\n\nimport \{ defineStrategy \} from '@tangle-network/agent-runtime/kernel'\nexport default defineStrategy('your-strategy-name', async (\{ surface, task, budget, shot, critique, listTools \}) =\> \{\n // your composition (listTools comes from the destructured context — it is NOT a global)\n\})\n" @@ -23270,6 +23293,26 @@ a forked copy). *** +### isTerminalDecision() + +> **isTerminalDecision**(`decision`): decision is "stop" \| "done" \| "pick-winner" \| "fail" + +**`Stable`** + +True when the kernel stops the loop for this decision value. + +#### Parameters + +##### decision + +`unknown` + +#### Returns + +decision is "stop" \| "done" \| "pick-winner" \| "fail" + +*** + ### acquireSandbox() > **acquireSandbox**(`client`, `options`, `acquire?`): `Promise`\<`SandboxInstance`\> diff --git a/docs/improve.md b/docs/improve.md new file mode 100644 index 00000000..c79b7956 --- /dev/null +++ b/docs/improve.md @@ -0,0 +1,288 @@ +# Improve an agent + +`improve` runs one complete optimization method against one profile field. +The method owns candidate generation and selection. +Runtime keeps the final test set out of the method, scores the baseline and the selected candidate on it, and returns `ship` only when the paired confidence interval clears `minimumLift`. +The profile is never changed. + +The runnable offline path is [`examples/improve`](../examples/improve). +This page is the reference for the production path. + +## The call + +```ts +import { improve, officialGepa } from '@tangle-network/agent-runtime' +import { profileOptimizerModelCall } from '@tangle-network/agent-runtime/kernel' +import { + type AgentProfile, + canonicalAgentProfileDigest, + canonicalCandidateDigest, +} from '@tangle-network/agent-interface' + +const executionRef = canonicalCandidateDigest({ + deployment: process.env.AGENT_DEPLOYMENT_SHA!, + model: process.env.AGENT_MODEL!, + tools: process.env.AGENT_TOOLSET_SHA!, +}) + +const result = await improve(baseProfile, { + surface: 'prompt', + executionRef, + method: officialGepa({ + objective: 'Improve the complete support-agent prompt.', + recipe: { kind: 'engine', run: { engine: 'gepa', maxEvaluations: 40, maxProposerCostUsd: 10 } }, + optimizer, + resume: 'if-compatible', + trustResumeState: true, + describeScenario: ({ input }) => ({ input }), + }), + findings, + trainScenarios, + selectionScenarios, + testScenarios, + judges: [judge], + agent: (candidateProfile, scenario, ctx) => runProfile(candidateProfile, scenario, ctx), + runDir: '.runs/support-prompt', + costCeiling: 25, +}) + +if (result.decision === 'ship') console.log(result.candidate.profile, result.liftInterval) +``` + +## The optimizer object + +SkillOpt and GEPA's standard reflection engine require `optimizer: { model, call, callRef, budget }`. +Agent-based GEPA engines may own their model connection instead. +Runtime owns those model calls through one exact `AgentProfile`. +Agent Eval enforces the nested budget and records the measured cost and execution evidence without receiving provider credentials. + +```ts +const optimizerProfile = { + name: 'support-prompt-optimizer', + harness: 'cli-base', + model: { + provider: 'tangle-router', + default: process.env.OPTIMIZER_MODEL!, + metadata: { maxTokens: 16_384 }, + }, +} satisfies AgentProfile + +const optimizerPricing = { + inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION), + outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION), +} + +const optimizer = { + model: optimizerProfile.model.default, + call: profileOptimizerModelCall({ + profile: optimizerProfile, + context: 'support-prompt optimizer', + executor: { + backend: 'router', + routerBaseUrl: process.env.OPTIMIZER_BASE_URL!, + routerKey: process.env.OPTIMIZER_API_KEY!, + }, + pricing: optimizerPricing, + }), + callRef: canonicalCandidateDigest({ + profile: canonicalAgentProfileDigest(optimizerProfile), + deployment: process.env.OPTIMIZER_DEPLOYMENT_SHA!, + }), + budget: { + maxCostUsd: 10, + maxRequests: 50, + maxRequestBytes: 2_000_000, + maxResponseBytes: 2_000_000, + maxOutputTokensPerRequest: 16_384, + pricing: optimizerPricing, + }, +} +``` + +`costCeiling` is the total limit for optimizer calls, candidate runs, judges, and final scoring. +Runtime returns `hold` when any part of that cost is unknown. +Runtime rejects a reported total above the limit. + +## Official optimizers + +`officialGepa(...)` delegates the complete search to GEPA's upstream Optimize Anything API through agent-eval. +Pass one explicit `engine`, `sequential`, `adaptive-sequential`, `best-of`, `vote`, or `omni` recipe. +There is no local fallback. +Install its optional Python process first: + +```bash +python -m pip install "agent-eval-rpc==0.145.0" +python -m pip install "gepa[full]==0.1.4" +``` + +The published GEPA 0.1.4 wheel supports the direct `gepa` engine. +Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N require the tested official source revision: + +```bash +python -m pip install "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f" +``` + +Use `officialSkillOpt(...)` for Microsoft's SkillOpt: + +```bash +python -m pip install "agent-eval-rpc==0.145.0" +python -m pip install "skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90" +``` + +SkillOpt 0.2.0's published wheel omits prompt files that `ReflACTTrainer` requires, so the tested SkillOpt source revision stays necessary. + +### Resume and provenance + +Runtime derives the upstream resume identity from `executionRef`, the complete baseline profile, and the selected surface. +With `resume: 'if-compatible'`, agent-eval resumes only when the saved run identity matches the candidate, recipe, data, optimizer settings, runner, and derived execution identity. +Set `trustResumeState: true` only when that run directory is private to the current operator. +Use `resume: 'required'` to fail when no matching run exists. + +`result.provenance` reports the upstream package, run ID, resume status, evaluation count, and artifact directory. +`result.candidatePopulation` verifies and joins callback observations with an optimizer's official candidate graph. +It returns every unique candidate as a complete profile with ordered Interface diffs, or as an explicit materialization refusal. +GEPA candidates keep exact parent indices and selection scores; callback-only proposals report lineage as unavailable. +Methods without either artifact return `status: 'unavailable'` instead of treating the winner as the full population. + +## Surfaces + +SkillOpt accepts one text surface. +GEPA accepts text or named components. +Any complete method from `@tangle-network/agent-eval` uses the same call. + +- For a skill, set `surface: 'skills'` and `skills.resourceName`. +- For the complete profile, set `surface: 'agent-profile'`. +- To optimize several named profile fields together, also provide `profileComponents.read` and `profileComponents.apply`. + +Tools, MCP, hooks, subagents, curated instructions, and rollout policy are also exact profile coordinates. +Runtime does not choose an optimizer for them. + +The `agent` callback receives the complete immutable candidate profile, not a raw prompt or a component fragment. +Runtime uses that exact profile for every candidate run and returns the same measured profile in `result.candidate.profile`. +`executionRef` is a content digest of the agent callback, profile component mapping, model, tools, and closure settings. +Runtime combines it with the complete baseline profile and the selected surface for saved work. +A change to any of them runs the affected work again. + +Code is the exception. +It uses Runtime's isolated git worktrees and coding-agent candidate execution: + +```ts +const result = await improve({ + surface: 'code', + code: { repoRoot, baseRef, profile, generator }, + scenarios, + judge, + agent, + budget, +}) +``` + +## What leaves your process + +Without `describeScenario`, the external optimizer receives only each development case ID. +Without `describeArtifact`, evaluation feedback contains no artifact body. +When either descriptor is present, its result passes through `redact` together with findings, background text, profile name, and judge notes. +The built-in redactor removes common credentials and email addresses. +Supply a domain redactor for customer names, account IDs, or other private data the built-in rules cannot identify. +Runtime applies that hook first and then still applies its built-in scrubber. +Set `redact: false` only when every outbound value is public and already reviewed. + +The selected profile surface is the optimizer's candidate and cannot be redacted without changing the measured candidate. +Runtime always rejects recognized credentials in those bytes. +It also rejects structurally sensitive fields such as MCP env, headers, URLs, metadata, and extensions. +For `tools`, `mcp`, `hooks`, `subagents`, and `agent-profile`, Runtime treats the entire selected coordinate as execution-capable. +Use `authorizeSensitiveCandidate` to inspect and accept each exact immutable profile that contains public values or safe references. +The callback runs for the baseline and every distinct candidate before either reaches your agent. +Its `sensitivePaths` includes `$` when the whole coordinate requires review. + +## From search to production + +`improve` is the search call. +For production, `proposeAgentImprovement` adds trace analysis and reruns the exact frozen baseline and winner before it creates a reviewable proposal. +Runtime rejects a candidate bundle that differs from the search winner. + +```ts +import { + createAgentImprovementActivation, + executeAgentImprovementActivation, + proposeAgentImprovement, + reviewAgentImprovementProposal, +} from '@tangle-network/agent-runtime/intelligence' + +const baseline = freezeBaseline(liveProfile) +const result = await proposeAgentImprovement({ + runId, + profile: liveProfile, + analysis, + improvement: { + surface: 'prompt', + executionRef, + method, + trainScenarios, + selectionScenarios, + testScenarios, + judges: [judge], + agent, + }, + buildExperiment: ({ improvement }) => + buildExperimentMaterial({ + baseline, + candidate: compileCandidateBundle({ baseline, improvement: improvement.candidate }), + benchmark: heldOutBenchmark, + policy: comparisonPolicy, + }), + placeCell, +}) + +const review = reviewAgentImprovementProposal(result.proposal, { + decision: 'approve', + reviewedBy: user.id, + reason: 'The measured gain is worth the cost.', +}) +const activation = createAgentImprovementActivation(result.proposal, review, { + intent: 'activate-candidate', + targets: [{ surface: 'prompt', identity: profileId }], + fundingOwner: tenantId, + authorizedBy: user.id, + expiresAt, +}) +const outcome = await executeAgentImprovementActivation( + { proposal: result.proposal, review, activation }, + { transition: commitProfileTransaction, reconcile: readCommittedResult }, +) +``` + +`buildExperimentMaterial`, `placeCell`, and the transaction functions are application ports, because storage and compute differ by product. +The builder returns only baseline, candidate, tasks, and policy; Runtime adds the search ancestry and seals the final experiment. +Runtime owns candidate identity, measurement, review binding, expiry, retry identity, and result validation; the application owns its atomic write. +Official optimizer proposals carry the observed package versions, the optimizer model, evaluation and token usage, separate optimization and final-test costs, and the resumed-run identity. +`createOptimizationActivationReceipt(result)` exposes the same detached record for a caller that must inspect an `improve()` result before it builds a proposal. + +## Improve a knowledge base + +`runKnowledgeImprovementJob` runs KB, wiki, memory-backed, and RAG improvement jobs. +It creates a candidate copy, runs agents against it, checks it through `@tangle-network/agent-knowledge`, and returns frozen baseline and candidate snapshots with spend and timing. +It never changes the live knowledge base. + +Use `improve(profile, { surface: 'memory', ... })` for the agent's curated lesson document. +Use this job for source, retrieval, and knowledge-store changes. + +```ts +import { runKnowledgeImprovementJob } from '@tangle-network/agent-runtime/knowledge' + +const result = await runKnowledgeImprovementJob({ + root: './kb', + goal: 'Improve support refund-policy knowledge', + implementationRef: 'git:0123456789abcdef0123456789abcdef01234567', + readinessSpecs, + budget: { maxIterations: 8, maxTokens: 120_000, maxUsd: 10 }, + backend, +}) + +console.log(result.knowledge?.reference.candidateHash, result.measurement.supervisedSpent) +``` + +Set `implementationRef` to the deployed `git:<40 hex>` revision, or to a `sha256:<64 hex>` digest that covers every callback, model, index, and external setting that can change the result. +The same run ID resumes only when this identity still matches. +Measure the returned bundle pair, record the review, then activate through `executeAgentImprovementActivation`. +Activation is the only write path. diff --git a/docs/primeintellect.md b/docs/primeintellect.md new file mode 100644 index 00000000..c64ba1eb --- /dev/null +++ b/docs/primeintellect.md @@ -0,0 +1,76 @@ +# Run on PrimeIntellect + +`@tangle-network/agent-runtime/primeintellect` packages typed train and eval tasks as a PrimeIntellect Verifiers environment. +Prime launches your actual runtime program against an intercepted model endpoint, so `runPersonified`, `runAgentic`, product agents, tool calls, and multiple rounds stay intact. +Reference answers stay in Prime's task process and never enter the agent workspace. +The runner file must be one executable bundle that contains the app and its runtime dependencies. + +## Package the tasks + +```ts +import { readFile } from 'node:fs/promises' +import { + createPrimeIntellectPackage, + writePrimeIntellectPackage, +} from '@tangle-network/agent-runtime/primeintellect' + +const bundledRunner = await readFile('./dist/prime-runner.mjs', 'utf8') +const bundle = createPrimeIntellectPackage({ + name: 'support-agent', + version: '1.0.0', + tasks: [ + { + id: 'train-refund-policy', + split: 'train', + prompt: 'Can a subscription renewal be refunded?', + answer: 'No', + }, + { + id: 'eval-final-sale', + split: 'eval', + prompt: 'Can a final-sale order be refunded?', + answer: 'No', + }, + ], + scoring: { kind: 'exact', normalization: 'trim-casefold' }, + runner: { + image: 'node:22-bookworm-slim', + files: { 'runner.mjs': bundledRunner }, + command: ['node', 'runner.mjs'], + }, +}) + +await writePrimeIntellectPackage(bundle, './prime/support-agent') +``` + +## Write the runner + +The runner reads the episode and uses the normal runtime APIs. +Here, `runProductAgent` is the application's existing entry point, not another loop supplied by this adapter. + +```ts +import { + primeIntellectExecutorConfig, + runPrimeIntellectProgram, +} from '@tangle-network/agent-runtime/primeintellect' +import { + collectAgentTurn, + createExecutor, + streamAgentTurn, +} from '@tangle-network/agent-runtime/kernel' + +await runPrimeIntellectProgram(async (episode) => { + const profile = makeProductProfile({ model: episode.model.name }) + return collectAgentTurn( + streamAgentTurn( + { kind: 'executor', profile, factory: createExecutor(primeIntellectExecutorConfig(episode)) }, + episode.task.prompt, + ), + ) +}) +``` + +## Read the traces back + +Prime writes complete `traces.jsonl` rows. +Use `importPrimeIntellectTraces(...)` to convert them to agent-eval `RunRecord`s for the existing reports and release checks. diff --git a/examples/README.md b/examples/README.md index 85c2cbc4..1e6321de 100644 --- a/examples/README.md +++ b/examples/README.md @@ -5,17 +5,33 @@ budget, score them against a real check, and let them improve from their own run imports from the published package (`@tangle-network/agent-runtime`) exactly as your code would, and most run **offline with no API key** so you can see the machinery before spending anything. -New here? Run these three, in order (the first two cost $0): +## The front doors + +One directory per entry point, ordered by how often real products use it. Each carries a README with +three sections: when to use it, how to use it, and why it exists. + +| Front door | Call | Runs offline | +|---|---|---| +| [`stream-a-turn/`](./stream-a-turn/) | `runAgentTaskStream` — one turn, and you read the events | yes | +| [`chat-handler/`](./chat-handler/) | `handleChatTurn` — stream one turn over HTTP and save the reply | yes | +| [`stream-backends/`](./stream-backends/) | `AgentExecutionBackend` — choose where the tokens come from | first two sections | +| [`tool-loop/`](./tool-loop/) | `runToolLoop` — the model calls your tools, then answers | yes | +| [`runtime-run/`](./runtime-run/) | `startRuntimeRun` — the cost tally and one persisted row | yes | +| [`quickstart/`](./quickstart/) | `runAgentRounds` — several attempts under your own stop rule | yes | +| [`supervise/`](./supervise/) | `supervise` — a model decides the plan and drives other agents | needs `TANGLE_API_KEY` | +| [`retained-run/`](./retained-run/) | `startRetainedRun` — a job that outlives your process | compile-checked only | +| [`improve/`](./improve/) | `improve` — change one profile field and prove the gain | yes | + +New here? Run these three, in order (all cost $0): ```bash -pnpm tsx examples/driver-loop/driver-loop.ts # 1. one agent steering another — offline -pnpm tsx examples/improve/improve.ts # 2. an agent that rewrites its own prompt, safely — offline -TANGLE_API_KEY=... pnpm tsx examples/supervise/supervise.ts # 3. one function call = a supervisor over real workers +pnpm tsx examples/stream-a-turn/stream-a-turn.ts # 1. the smallest real path: one turn, its events +pnpm tsx examples/driver-loop/driver-loop.ts # 2. one agent steering another — see the fold +pnpm tsx examples/improve/improve.ts # 3. an agent that rewrites its own prompt, safely ``` -`driver-loop` is the core move everything else builds on; `improve` is the self-improvement primitive; -`supervise` is the one-call product entry point. [`quickstart/`](./quickstart/) is the same loop as -`driver-loop` compressed into one un-annotated file — it is the quickstart shown in the root README. +[`quickstart/`](./quickstart/) holds the root README's `minimal.ts` plus the same loop grown into a +refine loop; [`driver-loop/`](./driver-loop/) is that loop with every seam annotated. ## A few words that appear everywhere @@ -57,7 +73,6 @@ repeat. A failing validator prunes a bad candidate so the loop can't keep it. | # | Example | What it shows | |---|---|---| | 8 | [`researcher-loop/`](./researcher-loop/) | A research agent whose validator hard-fails if one tenant's data leaks into another's namespace, so the leak is pruned automatically. Needs the optional `@tangle-network/agent-knowledge` peer installed. | -| 9 | [`ui-audit/`](./ui-audit/) | The smallest end-to-end loop over a real browser (Playwright) with a stub judge, persisting the findings. | | 9b | [`coding-benchmark/`](./coding-benchmark/) | Rank coding agents (Claude Code, opencode, Codex, a bare CLI) on real tasks with an **anti-cheat**: each agent is graded on hidden tests it never saw, so it can't hardcode the answer. Includes a secondary quality judge and real significance stats. Offline by default; `--live` uses real agent boxes. | | 9c | [`webcode-matrix/`](./webcode-matrix/) | The real WebCode benchmark (Exa's 33-task dataset, graded by its own hidden tests) across a harness × model grid, rendered as a publishable leaderboard with charts, confidence bands, and pairwise significance. | @@ -65,6 +80,9 @@ repeat. A failing validator prunes a bad candidate so the loop can't keep it. | # | Example | What it shows | |---|---|---| +| 9d | [`stream-a-turn/`](./stream-a-turn/) | One turn end to end: write an `AgentExecutionBackend`, run it through `runAgentTaskStream`, and read the `RuntimeStreamEvent` union back. Offline. | +| 9e | [`tool-loop/`](./tool-loop/) | One chat turn where the model calls your tools, each result folds back, and the turn re-runs until it stops. Offline. | +| 9f | [`retained-run/`](./retained-run/) | A job the provider owns: persist the claim ticket, then rebuild control in a fresh process. Compile-checked; needs your own provider to run. | | 10 | [`knowledge-gating/`](./knowledge-gating/) | Stop an agent before it acts on facts it isn't confident about: the loop blocks when a required-knowledge confidence is below threshold. | | 11 | [`runtime-run/`](./runtime-run/) | The run-record + cost-ledger you persist for dashboards — one row per run, any database. | | 12 | [`stream-backends/`](./stream-backends/) | Pick where an agent's streaming output comes from (in-process iterator, cloud sandbox, or an OpenAI-compatible endpoint) behind one wire format. The OpenAI path needs `OPENAI_API_KEY`; the rest is offline. | @@ -115,6 +133,8 @@ From the repo root: ```bash # Start here +pnpm tsx examples/stream-a-turn/stream-a-turn.ts +pnpm tsx examples/tool-loop/tool-loop.ts pnpm tsx examples/chat-handler/chat-handler.ts pnpm tsx examples/strategy-suite/strategy-suite.ts # offline; TANGLE_API_KEY swaps in the real model pnpm tsx examples/recursive-supervisor/recursive-supervisor.ts diff --git a/examples/chat-handler/README.md b/examples/chat-handler/README.md index 19c4422e..9b2bf5df 100644 --- a/examples/chat-handler/README.md +++ b/examples/chat-handler/README.md @@ -1,75 +1,64 @@ -# Stream an agent's reply over HTTP, then save it — in one call +# Serve one chat turn over HTTP -You have an agent that emits its reply token-by-token. You want to serve that to a browser as a -live, streaming HTTP response, and once the reply finishes, save the final text to your database. -`handleChatTurn` (from the edge-safe `@tangle-network/agent-runtime/durable` entry point) does exactly that plumbing so you -don't hand-roll it. +## When to use it -Give it two things — a `produce()` function that yields your agent's token stream, and a -`persistAssistantMessage` hook — and it hands back a ready-to-return HTTP body that streams every -token as it arrives, wrapped in a clean start/finish envelope, and calls your save hook the instant -the stream ends. +Use this when a web route must stream one turn to a browser and save the reply. +`handleChatTurn` owns the framing every chat product hand-rolls: NDJSON lines, a start and finish envelope, and one persist call after the last token. +You give it how to produce the tokens and how to save the final text. -## Why it matters +Use a sibling instead when you do not need the HTTP layer. -Every chat product re-implements the same fiddly middle layer: turn an async stream of partial -tokens into a well-framed HTTP response, tell the client when the turn starts and ends, and persist -the final message exactly once after the last token. Get the framing wrong and the client hangs, -double-saves, or drops the tail. `handleChatTurn` is that layer, done once and correctly: - -- **Streams as NDJSON** — one JSON event per line (`application/x-ndjson`), so a browser or any HTTP - client reads tokens live instead of waiting for the whole reply. -- **Wraps each turn** in a `session.run.started` / `session.run.completed` envelope, so the client - always knows when a turn begins and ends. -- **Persists after the stream drains**, exactly once — your `persistAssistantMessage` hook fires - with the final text after the last token, the natural place to write to your DB. +| You need | Use | +|---|---| +| The event stream, with no HTTP framing | [`../stream-a-turn`](../stream-a-turn) | +| The model to call your tools inside the turn | [`../tool-loop`](../tool-loop) | +| A cost figure and one persisted row | [`../runtime-run`](../runtime-run) | +| A turn that survives the reader disconnecting | [`../retained-run`](../retained-run) | -## Run it — no key, no network +## How to use it ```bash -pnpm tsx examples/chat-handler/chat-handler.ts +pnpm build && pnpm tsx examples/chat-handler/chat-handler.ts ``` -The example scripts a tiny two-turn tax-assistant conversation so it runs fully offline. You'll see -each turn start, stream (one `.` per token chunk), finish, persist, and print its final text: +The example scripts a two-turn conversation, so it runs offline and needs no credentials. +It prints each turn as it starts, streams, finishes, and persists: -``` +```text [run started ] turn=0 ............... [run done ] turn=0 [persist ] turn=0 chars=52 [turn 0 text ] Acknowledged: "Where do I start with my 2026 return?". Drafting a reply. - -[run started ] turn=1 -................... -[run done ] turn=1 -[persist ] turn=1 chars=62 -[turn 1 text ] The 2026 return is missing Schedule B and one W-2. Please upload them. ``` -The bottom half of `chat-handler.ts` reads the stream back by hand only so the example is -self-contained — that reader is illustrative, not something to copy. In your product you just return -`result.body` from your HTTP route and any NDJSON reader on the client consumes it. - -## From offline to production — one swap - -The only scripted part is `produce()`. In a real product it wraps a profile-bound Runtime turn: +In a route you return `result.body` and let the client read the NDJSON. ```ts -produce: () => streamAgentTurn( - { kind: 'executor', profile, factory: createExecutor(executorConfig) }, - input, -) +const executionId = deriveExecutionId({ projectId, sessionId: threadId, turnIndex }) +const result = handleChatTurn({ + identity: { tenantId, sessionId: threadId, userId, turnIndex }, + hooks: { + produce: () => ({ stream: box.streamPrompt(userMessage, { sessionId: threadId, executionId, turnId: executionId, detach: true }), finalText: () => box.lastResponse() }), + persistAssistantMessage: async ({ identity, finalText }) => db.insertMessage(identity, finalText), + }, + waitUntil, +}) +return new Response(result.body, { headers: { 'content-type': result.contentType } }) ``` -The exact `AgentProfile` selects the model and tools; the Runtime executor owns credentials, -routing, retries, and usage evidence. +Only `produce()` is scripted in the example. +In production it wraps a profile-bound turn: `streamAgentTurn({ kind: 'executor', profile, factory: createExecutor(config) }, input)`. -Everything else — the NDJSON framing, the `session.run.*` envelope, the after-drain persist — stays -identical. That framing is the whole point of `handleChatTurn`. +Two identity rules matter. +For a stream reconnect, call `streamPrompt` again with the same `executionId` and the last event id the client received. +For a repeated first dispatch, reuse both `sessionId` and `turnId`, because `executionId` alone is not an idempotency key. -## Files +The bottom half of [`chat-handler.ts`](./chat-handler.ts) reads the stream back by hand only to keep the file self-contained. +Do not copy that reader. -| file | what it is | -|---|---| -| `chat-handler.ts` | the full example: a scripted producer, `handleChatTurn`, and a hand-written reader that prints the stream | +## Why this exists + +The middle layer of a chat product is small, fiddly, and always the same. +Bad framing makes the client hang, save twice, or drop the tail of a reply. +`handleChatTurn` does that layer once: one JSON event per line, a `session.run.started` and `session.run.completed` envelope, and one persist call after the stream drains. diff --git a/examples/improve/README.md b/examples/improve/README.md index 7bd6c027..e98701a4 100644 --- a/examples/improve/README.md +++ b/examples/improve/README.md @@ -1,52 +1,50 @@ -# Improve one agent profile field +# Improve one profile field -`improve()` runs a complete optimization method against one profile field. -The method receives train and selection cases. -Runtime keeps the final-test cases private, compares the selected candidate with the baseline, and returns a detached candidate. -It never changes the input profile. +## When to use it -```bash -pnpm tsx examples/improve/improve.ts -``` +Use this when you must change one part of an agent and prove the change is better. +`improve()` runs a complete optimization method against one profile field, such as the prompt, a skill, memory, or code. +Runtime keeps the final-test cases away from the method, compares the selected candidate with the baseline, and returns a detached candidate. +It never changes the input profile. -Runs offline, no credentials. +Use a sibling instead when the job is different. +[`../self-improving-loop`](../self-improving-loop) unrolls the same flow step by step, so you can see which part owns each phase. +[`../self-improving-coder`](../self-improving-coder) runs it on a coding task graded by real tests. +[`../strategy-evolution`](../strategy-evolution) searches coordination tactics instead of a profile field. +[`../intelligence-recommend`](../intelligence-recommend) starts from a run trace and ends at a reviewable proposal. -## What it does, step by step +## How to use it -1. Runtime extracts the exact profile field selected by `surface`. -2. Runtime binds saved work to `executionRef` plus the complete baseline profile. -3. The supplied `OptimizationMethod` generates and selects a candidate using train and selection cases. -4. Runtime scores the baseline and candidate on the untouched final-test cases. -5. Runtime returns `ship` only when the paired confidence interval clears the required lift. -6. Approval and activation remain separate operations. +```bash +pnpm build && pnpm tsx examples/improve/improve.ts +``` -## What you'll see +The example runs offline. +It supplies a deterministic method, agent, and judge: the method returns `PROMOTED`, and the judge scores that literal string as `1`. +The partition firewall, the final comparison, the cost receipts, and the confidence interval are production code. -``` +```text improve() proposed a detached prompt candidate and measured it on final-test scenarios decision: ship lift: 1.000 candidate prompt: PROMOTED live prompt unchanged: BASELINE ``` -The starting prompt is `BASELINE`; the candidate is `PROMOTED`. -The final-test lift is `1.000`. - -## How it stays offline - -The example supplies a deterministic complete method, agent, and judge. -The method returns `PROMOTED`; the judge scores that literal string as `1`. -The partition firewall, final comparison, cost receipts, and confidence interval are production code. +Six steps run inside the call. -## Going live - -Replace `scriptedWinner` with `officialGepa(...)`, `officialSkillOpt(...)`, or another complete method from `@tangle-network/agent-eval`. -The root README documents the optional Python installation for official GEPA. +1. Runtime extracts the exact profile field named by `surface`. +2. Runtime binds saved work to `executionRef` plus the complete baseline profile. +3. The method generates and selects a candidate from the train and selection cases. +4. Runtime scores the baseline and the candidate on the untouched final-test cases. +5. Runtime returns `ship` only when the paired confidence interval clears the required lift. +6. Approval and activation stay separate operations. -## Files +To go live, replace the scripted method with `officialGepa(...)`, `officialSkillOpt(...)`, or another complete method from `@tangle-network/agent-eval`. +Those methods need an optimizer object, an optional Python process, and a redaction review. +[`docs/improve.md`](../../docs/improve.md) is the reference for all of it, including the production proposal, review, and activation path. -| file | what it is | -|---|---| -| `improve.ts` | The profile, complete method, three partitions, agent, judge, and result | +## Why this exists -The same path is covered by `src/improvement/improve.test.ts`. +A prompt edit that looks better is not a measured gain. +This call separates the three sets — train, selection, and final test — so the method never sees the cases that decide the release. +The result is a detached candidate plus an interval, so a human approves a number instead of a hunch. diff --git a/examples/quickstart/README.md b/examples/quickstart/README.md index 2b098d07..f1c02694 100644 --- a/examples/quickstart/README.md +++ b/examples/quickstart/README.md @@ -1,13 +1,51 @@ -# quickstart +# Run a scripted loop -The smallest complete agent loop, in one file: a driver runs a worker, reads the worker's real output, and writes the next prompt from it until a check passes. -Offline and deterministic — the worker is a scripted stand-in (`inProcessSandboxClient`), so it runs with zero credentials. +## When to use it + +Use this when one prompt is not enough, and you must run several attempts under your own rules. +A driver runs a worker, reads its output, and writes the next prompt until a check passes. +You own the rules: `plan` returns the tasks to run this iteration, and `decide` says whether the loop continues. + +Use a sibling instead when the shape is different. + +| Your shape | Use | +|---|---| +| One turn, read the events yourself | [`../stream-a-turn`](../stream-a-turn) | +| One turn where the model calls your tools | [`../tool-loop`](../tool-loop) | +| A model decides what to do next, not your code | [`../supervise`](../supervise) | +| A job that must outlive your process | [`../retained-run`](../retained-run) | + +## How to use it + +Two files, both offline and deterministic. +The worker is a scripted stand-in, so neither needs credentials. ```bash -pnpm build && pnpm tsx examples/quickstart/quickstart.ts +pnpm build +pnpm tsx examples/quickstart/minimal.ts # the smallest call the types accept +pnpm tsx examples/quickstart/quickstart.ts # the same call, refining until a check passes +``` + +[`minimal.ts`](./minimal.ts) is the root README quickstart, kept compiling by `pnpm typecheck:examples`. +It prints: + +```text +decision: done — 1 iteration(s) +``` + +[`quickstart.ts`](./quickstart.ts) adds the fold: it reads the last output and writes the next prompt from it. + +```ts +plan: async (task, history) => { + const last = history[history.length - 1] + if (!last) return [task] // shot 0: run the task as written + if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots + // The core move: read the last worker's real output, write the next prompt FROM it. + return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] +}, ``` -Expected output: +It prints: ```text shot 0: reject — "Shipped one-click restore." @@ -15,5 +53,18 @@ shot 1: PASS — "Shipped one-click restore with an instant rollback path." decision: pick-winner — winner: shot 1 ``` -This is the loop the root README shows. -The fully annotated version, with every seam explained (the fold, terminal decisions, the output adapter, the validator), is [`../driver-loop`](../driver-loop). +Two rules decide when the loop stops. +`plan` returns `[]` when it has no more work. +`decide` returns a value: the four keywords in `TERMINAL_DECISIONS` (`stop`, `pick-winner`, `fail`, `done`) end the loop, and every other value is your own vocabulary and continues it. +Type the return as `'your-word' | TerminalDecision` to keep the two apart. + +`driver.name` is a trace label. +It never selects a strategy or a decision path. + +The annotated version of the same loop, with every seam explained, is [`../driver-loop`](../driver-loop). + +## Why this exists + +A retry that sends the same prompt again learns nothing. +This loop reads what the worker produced, scores it with your check, and writes the next prompt from the real output. +The kernel owns the parts that are easy to get wrong — one fresh worker per attempt, a hard iteration cap, and teardown of every worker at the end — so your driver stays a few lines of plain code. diff --git a/examples/quickstart/minimal.ts b/examples/quickstart/minimal.ts new file mode 100644 index 00000000..fbfad70a --- /dev/null +++ b/examples/quickstart/minimal.ts @@ -0,0 +1,42 @@ +/** + * minimal — the root-README quickstart, kept compiling by `pnpm typecheck:examples`. + * One worker attempt, parsed and returned. Offline, deterministic, no API keys. + * + * Run: pnpm build && pnpm tsx examples/quickstart/minimal.ts + */ + +import type { AgentProfile } from '@tangle-network/agent-interface' +import { + inProcessSandboxClient, + runAgentRounds, + type TerminalDecision, +} from '@tangle-network/agent-runtime/kernel' +import type { SandboxEvent } from '@tangle-network/sandbox' + +const profile = { + name: 'note-writer', + harness: 'cli-base', + model: { provider: 'scripted', default: 'scripted/note-writer' }, +} satisfies AgentProfile + +// A scripted worker. Swap in a sandbox, CLI-harness, or router backend later. +const worker = inProcessSandboxClient({ + onPrompt: (): SandboxEvent[] => [ + { type: 'result', data: { result: { note: 'Shipped one-click restore.' } } }, + ], +}) + +const result = await runAgentRounds({ + task: 'Write a one-line release note for one-click restore.', + driver: { + // plan returns the tasks to run this iteration; [] means no more work. + plan: async (task, history) => (history.length === 0 ? [task] : []), + // 'done' is one of the four kernel keywords in TERMINAL_DECISIONS. + decide: (): TerminalDecision => 'done', + }, + agentRun: { profile, taskToPrompt: (t) => t }, + output: { parse: (events) => events }, + ctx: { sandboxClient: worker }, +}) + +console.log(`decision: ${result.decision} — ${result.iterations.length} iteration(s)`) diff --git a/examples/quickstart/quickstart.ts b/examples/quickstart/quickstart.ts index 1fa6f0ad..e1d69989 100644 --- a/examples/quickstart/quickstart.ts +++ b/examples/quickstart/quickstart.ts @@ -1,5 +1,5 @@ /** - * quickstart — the smallest complete agent loop. Offline, deterministic, no API keys. + * quickstart — the smallest complete refine loop. Offline, deterministic, no API keys. * * One driver runs a worker, reads the worker's real output, and writes the next prompt * from it until a check passes. The worker here is a scripted stand-in so the loop runs @@ -11,7 +11,11 @@ */ import type { AgentProfile } from '@tangle-network/agent-interface' -import { inProcessSandboxClient, runAgentRounds } from '@tangle-network/agent-runtime/kernel' +import { + inProcessSandboxClient, + runAgentRounds, + type TerminalDecision, +} from '@tangle-network/agent-runtime/kernel' import type { SandboxEvent } from '@tangle-network/sandbox' type Task = { prompt: string } @@ -39,27 +43,11 @@ const worker = inProcessSandboxClient({ ], }) -const result = await runAgentRounds({ - task: { prompt: 'Write a one-line release note for one-click restore.' }, - driver: { - name: 'refine', - plan: async (task, history) => { - const last = history[history.length - 1] - if (!last) return [task] // shot 0: run the task as written - if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots - // The core move: read the last worker's real output, write the next prompt FROM it. - return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] - }, - decide: (history) => - history.some((shot) => shot.verdict?.valid) - ? 'pick-winner' - : history.length < 3 - ? 'refine' - : 'fail', - }, +const result = await runAgentRounds({ + task: { prompt: 'Write a one-line release note for one-click restore.' } as Task, agentRun: { profile: noteWriterProfile, taskToPrompt: (t) => t.prompt }, output: { - parse: (events) => { + parse: (events): Note => { for (const ev of events) { if (ev.type === 'result') { const r = (ev as { data?: { result?: unknown } }).data?.result @@ -75,8 +63,26 @@ const result = await runAgentRounds { + const last = history[history.length - 1] + if (!last) return [task] // shot 0: run the task as written + if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots + // The core move: read the last worker's real output, write the next prompt FROM it. + return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] + }, + // 'refine' is this driver's own word — any non-terminal value continues the + // loop. 'pick-winner' and 'fail' are kernel keywords from TERMINAL_DECISIONS. + decide: (history): 'refine' | TerminalDecision => + history.some((shot) => shot.verdict?.valid) + ? 'pick-winner' + : history.length < 3 + ? 'refine' + : 'fail', + }, ctx: { sandboxClient: worker }, - maxIterations: 3, }) for (const shot of result.iterations) { diff --git a/examples/retained-run/README.md b/examples/retained-run/README.md new file mode 100644 index 00000000..89156345 --- /dev/null +++ b/examples/retained-run/README.md @@ -0,0 +1,72 @@ +# Keep a run alive after your process dies + +## When to use it + +Use this when the job must outlive the process that started it. +The provider owns the job; you own a claim ticket that any process can present. + +The runtime has five ways to continue work. +Pick by what died. + +| What died | What continues it | +|---|---| +| The HTTP connection to the browser | Call `streamPrompt` again with the same `executionId` and the last event id | +| Nothing; you want the same box for the next turn | `openSandboxRun` | +| The coordinator process, mid-orchestration | `supervise({ runDir })`, which replays settled children | +| The user left and came back to a chat | The `/conversation` store adapters | +| Everything except the provider | **This example**: `startRetainedRun` and `reconnectRetainedRun` | + +The last row is the expensive one. +Use it only when a dropped reader or a restarted application must not lose the job. + +## How to use it + +```bash +pnpm typecheck:examples # this file is compile-checked, not runnable offline +``` + +`startRetainedRun` refuses a provider that cannot promise exact run identity, event and result identity, idempotent cancellation, detach, and replay. +No provider in this repository advertises those seven capabilities, so supply your own provider to run [`retained-run.ts`](./retained-run.ts). + +Process A starts the job and persists the ticket. + +```ts +const run = await startRetainedRun({ + provider, + environment: { idempotencyKey: 'workspace-42', profile }, + turn: { turnId: 'turn-7', prompt: 'Finish the migration and run its tests.' }, + identity: { sessionId: 'thread-42', executionId: 'execution-7' }, + onAdmission: async (admission) => { + await journal.write(admission) + }, +}) +``` + +The runtime awaits `onAdmission` twice: after the environment exists, and again after the dispatch is verified. +The start promise resolves only after the second record is durable. +Write each record inside the hook before the hook returns. + +Process B rebuilds control from the persisted ticket. + +```ts +const handle = await reconnectRetainedRun({ provider, controlRef: dispatched.controlRef }) +const snapshot = await handle.status({ waitMs: 30_000 }) +const result = await handle.result() +``` + +Three rules keep a restart safe. + +- Persist each event cursor and sequence before you show the event to a user. +- After a crash that landed only the environment record, call `recoverRetainedRun` with its coordinates. +- Never destroy an environment on the `unverifiable` outcome. Keep it, retry the reconnect later, or inspect it with the provider's tools. + +An unknown provider result stays unknown. +The runtime never reports it as success or as a confirmed cancellation. + +## Why this exists + +`runAgentRounds` is a loop your process runs. +It holds your `plan`, `decide`, `parse`, and `validate` functions in memory, so when your process dies the loop dies and nothing outside ever knew it existed. +A retained run is a job the provider runs: your process holds only a claim ticket — provider, environment, session, execution, and request digest — so any process holding the ticket can reattach, replay, or cancel, which is why every read is verified against the ticket instead of trusted from memory. +It is a separate call because it must refuse providers that cannot honor the ticket, and it must force you to save the ticket durably before it reports success. +Neither demand can be bolted onto the in-process loop without breaking its callers or reopening the crash-orphan bug it was built to close. diff --git a/examples/retained-run/retained-run.ts b/examples/retained-run/retained-run.ts new file mode 100644 index 00000000..6a5c4a1e --- /dev/null +++ b/examples/retained-run/retained-run.ts @@ -0,0 +1,112 @@ +/** + * retained-run — hold a claim ticket to a job the provider runs. + * + * `runAgentRounds` holds your closures, so the run dies with your process. A + * retained run lives in the provider. You keep only a plain-data control + * reference, and any process that holds it can replay, resume, or cancel. + * + * The two functions below are the copyable shape: start and persist the ticket, + * then rebuild control from the persisted ticket in a fresh process. + * + * This file is compile-checked by `pnpm typecheck:examples`. It does not run + * offline: `startRetainedRun` refuses a provider that cannot promise exact run + * identity, event and result identity, idempotent cancellation, detach, and + * replay, and no provider in this repo advertises them. Supply your own + * provider to run it. + */ + +import type { AgentProfile } from '@tangle-network/agent-interface' +import type { AgentEnvironmentProvider } from '@tangle-network/agent-interface/environment-provider' +import { + type RetainedRunAdmission, + type RetainedRunDispatchedAdmission, + reconnectRetainedRun, + recoverRetainedRun, + startRetainedRun, +} from '@tangle-network/agent-runtime/kernel' + +/** Your durable store for admission records. One row per record is enough. */ +interface AdmissionJournal { + write(admission: RetainedRunAdmission): Promise + readDispatched(turnId: string): Promise +} + +/** + * Process A. Start the job and stream it while this process lives. + * + * `onAdmission` is awaited twice: after the environment exists, and again after + * the dispatch is verified. The start promise resolves only after the second + * record is durable, so a crash can never lose a live run. + */ +export async function startAndStream( + provider: AgentEnvironmentProvider, + journal: AdmissionJournal, + profile: AgentProfile, +): Promise { + const run = await startRetainedRun({ + provider, + environment: { idempotencyKey: 'workspace-42', profile }, + turn: { turnId: 'turn-7', prompt: 'Finish the migration and run its tests.' }, + identity: { sessionId: 'thread-42', executionId: 'execution-7' }, + onAdmission: async (admission) => { + await journal.write(admission) + }, + }) + + // Every event carries its cursor and sequence. Persist them before you show + // the event to a user, so a replay resumes strictly after what the user saw. + for await (const envelope of run.events()) { + console.log(`${envelope.sequence}: ${envelope.event.type}`) + } +} + +/** + * Process B. Rebuild control from the persisted ticket. + * + * `reconnectRetainedRun` rejects any mismatch of provider, environment, + * session, execution, run, or request digest, so a wrong ticket fails loud + * instead of controlling somebody else's job. + */ +export async function reattach( + provider: AgentEnvironmentProvider, + journal: AdmissionJournal, +): Promise { + const dispatched = await journal.readDispatched('turn-7') + if (!dispatched) throw new Error('no dispatched admission record for turn-7') + + const handle = await reconnectRetainedRun({ provider, controlRef: dispatched.controlRef }) + if (!handle) throw new Error('the provider no longer retains this environment') + + const snapshot = await handle.status({ waitMs: 30_000 }) + console.log(`status: ${snapshot.status ?? 'unknown'} — effect: ${snapshot.effect}`) + + const result = await handle.result() + return result.text +} + +/** + * Process B, after a crash that landed only the environment record. + * + * Never destroy the environment on `unverifiable`. Keep it, retry the reconnect + * when a dispatched record appears, or inspect it with the provider's tools. + */ +export async function recoverFromEnvironmentRecord( + provider: AgentEnvironmentProvider, + environmentId: string, +): Promise { + const recovery = await recoverRetainedRun({ + provider, + environmentId, + sessionId: 'thread-42', + executionId: 'execution-7', + }) + if (recovery.outcome === 'recovered') console.log(await recovery.handle.result()) + if (recovery.outcome === 'not_found') console.log('the environment is gone; nothing to clean up') + if (recovery.outcome === 'unverifiable') console.log('keep the environment and retry later') +} + +if (process.argv[1]?.endsWith('retained-run.ts')) { + console.log( + 'Compile-checked template. Pass your own AgentEnvironmentProvider to startAndStream / reattach.', + ) +} diff --git a/examples/runtime-run/README.md b/examples/runtime-run/README.md index 0f9abb04..4796a9a9 100644 --- a/examples/runtime-run/README.md +++ b/examples/runtime-run/README.md @@ -1,78 +1,50 @@ -# A cost-and-audit ledger for any agent run — one row, any database +# Record what a run cost -Wrap a running agent task and this records what it cost and what happened: every model call rolls up -into a live tally (tokens in, tokens out, dollars, wall-clock, number of calls), and at the end you -get one durable row — run id, workspace, session, status, cost, start/finish timestamps — written to -whatever store you use through a single `upsert(row)` method. Point that method at Postgres, SQLite, -Cloudflare D1, or an in-memory array; the ledger doesn't care. +## When to use it -## Why it matters +Use this when you must record what a run cost and whether it succeeded. +`startRuntimeRun` opens a run handle, keeps a live tally from the stream, and writes one canonical row through your store adapter. +The tally counts model calls only, so you can pipe the whole stream through it. -Once agents run in production you need to answer "what did this run cost?" and "did it succeed?" for -every session — for dashboards, billing, and audit. Hand-rolling that means threading cost -accounting through your whole stream and inventing a row schema. This gives you both: a correct cost -tally that ignores everything except model-call events, and a canonical row shape you persist with -one method. Backend-agnostic (works over any agent backend) and store-agnostic (works over any DB). +Use a sibling instead when cost is not the question. +[`../stream-a-turn`](../stream-a-turn) is the same stream with no ledger. +[`../chat-handler`](../chat-handler) adds the HTTP framing and the message write. +[`../sanitized-telemetry-streaming`](../sanitized-telemetry-streaming) redacts user data before you log it. -## How it works - -Four calls make up the lifecycle: - -1. `startRuntimeRun({ workspaceId, sessionId, agentId, taskSpec, adapter })` opens a run. The - identity fields land in the persisted row's typed columns; the task spec carries only what - describes the *work*. -2. `run.observe(event)` on every streamed event keeps the cost tally in sync. Only `llm_call` events - add to it — everything else is a no-op, so you can safely pipe the *entire* stream through it. -3. `run.complete({ status, resultSummary, error? })` exactly once at end of stream. It's idempotent - for the same status and throws if you try to change a status after the fact. -4. `run.persist()` writes the row via your adapter; `run.cost()` returns the live tally any time. - -## Run — fully offline, no key, no network +## How to use it ```bash -pnpm tsx examples/runtime-run/runtime-run.ts +pnpm build && pnpm tsx examples/runtime-run/runtime-run.ts ``` -A toy backend emits two model calls and some text, so the ledger has real numbers to add up. You'll -see the accumulated cost, then the exact row that would hit your database: +The example runs offline. +A toy backend emits two model calls, so the ledger has real numbers: -``` -Cost ledger: { - tokensIn: 1800, - tokensOut: 390, - costUsd: 0.0060999999999999995, - wallMs: 1, - llmCalls: 2 -} +```text +Cost ledger: { tokensIn: 1800, tokensOut: 390, costUsd: 0.0061, wallMs: 1, llmCalls: 2 } Persisted row: { id: 'legal-chat:thread-42:puomxsx1', - workspaceId: 'ws-1', - sessionId: 'thread-42', - agentId: 'legal-chat-runtime', - domain: 'legal', - taskId: 'legal-chat:thread-42', - scenarioId: 'legal-chat:thread-42', - status: 'completed', - resultSummary: 'Reviewed', + workspaceId: 'ws-1', sessionId: 'thread-42', agentId: 'legal-chat-runtime', + status: 'completed', resultSummary: 'Reviewed', cost: { tokensIn: 1800, tokensOut: 390, costUsd: 0.0061, wallMs: 1, llmCalls: 2 }, - startedAt: '...', completedAt: '...', - metadata: { note: 'demo persistence metadata' } + startedAt: '...', completedAt: '...' } ``` -`1800` in / `390` out / `2` calls is the tally summed from the two `llm_call` events (1200+600 in, -280+110 out) — proof `observe` only counts model calls. +The tally sums the two `llm_call` events (1200+600 in, 280+110 out), which proves `observe` counts model calls only. + +Four calls make the lifecycle. -## Make it real +1. `startRuntimeRun({ workspaceId, sessionId, agentId, taskSpec, adapter })` opens the run. The identity lands in typed columns; the task spec describes only the work. +2. `run.observe(event)` on every streamed event keeps the tally correct. +3. `run.complete({ status, resultSummary, error })` once, at the end of the stream. It is idempotent for the same status and refuses a changed status. +4. `run.persist()` writes the row. `run.cost()` returns the live tally at any time. -- **Real work:** replace the toy backend with `createSandboxPromptBackend` (a cloud sandbox) or - another caller-owned `AgentExecutionBackend`. Paid model work uses `streamAgentTurn` with an - exact `AgentProfile` and Runtime executor instead of a provider transport in this lifecycle. -- **Real store:** implement `RuntimeRunPersistenceAdapter` — one `upsert(row)` method — against D1, - Postgres, or your existing runs table. The row shape doesn't change. +Implement `RuntimeRunPersistenceAdapter` — one `upsert(row)` method — against D1, Postgres, SQLite, or your existing runs table. +The row shape does not change. -## Files +## Why this exists -| file | what it is | -|---|---| -| `runtime-run.ts` | the full lifecycle: open → observe → complete → persist, with a toy backend and in-memory store | +Once agents run in production, two questions arrive for every session: what did this cost, and did it work. +Answering them by hand means threading cost accounting through the whole stream and inventing a row schema. +This gives you a correct tally and one canonical row, over any backend and any store. diff --git a/examples/stream-a-turn/README.md b/examples/stream-a-turn/README.md new file mode 100644 index 00000000..5549732d --- /dev/null +++ b/examples/stream-a-turn/README.md @@ -0,0 +1,60 @@ +# Stream one agent turn + +## When to use it + +Use this when you run one agent turn and read its events yourself. +This is the smallest complete path through the runtime, and it is the path most products take. +Two contracts carry it: you write an `AgentExecutionBackend`, and you read `RuntimeStreamEvent` values back. + +Use a sibling instead when you need more than the raw stream. + +| You also need | Use | +|---|---| +| An HTTP response and a saved reply | [`../chat-handler`](../chat-handler) | +| A cost figure and one persisted row | [`../runtime-run`](../runtime-run) | +| A ready-made backend for a model, a sandbox, or your own loop | [`../stream-backends`](../stream-backends) | +| The model to call your tools inside the same turn | [`../tool-loop`](../tool-loop) | +| Several attempts under your own stop rule | [`../quickstart`](../quickstart) | + +## How to use it + +```bash +pnpm build && pnpm tsx examples/stream-a-turn/stream-a-turn.ts +``` + +The example runs offline and needs no credentials. +It prints: + +```text +Checking the refund policy +A renewal is refundable within 14 days of the charge. + +status: completed — backend completed +tools: search_policy +cost: $0.0009 — reply chars: 81 +``` + +Read [`stream-a-turn.ts`](./stream-a-turn.ts) for the whole file. +The two contracts are these. + +```ts +const backend: AgentExecutionBackend = { + kind: 'scripted-support', + async *stream(input, ctx): AsyncIterable { + yield { type: 'text_delta', text: 'Checking the refund policy', timestamp: now() } + }, +} + +for await (const event of runAgentTaskStream({ task, backend, input, sessionStore, sessionId })) { + if (event.type === 'text_delta') process.stdout.write(event.text) +} +``` + +Replace the scripted backend with your model call, your sandbox, or your coding CLI. +The reader code does not change. + +## Why this exists + +`RuntimeStreamEvent` is one union for every event source, so your product code reads one shape. +The runtime adds the parts a turn always needs around your backend: readiness checks, session start or resume, and a terminal `final` event. +Your backend stays small because it only produces events. diff --git a/examples/stream-a-turn/stream-a-turn.ts b/examples/stream-a-turn/stream-a-turn.ts new file mode 100644 index 00000000..f625cf12 --- /dev/null +++ b/examples/stream-a-turn/stream-a-turn.ts @@ -0,0 +1,106 @@ +/** + * stream-a-turn — run one agent turn and read its events. + * + * Two contracts, and nothing else: you write an `AgentExecutionBackend` that + * yields `RuntimeStreamEvent` values, and you read the same union back out of + * `runAgentTaskStream`. Offline, deterministic, no API keys. + * + * Run: pnpm build && pnpm tsx examples/stream-a-turn/stream-a-turn.ts + */ + +import type { + AgentBackendContext, + AgentBackendInput, + AgentExecutionBackend, + AgentTaskSpec, + RuntimeStreamEvent, +} from '@tangle-network/agent-runtime' +import { InMemoryRuntimeSessionStore, runAgentTaskStream } from '@tangle-network/agent-runtime' + +// The unit of work. `id` is yours; `intent` and `domain` describe the work. +const task: AgentTaskSpec = { + id: 'support:thread-42', + intent: 'Answer one support question about refunds.', + domain: 'support', +} + +// The backend contract every product implements. `kind` names it in the trace. +// `stream` yields the turn's events. A real backend calls a model API, a +// sandbox, or a coding CLI here; this one is scripted so the file runs offline. +const backend: AgentExecutionBackend = { + kind: 'scripted-support', + async *stream( + input: AgentBackendInput, + ctx: AgentBackendContext, + ): AsyncIterable { + const now = () => new Date().toISOString() + yield { type: 'text_delta', text: 'Checking the refund policy', timestamp: now() } + yield { + type: 'tool_call', + toolName: 'search_policy', + toolCallId: 'call_1', + args: { query: input.message ?? '' }, + timestamp: now(), + } + yield { + type: 'tool_result', + toolName: 'search_policy', + toolCallId: 'call_1', + result: { section: 'renewals', window: '14 days' }, + timestamp: now(), + } + yield { + type: 'llm_call', + task: ctx.task, + session: ctx.session, + model: 'scripted/support-agent', + tokensIn: 320, + tokensOut: 64, + costUsd: 0.0009, + latencyMs: 120, + timestamp: now(), + } + yield { + type: 'text_delta', + text: '\nA renewal is refundable within 14 days of the charge.\n', + timestamp: now(), + } + }, +} + +// The session store keeps this turn's session and events. Swap in your own +// store to make the transcript durable; omit it entirely for a stateless turn. +const sessions = new InMemoryRuntimeSessionStore() + +let reply = '' +let costUsd = 0 +const toolCalls: string[] = [] + +for await (const event of runAgentTaskStream({ + task, + backend, + input: { message: 'Can I refund a subscription renewal?' }, + sessionStore: sessions, + sessionId: 'thread-42', +})) { + switch (event.type) { + case 'text_delta': + reply += event.text + process.stdout.write(event.text) + break + case 'tool_call': + toolCalls.push(event.toolName) + break + case 'llm_call': + costUsd += event.costUsd ?? 0 + break + case 'final': + console.log(`\nstatus: ${event.status} — ${event.reason}`) + break + default: + break + } +} + +console.log(`tools: ${toolCalls.join(', ') || 'none'}`) +console.log(`cost: $${costUsd.toFixed(4)} — reply chars: ${reply.length}`) diff --git a/examples/stream-backends/README.md b/examples/stream-backends/README.md index 79eb43dd..45c8314a 100644 --- a/examples/stream-backends/README.md +++ b/examples/stream-backends/README.md @@ -1,47 +1,30 @@ -# Three sources of an AI's streaming output, one wire format for your app +# Choose where the tokens come from -An agent's output arrives as a live stream of events: text as it's typed, tool calls as they -fire, tool results as they return. That stream can come from three very different places — a -function you wrote, a remote sandbox running a coding agent, or an exact `AgentProfile` executed -by Runtime through the Tangle Router. This example runs all three and shows they emit the **same typed events** and serialize -to the **same format a browser reads**, so the source is a swappable detail your UI never -sees. +## When to use it -## Why it matters +Use this when you must choose where the tokens come from. +Three sources feed the same `runAgentTaskStream` call and emit the same typed events, so your route never learns which one ran. -The painful coupling in agent apps is that the frontend ends up knowing which backend it's -talking to — mock in tests, a real model in prod, a sandbox for the heavy stuff — because -each streams a different shape. Here they don't. Every backend lands on one typed event -stream and one SSE serialization, so you swap transports (test → sandbox → hosted model) -without touching the route that streams to the browser or the code that collects the events. - -("SSE" is Server-Sent Events, the plain `data: ...\n\n` streaming format a browser reads with -`EventSource` — the standard way a web app receives a live token stream.) - -## The three backends - -| Backend | You'd use it for | +| Source | Use it for | |---|---| -| **Iterable** (`createIterableBackend`) | You own the loop: write an async generator that yields events directly. For tests, scripted demos, or wrapping a stream shape the others don't map. | -| **Sandbox** (`createSandboxPromptBackend`) | A remote `@tangle-network/sandbox` box runs the agent and streams back its native events (text updates, tool calls, tool results). The default mapper already understands them, so you write no translation code. | -| **Exact model turn** (`streamAgentTurn`) | A concrete profile whose prompt, provider, model, and generation controls Runtime preserves and meters. | +| `createIterableBackend` | You own the loop. Write an async generator that yields events. Good for tests and scripted demos. | +| `createSandboxPromptBackend` | A remote sandbox box runs the agent and streams its native events. The default mapper reads them, so you write no translation. | +| `streamAgentTurn` | An exact `AgentProfile` through Runtime. Runtime keeps the prompt, provider, model, and generation controls, and meters the spend. | -All three feed `runAgentTaskStream`, which emits a typed `RuntimeStreamEvent` stream, which -two helpers serialize to SSE (`runtimeStreamServerSentEvent` per event, plus -`readinessServerSentEvent` for a one-off "still waiting on required info" event a gated task -can emit). +Use a sibling instead when the backend is not the question. +[`../stream-a-turn`](../stream-a-turn) writes one backend by hand and reads the events. +[`../chat-handler`](../chat-handler) serves the same stream over HTTP. -## Run it +## How to use it ```bash -pnpm tsx examples/stream-backends/stream-backends.ts +pnpm build && pnpm tsx examples/stream-backends/stream-backends.ts ``` -No API key needed for the first two backends: the iterable and sandbox sections run offline -against a synthetic in-process box. You'll see each section stream SSE frames to stdout, -e.g.: +The first two sections run offline against an in-process box. +Each section serializes its events as Server-Sent Events, the `data: ...` format a browser reads with `EventSource`: -``` +```text --- iterable backend --- data: {"type":"text_delta","text":"you said: hello\n"} @@ -51,11 +34,18 @@ data: {"type":"tool_call","toolName":"Read","toolCallId":"call_1", ...} data: {"type":"tool_result","toolName":"Read", ...} ``` -The third section is skipped unless you provide a Tangle Router key: +The third section needs a router key: ```bash TANGLE_API_KEY=sk-... pnpm tsx examples/stream-backends/stream-backends.ts ``` -`MODEL` and `MODEL_PROVIDER` override the concrete DeepSeek defaults, while `ROUTER_BASE` changes -only the transport endpoint. The output is the same SSE shape as the offline sections. +`MODEL` and `MODEL_PROVIDER` override the defaults. +`ROUTER_BASE` changes only the transport endpoint. +The output keeps the same shape as the offline sections. + +## Why this exists + +The usual coupling in an agent app is that the frontend learns which backend it talks to, because each one streams a different shape. +Here every source lands on one typed event union and one SSE serialization. +You swap test, sandbox, and hosted model without touching the route or the collector. diff --git a/examples/supervise/README.md b/examples/supervise/README.md index 44edbcbf..b13deb8f 100644 --- a/examples/supervise/README.md +++ b/examples/supervise/README.md @@ -1,66 +1,46 @@ -# Run a supervisor agent that delegates the work, in one function call +# Let one agent run other agents -Give a goal to a supervisor agent and it breaks the work off to worker agents, waits for them -to finish, and only calls itself done when a real check passes on a worker's actual output — -not when a worker *claims* success. This whole pattern is one call: `supervise(profile, task, -opts)`. Everything else (bookkeeping, worker plumbing, depth limits) is defaulted for you. +## When to use it -## Why it matters +Use this when a model must decide the plan, not your code. +One supervisor spawns workers, steers them, and stops when a check passes on a worker's real output. +Budget, journaling, and depth limits are defaulted. -Multi-agent orchestration usually means a pile of glue: spawning workers, tracking who's -running, collecting results, deciding when it's truly done. Two things here make that -tractable. First, it's a single call with sane defaults, so you write a goal and a profile, -not a framework. Second, "done" is a **check you provide** that runs against the worker's -output — so a worker can't lie its way to completion, and a failure reports the real reason -and the spend instead of a silent "no winner." +Use a sibling instead when your code owns the plan. +[`../quickstart`](../quickstart) is the loop you write yourself with `plan` and `decide`. +[`../graphs`](../graphs) is a fixed topology authored as data. +[`../supervisor-loop`](../supervisor-loop) is this same call with a real worker backend, such as a sandbox or a coding CLI. +[`../delegate`](../delegate) is the zero-configuration entry: one intent string in, one result out. -## What runs - -The goal is deliberately tiny: *produce the exact line `READY`*. It's a stand-in for real -delegated work so the mechanics are visible. - -1. The supervisor (its brain is a model reasoning over `spawn_agent` / `await_event` / `stop` - tools) is told to **delegate, not solve**: spawn a worker, wait for it to settle, then stop. -2. A worker agent runs and produces its output. -3. The completion check `out => ...includes('READY')` runs against that output. Only if it - passes is the run a win. If no worker ever delivers, the run ends with a typed reason plus - token/dollar spend. - -Three knobs are worth knowing: - -- **`profile.harness`** picks what drives the supervisor's brain: `null` (this example) uses - an in-process model tool-loop; `'opencode'` / `'claude-code'` / `'codex'` run a real coding - CLI in a sandbox instead. -- **`backend`** is *where the workers run* — one value to swap. Here it's `router-tools` - (off-box model agents); change it to `sandbox` + a harness to run each worker as a coding - agent in a real box. -- **`deliverable`** is the completion check described above. Optional, but it's what makes - "done" mean *verified* rather than *self-reported*. - -## Run it +## How to use it ```bash -TANGLE_API_KEY=sk-tan-... pnpm tsx examples/supervise/supervise.ts +TANGLE_API_KEY=sk-tan-... pnpm build && pnpm tsx examples/supervise/supervise.ts ``` -A key is required (the supervisor's brain and the worker both call the Tangle router). -Optional: `MODEL` (default `gemini-2.5-pro`), `TANGLE_ROUTER_URL`. +A key is required, because the supervisor's brain and the worker both call the router. +`MODEL` and `TANGLE_ROUTER_URL` are optional. On success it prints the delivered output: -``` +```text [OK] delivered: {"content":"READY"} ``` -If no worker delivers, it prints the reason and what it cost instead: +When no worker delivers, it prints the reason and the spend instead: -``` +```text [--] no winner (budget-exhausted) — 1 child(ren) down, spent 4210 tokens / $0.0031 ``` -## Going further +Three settings are worth knowing. + +- `profile.harness` picks what drives the supervisor's brain. This example uses an in-process model tool-loop. Set `opencode`, `claude-code`, or `codex` to run a coding CLI in a sandbox. +- `backend` is where the workers run. This example uses `router-tools`. Change it to `sandbox` plus a harness to run each worker as a coding agent in a real box. +- `deliverable` is the completion check. It is optional, and it is what makes "done" mean verified. + +## Why this exists -This is the smallest possible call — model brain, off-box workers, everything defaulted. When -your workers need a real backend (a sandbox box, a local coding CLI, or an MCP tool server), -go to [`../supervisor-loop/`](../supervisor-loop/): the same `supervise()` call with the -worker backend swapped in as the only change. +Multi-agent orchestration usually becomes glue code: spawn, track, collect, and guess when the work is done. +This is one call with defaults, so you write a goal and a profile instead of a framework. +"Done" is your check against a worker's output, so a worker cannot claim success, and a failure reports the real reason and the spend. diff --git a/examples/tool-loop/README.md b/examples/tool-loop/README.md new file mode 100644 index 00000000..502d5f2e --- /dev/null +++ b/examples/tool-loop/README.md @@ -0,0 +1,55 @@ +# Run a tool-calling turn + +## When to use it + +Use this when the model must call your tools and then answer in the same turn. +The loop runs one model turn, executes each requested tool, folds the results back, and runs the turn again. +It stops when the model finishes, or when a turn cap, a deadline, a cost limit, or a stuck-loop check fires. + +Use a sibling instead when the shape is different. + +| Your shape | Use | +|---|---| +| One turn with no tool round trip | [`../stream-a-turn`](../stream-a-turn) | +| One turn served over HTTP and saved | [`../chat-handler`](../chat-handler) | +| Several worker attempts under your own stop rule | [`../quickstart`](../quickstart) | +| One agent that spawns other agents | [`../supervise`](../supervise) | + +## How to use it + +```bash +pnpm build && pnpm tsx examples/tool-loop/tool-loop.ts +``` + +The example runs offline and needs no credentials. +It prints: + +```text +Looking up the invoice. +Invoice inv-42 is $120 and already paid. +tool get_invoice → ok +turns: 2 — stopReason: completed +``` + +Read [`tool-loop.ts`](./tool-loop.ts) for the whole file. +You supply two functions and the loop owns the rest. + +```ts +const result = await runToolLoop({ + systemPrompt: 'You answer billing questions. Use the tools before you answer.', + userMessage: 'Is invoice inv-42 paid?', + streamTurn, // one model turn: yields text and tool calls + executeToolCall, // your executors: one call in, one typed outcome out + isExecutableTool: (name) => tools.some((tool) => tool.function.name === name), + maxToolTurns: 8, +}) +``` + +Read `result.stopReason` before you score the turn. +Only `completed` means the model finished; `stuck-loop`, `backstop`, `deadline`, and `budget` are resource outcomes. + +## Why this exists + +Every agent product writes this loop, and the hard parts are the same each time. +The loop keeps the OpenAI tool history correct, so a strict model reads its own tool use back instead of repeating the call. +A tool failure returns a typed outcome instead of throwing, so the model reads the reason and can recover. diff --git a/examples/tool-loop/tool-loop.ts b/examples/tool-loop/tool-loop.ts new file mode 100644 index 00000000..98bb7ca1 --- /dev/null +++ b/examples/tool-loop/tool-loop.ts @@ -0,0 +1,83 @@ +/** + * tool-loop — one chat turn that calls tools until the model stops. + * + * The loop is the runtime's; the model and the tools stay yours. You supply + * `streamTurn` (one model turn) and `executeToolCall` (your executors), and the + * loop folds each result back into the conversation and re-runs the turn. + * Offline, deterministic, no API keys. + * + * Run: pnpm build && pnpm tsx examples/tool-loop/tool-loop.ts + */ + +import type { OpenAIChatTool } from '@tangle-network/agent-runtime' +import { + runToolLoop, + type ToolCallOutcome, + type ToolLoopCall, + type ToolLoopEvent, + type ToolLoopMessage, +} from '@tangle-network/agent-runtime/tool-loop' + +// The tool declarations you send to the model. This is the OpenAI function +// shape every OpenAI-compatible provider accepts. +const tools: OpenAIChatTool[] = [ + { + type: 'function', + function: { + name: 'get_invoice', + description: 'Read one invoice by id.', + parameters: { + type: 'object', + properties: { invoiceId: { type: 'string' } }, + required: ['invoiceId'], + }, + }, + }, +] + +const invoices: Record = { + 'inv-42': { amountUsd: 120, status: 'paid' }, +} + +// Your executors. One call in, one typed outcome out. A failure is a value, +// not a thrown error, so the model reads the reason and can recover. +async function executeToolCall(call: ToolLoopCall): Promise { + if (call.toolName !== 'get_invoice') { + return { ok: false, code: 'unknown_tool', message: `no tool named ${call.toolName}` } + } + const invoice = invoices[String(call.args.invoiceId)] + if (!invoice) return { ok: false, code: 'not_found', message: 'no such invoice', status: 404 } + return { ok: true, result: invoice } +} + +// One model turn. A real one calls your provider with `messages` and `tools`, +// then yields text and tool calls as they arrive. This one is scripted: it asks +// for the invoice first, then answers once the result is in the history. +async function* streamTurn(messages: ToolLoopMessage[]): AsyncIterable { + const sawToolResult = messages.some((message) => message.role === 'tool') + if (!sawToolResult) { + yield { type: 'text', text: 'Looking up the invoice.\n' } + yield { + type: 'tool_call', + call: { toolCallId: 'call_1', toolName: 'get_invoice', args: { invoiceId: 'inv-42' } }, + } + return + } + yield { type: 'text', text: 'Invoice inv-42 is $120 and already paid.\n' } +} + +const result = await runToolLoop({ + systemPrompt: 'You answer billing questions. Use the tools before you answer.', + userMessage: 'Is invoice inv-42 paid?', + streamTurn, + executeToolCall, + isExecutableTool: (name) => tools.some((tool) => tool.function.name === name), + // A watchdog, not a policy cap. Real limits come from deadlineMs / maxCostUsd. + maxToolTurns: 8, +}) + +console.log(result.finalText.trim()) +for (const executed of result.toolResults) { + console.log(`tool ${executed.label} → ${executed.outcome.ok ? 'ok' : executed.outcome.code}`) +} +console.log(`turns: ${result.turns} — stopReason: ${result.stopReason}`) diff --git a/src/runtime/index.ts b/src/runtime/index.ts index ffdbe884..9a0f882e 100644 --- a/src/runtime/index.ts +++ b/src/runtime/index.ts @@ -386,7 +386,14 @@ export { // `runAgentRounds` is the multi-agent fanout/vote/refine kernel over many sandbox sessions. // It is distinct from `runToolLoop`/`streamToolLoop`, which execute one chat turn and fold // tool results back into that same conversation. -export { defaultSelectWinner, type RunAgentRoundsOptions, runAgentRounds } from './run-loop' +export { + defaultSelectWinner, + isTerminalDecision, + type RunAgentRoundsOptions, + runAgentRounds, + TERMINAL_DECISIONS, + type TerminalDecision, +} from './run-loop' export { type AcquireOptions, acquireSandbox } from './sandbox-acquire' export { type CriuCapableClient, diff --git a/src/runtime/run-loop.ts b/src/runtime/run-loop.ts index 08ecbb0c..811c4f8d 100644 --- a/src/runtime/run-loop.ts +++ b/src/runtime/run-loop.ts @@ -1212,10 +1212,22 @@ function resolveAgentRuns( throw new ValidationError('runAgentRounds: `agentRun` or non-empty `agentRuns` is required') } -function isTerminalDecision(decision: unknown): boolean { - return ( - decision === 'stop' || decision === 'pick-winner' || decision === 'fail' || decision === 'done' - ) +/** + * Decision values the kernel treats as terminal. Every other value returned by + * `decide` continues the loop. Type a driver's `decide` return as + * `'your-word' | TerminalDecision` so caller vocabulary and kernel keywords + * stay visibly distinct. + * + * @stable + */ +export const TERMINAL_DECISIONS = ['stop', 'pick-winner', 'fail', 'done'] as const + +/** One of the kernel's terminal decision values. @stable */ +export type TerminalDecision = (typeof TERMINAL_DECISIONS)[number] + +/** True when the kernel stops the loop for this decision value. @stable */ +export function isTerminalDecision(decision: unknown): decision is TerminalDecision { + return (TERMINAL_DECISIONS as readonly unknown[]).includes(decision) } function emitRunLoopHook( diff --git a/src/runtime/types.ts b/src/runtime/types.ts index d248d401..0cd44311 100644 --- a/src/runtime/types.ts +++ b/src/runtime/types.ts @@ -233,7 +233,8 @@ export interface Iteration { /** @stable */ export interface Driver { /** - * Stable identifier surfaced in trace events. Default `'driver'`. + * Trace label surfaced in trace events. No behavioral effect: it never + * selects a strategy or a decision path. Default `'driver'`. */ readonly name?: string /** @@ -243,9 +244,11 @@ export interface Driver { plan(task: Task, history: ReadonlyArray>): Promise /** * Inspect history and return the next state. The kernel terminates the - * loop when `decide` returns a value listed in `isTerminalDecision` - * (`'stop' | 'pick-winner' | 'fail' | 'done'`), when `maxIterations` - * is hit, or when the abort signal fires. + * loop when `decide` returns a `TerminalDecision` + * (`'stop' | 'pick-winner' | 'fail' | 'done'`, exported as + * `TERMINAL_DECISIONS` with the `isTerminalDecision` guard), when + * `maxIterations` is hit, or when the abort signal fires. Every other + * value is caller vocabulary and continues the loop. */ decide(history: ReadonlyArray>): Decision | Promise /** diff --git a/tsconfig.examples.json b/tsconfig.examples.json index f211c0fd..348cf602 100644 --- a/tsconfig.examples.json +++ b/tsconfig.examples.json @@ -9,6 +9,7 @@ "@tangle-network/agent-runtime/durable": ["./src/durable/index.ts"], "@tangle-network/agent-runtime/intelligence": ["./src/intelligence/index.ts"], "@tangle-network/agent-runtime/kernel": ["./src/runtime/index.ts"], + "@tangle-network/agent-runtime/tool-loop": ["./src/tool-loop.ts"], "@tangle-network/agent-runtime/environment-provider": [ "./src/runtime/environment-provider.ts" ], From 285d1d71a67777cf767d8eac94c31d0055916bf6 Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Sat, 15 Aug 2026 01:04:23 -0600 Subject: [PATCH 2/5] docs(examples): keep the audited handleChatTurn one-line intro --- examples/chat-handler/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/chat-handler/README.md b/examples/chat-handler/README.md index 9b2bf5df..e3e0738a 100644 --- a/examples/chat-handler/README.md +++ b/examples/chat-handler/README.md @@ -4,7 +4,7 @@ Use this when a web route must stream one turn to a browser and save the reply. `handleChatTurn` owns the framing every chat product hand-rolls: NDJSON lines, a start and finish envelope, and one persist call after the last token. -You give it how to produce the tokens and how to save the final text. +You give it how to produce the response and how to persist it; it streams, traces, and persists. Use a sibling instead when you do not need the HTTP layer. From 8b570a0873ce5c87bd7aacd5881be986301897d1 Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Sat, 15 Aug 2026 01:05:08 -0600 Subject: [PATCH 3/5] docs(index): list the improvement and PrimeIntellect references --- docs/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index bc915ad0..70b03c70 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,7 +4,7 @@ The map of every doc. **Start here** if you're new; the deeper tracks follow. ## Start here -1. [The README](../README.md) — what the package is, a runnable offline quickstart, and the primitives catalog. +1. [The README](../README.md) — what the package is, a runnable offline quickstart, and the front-door table. 2. [concepts.md](./concepts.md) — the mental model (chat turns, tasks, runs) in plain terms. 3. [canonical-api.md](./canonical-api.md) — find the right primitive: "I want to ___ → use ___". 4. [../examples/](../examples) — copy a runnable example near your task. @@ -35,6 +35,8 @@ These are internal working documents: design theses, research narrative, and roa | [STABILITY.md](./STABILITY.md) | stability contract | What `@stable` / `@experimental` promise consumers, the graduation bar, and the demotion/removal policy. | | [concepts.md](./concepts.md) | mental model | The product-API layer cake (chat turns, tasks, runs) — the onramp before the loop/strategy docs. | | [glossary.md](./glossary.md) | canonical vocabulary | One definition per term, grounded to `file:line`; drifted synonyms flagged. | +| [improve.md](./improve.md) | improvement reference | The `improve()` call, the optimizer object, official GEPA and SkillOpt installs, surfaces, redaction, and the proposal→review→activation path. | +| [primeintellect.md](./primeintellect.md) | training/eval adapter | Package the same runtime program as a PrimeIntellect Verifiers environment and import its traces back. | | [execution-model.md](./execution-model.md) | the picture | The unified `Executor` port (router/bridge/cli/sandbox/BYO) + two engines, driver vs worker, spawn mechanics. | | [agent-bus-protocol.md](./agent-bus-protocol.md) | normative protocol | The multi-agent call bus — depth limits, headers, refusal contract. | | [durability-adapters.md](./durability-adapters.md) | subsystem | SQL-backed journal and restart behavior for conversations. Supervised-tree recovery is not implemented. | From 07186d456ee2d5fdf3b5c1650fb0e8b0c875144e Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Sat, 15 Aug 2026 01:06:40 -0600 Subject: [PATCH 4/5] docs(readme): split two long sentences into lists --- README.md | 8 +++++++- examples/quickstart/README.md | 3 ++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 13ef6b0f..ca662b9e 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,13 @@ Each row links to a runnable example. | **`improve`** · [example](./examples/improve) | You must change one part of an agent and prove the gain. | a profile field, three case sets, a judge | a detached candidate, a lift interval, ship or hold | Five mechanisms continue interrupted work. -Pick by what died: the HTTP connection (`streamPrompt` with the same `executionId`), nothing but you want the same box (`openSandboxRun`), the coordinator process (`supervise({ runDir })`), the user's chat session (`/conversation` stores), or everything except the provider ([retained runs](./examples/retained-run)). +Pick by what died. + +- The HTTP connection — call `streamPrompt` again with the same `executionId`. +- Nothing, but you want the same box for the next turn — `openSandboxRun`. +- The coordinator process, mid-orchestration — `supervise({ runDir })`. +- The user's chat session — the `/conversation` store adapters. +- Everything except the provider — [retained runs](./examples/retained-run). ## Also in the box diff --git a/examples/quickstart/README.md b/examples/quickstart/README.md index f1c02694..b9f2ebe8 100644 --- a/examples/quickstart/README.md +++ b/examples/quickstart/README.md @@ -67,4 +67,5 @@ The annotated version of the same loop, with every seam explained, is [`../drive A retry that sends the same prompt again learns nothing. This loop reads what the worker produced, scores it with your check, and writes the next prompt from the real output. -The kernel owns the parts that are easy to get wrong — one fresh worker per attempt, a hard iteration cap, and teardown of every worker at the end — so your driver stays a few lines of plain code. +The kernel owns the parts that are easy to get wrong: one fresh worker per attempt, a hard iteration cap, and teardown of every worker at the end. +Your driver stays a few lines of plain code. From 3d4abe1cc191f3875abef716ee1d2b35957fbe24 Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Sat, 15 Aug 2026 01:33:08 -0600 Subject: [PATCH 5/5] docs: correct the cli-base harness gloss --- README.md | 2 +- examples/supervise/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ca662b9e..6af80e30 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ Five words appear everywhere: | **driver** | Your code. It runs a worker, reads the output, and writes the next prompt. | | **decision** | What `decide` returns. The four keywords in `TERMINAL_DECISIONS` (`stop`, `pick-winner`, `fail`, `done`) end the loop; every other value is your own vocabulary and continues it. | | **verdict** | What a validator returns: valid or not, with a score. | -| **harness** | What drives an agent: an in-process model loop (`cli-base`), or a coding CLI such as `claude-code`, `codex`, or `opencode`. | +| **harness** | What drives an agent. `cli-base` is the router-backed mode with no coding agent behind it; `claude-code`, `codex`, and `opencode` each run a real coding CLI. | ## Which front door diff --git a/examples/supervise/README.md b/examples/supervise/README.md index b13deb8f..38ae4b97 100644 --- a/examples/supervise/README.md +++ b/examples/supervise/README.md @@ -35,7 +35,7 @@ When no worker delivers, it prints the reason and the spend instead: Three settings are worth knowing. -- `profile.harness` picks what drives the supervisor's brain. This example uses an in-process model tool-loop. Set `opencode`, `claude-code`, or `codex` to run a coding CLI in a sandbox. +- `profile.harness` picks what drives the supervisor's brain. This example uses `cli-base`, the router-backed brain with no coding agent. Set `opencode`, `claude-code`, or `codex` to run a coding CLI in a sandbox. - `backend` is where the workers run. This example uses `router-tools`. Change it to `sandbox` plus a harness to run each worker as a coding agent in a real box. - `deliverable` is the completion check. It is optional, and it is what makes "done" mean verified.