diff --git a/README.md b/README.md index 5b0abe3f..6af80e30 100644 --- a/README.md +++ b/README.md @@ -9,526 +9,109 @@ Domain behavior (models, tools, knowledge) plugs in as adapters; the scoring sta pnpm add @tangle-network/agent-runtime @tangle-network/agent-eval @tangle-network/sandbox ``` -## Contents - -- [Quickstart](#quickstart-offline-no-api-keys) -- [What you do with it](#what-you-do-with-it) -- [Run a chat turn](#run-a-chat-turn) -- [Retain and reconnect a run](#retain-and-reconnect-a-run) -- [Supervise a team of agents](#supervise-a-team-of-agents) -- [Improve an agent](#improve-an-agent) -- [Improve a knowledge base](#improve-a-knowledge-base) -- [Run on PrimeIntellect](#run-on-primeintellect) -- [How it works](#how-it-works-the-short-version) -- [Primitives](#primitives) -- [Examples](#examples) -- [Where to go next](#where-to-go-next) +New here? Read [`docs/concepts.md`](./docs/concepts.md) for the mental model in plain terms, then pick a front door below. ## Quickstart (offline, no API keys) -A driver runs a worker, reads its output, and writes the next prompt until a check passes. -This excerpt shows the driver from the runnable [`examples/quickstart/quickstart.ts`](./examples/quickstart/quickstart.ts). -That file defines the scripted `worker`, `output`, and `validator` used below so it runs without credentials. -Replace the scripted worker with a sandbox, CLI bridge, or router backend without changing the driver. +One agent attempt, run by a loop you control. +This is [`examples/quickstart/minimal.ts`](./examples/quickstart/minimal.ts) in full: it compiles and runs as pasted, with no credentials. ```ts import type { AgentProfile } from '@tangle-network/agent-interface' -import { inProcessSandboxClient, runAgentRounds } from '@tangle-network/agent-runtime/kernel' +import { + inProcessSandboxClient, + runAgentRounds, + type TerminalDecision, +} from '@tangle-network/agent-runtime/kernel' +import type { SandboxEvent } from '@tangle-network/sandbox' -const noteWriterProfile = { +const profile = { name: 'note-writer', harness: 'cli-base', model: { provider: 'scripted', default: 'scripted/note-writer' }, } satisfies AgentProfile -const result = await runAgentRounds({ - task: { prompt: 'Write a one-line release note for one-click restore.' }, +// A scripted worker. Swap in a sandbox, CLI-harness, or router backend later. +const worker = inProcessSandboxClient({ + onPrompt: (): SandboxEvent[] => [ + { type: 'result', data: { result: { note: 'Shipped one-click restore.' } } }, + ], +}) + +const result = await runAgentRounds({ + task: 'Write a one-line release note for one-click restore.', driver: { - name: 'refine', - plan: async (task, history) => { - const last = history[history.length - 1] - if (!last) return [task] // shot 0: run the task as written - if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots - // The core move: read the last worker's real output, write the next prompt FROM it. - return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] - }, - decide: (history) => - history.some((shot) => shot.verdict?.valid) ? 'pick-winner' : history.length < 3 ? 'refine' : 'fail', + // plan returns the tasks to run this iteration; [] means no more work. + plan: async (task, history) => (history.length === 0 ? [task] : []), + // 'done' is one of the four kernel keywords in TERMINAL_DECISIONS. + decide: (): TerminalDecision => 'done', }, - agentRun: { profile: noteWriterProfile, taskToPrompt: (t) => t.prompt }, - output, // parses the worker's event stream into { note } - validator, // pass/fail check: does the note mention "rollback"? + agentRun: { profile, taskToPrompt: (t) => t }, + output: { parse: (events) => events }, ctx: { sandboxClient: worker }, - maxIterations: 3, }) + +console.log(`decision: ${result.decision} — ${result.iterations.length} iteration(s)`) ``` Run it from a clone of this repo and you get exactly this: ```bash $ pnpm i && pnpm build -$ pnpm tsx examples/quickstart/quickstart.ts -shot 0: reject — "Shipped one-click restore." -shot 1: PASS — "Shipped one-click restore with an instant rollback path." -decision: pick-winner — winner: shot 1 +$ pnpm tsx examples/quickstart/minimal.ts +decision: done — 1 iteration(s) ``` -The annotated version is [`examples/driver-loop`](./examples/driver-loop). +[`examples/quickstart`](./examples/quickstart) grows the same call into a loop that reads each output and writes the next prompt from it. -## What you do with it +Five words appear everywhere: -| You want to… | Call | +| Word | What it means | |---|---| -| Run a **chat turn** for a production product agent | `handleChatTurn(...)` | -| Have one agent **supervise a team of agents** toward a goal | `supervise(profile, task, opts)` | -| **Improve** an agent and prove the gain on fresh tasks | `improve(profile, opts)` | -| Produce a measured knowledge-base candidate with agents and checks | `runKnowledgeImprovementJob(...)` | -| Evaluate or train the same agent on **PrimeIntellect** | `createPrimeIntellectPackage(...)` | - -### Run a chat turn - -A product agent is one `handleChatTurn` call inside a route. You give it how to produce the response and how to persist it; it streams, traces, and persists. - -```ts -import { deriveExecutionId, handleChatTurn } from '@tangle-network/agent-runtime/durable' - -const turnIndex = 0 -const executionId = deriveExecutionId({ projectId, sessionId: threadId, turnIndex }) -const result = handleChatTurn({ - identity: { tenantId, sessionId: threadId, userId, turnIndex }, - hooks: { - produce: () => ({ - stream: box.streamPrompt(userMessage, { - sessionId: threadId, - executionId, - turnId: executionId, - detach: true, - }), - finalText: () => box.lastResponse(), - }), - persistAssistantMessage: async ({ identity, finalText }) => db.insertMessage(identity, finalText), - }, - waitUntil, -}) -return new Response(result.body, { headers: { 'content-type': result.contentType } }) -``` - -For a stream reconnect, call `streamPrompt` with the same `executionId` and the last event id the client received. -For a repeated initial dispatch, reuse both `sessionId` and `turnId`; `executionId` alone is not an idempotency key. - -### Retain and reconnect a run - -Use the retained-run API when the provider owns a job that must outlive one HTTP reader or application process. -The provider must advertise exact run identity, replay, result identity, and idempotent cancellation. - -```ts -import { - reconnectRetainedRun, - recoverRetainedRun, - startRetainedRun, -} from '@tangle-network/agent-runtime/kernel' - -const run = await startRetainedRun({ - provider, - environment: { idempotencyKey: 'workspace-42', profile }, - turn: { turnId: 'turn-7', prompt: 'Finish the migration and run its tests.' }, - identity: { sessionId: 'thread-42', executionId: 'execution-7' }, - onAdmission: async (admission) => { - await journal.write(admission) - }, -}) - -for await (const event of run.events()) { - await journal.write(event) -} - -const recovered = await reconnectRetainedRun({ - provider: freshProvider, - controlRef: (await journal.readDispatchedAdmission()).controlRef, -}) -if (!recovered) throw new Error('the provider no longer retains this environment') - -const snapshot = await recovered.status({ waitMs: 30_000 }) -const result = await recovered.result() -``` - -The runtime awaits `onAdmission` after environment creation and again after dispatch. -The start promise resolves only after the dispatched admission, so the exact reference is durable before any caller observes success. -Persist each admission record inside the hook before it returns. -A hook rejection fails the start with `RetainedRunAdmissionError` and keeps the environment for recovery. -When you omit `identity`, the runtime mints deterministic coordinates from the two keys, so every process derives the same values. -After dispatch, the runtime verifies the provider honored the requested identity and fails with `RetainedRunDispatchBindingError` when it did not. -Persist each event cursor and sequence before advancing the visible transcript. -`reconnectRetainedRun` reconstructs a client from a dispatched admission's `controlRef` and rejects any provider, environment, session, execution, run, or digest mismatch. -After a crash that left only the `environment` admission, call `recoverRetainedRun` with its coordinates. -It reports `recovered` with a handle, `not_found` when the environment is gone, or `unverifiable` when the provider cannot self-identify the session. -Never destroy an environment on `unverifiable`; keep it, retry `reconnectRetainedRun` later, or inspect it with provider-native tools. -An unknown provider result remains unknown; the runtime never converts it into success or confirmed cancellation. - -### Supervise a team of agents - -One supervisor spawns and steers workers toward a goal. Where the workers run (an in-process loop, or a sandboxed coding harness) is one data value; the budget, journaling, and stopping are handled for you. - -```ts -import { supervise } from '@tangle-network/agent-runtime/kernel' - -const result = await supervise( - { - name: 'supervisor', - harness: 'cli-base', - model: { provider: 'tangle-router', default: process.env.TANGLE_MODEL! }, - prompt: { - systemPrompt: 'Delegate to workers; do not solve the task yourself.', - }, - }, - 'Implement the feature and make the tests pass.', - { budget, router, backend }, // backend = where workers run: router-tools | sandbox+harness | bridge -) -``` - -### Improve an agent - -`improve` runs one complete `OptimizationMethod` against one profile field. -The method owns candidate generation and selection. -Runtime keeps the final test set out of the method, scores the baseline and selected candidate on it, and returns `ship` only when the paired confidence interval clears `minimumLift`. -The profile is never changed. - -```ts -import { improve, officialGepa } from '@tangle-network/agent-runtime' -import { profileOptimizerModelCall } from '@tangle-network/agent-runtime/kernel' -import { - type AgentProfile, - canonicalAgentProfileDigest, - canonicalCandidateDigest, -} from '@tangle-network/agent-interface' - -const executionRef = canonicalCandidateDigest({ - deployment: process.env.AGENT_DEPLOYMENT_SHA!, - model: process.env.AGENT_MODEL!, - tools: process.env.AGENT_TOOLSET_SHA!, -}) - -const optimizerProfile = { - name: 'support-prompt-optimizer', - harness: 'cli-base', - model: { - provider: 'tangle-router', - default: process.env.OPTIMIZER_MODEL!, - metadata: { maxTokens: 16_384 }, - }, -} satisfies AgentProfile -const optimizerPricing = { - inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION), - outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION), -} -const optimizer = { - model: optimizerProfile.model.default, - call: profileOptimizerModelCall({ - profile: optimizerProfile, - context: 'support-prompt optimizer', - executor: { - backend: 'router', - routerBaseUrl: process.env.OPTIMIZER_BASE_URL!, - routerKey: process.env.OPTIMIZER_API_KEY!, - }, - pricing: optimizerPricing, - }), - callRef: canonicalCandidateDigest({ - profile: canonicalAgentProfileDigest(optimizerProfile), - deployment: process.env.OPTIMIZER_DEPLOYMENT_SHA!, - }), - budget: { - maxCostUsd: 10, - maxRequests: 50, - maxRequestBytes: 2_000_000, - maxResponseBytes: 2_000_000, - maxOutputTokensPerRequest: 16_384, - pricing: optimizerPricing, - }, -} - -const result = await improve(baseProfile, { - surface: 'prompt', - executionRef, - method: officialGepa({ - objective: 'Improve the complete support-agent prompt.', - recipe: { - kind: 'engine', - run: { - engine: 'gepa', - maxEvaluations: 40, - maxProposerCostUsd: 10, - }, - }, - optimizer, - resume: 'if-compatible', - trustResumeState: true, - describeScenario: ({ input }) => ({ input }), - }), - findings, - trainScenarios, - selectionScenarios, - testScenarios, - judges: [judge], - agent: (candidateProfile, scenario, ctx) => - runProfile(candidateProfile, scenario, ctx), - runDir: '.runs/support-prompt', - costCeiling: 25, -}) - -if (result.decision === 'ship') { - console.log(result.candidate.profile, result.liftInterval) -} -``` - -`officialGepa(...)` delegates the complete search to GEPA's upstream Optimize Anything API through agent-eval. -Pass one explicit `engine`, `sequential`, `adaptive-sequential`, `best-of`, `vote`, or `omni` recipe. -Runtime derives the upstream resume identity from `executionRef`, the complete baseline profile, and the selected surface. -With `resume: 'if-compatible'`, agent-eval resumes only when the saved run identity matches the candidate, recipe, data, optimizer settings, runner, and derived execution identity. -Set `trustResumeState: true` only when that run directory is private to the current operator. -Use `resume: 'required'` to fail when no matching run exists. -`result.provenance` reports the upstream package, run ID, resume status, evaluation count, and artifact directory. -`result.candidatePopulation` verifies and joins callback observations with an optimizer's official candidate graph. -It returns every unique candidate as a complete profile with ordered Interface diffs, or as an explicit materialization refusal. -GEPA candidates retain exact parent indices and selection scores; callback-only proposals report lineage as unavailable. -Methods without either artifact return `status: 'unavailable'` instead of treating the winner as the full population. -There is no local fallback. -Install its optional Python process before using it: - -```bash -python -m pip install "agent-eval-rpc==0.145.0" -python -m pip install "gepa[full]==0.1.4" -``` - -The published GEPA 0.1.4 wheel supports the direct `gepa` engine. -Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N require the tested official source revision: - -```bash -python -m pip install "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f" -``` - -Use `officialSkillOpt(...)` for Microsoft's SkillOpt: - -```bash -python -m pip install "agent-eval-rpc==0.145.0" -python -m pip install "skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90" -``` - -SkillOpt 0.2.0's published wheel omits prompt files required by `ReflACTTrainer`, so the tested SkillOpt source revision remains necessary. -SkillOpt and GEPA's standard reflection engine require `optimizer: { model, call, callRef, budget }`. -Agent-based GEPA engines may own their model connection instead. -Runtime owns those model calls through one exact `AgentProfile`; Agent Eval enforces the nested budget and records their measured cost and execution evidence without receiving provider credentials. -`costCeiling` is the total limit for optimizer calls, candidate runs, judges, and final scoring. -Runtime returns `hold` when any part of that cost is unknown. -Runtime rejects a reported total above the limit. - -SkillOpt accepts one text surface. -GEPA accepts text or named components. -Any complete method from `@tangle-network/agent-eval` uses the same call. -The `agent` callback receives the complete immutable candidate profile, not a raw prompt or component fragment. -Runtime uses that exact profile for every candidate run and returns the same measured profile in `result.candidate.profile`. -`executionRef` is a content digest of the agent callback, profile component mapping, model, tools, and closure settings. -Runtime combines it with the complete baseline profile and selected surface for saved work. -Changing any of them runs the affected work again. -For a skill, set `surface: 'skills'` and `skills.resourceName`. -For the complete profile, set `surface: 'agent-profile'`. -To optimize several named profile fields together, also provide `profileComponents.read` and `profileComponents.apply`. -Tools, MCP, hooks, subagents, curated instructions, and rollout policy are also exact profile coordinates. -Runtime does not choose an optimizer for them. - -Without `describeScenario`, the external optimizer receives only each development case ID. -Without `describeArtifact`, evaluation feedback contains no artifact body. -When either descriptor is present, its result passes through `redact` together with findings, background text, profile name, and judge notes. -The built-in redactor removes common credentials and email addresses. -Supply a domain redactor for customer names, account IDs, or other private data the built-in rules cannot identify. -Runtime applies that hook first and then still applies its built-in scrubber. -Set `redact: false` only when every outbound value is public and already reviewed. - -The selected profile surface is the optimizer's candidate and cannot be redacted without changing the measured candidate. -Runtime always rejects recognized credentials in those bytes. -It also rejects structurally sensitive fields such as MCP env, headers, URLs, metadata, and extensions. -For `tools`, `mcp`, `hooks`, `subagents`, and `agent-profile`, Runtime treats the entire selected coordinate as execution-capable. -Use `authorizeSensitiveCandidate` to inspect and accept each exact immutable profile containing public values or safe references. -The callback runs for the baseline and every distinct candidate before either reaches your agent. -Its `sensitivePaths` includes `$` when the whole coordinate requires review. - -Code is the exception. -It uses Runtime's isolated git worktrees and coding-agent candidate execution: - -```ts -const result = await improve({ - surface: 'code', - code: { repoRoot, baseRef, profile, generator }, - scenarios, - judge, - agent, - budget, -}) -``` - -`improve` is the search call. -For production, `proposeAgentImprovement` adds trace analysis and reruns the exact frozen baseline and winner before creating a reviewable proposal. -Runtime rejects a candidate bundle that differs from the search winner. - -```ts -import { - createAgentImprovementActivation, - executeAgentImprovementActivation, - proposeAgentImprovement, - reviewAgentImprovementProposal, -} from '@tangle-network/agent-runtime/intelligence' - -const baseline = freezeBaseline(liveProfile) -const result = await proposeAgentImprovement({ - runId, - profile: liveProfile, - analysis, - improvement: { - surface: 'prompt', - executionRef, - method, - trainScenarios, - selectionScenarios, - testScenarios, - judges: [judge], - agent, - }, - buildExperiment: ({ improvement }) => - buildExperimentMaterial({ - baseline, - candidate: compileCandidateBundle({ baseline, improvement: improvement.candidate }), - benchmark: heldOutBenchmark, - policy: comparisonPolicy, - }), - placeCell, -}) - -const review = reviewAgentImprovementProposal(result.proposal, { - decision: 'approve', - reviewedBy: user.id, - reason: 'The measured gain is worth the cost.', -}) -const activation = createAgentImprovementActivation(result.proposal, review, { - intent: 'activate-candidate', - targets: [{ surface: 'prompt', identity: profileId }], - fundingOwner: tenantId, - authorizedBy: user.id, - expiresAt, -}) -const outcome = await executeAgentImprovementActivation( - { proposal: result.proposal, review, activation }, - { transition: commitProfileTransaction, reconcile: readCommittedResult }, -) -``` - -`buildExperimentMaterial`, `placeCell`, and the transaction functions are application ports because storage and compute differ by product. -The builder returns only baseline, candidate, tasks, and policy; Runtime adds the search ancestry and seals the final experiment. -Runtime owns candidate identity, measurement, review binding, expiry, retry identity, and result validation; the application owns its atomic write. -Official optimizer proposals carry the observed package versions, optimizer model, evaluation and token usage, separate optimization and final-test costs, and resumed-run identity. -`createOptimizationActivationReceipt(result)` exposes the same detached record for callers that need to inspect an `improve()` result before building a proposal. - -### Improve a knowledge base - -`runKnowledgeImprovementJob` runs KB, wiki, memory-backed, and RAG improvement jobs. -It creates a candidate copy, runs agents against it, checks it through `@tangle-network/agent-knowledge`, and returns frozen baseline and candidate snapshots with spend and timing. -It never changes the live knowledge base. -Use `improve(profile, { surface: 'memory', ... })` for the agent's curated lesson document. -Use this job for source, retrieval, and knowledge-store changes. - -```ts -import { runKnowledgeImprovementJob } from '@tangle-network/agent-runtime/knowledge' - -const result = await runKnowledgeImprovementJob({ - root: './kb', - goal: 'Improve support refund-policy knowledge', - implementationRef: 'git:0123456789abcdef0123456789abcdef01234567', - readinessSpecs, - budget: { maxIterations: 8, maxTokens: 120_000, maxUsd: 10 }, - backend, -}) - -console.log(result.knowledge?.reference.candidateHash, result.measurement.supervisedSpent) -``` - -Use it when the product needs one knob for "make this knowledge base better" instead of wiring `improveKnowledgeBase`, a runtime supervisor, candidate workspaces, and readiness checks by hand. -Set `implementationRef` to the deployed `git:<40 hex>` revision or a `sha256:<64 hex>` digest covering every callback, model, index, and external setting that can change the result. -The same run ID resumes only when this identity still matches. -Measure the returned bundle pair, record the review, then activate through `executeAgentImprovementActivation`; activation is the only write path. - -### Run on PrimeIntellect - -`@tangle-network/agent-runtime/primeintellect` packages typed train and eval tasks as a PrimeIntellect Verifiers environment. -Prime launches your actual runtime program against an intercepted model endpoint, so `runPersonified`, `runAgentic`, product agents, tool calls, and multiple rounds stay intact. -Reference answers remain in Prime's task process and never enter the agent workspace. -The runner file must be one executable bundle containing the app and its runtime dependencies. - -```ts -import { readFile } from 'node:fs/promises' -import { - createPrimeIntellectPackage, - writePrimeIntellectPackage, -} from '@tangle-network/agent-runtime/primeintellect' - -const bundledRunner = await readFile('./dist/prime-runner.mjs', 'utf8') -const bundle = createPrimeIntellectPackage({ - name: 'support-agent', - version: '1.0.0', - tasks: [ - { - id: 'train-refund-policy', - split: 'train', - prompt: 'Can a subscription renewal be refunded?', - answer: 'No', - }, - { - id: 'eval-final-sale', - split: 'eval', - prompt: 'Can a final-sale order be refunded?', - answer: 'No', - }, - ], - scoring: { kind: 'exact', normalization: 'trim-casefold' }, - runner: { - image: 'node:22-bookworm-slim', - files: { 'runner.mjs': bundledRunner }, - command: ['node', 'runner.mjs'], - }, -}) - -await writePrimeIntellectPackage(bundle, './prime/support-agent') -``` - -The runner reads the episode and uses the normal runtime APIs: -Here, `runProductAgent` is the application's existing entry point, not another loop supplied by this adapter. - -```ts -import { - primeIntellectExecutorConfig, - runPrimeIntellectProgram, -} from '@tangle-network/agent-runtime/primeintellect' -import { - collectAgentTurn, - createExecutor, - streamAgentTurn, -} from '@tangle-network/agent-runtime/kernel' - -await runPrimeIntellectProgram(async (episode) => { - const profile = makeProductProfile({ model: episode.model.name }) - return collectAgentTurn( - streamAgentTurn( - { - kind: 'executor', - profile, - factory: createExecutor(primeIntellectExecutorConfig(episode)), - }, - episode.task.prompt, - ), - ) -}) -``` - -Prime writes complete `traces.jsonl` rows. -Use `importPrimeIntellectTraces(...)` to convert them to agent-eval `RunRecord`s for the existing reports and release checks. +| **worker** | An agent that produces an answer. Here it is a `SandboxClient`. | +| **driver** | Your code. It runs a worker, reads the output, and writes the next prompt. | +| **decision** | What `decide` returns. The four keywords in `TERMINAL_DECISIONS` (`stop`, `pick-winner`, `fail`, `done`) end the loop; every other value is your own vocabulary and continues it. | +| **verdict** | What a validator returns: valid or not, with a score. | +| **harness** | What drives an agent. `cli-base` is the router-backed mode with no coding agent behind it; `claude-code`, `codex`, and `opencode` each run a real coding CLI. | + +## Which front door + +One row per entry point, ordered by how often real products use it. +Each row links to a runnable example. + +| Front door | When to call it | What you give it | What you get back | +|---|---|---|---| +| **`runAgentTaskStream`** · [example](./examples/stream-a-turn) | You run one agent turn and read its events yourself. | a task, a backend, a message | an async stream of `RuntimeStreamEvent` | +| **`handleChatTurn`** (`/durable`) · [example](./examples/chat-handler) | A web route must stream one turn to a browser and save the reply. | how to produce tokens, how to persist | an HTTP body plus a persist call after the last token | +| **`AgentExecutionBackend`** · [example](./examples/stream-backends) | You choose where the tokens come from: your loop, a sandbox, or an exact profile. | `kind` plus a `stream()` generator | the same event union from any source | +| **`runToolLoop`** (`/tool-loop`) · [example](./examples/tool-loop) | The model must call your tools and answer in the same turn. | one model turn, your executors | final text, every tool outcome, a stop reason | +| **`startRuntimeRun`** · [example](./examples/runtime-run) | You must record what a run cost and whether it succeeded. | run identity, a store adapter | a live cost tally and one persisted row | +| **`runAgentRounds`** (`/kernel`) · [example](./examples/quickstart) | One prompt is not enough, and your code owns the stop rule. | `plan`, `decide`, an output adapter, a sandbox client | every attempt, the verdicts, and a winner | +| **`supervise`** (`/kernel`) · [example](./examples/supervise) | A model must decide the plan and drive other agents. | a supervisor profile, a goal, a budget | the delivered result, or a typed reason and the spend | +| **`startRetainedRun`** (`/kernel`) · [example](./examples/retained-run) | The job must outlive the process that started it. | a provider, keys, a durable admission hook | a claim ticket any process can reattach to | +| **`improve`** · [example](./examples/improve) | You must change one part of an agent and prove the gain. | a profile field, three case sets, a judge | a detached candidate, a lift interval, ship or hold | + +Five mechanisms continue interrupted work. +Pick by what died. + +- The HTTP connection — call `streamPrompt` again with the same `executionId`. +- Nothing, but you want the same box for the next turn — `openSandboxRun`. +- The coordinator process, mid-orchestration — `supervise({ runDir })`. +- The user's chat session — the `/conversation` store adapters. +- Everything except the provider — [retained runs](./examples/retained-run). + +## Also in the box + +- **Benchmarks and leaderboards** — compare strategies with significance stats (`runBenchmark`), or stand up a harness×model board (`defineLeaderboard`): [`examples/coding-benchmark`](./examples/coding-benchmark), [`examples/webcode-matrix`](./examples/webcode-matrix). +- **Agent graphs** — fixed topologies authored as data and run through `runGraph`: [`examples/graphs`](./examples/graphs). +- **Improve a knowledge base** — a measured candidate copy of a KB, wiki, or RAG corpus: [`docs/improve.md`](./docs/improve.md). +- **PrimeIntellect** — package the same runtime program as a Verifiers environment: [`docs/primeintellect.md`](./docs/primeintellect.md). +- **Conversations** (`/conversation`) — multi-turn two-agent sessions with SQL-backed resume. +- **MCP servers** (`/mcp`) — give any agent a `delegate` tool plus live coordination tools. +- **Live run view** (`/tui`) — `agent-runtime-top` shows every supervisor run in a workspace, with steer and cancel. +- **Telemetry** — every loop emits `loop.*` trace events, exported as OpenTelemetry GenAI spans when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. + +All 33 examples live in [`examples/`](./examples). ## How it works (the short version) @@ -537,55 +120,14 @@ Use `importPrimeIntellectTraces(...)` to convert them to agent-eval `RunRecord`s - **Candidates face fresh tasks.** The optimizer uses train and selection tasks. Promotion uses a separate final set. - **Scores come from executed attempts.** Runtime recomputes results from the recorded cells and rejects incomplete cost or source evidence. -## Primitives - -The general-purpose pieces, by import path. Every export with its one-line summary lives in the generated [`docs/api/primitive-catalog.md`](./docs/api/primitive-catalog.md): check it before building anything new. - -| Primitive | What it does | Import | -|---|---|---| -| Chat-turn runtime | Stream and persist one production chat turn (`handleChatTurn`); derive its stable execution and turn identity (`deriveExecutionId`); normalize any backend's stream into one event shape (`streamAgentTurn`) | `/durable` · `/kernel` | -| Retained provider runs | Start one detached provider job with a durable admission hook, replay exact events, reconnect after restart, rebuild from pre-dispatch coordinates, continue its native context, and cancel idempotently (`startRetainedRun`, `reconnectRetainedRun`, `recoverRetainedRun`) | `/kernel` | -| Tool-call loop | Run one model turn, execute requested tools, feed results back, and stop on completion, repetition, time, or cost limits (`runToolLoop`, `streamToolLoop`) | `/tool-loop` | -| Supervision | One agent spawns, budgets, and steers workers toward a goal (`supervise`, `delegate`), on an in-process loop or a sandboxed coding harness | `/kernel` · `/mcp` | -| Loop kernel + combinators | Write a driver (`plan`/`decide`) and run it (`runAgentRounds`), or compose fixed shapes: refine (`loopUntil`), best-of-N (`fanout`), chain (`pipeline`), multi-judge (`panel`) | `/kernel` | -| Improvement driver | Optimize one part of an agent and ship only if it wins on tasks it never practiced on (`improve`); production proposal/review/activation flow | root · `/intelligence` | -| Benchmarks + leaderboards | Compare strategies with significance stats (`runBenchmark`), stand up a harness×model leaderboard (`defineLeaderboard`, `leaderboard`) | `/kernel` | -| Knowledge improvement | Produce a measured candidate copy of a KB/wiki/RAG corpus without touching the live one (`runKnowledgeImprovementJob`) | `/knowledge` | -| MCP tool servers | Give an agent a `delegate` tool or live worker-coordination tools over MCP | `/mcp` | -| Conversations + durability | Multi-turn two-agent sessions with SQL-backed resume (D1/pg/sqlite/libSQL adapters) | `/conversation` | -| Training/eval adapter | Package the same runtime program as a PrimeIntellect Verifiers environment; import its traces back | `/primeintellect` | - -| Watching live runs | A terminal view of every supervisor run in a workspace — workers, spend, tokens, latency, plus steer and cancel (`agent-runtime-top`) | `/tui` | - -Remaining subpaths: `/agent`, `/profiles`, `/platform`, `/analyst-loop`, `/environment-provider`, `/testing` (validated fixture records for consumer tests). - -## Examples - -Runnable, grouped by what they show. Copy the one nearest your task: - -| Do this | Example | -|---|---| -| The smallest complete loop (start here) | [`quickstart`](./examples/quickstart) · [`driver-loop`](./examples/driver-loop) | -| Run a product chat turn | [`chat-handler`](./examples/chat-handler) | -| Drive a team of agents to a goal | [`supervise`](./examples/supervise) · [`recursive-supervisor`](./examples/recursive-supervisor) | -| Benchmark strategies on your own domain | [`coding-benchmark`](./examples/coding-benchmark) | -| Benchmark **harnesses × models** over a real task suite (the real WebCode dataset) | [`webcode-matrix`](./examples/webcode-matrix) | -| Render a **multi-profile leaderboard** with ranked board, score matrix, and SVG/HTML charts | `leaderboard(records)` → `renderLeaderboardMarkdown` / `Svg` / `Html` | -| Trace + bill + effort-gate the WebCode benchmark (the Intelligence SDK) | [`intelligence-webcode`](./examples/intelligence-webcode) | -| Self-improve an agent, gated on a held-out set | [`improve`](./examples/improve) · [`self-improving-coder`](./examples/self-improving-coder) | -| Improve a KB, wiki, or RAG corpus with runtime agents | [`docs/canonical-api.md`](./docs/canonical-api.md) | -| Evaluate or train a runtime program on PrimeIntellect | `@tangle-network/agent-runtime/primeintellect` | -| Study coordination vs raw compute | [`ablation-suite`](./examples/ablation-suite) | - -All 29 live in [`examples/`](./examples). - ## Where to go next -- New here? [`docs/concepts.md`](./docs/concepts.md), the mental model in plain terms. +- [`docs/concepts.md`](./docs/concepts.md), the mental model in plain terms. - [`docs/canonical-api.md`](./docs/canonical-api.md), find the primitive: "I want to ___ → use ___". - [`docs/api/primitive-catalog.md`](./docs/api/primitive-catalog.md), every export in one generated, never-stale list with its import path. Check it before building anything new. -- [`docs/STABILITY.md`](./docs/STABILITY.md), what `@stable` / `@experimental` promise you, and how a symbol graduates. -- [`docs/design.md`](./docs/design.md), the design philosophy and the internal research docs behind it: background reading, not required to use the package. +- [`docs/improve.md`](./docs/improve.md), the improvement reference: optimizers, surfaces, redaction, proposal, review, activation. +- [`docs/STABILITY.md`](./docs/STABILITY.md), what `@stable` and `@experimental` promise you, and how a symbol graduates. +- [`docs/design.md`](./docs/design.md), the design philosophy and the research behind it: background reading, not required to use the package. - [`bench/HARNESS.md`](./bench/HARNESS.md), the experiment harness and how to run a benchmark. **Contributing:** `pnpm i && pnpm build && pnpm test` gets you running; the full local gate is the [`package.json`](./package.json) scripts (`lint`, `typecheck`, `docs:check`). diff --git a/docs/README.md b/docs/README.md index bc915ad0..70b03c70 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,7 +4,7 @@ The map of every doc. **Start here** if you're new; the deeper tracks follow. ## Start here -1. [The README](../README.md) — what the package is, a runnable offline quickstart, and the primitives catalog. +1. [The README](../README.md) — what the package is, a runnable offline quickstart, and the front-door table. 2. [concepts.md](./concepts.md) — the mental model (chat turns, tasks, runs) in plain terms. 3. [canonical-api.md](./canonical-api.md) — find the right primitive: "I want to ___ → use ___". 4. [../examples/](../examples) — copy a runnable example near your task. @@ -35,6 +35,8 @@ These are internal working documents: design theses, research narrative, and roa | [STABILITY.md](./STABILITY.md) | stability contract | What `@stable` / `@experimental` promise consumers, the graduation bar, and the demotion/removal policy. | | [concepts.md](./concepts.md) | mental model | The product-API layer cake (chat turns, tasks, runs) — the onramp before the loop/strategy docs. | | [glossary.md](./glossary.md) | canonical vocabulary | One definition per term, grounded to `file:line`; drifted synonyms flagged. | +| [improve.md](./improve.md) | improvement reference | The `improve()` call, the optimizer object, official GEPA and SkillOpt installs, surfaces, redaction, and the proposal→review→activation path. | +| [primeintellect.md](./primeintellect.md) | training/eval adapter | Package the same runtime program as a PrimeIntellect Verifiers environment and import its traces back. | | [execution-model.md](./execution-model.md) | the picture | The unified `Executor` port (router/bridge/cli/sandbox/BYO) + two engines, driver vs worker, spawn mechanics. | | [agent-bus-protocol.md](./agent-bus-protocol.md) | normative protocol | The multi-agent call bus — depth limits, headers, refusal contract. | | [durability-adapters.md](./durability-adapters.md) | subsystem | SQL-backed journal and restart behavior for conversations. Supervised-tree recovery is not implemented. | diff --git a/docs/api/index.md b/docs/api/index.md index bc27fc5b..94ce2014 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -8903,7 +8903,8 @@ The spawn label, when the node's `spawned` event is in this journal tree. > `readonly` `optional` **name?**: `string` -Stable identifier surfaced in trace events. Default `'driver'`. +Trace label surfaced in trace events. No behavioral effect: it never +selects a strategy or a decision path. Default `'driver'`. #### Methods @@ -8933,9 +8934,11 @@ readonly [`Iteration`](runtime.md#iteration-1)\<`Task`, `Output`\>[] > **decide**(`history`): `Decision` \| `Promise`\<`Decision`\> Inspect history and return the next state. The kernel terminates the -loop when `decide` returns a value listed in `isTerminalDecision` -(`'stop' | 'pick-winner' | 'fail' | 'done'`), when `maxIterations` -is hit, or when the abort signal fires. +loop when `decide` returns a `TerminalDecision` +(`'stop' | 'pick-winner' | 'fail' | 'done'`, exported as +`TERMINAL_DECISIONS` with the `isTerminalDecision` guard), when +`maxIterations` is hit, or when the abort signal fires. Every other +value is caller vocabulary and continues the loop. ###### Parameters diff --git a/docs/api/primitive-catalog.md b/docs/api/primitive-catalog.md index 33533c57..ed750ae7 100644 --- a/docs/api/primitive-catalog.md +++ b/docs/api/primitive-catalog.md @@ -525,7 +525,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. ### Execution kernel — recursive atom, supervision, executors, round-synchronous loop -Import from `@tangle-network/agent-runtime/kernel` — 767 exports. +Import from `@tangle-network/agent-runtime/kernel` — 770 exports. | Symbol | Kind | Summary | |---|---|---| @@ -624,6 +624,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 767 exports. | `harvestSurfaceDiffs` | function | Re-read every mounted (and watched) surface and report the ones whose settled state differs from | | `inlineSandboxClient` | function | Adapt an `ExecutorFactory` into a `SandboxClient` for `runAgentRounds`. The factory is | | `inProcessSandboxClient` | function | Adapt a single `onPrompt(prompt, ctx)` callback into a `SandboxClient` for | +| `isTerminalDecision` | function | True when the kernel stops the loop for this decision value. | | `isWaitOutcome` | function | Narrow a settlement's `out` to a wait outcome — a wait settles on the SAME cursor as workers, | | `jjWorkspace` | function | A jj-backed `Workspace` (Jujutsu, colocated with git for the durable remote). | | `kernelPromptRegistry` | function | The kernel's seeded registry: every surface the runtime's own builders derive from. A caller | @@ -782,6 +783,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 767 exports. | `strategyAuthorContract` | const | The compressed consumable a skill carries: everything an author needs to emit a loop. | | `strategyAuthorSystemPrompt` | const | Standing behavior callers put in the strategy-author AgentProfile. | | `supervisorPolicyPrompt` | const | THE supervisor policy — one stance, both front doors. The work-vs-delegate rule is conditional | +| `TERMINAL_DECISIONS` | const | Decision values the kernel treats as terminal. Every other value returned by | | `VERIFY_TAIL_CHARS` | const | Tail of the verify output — the failing assertion lives at the END of a test log. | | `WORKER_TOOL_TRACE_SCHEMA_VERSION` | const | Schema version for content-addressed worker tool-trace artifacts. | | `workerTraceSeamKey` | const | Seam key the `Scope` seeds a {@link TraceContext} under on each child's `ExecutorContext.seams`. | @@ -1130,6 +1132,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 767 exports. | `SupervisorSpanAttributes` | type | OTLP span attribute values. Exported because `SupervisorSpanOptions.attributes` is public and | | `SurfaceReader` | type | The read seam: fetch the current bytes at a mounted path. Implemented by a sandbox box's | | `SurfaceReadOutcome` | type | Outcome of reading one surface back at settle. `missing: true` means the path no longer exists | +| `TerminalDecision` | type | One of the kernel's terminal decision values. | | `ToolLoopCompactionOptions` | type | Public supervisor-facing compaction config: same knobs as the primitive, but `distill` is optional | | `ToolLoopMessageRecord` | type | Provider-neutral conversation record accepted by a tool-loop brain. | | `TrajectoryReportFn` | type | `trajectoryReport(...)` — the tree+cost reconstructor. Async (reads journal + optionally blobs). | diff --git a/docs/api/runtime.md b/docs/api/runtime.md index 09b69807..f76b3af9 100644 --- a/docs/api/runtime.md +++ b/docs/api/runtime.md @@ -19635,6 +19635,16 @@ A checkable task domain — implement these 5 hooks and the suite does the rest. *** +### TerminalDecision + +> **TerminalDecision** = *typeof* [`TERMINAL_DECISIONS`](#terminal_decisions)\[`number`\] + +**`Stable`** + +One of the kernel's terminal decision values. + +*** + ### Deliverable > **Deliverable**\<`Out`\> = \{ `kind`: `"events"`; `fromEvents`: (`events`) => `Out`; \} \| \{ `kind`: `"artifact"`; `path`: `string`; `fromArtifact`: (`raw`, `events`) => `Out`; \} @@ -21132,6 +21142,19 @@ The default registry `runPersonified` resolves a shape name against. Empty by co *** +### TERMINAL\_DECISIONS + +> `const` **TERMINAL\_DECISIONS**: readonly \[`"stop"`, `"pick-winner"`, `"fail"`, `"done"`\] + +**`Stable`** + +Decision values the kernel treats as terminal. Every other value returned by +`decide` continues the loop. Type a driver's `decide` return as +`'your-word' | TerminalDecision` so caller vocabulary and kernel keywords +stay visibly distinct. + +*** + ### strategyAuthorContract > `const` **strategyAuthorContract**: "\nYou author an OPTIMIZATION STRATEGY for an agentic loop system. A strategy decides how to\nspend a compute budget to beat a task's deployable check. You compose exactly two steps:\n\n shot(spec?: \{ handle?, messages?, steer?, persona?, tools? \}): Promise\\n Runs ONE worker attempt (a bounded tool loop) over an artifact.\n - omit handle =\> the shot opens its OWN fresh artifact and closes it after (a sample).\n - pass handle =\> the shot CONTINUES that artifact (state accumulates across shots).\n - messages =\> the carried conversation (pass the previous ShotResult.messages to continue).\n - steer =\> a corrective instruction injected before the shot.\n - persona =\> \{ systemPrompt?, model? \} — give THIS shot its own role and/or model\n (multi-agent strategies: a researcher shot then an engineer shot, a panel of k\n personas over one budget). On a fresh shot the systemPrompt replaces the task's; on\n a carried conversation it arrives as a hand-off message. Same conserved budget.\n - tools =\> string\[\] — restrict THIS shot to a subset of the task's tools by\n name (focus an explore shot on read-only tools, an execute shot on write tools).\n Restriction-only; unknown names make the shot fail. ALWAYS select from\n await listTools(handle) — never hardcode. Omitted =\> the shot sees every tool.\n ShotResult = \{ messages, score (0..1 on the task's check), passes, total, completions, toolErrors \}\n Returns null if the attempt failed infra-wise.\n\n critique(messages): Promise\\n A firewalled trace-analyst reads the attempt's trajectory and returns ONE corrective\n instruction (or null when it judges the work complete). Costs ~1 completion.\n\n consult(messages, instruction): Promise\\n The RAW analyst channel: the same firewalled critic answers YOUR instruction over the\n trajectory verbatim (no reformatting) — use it when you need a specific reply format\n (a decision, a prediction). Costs ~1 completion.\n\n surface.open(task) / surface.close(handle)\n Open a persistent artifact you manage yourself (remember to close in a finally).\n close is idempotent — closing an already-closed handle is a safe no-op.\n\n listTools(handle): Promise\\>\n The tools THIS task actually offers. TOOL SETS VARY PER TASK — if you restrict a\n shot with \`tools\`, you MUST pick names from await listTools(handle); hardcoding\n names from an example kills your shots on every task whose tools differ.\n\nRules:\n- ALWAYS await every shot/critique/surface call — a floating promise that rejects\n crashes the whole benchmark run.\n- Stay within ~budget total shots; every shot/critique spends from a conserved pool.\n- For a FRESH attempt OMIT \`messages\` entirely (never pass \`\[\]\` — an empty array is a\n fresh conversation too, but be explicit). To CONTINUE, pass the previous\n ShotResult.messages unchanged.\n- Return \{ score, resolved, completions, progression, shots \} — score = the BEST checkpoint\n you reached (keep-best, never final-state), progression = score after each shot.\n- The module must be EXACTLY this shape (no other imports, no commentary outside code):\n\nimport \{ defineStrategy \} from '@tangle-network/agent-runtime/kernel'\nexport default defineStrategy('your-strategy-name', async (\{ surface, task, budget, shot, critique, listTools \}) =\> \{\n // your composition (listTools comes from the destructured context — it is NOT a global)\n\})\n" @@ -23270,6 +23293,26 @@ a forked copy). *** +### isTerminalDecision() + +> **isTerminalDecision**(`decision`): decision is "stop" \| "done" \| "pick-winner" \| "fail" + +**`Stable`** + +True when the kernel stops the loop for this decision value. + +#### Parameters + +##### decision + +`unknown` + +#### Returns + +decision is "stop" \| "done" \| "pick-winner" \| "fail" + +*** + ### acquireSandbox() > **acquireSandbox**(`client`, `options`, `acquire?`): `Promise`\<`SandboxInstance`\> diff --git a/docs/improve.md b/docs/improve.md new file mode 100644 index 00000000..c79b7956 --- /dev/null +++ b/docs/improve.md @@ -0,0 +1,288 @@ +# Improve an agent + +`improve` runs one complete optimization method against one profile field. +The method owns candidate generation and selection. +Runtime keeps the final test set out of the method, scores the baseline and the selected candidate on it, and returns `ship` only when the paired confidence interval clears `minimumLift`. +The profile is never changed. + +The runnable offline path is [`examples/improve`](../examples/improve). +This page is the reference for the production path. + +## The call + +```ts +import { improve, officialGepa } from '@tangle-network/agent-runtime' +import { profileOptimizerModelCall } from '@tangle-network/agent-runtime/kernel' +import { + type AgentProfile, + canonicalAgentProfileDigest, + canonicalCandidateDigest, +} from '@tangle-network/agent-interface' + +const executionRef = canonicalCandidateDigest({ + deployment: process.env.AGENT_DEPLOYMENT_SHA!, + model: process.env.AGENT_MODEL!, + tools: process.env.AGENT_TOOLSET_SHA!, +}) + +const result = await improve(baseProfile, { + surface: 'prompt', + executionRef, + method: officialGepa({ + objective: 'Improve the complete support-agent prompt.', + recipe: { kind: 'engine', run: { engine: 'gepa', maxEvaluations: 40, maxProposerCostUsd: 10 } }, + optimizer, + resume: 'if-compatible', + trustResumeState: true, + describeScenario: ({ input }) => ({ input }), + }), + findings, + trainScenarios, + selectionScenarios, + testScenarios, + judges: [judge], + agent: (candidateProfile, scenario, ctx) => runProfile(candidateProfile, scenario, ctx), + runDir: '.runs/support-prompt', + costCeiling: 25, +}) + +if (result.decision === 'ship') console.log(result.candidate.profile, result.liftInterval) +``` + +## The optimizer object + +SkillOpt and GEPA's standard reflection engine require `optimizer: { model, call, callRef, budget }`. +Agent-based GEPA engines may own their model connection instead. +Runtime owns those model calls through one exact `AgentProfile`. +Agent Eval enforces the nested budget and records the measured cost and execution evidence without receiving provider credentials. + +```ts +const optimizerProfile = { + name: 'support-prompt-optimizer', + harness: 'cli-base', + model: { + provider: 'tangle-router', + default: process.env.OPTIMIZER_MODEL!, + metadata: { maxTokens: 16_384 }, + }, +} satisfies AgentProfile + +const optimizerPricing = { + inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION), + outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION), +} + +const optimizer = { + model: optimizerProfile.model.default, + call: profileOptimizerModelCall({ + profile: optimizerProfile, + context: 'support-prompt optimizer', + executor: { + backend: 'router', + routerBaseUrl: process.env.OPTIMIZER_BASE_URL!, + routerKey: process.env.OPTIMIZER_API_KEY!, + }, + pricing: optimizerPricing, + }), + callRef: canonicalCandidateDigest({ + profile: canonicalAgentProfileDigest(optimizerProfile), + deployment: process.env.OPTIMIZER_DEPLOYMENT_SHA!, + }), + budget: { + maxCostUsd: 10, + maxRequests: 50, + maxRequestBytes: 2_000_000, + maxResponseBytes: 2_000_000, + maxOutputTokensPerRequest: 16_384, + pricing: optimizerPricing, + }, +} +``` + +`costCeiling` is the total limit for optimizer calls, candidate runs, judges, and final scoring. +Runtime returns `hold` when any part of that cost is unknown. +Runtime rejects a reported total above the limit. + +## Official optimizers + +`officialGepa(...)` delegates the complete search to GEPA's upstream Optimize Anything API through agent-eval. +Pass one explicit `engine`, `sequential`, `adaptive-sequential`, `best-of`, `vote`, or `omni` recipe. +There is no local fallback. +Install its optional Python process first: + +```bash +python -m pip install "agent-eval-rpc==0.145.0" +python -m pip install "gepa[full]==0.1.4" +``` + +The published GEPA 0.1.4 wheel supports the direct `gepa` engine. +Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N require the tested official source revision: + +```bash +python -m pip install "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f" +``` + +Use `officialSkillOpt(...)` for Microsoft's SkillOpt: + +```bash +python -m pip install "agent-eval-rpc==0.145.0" +python -m pip install "skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90" +``` + +SkillOpt 0.2.0's published wheel omits prompt files that `ReflACTTrainer` requires, so the tested SkillOpt source revision stays necessary. + +### Resume and provenance + +Runtime derives the upstream resume identity from `executionRef`, the complete baseline profile, and the selected surface. +With `resume: 'if-compatible'`, agent-eval resumes only when the saved run identity matches the candidate, recipe, data, optimizer settings, runner, and derived execution identity. +Set `trustResumeState: true` only when that run directory is private to the current operator. +Use `resume: 'required'` to fail when no matching run exists. + +`result.provenance` reports the upstream package, run ID, resume status, evaluation count, and artifact directory. +`result.candidatePopulation` verifies and joins callback observations with an optimizer's official candidate graph. +It returns every unique candidate as a complete profile with ordered Interface diffs, or as an explicit materialization refusal. +GEPA candidates keep exact parent indices and selection scores; callback-only proposals report lineage as unavailable. +Methods without either artifact return `status: 'unavailable'` instead of treating the winner as the full population. + +## Surfaces + +SkillOpt accepts one text surface. +GEPA accepts text or named components. +Any complete method from `@tangle-network/agent-eval` uses the same call. + +- For a skill, set `surface: 'skills'` and `skills.resourceName`. +- For the complete profile, set `surface: 'agent-profile'`. +- To optimize several named profile fields together, also provide `profileComponents.read` and `profileComponents.apply`. + +Tools, MCP, hooks, subagents, curated instructions, and rollout policy are also exact profile coordinates. +Runtime does not choose an optimizer for them. + +The `agent` callback receives the complete immutable candidate profile, not a raw prompt or a component fragment. +Runtime uses that exact profile for every candidate run and returns the same measured profile in `result.candidate.profile`. +`executionRef` is a content digest of the agent callback, profile component mapping, model, tools, and closure settings. +Runtime combines it with the complete baseline profile and the selected surface for saved work. +A change to any of them runs the affected work again. + +Code is the exception. +It uses Runtime's isolated git worktrees and coding-agent candidate execution: + +```ts +const result = await improve({ + surface: 'code', + code: { repoRoot, baseRef, profile, generator }, + scenarios, + judge, + agent, + budget, +}) +``` + +## What leaves your process + +Without `describeScenario`, the external optimizer receives only each development case ID. +Without `describeArtifact`, evaluation feedback contains no artifact body. +When either descriptor is present, its result passes through `redact` together with findings, background text, profile name, and judge notes. +The built-in redactor removes common credentials and email addresses. +Supply a domain redactor for customer names, account IDs, or other private data the built-in rules cannot identify. +Runtime applies that hook first and then still applies its built-in scrubber. +Set `redact: false` only when every outbound value is public and already reviewed. + +The selected profile surface is the optimizer's candidate and cannot be redacted without changing the measured candidate. +Runtime always rejects recognized credentials in those bytes. +It also rejects structurally sensitive fields such as MCP env, headers, URLs, metadata, and extensions. +For `tools`, `mcp`, `hooks`, `subagents`, and `agent-profile`, Runtime treats the entire selected coordinate as execution-capable. +Use `authorizeSensitiveCandidate` to inspect and accept each exact immutable profile that contains public values or safe references. +The callback runs for the baseline and every distinct candidate before either reaches your agent. +Its `sensitivePaths` includes `$` when the whole coordinate requires review. + +## From search to production + +`improve` is the search call. +For production, `proposeAgentImprovement` adds trace analysis and reruns the exact frozen baseline and winner before it creates a reviewable proposal. +Runtime rejects a candidate bundle that differs from the search winner. + +```ts +import { + createAgentImprovementActivation, + executeAgentImprovementActivation, + proposeAgentImprovement, + reviewAgentImprovementProposal, +} from '@tangle-network/agent-runtime/intelligence' + +const baseline = freezeBaseline(liveProfile) +const result = await proposeAgentImprovement({ + runId, + profile: liveProfile, + analysis, + improvement: { + surface: 'prompt', + executionRef, + method, + trainScenarios, + selectionScenarios, + testScenarios, + judges: [judge], + agent, + }, + buildExperiment: ({ improvement }) => + buildExperimentMaterial({ + baseline, + candidate: compileCandidateBundle({ baseline, improvement: improvement.candidate }), + benchmark: heldOutBenchmark, + policy: comparisonPolicy, + }), + placeCell, +}) + +const review = reviewAgentImprovementProposal(result.proposal, { + decision: 'approve', + reviewedBy: user.id, + reason: 'The measured gain is worth the cost.', +}) +const activation = createAgentImprovementActivation(result.proposal, review, { + intent: 'activate-candidate', + targets: [{ surface: 'prompt', identity: profileId }], + fundingOwner: tenantId, + authorizedBy: user.id, + expiresAt, +}) +const outcome = await executeAgentImprovementActivation( + { proposal: result.proposal, review, activation }, + { transition: commitProfileTransaction, reconcile: readCommittedResult }, +) +``` + +`buildExperimentMaterial`, `placeCell`, and the transaction functions are application ports, because storage and compute differ by product. +The builder returns only baseline, candidate, tasks, and policy; Runtime adds the search ancestry and seals the final experiment. +Runtime owns candidate identity, measurement, review binding, expiry, retry identity, and result validation; the application owns its atomic write. +Official optimizer proposals carry the observed package versions, the optimizer model, evaluation and token usage, separate optimization and final-test costs, and the resumed-run identity. +`createOptimizationActivationReceipt(result)` exposes the same detached record for a caller that must inspect an `improve()` result before it builds a proposal. + +## Improve a knowledge base + +`runKnowledgeImprovementJob` runs KB, wiki, memory-backed, and RAG improvement jobs. +It creates a candidate copy, runs agents against it, checks it through `@tangle-network/agent-knowledge`, and returns frozen baseline and candidate snapshots with spend and timing. +It never changes the live knowledge base. + +Use `improve(profile, { surface: 'memory', ... })` for the agent's curated lesson document. +Use this job for source, retrieval, and knowledge-store changes. + +```ts +import { runKnowledgeImprovementJob } from '@tangle-network/agent-runtime/knowledge' + +const result = await runKnowledgeImprovementJob({ + root: './kb', + goal: 'Improve support refund-policy knowledge', + implementationRef: 'git:0123456789abcdef0123456789abcdef01234567', + readinessSpecs, + budget: { maxIterations: 8, maxTokens: 120_000, maxUsd: 10 }, + backend, +}) + +console.log(result.knowledge?.reference.candidateHash, result.measurement.supervisedSpent) +``` + +Set `implementationRef` to the deployed `git:<40 hex>` revision, or to a `sha256:<64 hex>` digest that covers every callback, model, index, and external setting that can change the result. +The same run ID resumes only when this identity still matches. +Measure the returned bundle pair, record the review, then activate through `executeAgentImprovementActivation`. +Activation is the only write path. diff --git a/docs/primeintellect.md b/docs/primeintellect.md new file mode 100644 index 00000000..c64ba1eb --- /dev/null +++ b/docs/primeintellect.md @@ -0,0 +1,76 @@ +# Run on PrimeIntellect + +`@tangle-network/agent-runtime/primeintellect` packages typed train and eval tasks as a PrimeIntellect Verifiers environment. +Prime launches your actual runtime program against an intercepted model endpoint, so `runPersonified`, `runAgentic`, product agents, tool calls, and multiple rounds stay intact. +Reference answers stay in Prime's task process and never enter the agent workspace. +The runner file must be one executable bundle that contains the app and its runtime dependencies. + +## Package the tasks + +```ts +import { readFile } from 'node:fs/promises' +import { + createPrimeIntellectPackage, + writePrimeIntellectPackage, +} from '@tangle-network/agent-runtime/primeintellect' + +const bundledRunner = await readFile('./dist/prime-runner.mjs', 'utf8') +const bundle = createPrimeIntellectPackage({ + name: 'support-agent', + version: '1.0.0', + tasks: [ + { + id: 'train-refund-policy', + split: 'train', + prompt: 'Can a subscription renewal be refunded?', + answer: 'No', + }, + { + id: 'eval-final-sale', + split: 'eval', + prompt: 'Can a final-sale order be refunded?', + answer: 'No', + }, + ], + scoring: { kind: 'exact', normalization: 'trim-casefold' }, + runner: { + image: 'node:22-bookworm-slim', + files: { 'runner.mjs': bundledRunner }, + command: ['node', 'runner.mjs'], + }, +}) + +await writePrimeIntellectPackage(bundle, './prime/support-agent') +``` + +## Write the runner + +The runner reads the episode and uses the normal runtime APIs. +Here, `runProductAgent` is the application's existing entry point, not another loop supplied by this adapter. + +```ts +import { + primeIntellectExecutorConfig, + runPrimeIntellectProgram, +} from '@tangle-network/agent-runtime/primeintellect' +import { + collectAgentTurn, + createExecutor, + streamAgentTurn, +} from '@tangle-network/agent-runtime/kernel' + +await runPrimeIntellectProgram(async (episode) => { + const profile = makeProductProfile({ model: episode.model.name }) + return collectAgentTurn( + streamAgentTurn( + { kind: 'executor', profile, factory: createExecutor(primeIntellectExecutorConfig(episode)) }, + episode.task.prompt, + ), + ) +}) +``` + +## Read the traces back + +Prime writes complete `traces.jsonl` rows. +Use `importPrimeIntellectTraces(...)` to convert them to agent-eval `RunRecord`s for the existing reports and release checks. diff --git a/examples/README.md b/examples/README.md index 85c2cbc4..1e6321de 100644 --- a/examples/README.md +++ b/examples/README.md @@ -5,17 +5,33 @@ budget, score them against a real check, and let them improve from their own run imports from the published package (`@tangle-network/agent-runtime`) exactly as your code would, and most run **offline with no API key** so you can see the machinery before spending anything. -New here? Run these three, in order (the first two cost $0): +## The front doors + +One directory per entry point, ordered by how often real products use it. Each carries a README with +three sections: when to use it, how to use it, and why it exists. + +| Front door | Call | Runs offline | +|---|---|---| +| [`stream-a-turn/`](./stream-a-turn/) | `runAgentTaskStream` — one turn, and you read the events | yes | +| [`chat-handler/`](./chat-handler/) | `handleChatTurn` — stream one turn over HTTP and save the reply | yes | +| [`stream-backends/`](./stream-backends/) | `AgentExecutionBackend` — choose where the tokens come from | first two sections | +| [`tool-loop/`](./tool-loop/) | `runToolLoop` — the model calls your tools, then answers | yes | +| [`runtime-run/`](./runtime-run/) | `startRuntimeRun` — the cost tally and one persisted row | yes | +| [`quickstart/`](./quickstart/) | `runAgentRounds` — several attempts under your own stop rule | yes | +| [`supervise/`](./supervise/) | `supervise` — a model decides the plan and drives other agents | needs `TANGLE_API_KEY` | +| [`retained-run/`](./retained-run/) | `startRetainedRun` — a job that outlives your process | compile-checked only | +| [`improve/`](./improve/) | `improve` — change one profile field and prove the gain | yes | + +New here? Run these three, in order (all cost $0): ```bash -pnpm tsx examples/driver-loop/driver-loop.ts # 1. one agent steering another — offline -pnpm tsx examples/improve/improve.ts # 2. an agent that rewrites its own prompt, safely — offline -TANGLE_API_KEY=... pnpm tsx examples/supervise/supervise.ts # 3. one function call = a supervisor over real workers +pnpm tsx examples/stream-a-turn/stream-a-turn.ts # 1. the smallest real path: one turn, its events +pnpm tsx examples/driver-loop/driver-loop.ts # 2. one agent steering another — see the fold +pnpm tsx examples/improve/improve.ts # 3. an agent that rewrites its own prompt, safely ``` -`driver-loop` is the core move everything else builds on; `improve` is the self-improvement primitive; -`supervise` is the one-call product entry point. [`quickstart/`](./quickstart/) is the same loop as -`driver-loop` compressed into one un-annotated file — it is the quickstart shown in the root README. +[`quickstart/`](./quickstart/) holds the root README's `minimal.ts` plus the same loop grown into a +refine loop; [`driver-loop/`](./driver-loop/) is that loop with every seam annotated. ## A few words that appear everywhere @@ -57,7 +73,6 @@ repeat. A failing validator prunes a bad candidate so the loop can't keep it. | # | Example | What it shows | |---|---|---| | 8 | [`researcher-loop/`](./researcher-loop/) | A research agent whose validator hard-fails if one tenant's data leaks into another's namespace, so the leak is pruned automatically. Needs the optional `@tangle-network/agent-knowledge` peer installed. | -| 9 | [`ui-audit/`](./ui-audit/) | The smallest end-to-end loop over a real browser (Playwright) with a stub judge, persisting the findings. | | 9b | [`coding-benchmark/`](./coding-benchmark/) | Rank coding agents (Claude Code, opencode, Codex, a bare CLI) on real tasks with an **anti-cheat**: each agent is graded on hidden tests it never saw, so it can't hardcode the answer. Includes a secondary quality judge and real significance stats. Offline by default; `--live` uses real agent boxes. | | 9c | [`webcode-matrix/`](./webcode-matrix/) | The real WebCode benchmark (Exa's 33-task dataset, graded by its own hidden tests) across a harness × model grid, rendered as a publishable leaderboard with charts, confidence bands, and pairwise significance. | @@ -65,6 +80,9 @@ repeat. A failing validator prunes a bad candidate so the loop can't keep it. | # | Example | What it shows | |---|---|---| +| 9d | [`stream-a-turn/`](./stream-a-turn/) | One turn end to end: write an `AgentExecutionBackend`, run it through `runAgentTaskStream`, and read the `RuntimeStreamEvent` union back. Offline. | +| 9e | [`tool-loop/`](./tool-loop/) | One chat turn where the model calls your tools, each result folds back, and the turn re-runs until it stops. Offline. | +| 9f | [`retained-run/`](./retained-run/) | A job the provider owns: persist the claim ticket, then rebuild control in a fresh process. Compile-checked; needs your own provider to run. | | 10 | [`knowledge-gating/`](./knowledge-gating/) | Stop an agent before it acts on facts it isn't confident about: the loop blocks when a required-knowledge confidence is below threshold. | | 11 | [`runtime-run/`](./runtime-run/) | The run-record + cost-ledger you persist for dashboards — one row per run, any database. | | 12 | [`stream-backends/`](./stream-backends/) | Pick where an agent's streaming output comes from (in-process iterator, cloud sandbox, or an OpenAI-compatible endpoint) behind one wire format. The OpenAI path needs `OPENAI_API_KEY`; the rest is offline. | @@ -115,6 +133,8 @@ From the repo root: ```bash # Start here +pnpm tsx examples/stream-a-turn/stream-a-turn.ts +pnpm tsx examples/tool-loop/tool-loop.ts pnpm tsx examples/chat-handler/chat-handler.ts pnpm tsx examples/strategy-suite/strategy-suite.ts # offline; TANGLE_API_KEY swaps in the real model pnpm tsx examples/recursive-supervisor/recursive-supervisor.ts diff --git a/examples/chat-handler/README.md b/examples/chat-handler/README.md index 19c4422e..e3e0738a 100644 --- a/examples/chat-handler/README.md +++ b/examples/chat-handler/README.md @@ -1,75 +1,64 @@ -# Stream an agent's reply over HTTP, then save it — in one call +# Serve one chat turn over HTTP -You have an agent that emits its reply token-by-token. You want to serve that to a browser as a -live, streaming HTTP response, and once the reply finishes, save the final text to your database. -`handleChatTurn` (from the edge-safe `@tangle-network/agent-runtime/durable` entry point) does exactly that plumbing so you -don't hand-roll it. +## When to use it -Give it two things — a `produce()` function that yields your agent's token stream, and a -`persistAssistantMessage` hook — and it hands back a ready-to-return HTTP body that streams every -token as it arrives, wrapped in a clean start/finish envelope, and calls your save hook the instant -the stream ends. +Use this when a web route must stream one turn to a browser and save the reply. +`handleChatTurn` owns the framing every chat product hand-rolls: NDJSON lines, a start and finish envelope, and one persist call after the last token. +You give it how to produce the response and how to persist it; it streams, traces, and persists. -## Why it matters +Use a sibling instead when you do not need the HTTP layer. -Every chat product re-implements the same fiddly middle layer: turn an async stream of partial -tokens into a well-framed HTTP response, tell the client when the turn starts and ends, and persist -the final message exactly once after the last token. Get the framing wrong and the client hangs, -double-saves, or drops the tail. `handleChatTurn` is that layer, done once and correctly: - -- **Streams as NDJSON** — one JSON event per line (`application/x-ndjson`), so a browser or any HTTP - client reads tokens live instead of waiting for the whole reply. -- **Wraps each turn** in a `session.run.started` / `session.run.completed` envelope, so the client - always knows when a turn begins and ends. -- **Persists after the stream drains**, exactly once — your `persistAssistantMessage` hook fires - with the final text after the last token, the natural place to write to your DB. +| You need | Use | +|---|---| +| The event stream, with no HTTP framing | [`../stream-a-turn`](../stream-a-turn) | +| The model to call your tools inside the turn | [`../tool-loop`](../tool-loop) | +| A cost figure and one persisted row | [`../runtime-run`](../runtime-run) | +| A turn that survives the reader disconnecting | [`../retained-run`](../retained-run) | -## Run it — no key, no network +## How to use it ```bash -pnpm tsx examples/chat-handler/chat-handler.ts +pnpm build && pnpm tsx examples/chat-handler/chat-handler.ts ``` -The example scripts a tiny two-turn tax-assistant conversation so it runs fully offline. You'll see -each turn start, stream (one `.` per token chunk), finish, persist, and print its final text: +The example scripts a two-turn conversation, so it runs offline and needs no credentials. +It prints each turn as it starts, streams, finishes, and persists: -``` +```text [run started ] turn=0 ............... [run done ] turn=0 [persist ] turn=0 chars=52 [turn 0 text ] Acknowledged: "Where do I start with my 2026 return?". Drafting a reply. - -[run started ] turn=1 -................... -[run done ] turn=1 -[persist ] turn=1 chars=62 -[turn 1 text ] The 2026 return is missing Schedule B and one W-2. Please upload them. ``` -The bottom half of `chat-handler.ts` reads the stream back by hand only so the example is -self-contained — that reader is illustrative, not something to copy. In your product you just return -`result.body` from your HTTP route and any NDJSON reader on the client consumes it. - -## From offline to production — one swap - -The only scripted part is `produce()`. In a real product it wraps a profile-bound Runtime turn: +In a route you return `result.body` and let the client read the NDJSON. ```ts -produce: () => streamAgentTurn( - { kind: 'executor', profile, factory: createExecutor(executorConfig) }, - input, -) +const executionId = deriveExecutionId({ projectId, sessionId: threadId, turnIndex }) +const result = handleChatTurn({ + identity: { tenantId, sessionId: threadId, userId, turnIndex }, + hooks: { + produce: () => ({ stream: box.streamPrompt(userMessage, { sessionId: threadId, executionId, turnId: executionId, detach: true }), finalText: () => box.lastResponse() }), + persistAssistantMessage: async ({ identity, finalText }) => db.insertMessage(identity, finalText), + }, + waitUntil, +}) +return new Response(result.body, { headers: { 'content-type': result.contentType } }) ``` -The exact `AgentProfile` selects the model and tools; the Runtime executor owns credentials, -routing, retries, and usage evidence. +Only `produce()` is scripted in the example. +In production it wraps a profile-bound turn: `streamAgentTurn({ kind: 'executor', profile, factory: createExecutor(config) }, input)`. -Everything else — the NDJSON framing, the `session.run.*` envelope, the after-drain persist — stays -identical. That framing is the whole point of `handleChatTurn`. +Two identity rules matter. +For a stream reconnect, call `streamPrompt` again with the same `executionId` and the last event id the client received. +For a repeated first dispatch, reuse both `sessionId` and `turnId`, because `executionId` alone is not an idempotency key. -## Files +The bottom half of [`chat-handler.ts`](./chat-handler.ts) reads the stream back by hand only to keep the file self-contained. +Do not copy that reader. -| file | what it is | -|---|---| -| `chat-handler.ts` | the full example: a scripted producer, `handleChatTurn`, and a hand-written reader that prints the stream | +## Why this exists + +The middle layer of a chat product is small, fiddly, and always the same. +Bad framing makes the client hang, save twice, or drop the tail of a reply. +`handleChatTurn` does that layer once: one JSON event per line, a `session.run.started` and `session.run.completed` envelope, and one persist call after the stream drains. diff --git a/examples/improve/README.md b/examples/improve/README.md index 7bd6c027..e98701a4 100644 --- a/examples/improve/README.md +++ b/examples/improve/README.md @@ -1,52 +1,50 @@ -# Improve one agent profile field +# Improve one profile field -`improve()` runs a complete optimization method against one profile field. -The method receives train and selection cases. -Runtime keeps the final-test cases private, compares the selected candidate with the baseline, and returns a detached candidate. -It never changes the input profile. +## When to use it -```bash -pnpm tsx examples/improve/improve.ts -``` +Use this when you must change one part of an agent and prove the change is better. +`improve()` runs a complete optimization method against one profile field, such as the prompt, a skill, memory, or code. +Runtime keeps the final-test cases away from the method, compares the selected candidate with the baseline, and returns a detached candidate. +It never changes the input profile. -Runs offline, no credentials. +Use a sibling instead when the job is different. +[`../self-improving-loop`](../self-improving-loop) unrolls the same flow step by step, so you can see which part owns each phase. +[`../self-improving-coder`](../self-improving-coder) runs it on a coding task graded by real tests. +[`../strategy-evolution`](../strategy-evolution) searches coordination tactics instead of a profile field. +[`../intelligence-recommend`](../intelligence-recommend) starts from a run trace and ends at a reviewable proposal. -## What it does, step by step +## How to use it -1. Runtime extracts the exact profile field selected by `surface`. -2. Runtime binds saved work to `executionRef` plus the complete baseline profile. -3. The supplied `OptimizationMethod` generates and selects a candidate using train and selection cases. -4. Runtime scores the baseline and candidate on the untouched final-test cases. -5. Runtime returns `ship` only when the paired confidence interval clears the required lift. -6. Approval and activation remain separate operations. +```bash +pnpm build && pnpm tsx examples/improve/improve.ts +``` -## What you'll see +The example runs offline. +It supplies a deterministic method, agent, and judge: the method returns `PROMOTED`, and the judge scores that literal string as `1`. +The partition firewall, the final comparison, the cost receipts, and the confidence interval are production code. -``` +```text improve() proposed a detached prompt candidate and measured it on final-test scenarios decision: ship lift: 1.000 candidate prompt: PROMOTED live prompt unchanged: BASELINE ``` -The starting prompt is `BASELINE`; the candidate is `PROMOTED`. -The final-test lift is `1.000`. - -## How it stays offline - -The example supplies a deterministic complete method, agent, and judge. -The method returns `PROMOTED`; the judge scores that literal string as `1`. -The partition firewall, final comparison, cost receipts, and confidence interval are production code. +Six steps run inside the call. -## Going live - -Replace `scriptedWinner` with `officialGepa(...)`, `officialSkillOpt(...)`, or another complete method from `@tangle-network/agent-eval`. -The root README documents the optional Python installation for official GEPA. +1. Runtime extracts the exact profile field named by `surface`. +2. Runtime binds saved work to `executionRef` plus the complete baseline profile. +3. The method generates and selects a candidate from the train and selection cases. +4. Runtime scores the baseline and the candidate on the untouched final-test cases. +5. Runtime returns `ship` only when the paired confidence interval clears the required lift. +6. Approval and activation stay separate operations. -## Files +To go live, replace the scripted method with `officialGepa(...)`, `officialSkillOpt(...)`, or another complete method from `@tangle-network/agent-eval`. +Those methods need an optimizer object, an optional Python process, and a redaction review. +[`docs/improve.md`](../../docs/improve.md) is the reference for all of it, including the production proposal, review, and activation path. -| file | what it is | -|---|---| -| `improve.ts` | The profile, complete method, three partitions, agent, judge, and result | +## Why this exists -The same path is covered by `src/improvement/improve.test.ts`. +A prompt edit that looks better is not a measured gain. +This call separates the three sets — train, selection, and final test — so the method never sees the cases that decide the release. +The result is a detached candidate plus an interval, so a human approves a number instead of a hunch. diff --git a/examples/quickstart/README.md b/examples/quickstart/README.md index 2b098d07..b9f2ebe8 100644 --- a/examples/quickstart/README.md +++ b/examples/quickstart/README.md @@ -1,13 +1,51 @@ -# quickstart +# Run a scripted loop -The smallest complete agent loop, in one file: a driver runs a worker, reads the worker's real output, and writes the next prompt from it until a check passes. -Offline and deterministic — the worker is a scripted stand-in (`inProcessSandboxClient`), so it runs with zero credentials. +## When to use it + +Use this when one prompt is not enough, and you must run several attempts under your own rules. +A driver runs a worker, reads its output, and writes the next prompt until a check passes. +You own the rules: `plan` returns the tasks to run this iteration, and `decide` says whether the loop continues. + +Use a sibling instead when the shape is different. + +| Your shape | Use | +|---|---| +| One turn, read the events yourself | [`../stream-a-turn`](../stream-a-turn) | +| One turn where the model calls your tools | [`../tool-loop`](../tool-loop) | +| A model decides what to do next, not your code | [`../supervise`](../supervise) | +| A job that must outlive your process | [`../retained-run`](../retained-run) | + +## How to use it + +Two files, both offline and deterministic. +The worker is a scripted stand-in, so neither needs credentials. ```bash -pnpm build && pnpm tsx examples/quickstart/quickstart.ts +pnpm build +pnpm tsx examples/quickstart/minimal.ts # the smallest call the types accept +pnpm tsx examples/quickstart/quickstart.ts # the same call, refining until a check passes +``` + +[`minimal.ts`](./minimal.ts) is the root README quickstart, kept compiling by `pnpm typecheck:examples`. +It prints: + +```text +decision: done — 1 iteration(s) +``` + +[`quickstart.ts`](./quickstart.ts) adds the fold: it reads the last output and writes the next prompt from it. + +```ts +plan: async (task, history) => { + const last = history[history.length - 1] + if (!last) return [task] // shot 0: run the task as written + if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots + // The core move: read the last worker's real output, write the next prompt FROM it. + return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] +}, ``` -Expected output: +It prints: ```text shot 0: reject — "Shipped one-click restore." @@ -15,5 +53,19 @@ shot 1: PASS — "Shipped one-click restore with an instant rollback path." decision: pick-winner — winner: shot 1 ``` -This is the loop the root README shows. -The fully annotated version, with every seam explained (the fold, terminal decisions, the output adapter, the validator), is [`../driver-loop`](../driver-loop). +Two rules decide when the loop stops. +`plan` returns `[]` when it has no more work. +`decide` returns a value: the four keywords in `TERMINAL_DECISIONS` (`stop`, `pick-winner`, `fail`, `done`) end the loop, and every other value is your own vocabulary and continues it. +Type the return as `'your-word' | TerminalDecision` to keep the two apart. + +`driver.name` is a trace label. +It never selects a strategy or a decision path. + +The annotated version of the same loop, with every seam explained, is [`../driver-loop`](../driver-loop). + +## Why this exists + +A retry that sends the same prompt again learns nothing. +This loop reads what the worker produced, scores it with your check, and writes the next prompt from the real output. +The kernel owns the parts that are easy to get wrong: one fresh worker per attempt, a hard iteration cap, and teardown of every worker at the end. +Your driver stays a few lines of plain code. diff --git a/examples/quickstart/minimal.ts b/examples/quickstart/minimal.ts new file mode 100644 index 00000000..fbfad70a --- /dev/null +++ b/examples/quickstart/minimal.ts @@ -0,0 +1,42 @@ +/** + * minimal — the root-README quickstart, kept compiling by `pnpm typecheck:examples`. + * One worker attempt, parsed and returned. Offline, deterministic, no API keys. + * + * Run: pnpm build && pnpm tsx examples/quickstart/minimal.ts + */ + +import type { AgentProfile } from '@tangle-network/agent-interface' +import { + inProcessSandboxClient, + runAgentRounds, + type TerminalDecision, +} from '@tangle-network/agent-runtime/kernel' +import type { SandboxEvent } from '@tangle-network/sandbox' + +const profile = { + name: 'note-writer', + harness: 'cli-base', + model: { provider: 'scripted', default: 'scripted/note-writer' }, +} satisfies AgentProfile + +// A scripted worker. Swap in a sandbox, CLI-harness, or router backend later. +const worker = inProcessSandboxClient({ + onPrompt: (): SandboxEvent[] => [ + { type: 'result', data: { result: { note: 'Shipped one-click restore.' } } }, + ], +}) + +const result = await runAgentRounds({ + task: 'Write a one-line release note for one-click restore.', + driver: { + // plan returns the tasks to run this iteration; [] means no more work. + plan: async (task, history) => (history.length === 0 ? [task] : []), + // 'done' is one of the four kernel keywords in TERMINAL_DECISIONS. + decide: (): TerminalDecision => 'done', + }, + agentRun: { profile, taskToPrompt: (t) => t }, + output: { parse: (events) => events }, + ctx: { sandboxClient: worker }, +}) + +console.log(`decision: ${result.decision} — ${result.iterations.length} iteration(s)`) diff --git a/examples/quickstart/quickstart.ts b/examples/quickstart/quickstart.ts index 1fa6f0ad..e1d69989 100644 --- a/examples/quickstart/quickstart.ts +++ b/examples/quickstart/quickstart.ts @@ -1,5 +1,5 @@ /** - * quickstart — the smallest complete agent loop. Offline, deterministic, no API keys. + * quickstart — the smallest complete refine loop. Offline, deterministic, no API keys. * * One driver runs a worker, reads the worker's real output, and writes the next prompt * from it until a check passes. The worker here is a scripted stand-in so the loop runs @@ -11,7 +11,11 @@ */ import type { AgentProfile } from '@tangle-network/agent-interface' -import { inProcessSandboxClient, runAgentRounds } from '@tangle-network/agent-runtime/kernel' +import { + inProcessSandboxClient, + runAgentRounds, + type TerminalDecision, +} from '@tangle-network/agent-runtime/kernel' import type { SandboxEvent } from '@tangle-network/sandbox' type Task = { prompt: string } @@ -39,27 +43,11 @@ const worker = inProcessSandboxClient({ ], }) -const result = await runAgentRounds({ - task: { prompt: 'Write a one-line release note for one-click restore.' }, - driver: { - name: 'refine', - plan: async (task, history) => { - const last = history[history.length - 1] - if (!last) return [task] // shot 0: run the task as written - if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots - // The core move: read the last worker's real output, write the next prompt FROM it. - return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] - }, - decide: (history) => - history.some((shot) => shot.verdict?.valid) - ? 'pick-winner' - : history.length < 3 - ? 'refine' - : 'fail', - }, +const result = await runAgentRounds({ + task: { prompt: 'Write a one-line release note for one-click restore.' } as Task, agentRun: { profile: noteWriterProfile, taskToPrompt: (t) => t.prompt }, output: { - parse: (events) => { + parse: (events): Note => { for (const ev of events) { if (ev.type === 'result') { const r = (ev as { data?: { result?: unknown } }).data?.result @@ -75,8 +63,26 @@ const result = await runAgentRounds { + const last = history[history.length - 1] + if (!last) return [task] // shot 0: run the task as written + if (last.verdict?.valid || history.length >= 3) return [] // done, or out of shots + // The core move: read the last worker's real output, write the next prompt FROM it. + return [{ prompt: `Rewrite "${last.output?.note}" to mention the rollback path.` }] + }, + // 'refine' is this driver's own word — any non-terminal value continues the + // loop. 'pick-winner' and 'fail' are kernel keywords from TERMINAL_DECISIONS. + decide: (history): 'refine' | TerminalDecision => + history.some((shot) => shot.verdict?.valid) + ? 'pick-winner' + : history.length < 3 + ? 'refine' + : 'fail', + }, ctx: { sandboxClient: worker }, - maxIterations: 3, }) for (const shot of result.iterations) { diff --git a/examples/retained-run/README.md b/examples/retained-run/README.md new file mode 100644 index 00000000..89156345 --- /dev/null +++ b/examples/retained-run/README.md @@ -0,0 +1,72 @@ +# Keep a run alive after your process dies + +## When to use it + +Use this when the job must outlive the process that started it. +The provider owns the job; you own a claim ticket that any process can present. + +The runtime has five ways to continue work. +Pick by what died. + +| What died | What continues it | +|---|---| +| The HTTP connection to the browser | Call `streamPrompt` again with the same `executionId` and the last event id | +| Nothing; you want the same box for the next turn | `openSandboxRun` | +| The coordinator process, mid-orchestration | `supervise({ runDir })`, which replays settled children | +| The user left and came back to a chat | The `/conversation` store adapters | +| Everything except the provider | **This example**: `startRetainedRun` and `reconnectRetainedRun` | + +The last row is the expensive one. +Use it only when a dropped reader or a restarted application must not lose the job. + +## How to use it + +```bash +pnpm typecheck:examples # this file is compile-checked, not runnable offline +``` + +`startRetainedRun` refuses a provider that cannot promise exact run identity, event and result identity, idempotent cancellation, detach, and replay. +No provider in this repository advertises those seven capabilities, so supply your own provider to run [`retained-run.ts`](./retained-run.ts). + +Process A starts the job and persists the ticket. + +```ts +const run = await startRetainedRun({ + provider, + environment: { idempotencyKey: 'workspace-42', profile }, + turn: { turnId: 'turn-7', prompt: 'Finish the migration and run its tests.' }, + identity: { sessionId: 'thread-42', executionId: 'execution-7' }, + onAdmission: async (admission) => { + await journal.write(admission) + }, +}) +``` + +The runtime awaits `onAdmission` twice: after the environment exists, and again after the dispatch is verified. +The start promise resolves only after the second record is durable. +Write each record inside the hook before the hook returns. + +Process B rebuilds control from the persisted ticket. + +```ts +const handle = await reconnectRetainedRun({ provider, controlRef: dispatched.controlRef }) +const snapshot = await handle.status({ waitMs: 30_000 }) +const result = await handle.result() +``` + +Three rules keep a restart safe. + +- Persist each event cursor and sequence before you show the event to a user. +- After a crash that landed only the environment record, call `recoverRetainedRun` with its coordinates. +- Never destroy an environment on the `unverifiable` outcome. Keep it, retry the reconnect later, or inspect it with the provider's tools. + +An unknown provider result stays unknown. +The runtime never reports it as success or as a confirmed cancellation. + +## Why this exists + +`runAgentRounds` is a loop your process runs. +It holds your `plan`, `decide`, `parse`, and `validate` functions in memory, so when your process dies the loop dies and nothing outside ever knew it existed. +A retained run is a job the provider runs: your process holds only a claim ticket — provider, environment, session, execution, and request digest — so any process holding the ticket can reattach, replay, or cancel, which is why every read is verified against the ticket instead of trusted from memory. +It is a separate call because it must refuse providers that cannot honor the ticket, and it must force you to save the ticket durably before it reports success. +Neither demand can be bolted onto the in-process loop without breaking its callers or reopening the crash-orphan bug it was built to close. diff --git a/examples/retained-run/retained-run.ts b/examples/retained-run/retained-run.ts new file mode 100644 index 00000000..6a5c4a1e --- /dev/null +++ b/examples/retained-run/retained-run.ts @@ -0,0 +1,112 @@ +/** + * retained-run — hold a claim ticket to a job the provider runs. + * + * `runAgentRounds` holds your closures, so the run dies with your process. A + * retained run lives in the provider. You keep only a plain-data control + * reference, and any process that holds it can replay, resume, or cancel. + * + * The two functions below are the copyable shape: start and persist the ticket, + * then rebuild control from the persisted ticket in a fresh process. + * + * This file is compile-checked by `pnpm typecheck:examples`. It does not run + * offline: `startRetainedRun` refuses a provider that cannot promise exact run + * identity, event and result identity, idempotent cancellation, detach, and + * replay, and no provider in this repo advertises them. Supply your own + * provider to run it. + */ + +import type { AgentProfile } from '@tangle-network/agent-interface' +import type { AgentEnvironmentProvider } from '@tangle-network/agent-interface/environment-provider' +import { + type RetainedRunAdmission, + type RetainedRunDispatchedAdmission, + reconnectRetainedRun, + recoverRetainedRun, + startRetainedRun, +} from '@tangle-network/agent-runtime/kernel' + +/** Your durable store for admission records. One row per record is enough. */ +interface AdmissionJournal { + write(admission: RetainedRunAdmission): Promise + readDispatched(turnId: string): Promise +} + +/** + * Process A. Start the job and stream it while this process lives. + * + * `onAdmission` is awaited twice: after the environment exists, and again after + * the dispatch is verified. The start promise resolves only after the second + * record is durable, so a crash can never lose a live run. + */ +export async function startAndStream( + provider: AgentEnvironmentProvider, + journal: AdmissionJournal, + profile: AgentProfile, +): Promise { + const run = await startRetainedRun({ + provider, + environment: { idempotencyKey: 'workspace-42', profile }, + turn: { turnId: 'turn-7', prompt: 'Finish the migration and run its tests.' }, + identity: { sessionId: 'thread-42', executionId: 'execution-7' }, + onAdmission: async (admission) => { + await journal.write(admission) + }, + }) + + // Every event carries its cursor and sequence. Persist them before you show + // the event to a user, so a replay resumes strictly after what the user saw. + for await (const envelope of run.events()) { + console.log(`${envelope.sequence}: ${envelope.event.type}`) + } +} + +/** + * Process B. Rebuild control from the persisted ticket. + * + * `reconnectRetainedRun` rejects any mismatch of provider, environment, + * session, execution, run, or request digest, so a wrong ticket fails loud + * instead of controlling somebody else's job. + */ +export async function reattach( + provider: AgentEnvironmentProvider, + journal: AdmissionJournal, +): Promise { + const dispatched = await journal.readDispatched('turn-7') + if (!dispatched) throw new Error('no dispatched admission record for turn-7') + + const handle = await reconnectRetainedRun({ provider, controlRef: dispatched.controlRef }) + if (!handle) throw new Error('the provider no longer retains this environment') + + const snapshot = await handle.status({ waitMs: 30_000 }) + console.log(`status: ${snapshot.status ?? 'unknown'} — effect: ${snapshot.effect}`) + + const result = await handle.result() + return result.text +} + +/** + * Process B, after a crash that landed only the environment record. + * + * Never destroy the environment on `unverifiable`. Keep it, retry the reconnect + * when a dispatched record appears, or inspect it with the provider's tools. + */ +export async function recoverFromEnvironmentRecord( + provider: AgentEnvironmentProvider, + environmentId: string, +): Promise { + const recovery = await recoverRetainedRun({ + provider, + environmentId, + sessionId: 'thread-42', + executionId: 'execution-7', + }) + if (recovery.outcome === 'recovered') console.log(await recovery.handle.result()) + if (recovery.outcome === 'not_found') console.log('the environment is gone; nothing to clean up') + if (recovery.outcome === 'unverifiable') console.log('keep the environment and retry later') +} + +if (process.argv[1]?.endsWith('retained-run.ts')) { + console.log( + 'Compile-checked template. Pass your own AgentEnvironmentProvider to startAndStream / reattach.', + ) +} diff --git a/examples/runtime-run/README.md b/examples/runtime-run/README.md index 0f9abb04..4796a9a9 100644 --- a/examples/runtime-run/README.md +++ b/examples/runtime-run/README.md @@ -1,78 +1,50 @@ -# A cost-and-audit ledger for any agent run — one row, any database +# Record what a run cost -Wrap a running agent task and this records what it cost and what happened: every model call rolls up -into a live tally (tokens in, tokens out, dollars, wall-clock, number of calls), and at the end you -get one durable row — run id, workspace, session, status, cost, start/finish timestamps — written to -whatever store you use through a single `upsert(row)` method. Point that method at Postgres, SQLite, -Cloudflare D1, or an in-memory array; the ledger doesn't care. +## When to use it -## Why it matters +Use this when you must record what a run cost and whether it succeeded. +`startRuntimeRun` opens a run handle, keeps a live tally from the stream, and writes one canonical row through your store adapter. +The tally counts model calls only, so you can pipe the whole stream through it. -Once agents run in production you need to answer "what did this run cost?" and "did it succeed?" for -every session — for dashboards, billing, and audit. Hand-rolling that means threading cost -accounting through your whole stream and inventing a row schema. This gives you both: a correct cost -tally that ignores everything except model-call events, and a canonical row shape you persist with -one method. Backend-agnostic (works over any agent backend) and store-agnostic (works over any DB). +Use a sibling instead when cost is not the question. +[`../stream-a-turn`](../stream-a-turn) is the same stream with no ledger. +[`../chat-handler`](../chat-handler) adds the HTTP framing and the message write. +[`../sanitized-telemetry-streaming`](../sanitized-telemetry-streaming) redacts user data before you log it. -## How it works - -Four calls make up the lifecycle: - -1. `startRuntimeRun({ workspaceId, sessionId, agentId, taskSpec, adapter })` opens a run. The - identity fields land in the persisted row's typed columns; the task spec carries only what - describes the *work*. -2. `run.observe(event)` on every streamed event keeps the cost tally in sync. Only `llm_call` events - add to it — everything else is a no-op, so you can safely pipe the *entire* stream through it. -3. `run.complete({ status, resultSummary, error? })` exactly once at end of stream. It's idempotent - for the same status and throws if you try to change a status after the fact. -4. `run.persist()` writes the row via your adapter; `run.cost()` returns the live tally any time. - -## Run — fully offline, no key, no network +## How to use it ```bash -pnpm tsx examples/runtime-run/runtime-run.ts +pnpm build && pnpm tsx examples/runtime-run/runtime-run.ts ``` -A toy backend emits two model calls and some text, so the ledger has real numbers to add up. You'll -see the accumulated cost, then the exact row that would hit your database: +The example runs offline. +A toy backend emits two model calls, so the ledger has real numbers: -``` -Cost ledger: { - tokensIn: 1800, - tokensOut: 390, - costUsd: 0.0060999999999999995, - wallMs: 1, - llmCalls: 2 -} +```text +Cost ledger: { tokensIn: 1800, tokensOut: 390, costUsd: 0.0061, wallMs: 1, llmCalls: 2 } Persisted row: { id: 'legal-chat:thread-42:puomxsx1', - workspaceId: 'ws-1', - sessionId: 'thread-42', - agentId: 'legal-chat-runtime', - domain: 'legal', - taskId: 'legal-chat:thread-42', - scenarioId: 'legal-chat:thread-42', - status: 'completed', - resultSummary: 'Reviewed', + workspaceId: 'ws-1', sessionId: 'thread-42', agentId: 'legal-chat-runtime', + status: 'completed', resultSummary: 'Reviewed', cost: { tokensIn: 1800, tokensOut: 390, costUsd: 0.0061, wallMs: 1, llmCalls: 2 }, - startedAt: '...', completedAt: '...', - metadata: { note: 'demo persistence metadata' } + startedAt: '...', completedAt: '...' } ``` -`1800` in / `390` out / `2` calls is the tally summed from the two `llm_call` events (1200+600 in, -280+110 out) — proof `observe` only counts model calls. +The tally sums the two `llm_call` events (1200+600 in, 280+110 out), which proves `observe` counts model calls only. + +Four calls make the lifecycle. -## Make it real +1. `startRuntimeRun({ workspaceId, sessionId, agentId, taskSpec, adapter })` opens the run. The identity lands in typed columns; the task spec describes only the work. +2. `run.observe(event)` on every streamed event keeps the tally correct. +3. `run.complete({ status, resultSummary, error })` once, at the end of the stream. It is idempotent for the same status and refuses a changed status. +4. `run.persist()` writes the row. `run.cost()` returns the live tally at any time. -- **Real work:** replace the toy backend with `createSandboxPromptBackend` (a cloud sandbox) or - another caller-owned `AgentExecutionBackend`. Paid model work uses `streamAgentTurn` with an - exact `AgentProfile` and Runtime executor instead of a provider transport in this lifecycle. -- **Real store:** implement `RuntimeRunPersistenceAdapter` — one `upsert(row)` method — against D1, - Postgres, or your existing runs table. The row shape doesn't change. +Implement `RuntimeRunPersistenceAdapter` — one `upsert(row)` method — against D1, Postgres, SQLite, or your existing runs table. +The row shape does not change. -## Files +## Why this exists -| file | what it is | -|---|---| -| `runtime-run.ts` | the full lifecycle: open → observe → complete → persist, with a toy backend and in-memory store | +Once agents run in production, two questions arrive for every session: what did this cost, and did it work. +Answering them by hand means threading cost accounting through the whole stream and inventing a row schema. +This gives you a correct tally and one canonical row, over any backend and any store. diff --git a/examples/stream-a-turn/README.md b/examples/stream-a-turn/README.md new file mode 100644 index 00000000..5549732d --- /dev/null +++ b/examples/stream-a-turn/README.md @@ -0,0 +1,60 @@ +# Stream one agent turn + +## When to use it + +Use this when you run one agent turn and read its events yourself. +This is the smallest complete path through the runtime, and it is the path most products take. +Two contracts carry it: you write an `AgentExecutionBackend`, and you read `RuntimeStreamEvent` values back. + +Use a sibling instead when you need more than the raw stream. + +| You also need | Use | +|---|---| +| An HTTP response and a saved reply | [`../chat-handler`](../chat-handler) | +| A cost figure and one persisted row | [`../runtime-run`](../runtime-run) | +| A ready-made backend for a model, a sandbox, or your own loop | [`../stream-backends`](../stream-backends) | +| The model to call your tools inside the same turn | [`../tool-loop`](../tool-loop) | +| Several attempts under your own stop rule | [`../quickstart`](../quickstart) | + +## How to use it + +```bash +pnpm build && pnpm tsx examples/stream-a-turn/stream-a-turn.ts +``` + +The example runs offline and needs no credentials. +It prints: + +```text +Checking the refund policy +A renewal is refundable within 14 days of the charge. + +status: completed — backend completed +tools: search_policy +cost: $0.0009 — reply chars: 81 +``` + +Read [`stream-a-turn.ts`](./stream-a-turn.ts) for the whole file. +The two contracts are these. + +```ts +const backend: AgentExecutionBackend = { + kind: 'scripted-support', + async *stream(input, ctx): AsyncIterable { + yield { type: 'text_delta', text: 'Checking the refund policy', timestamp: now() } + }, +} + +for await (const event of runAgentTaskStream({ task, backend, input, sessionStore, sessionId })) { + if (event.type === 'text_delta') process.stdout.write(event.text) +} +``` + +Replace the scripted backend with your model call, your sandbox, or your coding CLI. +The reader code does not change. + +## Why this exists + +`RuntimeStreamEvent` is one union for every event source, so your product code reads one shape. +The runtime adds the parts a turn always needs around your backend: readiness checks, session start or resume, and a terminal `final` event. +Your backend stays small because it only produces events. diff --git a/examples/stream-a-turn/stream-a-turn.ts b/examples/stream-a-turn/stream-a-turn.ts new file mode 100644 index 00000000..f625cf12 --- /dev/null +++ b/examples/stream-a-turn/stream-a-turn.ts @@ -0,0 +1,106 @@ +/** + * stream-a-turn — run one agent turn and read its events. + * + * Two contracts, and nothing else: you write an `AgentExecutionBackend` that + * yields `RuntimeStreamEvent` values, and you read the same union back out of + * `runAgentTaskStream`. Offline, deterministic, no API keys. + * + * Run: pnpm build && pnpm tsx examples/stream-a-turn/stream-a-turn.ts + */ + +import type { + AgentBackendContext, + AgentBackendInput, + AgentExecutionBackend, + AgentTaskSpec, + RuntimeStreamEvent, +} from '@tangle-network/agent-runtime' +import { InMemoryRuntimeSessionStore, runAgentTaskStream } from '@tangle-network/agent-runtime' + +// The unit of work. `id` is yours; `intent` and `domain` describe the work. +const task: AgentTaskSpec = { + id: 'support:thread-42', + intent: 'Answer one support question about refunds.', + domain: 'support', +} + +// The backend contract every product implements. `kind` names it in the trace. +// `stream` yields the turn's events. A real backend calls a model API, a +// sandbox, or a coding CLI here; this one is scripted so the file runs offline. +const backend: AgentExecutionBackend = { + kind: 'scripted-support', + async *stream( + input: AgentBackendInput, + ctx: AgentBackendContext, + ): AsyncIterable { + const now = () => new Date().toISOString() + yield { type: 'text_delta', text: 'Checking the refund policy', timestamp: now() } + yield { + type: 'tool_call', + toolName: 'search_policy', + toolCallId: 'call_1', + args: { query: input.message ?? '' }, + timestamp: now(), + } + yield { + type: 'tool_result', + toolName: 'search_policy', + toolCallId: 'call_1', + result: { section: 'renewals', window: '14 days' }, + timestamp: now(), + } + yield { + type: 'llm_call', + task: ctx.task, + session: ctx.session, + model: 'scripted/support-agent', + tokensIn: 320, + tokensOut: 64, + costUsd: 0.0009, + latencyMs: 120, + timestamp: now(), + } + yield { + type: 'text_delta', + text: '\nA renewal is refundable within 14 days of the charge.\n', + timestamp: now(), + } + }, +} + +// The session store keeps this turn's session and events. Swap in your own +// store to make the transcript durable; omit it entirely for a stateless turn. +const sessions = new InMemoryRuntimeSessionStore() + +let reply = '' +let costUsd = 0 +const toolCalls: string[] = [] + +for await (const event of runAgentTaskStream({ + task, + backend, + input: { message: 'Can I refund a subscription renewal?' }, + sessionStore: sessions, + sessionId: 'thread-42', +})) { + switch (event.type) { + case 'text_delta': + reply += event.text + process.stdout.write(event.text) + break + case 'tool_call': + toolCalls.push(event.toolName) + break + case 'llm_call': + costUsd += event.costUsd ?? 0 + break + case 'final': + console.log(`\nstatus: ${event.status} — ${event.reason}`) + break + default: + break + } +} + +console.log(`tools: ${toolCalls.join(', ') || 'none'}`) +console.log(`cost: $${costUsd.toFixed(4)} — reply chars: ${reply.length}`) diff --git a/examples/stream-backends/README.md b/examples/stream-backends/README.md index 79eb43dd..45c8314a 100644 --- a/examples/stream-backends/README.md +++ b/examples/stream-backends/README.md @@ -1,47 +1,30 @@ -# Three sources of an AI's streaming output, one wire format for your app +# Choose where the tokens come from -An agent's output arrives as a live stream of events: text as it's typed, tool calls as they -fire, tool results as they return. That stream can come from three very different places — a -function you wrote, a remote sandbox running a coding agent, or an exact `AgentProfile` executed -by Runtime through the Tangle Router. This example runs all three and shows they emit the **same typed events** and serialize -to the **same format a browser reads**, so the source is a swappable detail your UI never -sees. +## When to use it -## Why it matters +Use this when you must choose where the tokens come from. +Three sources feed the same `runAgentTaskStream` call and emit the same typed events, so your route never learns which one ran. -The painful coupling in agent apps is that the frontend ends up knowing which backend it's -talking to — mock in tests, a real model in prod, a sandbox for the heavy stuff — because -each streams a different shape. Here they don't. Every backend lands on one typed event -stream and one SSE serialization, so you swap transports (test → sandbox → hosted model) -without touching the route that streams to the browser or the code that collects the events. - -("SSE" is Server-Sent Events, the plain `data: ...\n\n` streaming format a browser reads with -`EventSource` — the standard way a web app receives a live token stream.) - -## The three backends - -| Backend | You'd use it for | +| Source | Use it for | |---|---| -| **Iterable** (`createIterableBackend`) | You own the loop: write an async generator that yields events directly. For tests, scripted demos, or wrapping a stream shape the others don't map. | -| **Sandbox** (`createSandboxPromptBackend`) | A remote `@tangle-network/sandbox` box runs the agent and streams back its native events (text updates, tool calls, tool results). The default mapper already understands them, so you write no translation code. | -| **Exact model turn** (`streamAgentTurn`) | A concrete profile whose prompt, provider, model, and generation controls Runtime preserves and meters. | +| `createIterableBackend` | You own the loop. Write an async generator that yields events. Good for tests and scripted demos. | +| `createSandboxPromptBackend` | A remote sandbox box runs the agent and streams its native events. The default mapper reads them, so you write no translation. | +| `streamAgentTurn` | An exact `AgentProfile` through Runtime. Runtime keeps the prompt, provider, model, and generation controls, and meters the spend. | -All three feed `runAgentTaskStream`, which emits a typed `RuntimeStreamEvent` stream, which -two helpers serialize to SSE (`runtimeStreamServerSentEvent` per event, plus -`readinessServerSentEvent` for a one-off "still waiting on required info" event a gated task -can emit). +Use a sibling instead when the backend is not the question. +[`../stream-a-turn`](../stream-a-turn) writes one backend by hand and reads the events. +[`../chat-handler`](../chat-handler) serves the same stream over HTTP. -## Run it +## How to use it ```bash -pnpm tsx examples/stream-backends/stream-backends.ts +pnpm build && pnpm tsx examples/stream-backends/stream-backends.ts ``` -No API key needed for the first two backends: the iterable and sandbox sections run offline -against a synthetic in-process box. You'll see each section stream SSE frames to stdout, -e.g.: +The first two sections run offline against an in-process box. +Each section serializes its events as Server-Sent Events, the `data: ...` format a browser reads with `EventSource`: -``` +```text --- iterable backend --- data: {"type":"text_delta","text":"you said: hello\n"} @@ -51,11 +34,18 @@ data: {"type":"tool_call","toolName":"Read","toolCallId":"call_1", ...} data: {"type":"tool_result","toolName":"Read", ...} ``` -The third section is skipped unless you provide a Tangle Router key: +The third section needs a router key: ```bash TANGLE_API_KEY=sk-... pnpm tsx examples/stream-backends/stream-backends.ts ``` -`MODEL` and `MODEL_PROVIDER` override the concrete DeepSeek defaults, while `ROUTER_BASE` changes -only the transport endpoint. The output is the same SSE shape as the offline sections. +`MODEL` and `MODEL_PROVIDER` override the defaults. +`ROUTER_BASE` changes only the transport endpoint. +The output keeps the same shape as the offline sections. + +## Why this exists + +The usual coupling in an agent app is that the frontend learns which backend it talks to, because each one streams a different shape. +Here every source lands on one typed event union and one SSE serialization. +You swap test, sandbox, and hosted model without touching the route or the collector. diff --git a/examples/supervise/README.md b/examples/supervise/README.md index 44edbcbf..38ae4b97 100644 --- a/examples/supervise/README.md +++ b/examples/supervise/README.md @@ -1,66 +1,46 @@ -# Run a supervisor agent that delegates the work, in one function call +# Let one agent run other agents -Give a goal to a supervisor agent and it breaks the work off to worker agents, waits for them -to finish, and only calls itself done when a real check passes on a worker's actual output — -not when a worker *claims* success. This whole pattern is one call: `supervise(profile, task, -opts)`. Everything else (bookkeeping, worker plumbing, depth limits) is defaulted for you. +## When to use it -## Why it matters +Use this when a model must decide the plan, not your code. +One supervisor spawns workers, steers them, and stops when a check passes on a worker's real output. +Budget, journaling, and depth limits are defaulted. -Multi-agent orchestration usually means a pile of glue: spawning workers, tracking who's -running, collecting results, deciding when it's truly done. Two things here make that -tractable. First, it's a single call with sane defaults, so you write a goal and a profile, -not a framework. Second, "done" is a **check you provide** that runs against the worker's -output — so a worker can't lie its way to completion, and a failure reports the real reason -and the spend instead of a silent "no winner." +Use a sibling instead when your code owns the plan. +[`../quickstart`](../quickstart) is the loop you write yourself with `plan` and `decide`. +[`../graphs`](../graphs) is a fixed topology authored as data. +[`../supervisor-loop`](../supervisor-loop) is this same call with a real worker backend, such as a sandbox or a coding CLI. +[`../delegate`](../delegate) is the zero-configuration entry: one intent string in, one result out. -## What runs - -The goal is deliberately tiny: *produce the exact line `READY`*. It's a stand-in for real -delegated work so the mechanics are visible. - -1. The supervisor (its brain is a model reasoning over `spawn_agent` / `await_event` / `stop` - tools) is told to **delegate, not solve**: spawn a worker, wait for it to settle, then stop. -2. A worker agent runs and produces its output. -3. The completion check `out => ...includes('READY')` runs against that output. Only if it - passes is the run a win. If no worker ever delivers, the run ends with a typed reason plus - token/dollar spend. - -Three knobs are worth knowing: - -- **`profile.harness`** picks what drives the supervisor's brain: `null` (this example) uses - an in-process model tool-loop; `'opencode'` / `'claude-code'` / `'codex'` run a real coding - CLI in a sandbox instead. -- **`backend`** is *where the workers run* — one value to swap. Here it's `router-tools` - (off-box model agents); change it to `sandbox` + a harness to run each worker as a coding - agent in a real box. -- **`deliverable`** is the completion check described above. Optional, but it's what makes - "done" mean *verified* rather than *self-reported*. - -## Run it +## How to use it ```bash -TANGLE_API_KEY=sk-tan-... pnpm tsx examples/supervise/supervise.ts +TANGLE_API_KEY=sk-tan-... pnpm build && pnpm tsx examples/supervise/supervise.ts ``` -A key is required (the supervisor's brain and the worker both call the Tangle router). -Optional: `MODEL` (default `gemini-2.5-pro`), `TANGLE_ROUTER_URL`. +A key is required, because the supervisor's brain and the worker both call the router. +`MODEL` and `TANGLE_ROUTER_URL` are optional. On success it prints the delivered output: -``` +```text [OK] delivered: {"content":"READY"} ``` -If no worker delivers, it prints the reason and what it cost instead: +When no worker delivers, it prints the reason and the spend instead: -``` +```text [--] no winner (budget-exhausted) — 1 child(ren) down, spent 4210 tokens / $0.0031 ``` -## Going further +Three settings are worth knowing. + +- `profile.harness` picks what drives the supervisor's brain. This example uses `cli-base`, the router-backed brain with no coding agent. Set `opencode`, `claude-code`, or `codex` to run a coding CLI in a sandbox. +- `backend` is where the workers run. This example uses `router-tools`. Change it to `sandbox` plus a harness to run each worker as a coding agent in a real box. +- `deliverable` is the completion check. It is optional, and it is what makes "done" mean verified. + +## Why this exists -This is the smallest possible call — model brain, off-box workers, everything defaulted. When -your workers need a real backend (a sandbox box, a local coding CLI, or an MCP tool server), -go to [`../supervisor-loop/`](../supervisor-loop/): the same `supervise()` call with the -worker backend swapped in as the only change. +Multi-agent orchestration usually becomes glue code: spawn, track, collect, and guess when the work is done. +This is one call with defaults, so you write a goal and a profile instead of a framework. +"Done" is your check against a worker's output, so a worker cannot claim success, and a failure reports the real reason and the spend. diff --git a/examples/tool-loop/README.md b/examples/tool-loop/README.md new file mode 100644 index 00000000..502d5f2e --- /dev/null +++ b/examples/tool-loop/README.md @@ -0,0 +1,55 @@ +# Run a tool-calling turn + +## When to use it + +Use this when the model must call your tools and then answer in the same turn. +The loop runs one model turn, executes each requested tool, folds the results back, and runs the turn again. +It stops when the model finishes, or when a turn cap, a deadline, a cost limit, or a stuck-loop check fires. + +Use a sibling instead when the shape is different. + +| Your shape | Use | +|---|---| +| One turn with no tool round trip | [`../stream-a-turn`](../stream-a-turn) | +| One turn served over HTTP and saved | [`../chat-handler`](../chat-handler) | +| Several worker attempts under your own stop rule | [`../quickstart`](../quickstart) | +| One agent that spawns other agents | [`../supervise`](../supervise) | + +## How to use it + +```bash +pnpm build && pnpm tsx examples/tool-loop/tool-loop.ts +``` + +The example runs offline and needs no credentials. +It prints: + +```text +Looking up the invoice. +Invoice inv-42 is $120 and already paid. +tool get_invoice → ok +turns: 2 — stopReason: completed +``` + +Read [`tool-loop.ts`](./tool-loop.ts) for the whole file. +You supply two functions and the loop owns the rest. + +```ts +const result = await runToolLoop({ + systemPrompt: 'You answer billing questions. Use the tools before you answer.', + userMessage: 'Is invoice inv-42 paid?', + streamTurn, // one model turn: yields text and tool calls + executeToolCall, // your executors: one call in, one typed outcome out + isExecutableTool: (name) => tools.some((tool) => tool.function.name === name), + maxToolTurns: 8, +}) +``` + +Read `result.stopReason` before you score the turn. +Only `completed` means the model finished; `stuck-loop`, `backstop`, `deadline`, and `budget` are resource outcomes. + +## Why this exists + +Every agent product writes this loop, and the hard parts are the same each time. +The loop keeps the OpenAI tool history correct, so a strict model reads its own tool use back instead of repeating the call. +A tool failure returns a typed outcome instead of throwing, so the model reads the reason and can recover. diff --git a/examples/tool-loop/tool-loop.ts b/examples/tool-loop/tool-loop.ts new file mode 100644 index 00000000..98bb7ca1 --- /dev/null +++ b/examples/tool-loop/tool-loop.ts @@ -0,0 +1,83 @@ +/** + * tool-loop — one chat turn that calls tools until the model stops. + * + * The loop is the runtime's; the model and the tools stay yours. You supply + * `streamTurn` (one model turn) and `executeToolCall` (your executors), and the + * loop folds each result back into the conversation and re-runs the turn. + * Offline, deterministic, no API keys. + * + * Run: pnpm build && pnpm tsx examples/tool-loop/tool-loop.ts + */ + +import type { OpenAIChatTool } from '@tangle-network/agent-runtime' +import { + runToolLoop, + type ToolCallOutcome, + type ToolLoopCall, + type ToolLoopEvent, + type ToolLoopMessage, +} from '@tangle-network/agent-runtime/tool-loop' + +// The tool declarations you send to the model. This is the OpenAI function +// shape every OpenAI-compatible provider accepts. +const tools: OpenAIChatTool[] = [ + { + type: 'function', + function: { + name: 'get_invoice', + description: 'Read one invoice by id.', + parameters: { + type: 'object', + properties: { invoiceId: { type: 'string' } }, + required: ['invoiceId'], + }, + }, + }, +] + +const invoices: Record = { + 'inv-42': { amountUsd: 120, status: 'paid' }, +} + +// Your executors. One call in, one typed outcome out. A failure is a value, +// not a thrown error, so the model reads the reason and can recover. +async function executeToolCall(call: ToolLoopCall): Promise { + if (call.toolName !== 'get_invoice') { + return { ok: false, code: 'unknown_tool', message: `no tool named ${call.toolName}` } + } + const invoice = invoices[String(call.args.invoiceId)] + if (!invoice) return { ok: false, code: 'not_found', message: 'no such invoice', status: 404 } + return { ok: true, result: invoice } +} + +// One model turn. A real one calls your provider with `messages` and `tools`, +// then yields text and tool calls as they arrive. This one is scripted: it asks +// for the invoice first, then answers once the result is in the history. +async function* streamTurn(messages: ToolLoopMessage[]): AsyncIterable { + const sawToolResult = messages.some((message) => message.role === 'tool') + if (!sawToolResult) { + yield { type: 'text', text: 'Looking up the invoice.\n' } + yield { + type: 'tool_call', + call: { toolCallId: 'call_1', toolName: 'get_invoice', args: { invoiceId: 'inv-42' } }, + } + return + } + yield { type: 'text', text: 'Invoice inv-42 is $120 and already paid.\n' } +} + +const result = await runToolLoop({ + systemPrompt: 'You answer billing questions. Use the tools before you answer.', + userMessage: 'Is invoice inv-42 paid?', + streamTurn, + executeToolCall, + isExecutableTool: (name) => tools.some((tool) => tool.function.name === name), + // A watchdog, not a policy cap. Real limits come from deadlineMs / maxCostUsd. + maxToolTurns: 8, +}) + +console.log(result.finalText.trim()) +for (const executed of result.toolResults) { + console.log(`tool ${executed.label} → ${executed.outcome.ok ? 'ok' : executed.outcome.code}`) +} +console.log(`turns: ${result.turns} — stopReason: ${result.stopReason}`) diff --git a/src/runtime/index.ts b/src/runtime/index.ts index ffdbe884..9a0f882e 100644 --- a/src/runtime/index.ts +++ b/src/runtime/index.ts @@ -386,7 +386,14 @@ export { // `runAgentRounds` is the multi-agent fanout/vote/refine kernel over many sandbox sessions. // It is distinct from `runToolLoop`/`streamToolLoop`, which execute one chat turn and fold // tool results back into that same conversation. -export { defaultSelectWinner, type RunAgentRoundsOptions, runAgentRounds } from './run-loop' +export { + defaultSelectWinner, + isTerminalDecision, + type RunAgentRoundsOptions, + runAgentRounds, + TERMINAL_DECISIONS, + type TerminalDecision, +} from './run-loop' export { type AcquireOptions, acquireSandbox } from './sandbox-acquire' export { type CriuCapableClient, diff --git a/src/runtime/run-loop.ts b/src/runtime/run-loop.ts index 08ecbb0c..811c4f8d 100644 --- a/src/runtime/run-loop.ts +++ b/src/runtime/run-loop.ts @@ -1212,10 +1212,22 @@ function resolveAgentRuns( throw new ValidationError('runAgentRounds: `agentRun` or non-empty `agentRuns` is required') } -function isTerminalDecision(decision: unknown): boolean { - return ( - decision === 'stop' || decision === 'pick-winner' || decision === 'fail' || decision === 'done' - ) +/** + * Decision values the kernel treats as terminal. Every other value returned by + * `decide` continues the loop. Type a driver's `decide` return as + * `'your-word' | TerminalDecision` so caller vocabulary and kernel keywords + * stay visibly distinct. + * + * @stable + */ +export const TERMINAL_DECISIONS = ['stop', 'pick-winner', 'fail', 'done'] as const + +/** One of the kernel's terminal decision values. @stable */ +export type TerminalDecision = (typeof TERMINAL_DECISIONS)[number] + +/** True when the kernel stops the loop for this decision value. @stable */ +export function isTerminalDecision(decision: unknown): decision is TerminalDecision { + return (TERMINAL_DECISIONS as readonly unknown[]).includes(decision) } function emitRunLoopHook( diff --git a/src/runtime/types.ts b/src/runtime/types.ts index d248d401..0cd44311 100644 --- a/src/runtime/types.ts +++ b/src/runtime/types.ts @@ -233,7 +233,8 @@ export interface Iteration { /** @stable */ export interface Driver { /** - * Stable identifier surfaced in trace events. Default `'driver'`. + * Trace label surfaced in trace events. No behavioral effect: it never + * selects a strategy or a decision path. Default `'driver'`. */ readonly name?: string /** @@ -243,9 +244,11 @@ export interface Driver { plan(task: Task, history: ReadonlyArray>): Promise /** * Inspect history and return the next state. The kernel terminates the - * loop when `decide` returns a value listed in `isTerminalDecision` - * (`'stop' | 'pick-winner' | 'fail' | 'done'`), when `maxIterations` - * is hit, or when the abort signal fires. + * loop when `decide` returns a `TerminalDecision` + * (`'stop' | 'pick-winner' | 'fail' | 'done'`, exported as + * `TERMINAL_DECISIONS` with the `isTerminalDecision` guard), when + * `maxIterations` is hit, or when the abort signal fires. Every other + * value is caller vocabulary and continues the loop. */ decide(history: ReadonlyArray>): Decision | Promise /** diff --git a/tsconfig.examples.json b/tsconfig.examples.json index f211c0fd..348cf602 100644 --- a/tsconfig.examples.json +++ b/tsconfig.examples.json @@ -9,6 +9,7 @@ "@tangle-network/agent-runtime/durable": ["./src/durable/index.ts"], "@tangle-network/agent-runtime/intelligence": ["./src/intelligence/index.ts"], "@tangle-network/agent-runtime/kernel": ["./src/runtime/index.ts"], + "@tangle-network/agent-runtime/tool-loop": ["./src/tool-loop.ts"], "@tangle-network/agent-runtime/environment-provider": [ "./src/runtime/environment-provider.ts" ],