diff --git a/README.md b/README.md index 211033b3..799c65cd 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,16 @@ # `@tangle-network/agent-eval` -Measure agent behavior, compare changes on the same cases, and improve prompts or skills without exposing final test cases to the optimizer. +Measure agent behavior, compare changes on the same cases, and improve prompts or skills without showing the final test cases to the optimizer. [![npm](https://img.shields.io/npm/v/@tangle-network/agent-eval.svg)](https://www.npmjs.com/package/@tangle-network/agent-eval) [![pypi](https://img.shields.io/pypi/v/agent-eval-rpc.svg)](https://pypi.org/project/agent-eval-rpc/) [![tests](https://github.com/tangle-network/agent-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/tangle-network/agent-eval/actions/workflows/ci.yml) [![license: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE) -Use this package to: - -- run an agent over representative cases and score every result, -- compare a candidate with a baseline using paired statistics, -- analyze existing runs, traces, or human feedback, -- optimize a prompt or skill with official GEPA or SkillOpt, -- supply your own candidate generator for product-specific changes. - The evaluation path runs in your TypeScript process. -Model calls occur only through the clients and agents you configure. +Model calls happen only through the clients and agents you configure. + +New to the package? Read [concepts](./docs/concepts.md) first — it takes five minutes and defines every word used here. ## Install @@ -24,56 +18,10 @@ Model calls occur only through the clients and agents you configure. pnpm add @tangle-network/agent-eval ``` -## Configure Model Calls - -Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters accept the same `ChatClient`. - -```ts -import { createChatClient } from '@tangle-network/agent-eval' - -const chat = createChatClient({ - transport: 'router', - apiKey: process.env.TANGLE_API_KEY!, - defaultModel: 'openai/gpt-4.1', - maximumAttempts: 3, -}) -``` - -Use `direct-provider` for an OpenAI-compatible endpoint, `cli-bridge` for a local subscription, `sandbox-sdk` for Sandbox, or `custom` to adapt another SDK. -A custom adapter must return `ChatResponse` and declare `maximumAttempts` before a capped cost ledger can dispatch it. - -The official optimizers use the Python bridge. -Install only the optimizer you plan to run: - -```sh -# Microsoft SkillOpt at the tested source revision -python -m pip install agent-eval-rpc -python -m pip install \ - "skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90" - -# Standard GEPA engine from the published package -python -m pip install agent-eval-rpc -python -m pip install "gepa[full]==0.1.4" - -# GEPA Omni and source-only engines -python -m pip install \ - "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f" - -# DSPy 3.2.1 with Agent Eval metrics -python -m pip install "agent-eval-rpc[dspy]" -``` - -The published GEPA package supports the standard `gepa` engine. -Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N currently require the tested official source revision. -Move that revision only after both release and source compatibility tests pass. -The published `skillopt==0.2.0` wheel omits the prompt files required by `ReflACTTrainer`, so the tested SkillOpt source revision is also intentional. -DSPy 3.2.1 requires GEPA 0.0.27, while the general bridge requires GEPA 0.1.4. -Install the DSPy adapter and the general GEPA bridge in separate Python environments. +## Quickstart -## Evaluate An Agent - -This example is offline. -Replace the agent and judge functions with your product code. +This example is offline and complete. +Copy it, run it, then replace the agent and the judge with your product code. ```ts import { defineAgentEval } from '@tangle-network/agent-eval/contract' @@ -110,371 +58,106 @@ console.log( ) ``` -Each call runs every case, records the artifact, applies the same judge, and returns score distributions. -The surface is the value being changed, such as a prompt, skill, or serialized configuration. - -### Inspect cached cells before rerunning - -`runCampaign()` refuses to start when an existing cache file is unreadable or lacks trustworthy cost data. -This check covers the full schedule before concurrent work begins, so one bad cache cannot waste paid calls from earlier cells. -Use `planCampaignRun()` to see which cells are reusable, runnable, or blocked: - -```ts -const plan = planCampaignRun({ - scenarios, - dispatch, - judges: [judge], - runDir: 'release-candidate', -}) - -console.table(plan.cells) -``` - -After inspecting the plan, set `rerunInvalidCachedCells: true` to rerun only blocked cells while retaining valid cached cells. -Set `resumable: false` only when you intend to rerun every cell. -When execution uses a shared `costLedger` or custom `costTags`, pass those same values to `planCampaignRun()` so both calls inspect the same receipts. - -### Stop after the first failed cell - -`runCampaign()` normally records a dispatch or judge error on that cell and continues the remaining cases. -Set `abortOnCellError: true` when another failed cell would only waste time or money: - -```ts -await runCampaign({ - scenarios, - dispatch, - judges: [judge], - runDir: 'release-candidate', - abortOnCellError: true, -}) -``` - -The failed cell is written first to `//failure-receipt.json`. -That receipt contains the original error, the cell result, exact call IDs, and settled agent-plus-judge cost and token totals. -Active sibling cells are cancelled and allowed to finish recording their own receipts before the campaign rejects with the original cell error. -Leaving `abortOnCellError` unset preserves continue-on-error behavior. - -When an external text optimizer uses that continue-on-error behavior, a failed evaluation returns a finite optimizer penalty of `0` with `info.status: 'failed'`. -The campaign cell remains failed and unscored, so this protocol penalty is not a measured zero. -The failed cell is not cached; a later request retries it with a fresh `ctx.runAttemptId`, while completed cells remain resumable. -Set `abortOnCellError: true` to keep fail-fast behavior. - -`runProfileMatrix()` retains a failed cell when a moving model alias produced no served model snapshot. -The row carries `model: 'unknown'`, uncaptured cost, and incomplete token usage so downstream reports can count the failure without inventing measurements. -Successful moving-model cells still require one immutable served snapshot. - -## Adapt Another Text Optimizer - -Use `externalTextOptimizationMethod()` when an existing package owns search and selection for a text prompt or named text components. -Its `run` callback receives the starting candidate plus serialized train and selection cases, but it never receives final test cases. -The optimizer must score candidates through `context.evaluate()` so Agent Eval can enforce the evaluation limit and use the configured execution and judges. -Every optimizer-owned paid call must use `context.cost.runPaidCall()`. -Set `source` to the package version and revision, and set `evaluationId` to a commit, content hash, or other stable identity for the execution and scoring behavior. -Agent Eval derives the run identity from those values, the exact dispatch identity, the optimizer settings, the starting surface, the described data, and the seed. -The callback returns the selected candidate, whether compatible state was restored, and how optimizer spend was recorded. - -See [Adapt A Third-Party Text Optimizer](./docs/campaign-proposers.md#adapt-a-third-party-text-optimizer) for a complete minimal adapter. - -## Optimize With Official GEPA - -`gepaOptimizationMethod()` delegates candidate search and recipe composition to the installed GEPA package. -Agent Eval supplies the train and selection cases, executes candidates, records cost, and evaluates the selected result on final cases after GEPA exits. - -```ts -import { gepaOptimizationMethod } from '@tangle-network/agent-eval/campaign' - -// Supplied by the package that owns model execution. Discovery uses Runtime -// with one exact AgentProfile; Agent Eval receives no provider credential. -declare const optimizerExecution: { - call: import('@tangle-network/agent-eval/campaign').ExternalOptimizerModelCall - callRef: string -} - -const optimizerPricing = { - inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION), - outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION), -} - -const gepa = gepaOptimizationMethod({ - objective: 'Improve the instructions so the agent returns valid, complete JSON.', - evaluationId: 'json-agent', - recipe: { - kind: 'engine', - run: { - engine: 'gepa', - maxEvaluations: 40, - maxProposerCostUsd: 5, - }, - }, - optimizer: { - model: 'gpt-4.1-mini', - ...optimizerExecution, - budget: { - maxCostUsd: 5, - maxRequests: 100, - maxRequestBytes: 2_000_000, - maxResponseBytes: 2_000_000, - maxOutputTokensPerRequest: 32_768, - pricing: optimizerPricing, - }, - }, - describeScenario: (scenario) => ({ input: scenario.input }), - describeArtifact: (artifact) => ({ output: artifact.output }), -}) -``` - -GEPA also supports official sequential, adaptive, best-of, vote, and Omni recipes through the same factory. -When every recipe stage uses the standard GEPA engine, Agent Eval gives Python only a loopback address and asks the caller-owned execution package to make every model call. -Every call must return measured usage and finite JSON execution evidence; Agent Eval stores both and fails the attempt when either is missing. -Other official GEPA engines can still run through `engineConfig`, but their model spend remains incomplete unless the engine reports it. -Custom engines can register through GEPA's official registry by listing their Python modules in `engineModules`. - -Run the repository example: - -```sh -OPTIMIZERS=gepa \ -LLM_API_KEY="$OPENAI_API_KEY" \ -OPTIMIZER_EXECUTION_OWNER_MODULE=@acme/runtime-optimizer-owner \ -GEPA_PRICE_IN_PER_M=0.4 \ -GEPA_PRICE_OUT_PER_M=1.6 \ -pnpm tsx examples/compare-optimization-methods/index.ts -``` - -Replace the example rates with the exact endpoint rates. - -## Optimize With Official SkillOpt - -`skillOptOptimizationMethod()` runs Microsoft's `ReflACTTrainer` against the same TypeScript execution and scoring path. -SkillOpt receives train and selection cases but never receives final cases. - -```ts -import { skillOptOptimizationMethod } from '@tangle-network/agent-eval/campaign' - -declare const optimizerExecution: { - call: import('@tangle-network/agent-eval/campaign').ExternalOptimizerModelCall - callRef: string -} - -const optimizerPricing = { - inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION), - outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION), -} - -const skillopt = skillOptOptimizationMethod({ - objective: 'Improve the skill so the agent returns valid, complete JSON.', - evaluationId: 'json-agent', - trainer: { - epochs: 2, - batchSize: 4, - }, - optimizer: { - model: 'gpt-4.1-mini', - ...optimizerExecution, - budget: { - maxCostUsd: 5, - maxRequests: 100, - maxRequestBytes: 2_000_000, - maxResponseBytes: 2_000_000, - maxOutputTokensPerRequest: 32_768, - pricing: optimizerPricing, - }, - }, - maxEvaluations: 80, - describeScenario: (scenario) => ({ input: scenario.input }), - describeArtifact: (artifact) => ({ output: artifact.output }), -}) -``` - -Replace the example rates with the exact rates for your endpoint. -Agent Eval places a local OpenAI-compatible proxy between each standard optimizer and the caller-owned execution package. -The proxy enforces request, byte, and token limits before one callback invocation, records the caller's measured receipt and execution evidence, and does not accept a provider URL or credential. -Dollar limits and pricing are optional; omit both when billed USD is unknown instead of supplying a catalog estimate as if it were observed billing. - -## Optimize A DSPy Program - -DSPy owns its program optimizers. -`DspyJudgeMetric` lets them use the same Agent Eval rubric as TypeScript agents. - -```python -import dspy - -from agent_eval_rpc import DspyJudgeMetric - -dspy.configure(lm=dspy.LM("openai/gpt-4.1-mini")) -dspy.configure_cache(restrict_pickle=True) -metric = DspyJudgeMetric(rubric_name="answer-quality") - -# GEPA needs both the numeric score and diagnostic feedback. -optimizer = dspy.GEPA( - metric=metric.feedback, - reflection_lm=dspy.LM("openai/gpt-4.1-mini"), - max_metric_calls=100, -) -optimized_program = optimizer.compile(program, trainset=train, valset=selection) - -# MIPROv2, SIMBA, and few-shot optimizers use the numeric metric directly. -mipro = dspy.MIPROv2(metric=metric, auto="light") -``` - -Use official DSPy directly for DSPy programs. -Use `gepaOptimizationMethod()` for text or named component surfaces in non-DSPy agents. -Use agent-runtime's worktree path for executable code changes. - -## Compare Complete Methods - -`compareOptimizationMethods()` gives each method the same starting surface, execution function, judges, train cases, and selection cases. -It waits for optimization to finish before it evaluates any selected surface on the final cases. - -```ts -import { compareOptimizationMethods } from '@tangle-network/agent-eval/campaign' - -const result = await compareOptimizationMethods({ - methods: [gepa, skillopt], - baselineSurface, - trainScenarios, - selectionScenarios, - testScenarios, - dispatchWithSurface, - judges, - runDir: '.agent-eval/optimizer-comparison', -}) - -console.table(result.scores) -``` - -Read `scores` for final-case lift and intervals. -Read `pairwise` before claiming one method beat another. -Read `totalCost.accountingComplete` before using the reported dollars as a complete total. -Each official method score records the optimizer and bridge package versions, source revisions and source-tree hashes, Python runtime, configured optimizer model when present, custom engine module hashes, compatible run ID, exact attempt ID, resume status, evaluation count, artifact directory, and available optimizer token usage in `provenance`. -Call `readExternalOptimizerObservationArtifact()` with `provenance.observations` to read every distinct callback-submitted candidate. -The reader verifies the artifact digest, canonical rows, sequence, candidate identities, and summary counts before returning candidates. -This verification proves that the bytes match the supplied summary; use a summary from trusted method provenance when authenticity matters. -For a direct standard GEPA run, call `readGepaCandidatePopulationArtifact()` with `provenance.gepaCandidatePopulation`. -It returns GEPA's accepted candidates with exact parent indices, aggregate scores, per-case selection scores, and discovery evaluation counts. -The callback artifact remains the complete source for rejected or refused proposals that GEPA did not add to its accepted population. - -The [optimizer guide](./docs/campaign-proposers.md) covers recipes, budgets, resuming, and data separation. -The [runnable comparison](./examples/compare-optimization-methods/) can run GEPA, SkillOpt, or both. +Each call runs every case, records what the agent produced, applies the same judge, and returns score distributions. + +Three words carry this example. +A **case** is one task the agent must do. +A **surface** is the value being changed: a prompt, a skill, or a serialized configuration. +A **judge** is a function that scores one produced result. + +`expectUsage: 'off'` is set because this agent makes no paid calls. +The default, `'assert'`, fails a run whose cells report no cost receipt. +Keep the default whenever real model calls happen. + +Runnable copy: [`examples/evaluate-a-change`](./examples/evaluate-a-change/). + +## Which Front Door + +Every row is a function you call. Each links to a runnable example. + +| When to call it | What you give it | What you get back | +|---|---|---| +| [`defineAgentEval()`](./examples/evaluate-a-change/) — you changed a surface and must know whether it helped | cases, an agent, a judge, a starting surface | `evaluate()` for scores, `improve()` for a search plus a release decision | +| [`selfImprove()`](./examples/selfimprove-quickstart/) — you want candidate generation, scoring, and a release decision in one call | cases, an agent, a judge, a starting surface | a report, a winner surface, and a `gateDecision` | +| [`analyzeRuns()`](./examples/analyze-existing-runs/) — the runs already happened and no agent needs to run again | `RunRecord[]` | an `InsightReport`: distributions, paired lift, judge agreement, cost, failure clusters | +| `fromFeedbackTable()` ([example](./examples/customer-feedback-loop/)) / `fromOtelSpans()` ([example](./examples/customer-otel-traces/)) — your data is in a table or an OTel collector, not in `RunRecord` shape | source rows or spans | `RunRecord[]` ready for `analyzeRuns()` | +| [`planCampaignRun()` / `runCampaign()`](./examples/plan-before-you-spend/) — you need direct control of the case grid, or you must see it before paying for it | cases, a dispatch function, judges, a run directory | a per-cell schedule, then a campaign result with cached cells | +| [`loadEvalFixtureScenarios()`](./examples/eval-fixtures-quickstart/) — agents should add cases as folders on disk | `evals//PROMPT.md` plus checks | `Scenario[]` for `runCampaign()` | +| [`compareOptimizationMethods()`](./examples/compare-optimization-methods/) — two search methods must be compared at equal budget | methods, a starting surface, train, selection, and final cases | per-method final lift, intervals, pairwise contrasts, and cost | +| [`gepaOptimizationMethod()` / `skillOptOptimizationMethod()`](./examples/compare-optimization-methods/) — official GEPA or Microsoft SkillOpt should own the search | an objective, a recipe or trainer, an optimizer budget | an optimization method for the comparison above | +| [`externalTextOptimizationMethod()`](./examples/adapt-a-text-optimizer/) — another package owns text search and you keep the scoring | the package identity, limits, and a `run` callback | the same, with the final cases never exposed | +| [`SurfaceProposer`](./examples/selfimprove-quickstart/) — candidate generation belongs to your product | a `propose()` function | candidates the campaign executes, scores, and gates | +| [`runProfileMatrix()`](./docs/eval-surface-map.md) — the same cases must run across models or profiles | axes of models and profiles, cases | one row per cell, with an explicit `unknown` model rather than an invented one | +| [`sealExperiment()` / `openSealedExperiment()`](./examples/sealed-experiment/) — the result must convince someone who does not trust you | arms, an admission funnel, an estimand, an interval, a decision table | a hashed rule tree, and executors that can run no other rule | +| [`runEquivalenceCheck()` / `VERIFICATION_STRATEGIES`](./examples/verify-without-an-answer-key/) — the work has no held-out test suite | a claim, two blind arms, an injected checker | a certification that names who vouched and how it can fail | +| [`AnalystRegistry.runExact()`](./examples/custom-trace-analyst/) — a batch of runs failed and you need cited findings | recorded evidence, a declared analyst list | findings with evidence references, an execution plan, and a receipt | +| [`runAnalystBenchmark()`](./docs/trace-analysis.md) — an analyst's accuracy must be measured, not assumed | labeled issues and exact span locations | scored findings, trace reads, model calls, tokens, cost, and runtime | +| [`deltaRepair()`](./docs/trace-repair-grader.md) — a finding must be graded by executing the repair it proposes | a trajectory, an analyst finding, a sandbox | the repair's measured effect against a no-fix control | +| [`replayVerify()`](./docs/trajectory-replay.md) — you must know whether a recorded failure still reproduces | a recorded shell trajectory and its pinned image | a re-execution verdict and the divergences found | +| [`analyzeSupervisorRun()`](./docs/adapters-observability.md) — a recursive or supervised run directory must be read | a run directory | counts that stay missing when a measurement is missing, never zero | +| [`buildRlDataset()`](./examples/publish-rl-dataset/) — scored runs should become training data | run records and preferences | reward, preference, and supervised rows | -## Supply Your Own Candidate Generator +## Configure Model Calls -Use `SurfaceProposer` when candidate creation belongs to your product or an agent runtime. -The campaign still owns execution, scoring, history, stopping, and release decisions. +Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters all take the same `ChatClient`. ```ts -import { - defineAgentEval, - type SurfaceProposer, -} from '@tangle-network/agent-eval/contract' - -const proposer: SurfaceProposer = { - kind: 'product-rules', - async propose({ currentSurface, populationSize }) { - const prompt = String(currentSurface) - return [ - { - surface: `${prompt}\nReturn JSON only.`, - label: 'json-only', - rationale: 'Training failures contained prose around the JSON object.', - }, - ].slice(0, populationSize) - }, -} - -const result = await defineAgentEval({ - scenarios, - agent, - model: 'gpt-4.1-2025-04-14', - judge, - baselineSurface, - proposer, - budget: { generations: 1, populationSize: 1, holdoutFraction: 0.3 }, -}).improve() -``` - -`model` supplies the worker snapshot when the agent does not report paid calls through `ctx.cost.runPaidCall()`. -When every cell reports a concrete model receipt, omit it. - -Run the complete offline example: - -```sh -pnpm tsx examples/selfimprove-quickstart/index.ts -``` - -## Start From Existing Runs - -You do not need a runnable agent to analyze data you already captured. -Use `analyzeRuns()` for `RunRecord[]`. -For traces, run a registry of built-in or custom analysts, measure it on labeled issues and exact span locations, then turn only reviewed findings into eval data. -For a public quality check, convert CodeTraceBench with `traces import-codetracebench`, then run `agent-eval analyst-benchmark` against pinned labels. -The command compares an empty baseline with the official DSPy `RLM` trace analyst and records its trace reads, model calls, tokens, cost, runtime, and cited findings. - -Use `AnalystRegistry.runExact()` when the caller, rather than registry defaults, must own every execution choice. -The ordered `analystIds` array is the execution order, and `null` explicitly disables optional budget, timeout, cancellation, cost, tag, or prior-finding channels. -Exact runs are serial; callers that need recursive or concurrent scheduling compose them through their runtime rather than adding a second scheduler here. +import { createChatClient } from '@tangle-network/agent-eval' -```ts -const result = await registry.runExact('analysis-1', inputs, { - analystIds: ['failure-mode', 'improvement'], - budget: { kind: 'equal', totalUsd: 2 }, - totalTimeoutMs: 30_000, - signal: null, - costLedger: null, - costLedgerIdentity: null, - costPhase: null, - tags: null, - priorFindings: null, - chainFindings: true, - missingInputMode: 'abort', - applyRegistryHooks: false, - useRegistryChat: false, +const chat = createChatClient({ + transport: 'router', + apiKey: process.env.TANGLE_API_KEY!, + defaultModel: 'openai/gpt-4.1', + maximumAttempts: 3, }) ``` -Custom analysts passed to `runExact()` declare canonical `executionConfig`. -The same `defineTraceAnalyst()` helper returns an exact-capable analyst when that field is present. -Built-in analysts already declare it. -Trace analysts selected by `runExact()` also require `aiIdentity`, using the same non-secret `id`, `version`, and canonical `config` shape as cost ledgers, registry hooks, and registry chat clients. -Exact lifecycle hooks receive frozen snapshots for observation; they cannot rewrite the planned context. -Persisted results store configuration digests, not raw configuration. -The persisted plan records the exact equal or weighted allocation for every routed analyst, and archival validates summaries against that same plan. -Every exact receipt says whether it is `complete` or `failed`; a complete receipt must cover the full plan, while a failed receipt may contain only the executed prefix. -Any failure after an exact run starts rejects with `ExactAnalystRunExecutionError`; its immutable failed receipt preserves valid completed summaries, findings, usage, and cost. +Use `direct-provider` for an OpenAI-compatible endpoint, `cli-bridge` for a local subscription, `sandbox-sdk` for Sandbox, or `custom` to adapt another SDK. +A custom adapter must return a `ChatResponse` and declare `maximumAttempts` before a capped cost account can dispatch it. -See [concepts](./docs/concepts.md), [customer paths](./docs/customer-journeys.md), and [trace analysis](./docs/trace-analysis.md). +The official GEPA and SkillOpt optimizers run through a Python bridge. +Install commands, version pins, and the reason for each pin: +[GEPA](./docs/campaign-proposers.md#install-official-gepa), +[SkillOpt](./docs/campaign-proposers.md#install-official-skillopt), +and [DSPy](./docs/campaign-proposers.md#use-official-dspy-optimizers). ## Entry Points | Import | Use | |---|---| -| `@tangle-network/agent-eval/contract` | Define an evaluation, run it, improve with a custom candidate generator, and analyze runs. | -| `@tangle-network/agent-eval/campaign` | Control campaigns, official optimization methods, comparisons, storage, and release rules. | -| `@tangle-network/agent-eval/profile-cell` | Create and validate portable agent-profile identities. | -| `@tangle-network/agent-eval/ledger-core` | Generic hash-chained append-only journal: idempotent append, chain verification, trusted-head pinning, replay-to-projection, cross-process locking. | -| `@tangle-network/agent-eval/reporting` | Statistical comparisons and report rendering. | +| `@tangle-network/agent-eval/contract` | Define an evaluation, run it, improve it, and analyze existing runs. | +| `@tangle-network/agent-eval/campaign` | Campaigns, optimization methods, comparisons, storage, and release rules. | +| `@tangle-network/agent-eval/experiment` | Experiments as sealed objects: registered rules, funnels, estimands, refusals. | | `@tangle-network/agent-eval/analyst` | Built-in and custom trace analysts, labeled comparison, costs, and reports. | +| `@tangle-network/agent-eval/trace-repair` | Grade one analyst finding by executing the repair it proposes. | +| `@tangle-network/agent-eval/trajectory-replay` | Re-execute a recorded shell trajectory and check whether its failure reproduces. | | `@tangle-network/agent-eval/traces` | Store, replay, and inspect structured traces. | -| `@tangle-network/agent-eval/supervisor-run` | Read loops or agent-runtime recursive run directories without collapsing missing measurements to zero. | -| `@tangle-network/agent-eval/trajectory-replay` | Re-execute a recorded shell trajectory in its own image and check whether the recorded failure reproduces. | -| `@tangle-network/agent-eval/trace-repair` | Grade one analyst finding by executing the repair it proposes, and measure Delta-repair against a no-fix control. | +| `@tangle-network/agent-eval/reporting` | Statistical comparisons and report rendering. | +| `@tangle-network/agent-eval/supervisor-run` | Read recursive run directories without collapsing missing measurements to zero. | +| `@tangle-network/agent-eval/profile-cell` | Create and validate portable agent-profile identities. | +| `@tangle-network/agent-eval/ledger-core` | Append-only hash-chained journal with idempotent append and chain verification. | | `@tangle-network/agent-eval/benchmarks` | Benchmark adapters and retrieval metrics. | | `@tangle-network/agent-eval/rl` | Export rewards, preferences, and training rows. | | `@tangle-network/agent-eval/wire` | HTTP and RPC schemas for other languages. | -Use subpaths when you want an explicit capability boundary. Use the root import for common primitives. +Use a subpath when you want an explicit capability boundary. -## Examples +## Documentation -| Goal | Example | +| Question | Read | |---|---| -| Evaluate and improve with a custom candidate generator | [`selfimprove-quickstart`](./examples/selfimprove-quickstart/) | -| Run official GEPA or SkillOpt | [`compare-optimization-methods`](./examples/compare-optimization-methods/) | -| Analyze human feedback | [`customer-feedback-loop`](./examples/customer-feedback-loop/) | -| Analyze OpenTelemetry traces | [`customer-otel-traces`](./examples/customer-otel-traces/) | -| Run public benchmark adapters | [`benchmarks`](./examples/benchmarks/) | - -See the [example index](./examples/README.md) for the full list. +| What do these words mean? | [`docs/concepts.md`](./docs/concepts.md) | +| Why does this package exist, and where is it going? | [`docs/charter.md`](./docs/charter.md) | +| Which `run*` function do I want? | [`docs/eval-surface-map.md`](./docs/eval-surface-map.md) | +| How do I choose a candidate-generation method? | [`docs/campaign-proposers.md`](./docs/campaign-proposers.md) | +| What is in an `InsightReport`? | [`docs/insight-report.md`](./docs/insight-report.md) | +| How do I register an experiment as a sealed object? | [`docs/experiment.md`](./docs/experiment.md) | +| How is something certified without an answer key? | [`docs/verification-strategies.md`](./docs/verification-strategies.md) | +| Where does every verifier land its result? | [`docs/verdicts.md`](./docs/verdicts.md) | +| How do I score a string from another language? | [`docs/wire-protocol.md`](./docs/wire-protocol.md) | + +The [example index](./examples/README.md) lists every runnable example. ## Development diff --git a/docs/campaign-proposers.md b/docs/campaign-proposers.md index 14a29fe4..bc1c28be 100644 --- a/docs/campaign-proposers.md +++ b/docs/campaign-proposers.md @@ -223,6 +223,10 @@ uv sync --frozen --group gepa-release uv sync --frozen --group gepa-source ``` +The published package supports the standard `gepa` engine. +The composed recipes below — `sequential`, `adaptive-sequential`, `best-of`, `vote`, and `omni` — need the tested official source revision. +Move that revision only after both the release and the source compatibility tests pass. + ## Configure GEPA `gepaOptimizationMethod()` accepts text surfaces and component surfaces. diff --git a/docs/concepts.md b/docs/concepts.md index cdeba2d9..a700d91c 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -13,12 +13,38 @@ Use the lower-level functions when you need direct control over execution, stora | Function | When to call it | What you give it | What you get back | |---|---|---|---| | **`defineAgentEval()`** | You have scenarios, an agent, a judge, and a baseline surface, and you want one object you can score or improve. | scenarios, agent, judge, baseline surface | `{ evaluate(), improve() }` where `evaluate()` returns a campaign result and `improve()` returns a report | -| **`selfImprove()`** | You want candidate generation, scoring, and a release decision in one call. | scenarios, agent, judge, baseline surface | report, ship/hold decision, winner surface | +| **`selfImprove()`** | You want candidate generation, scoring, and a release decision in one call. | scenarios, agent, judge, baseline surface | report, winner surface, and a `gateDecision` (see below) | | **`loadEvalFixtureScenarios()`** | You want agents to add evals as folders with `PROMPT.md`, checks, and starter files. | `evals//PROMPT.md + EVAL.ts + package.json` | `Scenario[]` that runs through `runCampaign`; pair with `planEvalFixtureRun()` before spending tokens | | **`analyzeRuns()`** | You have existing runs and do not need to invoke an agent. | `RunRecord[]` and options | `InsightReport` | | **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Your data isn't already in `RunRecord` shape: it's in Obsidian, Sheets, an OTel collector, etc. | source-specific input | `RunRecord[]` ready to pipe into `analyzeRuns()` | +| **`sealExperiment()` / `openSealedExperiment()`** | The result must convince a reader who does not trust you, so the rules must be fixed before the data arrives. | arms, admission funnel, estimand, interval, decision table | a hashed rule tree plus executors that can run no other rule ([`experiment.md`](./experiment.md)) | +| **`runEquivalenceCheck()`** | The work has no held-out test suite, so no answer key exists to grade against. | a claim, two blind arms, an injected checker | a certification naming who vouched and how it can fail ([`verification-strategies.md`](./verification-strategies.md)) | +| **`AnalystRegistry.runExact()`** | A batch of runs failed and you need cited findings, with the caller owning every execution choice. | recorded evidence, a declared analyst list | findings with evidence references, an execution plan, and a receipt ([`trace-analysis.md`](./trace-analysis.md)) | See [`customer-journeys.md`](./customer-journeys.md) for runnable paths from existing logs, human ratings, and a callable agent. +The [README front-door table](../README.md#which-front-door) lists every callable entry point with a runnable example. + +### The five release decisions + +`selfImprove()` and every gate return a `GateDecision`, not a two-way ship/hold flag. +Folding the last three into `hold` throws away the action each one names. + +| Decision | What it means | What to do next | +|---|---|---| +| `ship` | Every gate passed on sufficient evidence. | Release the candidate. | +| `hold` | A gate failed on sufficient evidence. | Reject this candidate. | +| `need_more_work` | A gate could not decide: the evidence was missing, or the paired sample was too small to claim significance. | Gather more runs, then gate again. | +| `model_ceiling` | Reserved for a caller-supplied gate that attributes the limit to the model. | Handle it; no gate in this package emits it. | +| `arch_ceiling` | Reserved for a caller-supplied gate that attributes the limit to the architecture. | Handle it; no gate in this package emits it. | + +The last two are part of the taxonomy and of the composition order, but no built-in gate returns them today. +Handle all five anyway: a caller's own gate may return either, and the type will not let you ignore them. + +`need_more_work` is not a quiet `hold`. +"Gather more evidence" and "reject this candidate" are different actions, and folding the first into the second abandons a real gain that was only underpowered. + +When gates are composed, `ship` requires every gate to ship. +Otherwise the strongest hold wins, in this order: `arch_ceiling`, `model_ceiling`, `hold`, `need_more_work`. `analyzeRuns()` and the high-level contract return the same `InsightReport` shape. It contains score distributions, paired lift intervals, judge agreement, cost, failure clusters, contamination checks, outcome correlation, and recommendations. @@ -92,6 +118,48 @@ that can seed memory, replay scenarios, and optimization. | **Composite score** | A 0..1 number combining all dimensions. The single number you gate on. | | **Rubric version** | A stable hash of the rubric. Scores from different rubric versions are not comparable. | +### Running an evaluation + +| Term | Plain English | +|---|---| +| **Case** (`Scenario`) | One task the agent must do. The unit every score is per. | +| **Surface** | The value being changed: a prompt, a skill, or a serialized configuration. | +| **Dispatch** | The function that runs your agent on one case and returns the artifact. | +| **Campaign** | One complete pass of every case, executed, scored, and cached under a run directory. | +| **Cell** | One (case × replicate) of a campaign. Cells are cached, so a rerun skips the ones that finished. | +| **Receipt** | The record of what one paid call actually cost, in dollars and tokens. Absent when nothing measured it. | +| **Cost ledger** | The spend account receipts are written to. A capped ledger refuses a call that would exceed the cap. | +| **Provenance** | Where a number came from: the package version, the source revision, the run identity, the exact attempt. | +| **`RunRecord`** | The analysis-time projection of one run: who ran, on what, with which seed, at what cost, and what it scored. | + +### Improving a surface + +| Term | Plain English | +|---|---| +| **Optimizer** | Any procedure that writes candidate surfaces and picks one. | +| **GEPA** | An open-source optimizer that mutates text using reflection over failures. It searches; this package executes and scores. | +| **SkillOpt** | Microsoft's skill optimizer. Same division of labour. | +| **Engine** | One named search procedure inside GEPA. | +| **Recipe** | How several engines are composed: in order, adaptively, best-of, or by vote. | +| **Train cases** | Evidence the optimizer reads to write candidates. | +| **Selection cases** | Evidence the optimizer reads to choose among its candidates. | +| **Final cases** | Held back from the optimizer entirely. They produce the reported lift. | + +The three-way split is the reason a reported lift means anything. +An optimizer that saw the final cases can score well on them without the agent getting better. + +### Proving a result + +| Term | Plain English | +|---|---| +| **Experiment** | The rules — arms, funnel, estimand, interval, decision — written as data before the data arrives. | +| **Seal** | A hash of that whole rule tree. The execution surface accepts no rule outside it. | +| **Estimand** | The exact quantity being measured, for example the paired difference in pass rate. | +| **Funnel** | The denominator chain: how many rows entered, what each stage removed, and how many remain. | +| **Verification strategy** | One of ten ways to certify a result, each with a documented way it can certify a wrong one. | +| **Certification** | Who vouched for a verdict, with what checker version, and what the checker did not check. | +| **Analyst** | A function that reads recorded evidence and returns findings that cite it. | + ## The feedback trajectory loop Normal review activity can provide labels without a separate labeling interface: @@ -154,19 +222,21 @@ When you have a multi-step pipeline (install → typecheck → build → lint ```ts const verifier = new MultiLayerVerifier([ - installLayer, // runs `pnpm install` - typecheckLayer, // runs `tsc --noEmit`, depends on install - buildLayer, // runs `pnpm build`, depends on typecheck - semanticLayer, // LLM judge, weight 3, depends on build + installLayer, // runs `pnpm install` + typecheckLayer, // runs `tsc --noEmit`, depends on install + buildLayer, // runs `pnpm build`, depends on typecheck + semanticLayer, // LLM judge, weight 3, depends on build ]) -const report = await verifier.run({ env: { runner, workdir, ... } }) -report.allPass // boolean: every layer passed -report.taskScore // complete task score, or undefined -report.blendedScore // diagnostic weighted aggregate, possibly partial -report.layers // per-layer status, findings, duration +const report = await verifier.run({ env }) +report.allPass // boolean: every layer passed +report.taskScore // complete task score, or undefined +report.blendedScore // diagnostic weighted aggregate, possibly partial +report.layers // per-layer status, findings, duration ``` +`env` carries the sandbox driver, the working directory, and the harness commands each layer runs. + Use `taskScore` when creating task labels or training data. An errored, timed-out, skipped, or incomplete scoring panel leaves `taskScore` undefined. Use `blendedScore` only to inspect the measurements that did complete. @@ -184,9 +254,31 @@ Two questions to answer before trusting any LLM judge: 1. **Does it agree with humans?** `calibrateJudge(golden, candidate)` reports Pearson, MAE, integer-rounded κ, and worst-N miscalibrations vs a human golden set. 2. **Does it agree with itself / other judges?** `continuousAgreement(scores)` and `calibrateJudgeContinuous(golden, candidate)` report κ_w + ICC(2,1) + Pearson + Spearman with bootstrap 95% CIs on the raw [0,1] scores. -Why two κ flavours: the original `calibrateJudge` rounds scores to ints before computing κ. For fine-grained judges that loses information: 0.78 vs 0.81 both round to "1" and look perfectly agreed. Use `calibrateJudgeContinuous` (or `continuousAgreement` for N≥2 raters) when scores are continuous. ICC(2,1) catches systematic bias that Pearson misses: if judge B scores 2× judge A, Pearson stays ≈ 1 while ICC drops: that's the signal. +Each statistic answers a different question: + +| Statistic | What it answers | What it misses | +|---|---|---| +| Pearson | Do the two raters move together? | Constant offset and constant scaling | +| Spearman | Do they rank the same way? | The size of any gap | +| MAE (mean absolute error) | How far apart are they, on average? | Whether the gap is systematic | +| κ (Cohen's kappa) | Do they agree more than chance? | Everything below the rounding step | +| ICC(2,1) | Do they agree in absolute value, not just in shape? | — | + +Use two flavours of κ for one reason. +`calibrateJudge` rounds each score to an integer first. +For a fine-grained judge that throws information away: 0.78 and 0.81 both round to 1 and look perfectly agreed. +Use `calibrateJudgeContinuous`, or `continuousAgreement` for two or more raters, when the scores are continuous. + +ICC(2,1) catches a bias Pearson cannot see. +If judge B always scores twice judge A, the two move together perfectly and Pearson stays near 1, while ICC drops. +That drop is the signal. + +Every reported interval is a bootstrap 95 % interval: the statistic is recomputed on many resamples of the data, and the middle 95 % of those values is the interval. -Bias probes (`positionalBias`, `verbosityBias`, `selfPreference`) cover the orthogonal failure modes: position-dependent scoring, length-correlated scoring, and judge-prefers-its-own-family. +Three bias probes cover three separate failure modes. +`positionalBias` finds a judge that scores by position. +`verbosityBias` finds one that scores by length. +`selfPreference` finds one that prefers output from its own model family. ## Trace Model @@ -222,3 +314,8 @@ release decision. - **Building a code-generator eval?** → Start with `BuilderSession`, `SandboxHarness`, and `MultiLayerVerifier`. - **Multi-layer verifier?** → Use [control-runtime.md](./control-runtime.md) and `MultiLayerVerifier` for ordered gates with dependencies. - **Adding a new judge or rubric?** → `src/wire/rubrics.ts` for the cross-language path; `src/anti-slop.ts` and `src/judges.ts` for the in-process path. +- **Registering an experiment before the data arrives?** Read [experiment.md](./experiment.md) for the rule AST, the seal, the funnel, and the refusals. +- **Certifying a result with no answer key?** Read [verification-strategies.md](./verification-strategies.md) for the ten-member family and the blind two-arm protocol. +- **Reading a verdict someone else produced?** Read [verdicts.md](./verdicts.md) for what `certification` carries and what an absent one means. +- **Grading a finding by executing its repair?** Read [trace-repair-grader.md](./trace-repair-grader.md), and [trajectory-replay.md](./trajectory-replay.md) for re-executing a recorded failure. +- **Wondering why this package exists at all?** Read [charter.md](./charter.md) for the four end-states it is built against. diff --git a/docs/feature-guide.md b/docs/feature-guide.md index a16fa5a7..bfd5c5b8 100644 --- a/docs/feature-guide.md +++ b/docs/feature-guide.md @@ -34,7 +34,7 @@ Evaluation measures whether the result met its requirements, whether another att | “I need train/dev/test/holdout examples.” | `Dataset` plus feedback trajectory conversion | Stable splits and contamination control. | | “Which optimization procedure wins?” | `compareOptimizationMethods` | Runs complete methods on shared train and selection cases, then compares them on separate final cases. | | “Improve a multi-turn agent with candidates from my runtime.” | `runImprovementLoop` | Evaluates caller-generated candidates and applies a separate release rule. | -| “Improve prompts, then code if prompts plateau.” | `runPromptEvolution`, composite mutator, code mutator | Bounded evolution with telemetry and lineage. | +| “Improve prompts, then code if prompts plateau.” | `gepaOptimizationMethod` or `externalTextOptimizationMethod` for the prompt; agent-runtime's worktree path for the code | Text search stays here; executable code changes belong to the runtime. | | “Find why a regression happened.” | bisector, traces, run records | Narrows changes and preserves evidence. | | “Expose evals to another language.” | Wire protocol and Python client | HTTP/RPC boundary for non-TypeScript apps. | diff --git a/docs/trace-analysis.md b/docs/trace-analysis.md index 78746a25..d41a2fde 100644 --- a/docs/trace-analysis.md +++ b/docs/trace-analysis.md @@ -241,6 +241,32 @@ A definition a strategy cannot compile fails loud with `AnalystExpressivenessErr `AnalystContext.probe` (`ExecutionProbe`) is the optional live-execution port: a runtime that owns a sandbox or checkout fills it so an analyst can run a bounded command against the run's produced state and read a typed outcome. This package defines only the port; an absent probe means the analyst works from recorded evidence. +## Exact Runs + +Call `AnalystRegistry.runExact()` when the caller, and not the registry, must own every execution choice. +The runnable minimum is [`examples/custom-trace-analyst`](../examples/custom-trace-analyst/). + +The `analystIds` array is the execution order, and exact runs are serial. +A caller that needs recursive or concurrent scheduling composes exact runs through its own runtime rather than adding a second scheduler here. +Every option must be present, and `null` disables a channel on purpose, so a missing budget can never be read as an unlimited one. + +An analyst reaches an exact run only when it declares `executionConfig`: canonical JSON for every behavior knob that `version` does not already bind. +The receipt stores a digest of it, so two runs that behaved differently cannot look identical. +`defineCustomAnalyst()` returns an exact-capable analyst when that field is present, and the built-in analysts already declare it. + +Every other live component admitted to an exact run — the cost ledger, a registry hook, a registry chat client — carries an `ExactExecutionComponentIdentity`: a non-secret `id`, a `version`, and a canonical `config` object. +`snapshotExactExecutionComponentIdentity()` reduces each one to an `ExactExecutionComponentSnapshot`, which keeps `id` and `version` and replaces `config` with `config_digest`. +That is how a receipt names what ran without ever storing a credential. + +Three rules govern what a finished exact run may claim. + +1. Lifecycle hooks receive frozen snapshots. A hook observes the planned context; it cannot rewrite it. +2. Persisted results store configuration digests, never raw configuration. The plan records the exact equal or weighted allocation for every routed analyst, and archival validates each summary against that same plan. +3. Every receipt says whether it is `complete` or `failed`. A complete receipt must cover the whole plan. A failed receipt may cover only the prefix that ran. + +Any failure after an exact run starts rejects with `ExactAnalystRunExecutionError`. +Its immutable failed receipt keeps the summaries, findings, usage, and cost that were already valid, so a late failure does not erase what was measured before it. + ## Measure Analyst Quality Measure the analyst on labeled traces before using its findings for automated changes. diff --git a/examples/README.md b/examples/README.md index 1900bcf9..1247606b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,24 +1,36 @@ # Examples -Start with the [offline quickstart](./selfimprove-quickstart/). -It defines cases, an agent, a judge, a starting prompt, and a custom candidate generator in one file. +Every directory holds one runnable file and a README that answers three questions: when to use it, how to run it, and why it is built that way. +Every expected output printed in a README is the output that file produces. -## Evaluation And Improvement +Start with [`evaluate-a-change`](./evaluate-a-change/). +It is the smallest complete path: cases in, scores out. + +Run any offline example from the repository root: + +```sh +pnpm tsx examples/evaluate-a-change/index.ts +``` + +## Measure A Change | Goal | Example | Requirements | |---|---|---| -| Improve with a custom `SurfaceProposer` | [`selfimprove-quickstart`](./selfimprove-quickstart/) | Offline | -| Wrap an existing agent | [`foreign-agent-quickstart`](./foreign-agent-quickstart/) | Offline or an OpenAI-compatible endpoint | -| Compare official GEPA and SkillOpt | [`compare-optimization-methods`](./compare-optimization-methods/) | Python optimizer packages and an LLM endpoint | +| Score one change on the same cases | [`evaluate-a-change`](./evaluate-a-change/) | Offline | +| See the case grid before you pay for it | [`plan-before-you-spend`](./plan-before-you-spend/) | Offline | +| Wrap an existing agent | [`foreign-agent-quickstart`](./foreign-agent-quickstart/) | Offline, or an OpenAI-compatible endpoint | | Evaluate several attempts per case | [`multi-shot-optimization`](./multi-shot-optimization/) | Offline | -| Apply a release rule without search | [`held-out-gate`](./held-out-gate/) | Offline | -| Load folder-based cases | [`eval-fixtures-quickstart`](./eval-fixtures-quickstart/) | Offline | +| Apply a release rule without any search | [`held-out-gate`](./held-out-gate/) | Offline | +| Load cases from folders on disk | [`eval-fixtures-quickstart`](./eval-fixtures-quickstart/) | Offline | +| Record and compare scores over time | [`scorecard`](./scorecard/) | Offline | -Run an offline example from the repository root: +## Improve A Surface -```sh -pnpm tsx examples/selfimprove-quickstart/index.ts -``` +| Goal | Example | Requirements | +|---|---|---| +| Improve with your own candidate generator | [`selfimprove-quickstart`](./selfimprove-quickstart/) | Offline | +| Let another package own the text search | [`adapt-a-text-optimizer`](./adapt-a-text-optimizer/) | Offline | +| Compare official GEPA and SkillOpt | [`compare-optimization-methods`](./compare-optimization-methods/) | Python optimizer packages and an LLM endpoint | Run one official optimizer: @@ -30,18 +42,25 @@ GEPA_PRICE_OUT_PER_M=1.6 \ pnpm tsx examples/compare-optimization-methods/index.ts ``` -Replace the example rates with the exact endpoint rates. -Use `OPTIMIZERS=skillopt` for SkillOpt or `OPTIMIZERS=gepa,skillopt` for a shared comparison. +Replace the example rates with the exact rates for your endpoint. +Use `OPTIMIZERS=skillopt` for SkillOpt, or `OPTIMIZERS=gepa,skillopt` for a shared comparison. Read the [optimizer install instructions](./compare-optimization-methods/README.md) first. -## Existing Data +## Prove A Result -| Goal | Example | -|---|---| -| Analyze human approvals and rejections | [`customer-feedback-loop`](./customer-feedback-loop/) | -| Analyze OpenTelemetry spans | [`customer-otel-traces`](./customer-otel-traces/) | -| Record and compare scores over time | [`scorecard`](./scorecard/) | -| Reuse file-based cases and cached results | [`eval-fixtures-quickstart`](./eval-fixtures-quickstart/) | +| Goal | Example | Requirements | +|---|---|---| +| Register the rules before the data arrives | [`sealed-experiment`](./sealed-experiment/) | Offline | +| Certify a result that has no answer key | [`verify-without-an-answer-key`](./verify-without-an-answer-key/) | Offline | + +## Read Existing Data + +| Goal | Example | Requirements | +|---|---|---| +| Get a report from runs you already have | [`analyze-existing-runs`](./analyze-existing-runs/) | Offline | +| Get cited findings out of a failed batch | [`custom-trace-analyst`](./custom-trace-analyst/) | Offline | +| Analyze human approvals and rejections | [`customer-feedback-loop`](./customer-feedback-loop/) | Offline | +| Analyze OpenTelemetry spans | [`customer-otel-traces`](./customer-otel-traces/) | Offline | ## Benchmarks And Training @@ -60,5 +79,5 @@ Read the [optimizer install instructions](./compare-optimization-methods/README. | Run setup, execution, and scoring in one work directory | [`same-sandbox-harness`](./same-sandbox-harness/) | | Receive optional hosted events | [`hosted-ingest-server`](./hosted-ingest-server/) | -`_shared/` contains fixtures reused by multiple examples. +`_shared/` holds fixtures reused by several examples. It is not a standalone example. diff --git a/examples/adapt-a-text-optimizer/README.md b/examples/adapt-a-text-optimizer/README.md new file mode 100644 index 00000000..e3dd877a --- /dev/null +++ b/examples/adapt-a-text-optimizer/README.md @@ -0,0 +1,71 @@ +# Let Another Package Search, And Keep The Scoring Here + +## When to use this + +Use this example when a package already knows how to search over text, and you want its search without giving up control of execution, cost, and the final measurement. + +Use it also when you must show that the reported lift was not obtained by letting the optimizer see the final cases. + +For the official GEPA and SkillOpt bindings, use [`compare-optimization-methods`](../compare-optimization-methods/) instead. +This example is the general adapter for anything else. + +## How to run it + +```sh +pnpm tsx examples/adapt-a-text-optimizer/index.ts +``` + +No API key is required. +The optimizer in this file is a local hill climb. + +## What it does + +1. `externalTextOptimizationMethod()` wraps a `run` callback with the package's identity and limits. +2. The callback receives the starting candidate, train cases, and selection cases. +3. Every candidate is scored through `context.evaluate()`, which uses the configured execution and judges. +4. `compareOptimizationMethods()` runs the method, then scores the selected surface on final cases. + +The output is: + +```text +method: local-hill-climb +baseline: 0.000 +winner: 1.000 +final lift: 1.000 +lift interval: [1.000, 1.000] +cost is a complete total: true +``` + +## The three case sets + +| Set | Who sees it | What it decides | +|---|---|---| +| Train | The optimizer | Which candidates it writes | +| Selection | The optimizer | Which candidate it keeps | +| Final | Nobody until the search ends | The reported lift | + +The `run` callback never receives final cases. +That separation is the whole reason the reported number means anything. + +## Why it is built this way + +`context.evaluate()` is the only scoring path. +Calls are counted before execution and stop at `maxEvaluations`, and an unknown case id is rejected. +An optimizer that scored candidates its own way could report any number it liked. + +`maxOptimizerCostUsd` is required. +An adapter cannot be wired up without stating what the search may cost. + +Every optimizer-owned paid call must go through `context.cost.runPaidCall()`. +Declare `costAccounting: { kind: 'no-paid-work' }` only when the optimizer really makes none, as here. + +Read `totalCost.accountingComplete` before you treat the reported dollars as a complete total. +Read `pairwise` before you claim one method beat another. + +The train and selection campaigns are separate campaigns from the final scoring campaign. +Settings meant for both go in `optimizationRunOptions`. + +## Next + +- A complete adapter against a real package: [`docs/campaign-proposers.md`](../../docs/campaign-proposers.md#adapt-a-third-party-text-optimizer). +- Recipes, budgets, resuming, and data separation: [`docs/campaign-proposers.md`](../../docs/campaign-proposers.md). diff --git a/examples/adapt-a-text-optimizer/index.ts b/examples/adapt-a-text-optimizer/index.ts new file mode 100644 index 00000000..0989cc5c --- /dev/null +++ b/examples/adapt-a-text-optimizer/index.ts @@ -0,0 +1,157 @@ +/** + * Let another package own the search, and keep the scoring here. + * + * Run with: pnpm tsx examples/adapt-a-text-optimizer/index.ts + * + * `externalTextOptimizationMethod()` wraps a package that already knows how to + * search over text. Agent Eval supplies train and selection cases, executes + * every candidate, counts evaluations, records cost, and scores the winner on + * final cases the optimizer never saw. + * + * The optimizer below is a local hill climb so the example runs offline. + * Replace its body with a call into the real package. + */ + +import { mkdtempSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { + compareOptimizationMethods, + externalTextOptimizationMethod, +} from '../../src/campaign' +import type { JudgeConfig, Scenario } from '../../src/contract' + +interface SupportCase extends Scenario { + question: string + wanted: string +} + +interface SupportArtifact { + answer: string +} + +function caseOf(id: string, question: string, wanted: string): SupportCase { + return { id, kind: 'support', question, wanted } +} + +const trainScenarios = [ + caseOf('t1', 'Where is order 5512?', '5512'), + caseOf('t2', 'Refund invoice 8841?', '8841'), + caseOf('t3', 'Cancel ticket 2077?', '2077'), +] +const selectionScenarios = [ + caseOf('s1', 'Status of order 3310?', '3310'), + caseOf('s2', 'Reship parcel 6644?', '6644'), +] +const testScenarios = [ + caseOf('f1', 'Return item 9901?', '9901'), + caseOf('f2', 'Replace unit 4402?', '4402'), + caseOf('f3', 'Escalate case 7150?', '7150'), +] + +/** The agent under test. A surface that asks for the id makes it cite the id. */ +async function dispatchWithSurface( + surface: unknown, + scenario: SupportCase, +): Promise { + const prompt = String(surface) + const id = scenario.question.match(/\d+/)?.[0] ?? '' + return { answer: prompt.includes('reference number') ? `Reference ${id}. On it.` : 'On it.' } +} + +const judge: JudgeConfig = { + name: 'cites-the-id', + dimensions: [{ key: 'cited', description: 'The answer repeats the reference number' }], + score: ({ artifact, scenario }) => { + const cited = artifact.answer.includes(scenario.wanted) ? 1 : 0 + return { dimensions: { cited }, composite: cited, notes: '' } + }, +} + +/** Stand-in for the third-party package. Keeps the best of a few rewrites. */ +async function hillClimb(options: { + initial: string + candidates: string[] + evaluate: (candidate: string) => Promise +}): Promise<{ best: string; evaluations: number }> { + let best = options.initial + let bestScore = await options.evaluate(best) + let evaluations = 1 + for (const candidate of options.candidates) { + const score = await options.evaluate(candidate) + evaluations += 1 + if (score > bestScore) { + best = candidate + bestScore = score + } + } + return { best, evaluations } +} + +const method = externalTextOptimizationMethod({ + name: 'local-hill-climb', + source: { kind: 'package', package: 'local-hill-climb', version: '0.0.0' }, + objective: 'Make the answer repeat the reference number.', + // Identity of the execution and scoring behavior. Change it whenever that + // behavior changes, so two runs are never compared across a silent edit. + evaluationId: 'support-cites-id@1', + maxEvaluations: 12, + // A hard ceiling on optimizer-owned spend. It is required, so an adapter + // cannot be wired up without stating what the search may cost. + maxOptimizerCostUsd: 0, + describeScenario: (scenario) => ({ question: scenario.question }), + describeArtifact: (artifact) => ({ answer: artifact.answer }), + run: async (context) => { + const seed = String(context.seedCandidate) + const outcome = await hillClimb({ + initial: seed, + candidates: [ + `${seed}\nBe brief.`, + `${seed}\nAlways quote the reference number.`, + `${seed}\nQuote the reference number and confirm the next step.`, + ], + evaluate: async (candidate) => { + // One call scores one candidate on one case, through the configured + // execution and judges. Each call is counted against maxEvaluations + // before it runs, and an unknown case id is rejected. + let total = 0 + for (const example of context.trainSet) { + const response = await context.evaluate({ candidate, exampleId: example.id }) + total += response.score + } + return total / Math.max(1, context.trainSet.length) + }, + }) + return { + bestCandidate: outcome.best, + resumed: false, + // This optimizer makes no paid calls, so there is nothing to meter. + costAccounting: { kind: 'no-paid-work' }, + } + }, +}) + +const result = await compareOptimizationMethods({ + methods: [method], + baselineSurface: 'Answer politely.', + trainScenarios, + selectionScenarios, + testScenarios, + dispatchWithSurface, + judges: [judge], + runDir: mkdtempSync(join(tmpdir(), 'agent-eval-optimizer-')), + // The final scoring campaign makes no paid calls in this example. + expectUsage: 'off', + // Shared defaults for the train and selection campaigns each method runs. + // They are separate campaigns, so the setting above does not reach them. + optimizationRunOptions: { expectUsage: 'off' }, +}) + +for (const score of result.scores) { + console.log(`method: ${score.name}`) + console.log(`baseline: ${score.baselineComposite.toFixed(3)}`) + console.log(`winner: ${score.winnerComposite.toFixed(3)}`) + console.log(`final lift: ${score.lift.toFixed(3)}`) + console.log(`lift interval: [${score.liftCi.low.toFixed(3)}, ${score.liftCi.high.toFixed(3)}]`) +} +console.log('cost is a complete total:', result.totalCost.accountingComplete) diff --git a/examples/analyze-existing-runs/README.md b/examples/analyze-existing-runs/README.md new file mode 100644 index 00000000..771c237d --- /dev/null +++ b/examples/analyze-existing-runs/README.md @@ -0,0 +1,55 @@ +# Get A Report From Runs You Already Have + +## When to use this + +Use this example when the runs already happened. +You have logs, a feedback table, or exported rows, and you must know what they say. +No agent is invoked and no model is called. + +Use a different front door when you want to run the agent again: [`evaluate-a-change`](../evaluate-a-change/). + +## How to run it + +```sh +pnpm tsx examples/analyze-existing-runs/index.ts +``` + +No API key is required. + +## What it does + +1. Twelve `RunRecord` rows describe two candidates answering the same six cases. +2. `analyzeRuns()` reads them and returns one `InsightReport`. +3. The report carries score distributions, paired lift with an interval, judge agreement, cost, failure clusters, contamination checks, and recommendations. + +The output is: + +```text +runs analyzed: 12 +mean score: 0.5975 +paired lift: 0.1283333333333333 +lift interval: [ 0.10666666666666663, 0.15333333333333335 ] +paired n: 6 +recommendations: 2 +``` + +## Why it is built this way + +`baselineCandidateId` and `candidateCandidateId` make the lift paired. +Rows match on `(experimentId, scenarioId, seed)`, so each case is compared with itself, not with the mean of the other arm. +A row that finds no partner stays visible in the result instead of being dropped. + +Every field of the report is defined in [`docs/insight-report.md`](../../docs/insight-report.md). + +## Where the rows come from + +If your data is not already in `RunRecord` shape, convert it first: + +| Source | Adapter | +|---|---| +| Approvals and rejections in a table | `fromFeedbackTable` — see [`customer-feedback-loop`](../customer-feedback-loop/) | +| OpenTelemetry spans | `fromOtelSpans` — see [`customer-otel-traces`](../customer-otel-traces/) | + +## Next + +- Cluster the failures with a trace analyst: [`custom-trace-analyst`](../custom-trace-analyst/). diff --git a/examples/analyze-existing-runs/index.ts b/examples/analyze-existing-runs/index.ts new file mode 100644 index 00000000..d6c62384 --- /dev/null +++ b/examples/analyze-existing-runs/index.ts @@ -0,0 +1,57 @@ +/** + * Read a report out of runs you already captured. No agent runs here. + * + * Run with: pnpm tsx examples/analyze-existing-runs/index.ts + * + * A `RunRecord` is the analysis-time projection of one run: who ran, on what, + * with which seed and cost, and what it scored. `analyzeRuns()` turns a list + * of them into score distributions, paired lift, cost, and recommendations. + */ + +import { analyzeRuns } from '../../src/contract' +import type { RunRecord } from '../../src/run-record' + +/** One row per (candidate, case). Both arms answer the same six cases. */ +function record(candidateId: string, scenarioId: string, score: number): RunRecord { + return { + runId: `${candidateId}-${scenarioId}`, + experimentId: 'support-reply-v3', + candidateId, + seed: 42, + model: 'openai/gpt-4.1@2025-04-14', + promptHash: candidateId === 'baseline' ? 'a'.repeat(64) : 'b'.repeat(64), + configHash: 'c'.repeat(64), + commitSha: 'deadbeef', + wallMs: 1_200, + costUsd: 0.004, + costProvenance: { kind: 'observed', usd: 0.004 }, + tokenUsage: { input: 820, output: 240 }, + terminalOutcome: 'succeeded', + outcome: { holdoutScore: score, raw: { holdoutScore: score } }, + splitTag: 'holdout', + scenarioId, + } +} + +const cases = ['refund', 'shipping', 'cancel', 'address', 'invoice', 'warranty'] +const baselineScores = [0.51, 0.62, 0.48, 0.55, 0.6, 0.44] +const candidateScores = [0.68, 0.71, 0.63, 0.66, 0.7, 0.59] + +const runs: RunRecord[] = [ + ...cases.map((id, i) => record('baseline', id, baselineScores[i]!)), + ...cases.map((id, i) => record('cite-the-ticket', id, candidateScores[i]!)), +] + +const report = await analyzeRuns({ + runs, + // Naming both sides pairs the rows by (experimentId, scenarioId, seed). + baselineCandidateId: 'baseline', + candidateCandidateId: 'cite-the-ticket', +}) + +console.log('runs analyzed: ', report.n) +console.log('mean score: ', report.composite.mean) +console.log('paired lift: ', report.lift?.delta) +console.log('lift interval: ', report.lift?.ci95) +console.log('paired n: ', report.lift?.n) +console.log('recommendations:', report.recommendations.length) diff --git a/examples/custom-trace-analyst/README.md b/examples/custom-trace-analyst/README.md new file mode 100644 index 00000000..8c95e05c --- /dev/null +++ b/examples/custom-trace-analyst/README.md @@ -0,0 +1,63 @@ +# Get Cited Findings Out Of A Run + +## When to use this + +Use this example when a batch of runs failed and you must know why, in a form you can act on. +An analyst reads recorded evidence and returns findings. +Each finding carries a claim, a severity, a confidence, and the exact evidence it rests on, so a reader can check it instead of trusting it. + +Use `runExact()` when the caller, not the registry, must own every execution choice: which analysts run, in what order, and with what budget. + +## How to run it + +```sh +pnpm tsx examples/custom-trace-analyst/index.ts +``` + +No API key is required. +The analyst in this file is deterministic and calls no model. + +## What it does + +1. One analyst declares its id, version, cost class, and `executionConfig`. +2. `AnalystRegistry` registers it. +3. `runExact()` runs the declared list and returns findings, an execution plan, and a completion record. + +The output is: + +```text +run status: complete +analysts: 1 +high run_tests failed: exit 1, 3 failing specs [span:s2] +high run_tests failed: exit 1, 1 failing spec [span:s4] +``` + +## Why it is built this way + +Every option in `runExact()` must be present. +`null` disables a channel on purpose, so a missing budget cannot be read as an unlimited one. +The registry contributes no default, which is the difference between this call and `registry.run()`. + +Exact runs are serial. +A caller that needs concurrent or recursive scheduling composes exact runs in its own runtime rather than adding a second scheduler here. + +Every receipt says whether it is `complete` or `failed`. +A complete receipt must cover the whole plan. +A failed receipt may cover only the part that ran, and it keeps the completed summaries, findings, usage, and cost that were already valid. + +`cost: { kind: 'deterministic' }` is a promise the registry enforces: a deterministic analyst must not call a model. +Declare `{ kind: 'llm' }` when it does, and the budget channel becomes meaningful. + +## Related helpers + +| Goal | Call | +|---|---| +| Write a research question once and bind it to any engine later | `defineTraceAnalyst()` | +| Wrap an existing analyze function over a trace store | `defineCustomAnalyst()` | +| Start from the built-in analysts | `buildDefaultAnalystRegistry()` | +| Measure an analyst against labeled issues and exact spans | `runAnalystBenchmark()` | + +## Next + +- Read the analyst model and the public benchmark it is calibrated against: [`docs/trace-analysis.md`](../../docs/trace-analysis.md). +- Grade a finding by executing the repair it proposes: [`docs/trace-repair-grader.md`](../../docs/trace-repair-grader.md). diff --git a/examples/custom-trace-analyst/index.ts b/examples/custom-trace-analyst/index.ts new file mode 100644 index 00000000..58475aca --- /dev/null +++ b/examples/custom-trace-analyst/index.ts @@ -0,0 +1,101 @@ +/** + * Turn recorded evidence into cited findings, with every execution choice + * stated by the caller. + * + * Run with: pnpm tsx examples/custom-trace-analyst/index.ts + * + * An analyst reads recorded evidence and returns findings. Each finding names + * a severity, a claim, and the exact evidence it rests on. `runExact()` runs a + * declared list of analysts in a declared order and takes no default from the + * registry. + */ + +import { createHash } from 'node:crypto' +import { AnalystRegistry } from '../../src/analyst' +import type { AnalystFinding, ExactCapableAnalyst } from '../../src/analyst' + +interface ToolCall { + spanId: string + tool: string + ok: boolean + message: string +} + +const PRODUCED_AT = '2026-08-15T00:00:00.000Z' + +function findingId(analystId: string, claim: string): string { + return createHash('sha256').update(`${analystId}\n${claim}`).digest('hex').slice(0, 32) +} + +/** + * A deterministic analyst. It calls no model, so it costs nothing and returns + * the same findings for the same input every time. `executionConfig` is the + * canonical record of every behavior knob `version` does not already bind. + */ +const failedTools: ExactCapableAnalyst<{ toolCalls: ToolCall[] }> = { + id: 'failed-tools', + description: 'Report every tool call that failed, with the span that carries it.', + inputKind: 'custom', + cost: { kind: 'deterministic' }, + version: '1.0.0', + executionConfig: { kind: 'failed-tools', schemaVersion: '1' }, + async analyze(input): Promise { + return input.toolCalls + .filter((call) => !call.ok) + .map((call) => { + const claim = `${call.tool} failed: ${call.message}` + return { + schema_version: '1.0.0', + finding_id: findingId('failed-tools', claim), + analyst_id: 'failed-tools', + produced_at: PRODUCED_AT, + severity: 'high', + area: 'tool-use', + claim, + evidence_refs: [{ kind: 'span', uri: `span:${call.spanId}` }], + confidence: 1, + } satisfies AnalystFinding + }) + }, +} + +const registry = new AnalystRegistry() +registry.register(failedTools) + +const toolCalls: ToolCall[] = [ + { spanId: 's1', tool: 'read_file', ok: true, message: 'ok' }, + { spanId: 's2', tool: 'run_tests', ok: false, message: 'exit 1, 3 failing specs' }, + { spanId: 's3', tool: 'write_file', ok: true, message: 'ok' }, + { spanId: 's4', tool: 'run_tests', ok: false, message: 'exit 1, 1 failing spec' }, +] + +const result = await registry.runExact( + 'analysis-1', + // Custom input is keyed by analyst id: each analyst reads only its own. + { custom: { 'failed-tools': { toolCalls } } }, + { + // The array is the execution order. Exact runs are serial. + analystIds: ['failed-tools'], + // `null` disables a channel on purpose. There is no implicit default. + budget: null, + totalTimeoutMs: null, + signal: null, + costLedger: null, + costLedgerIdentity: null, + costPhase: null, + tags: null, + priorFindings: null, + chainFindings: false, + // Stop rather than run an analyst whose declared input is absent. + missingInputMode: 'abort', + applyRegistryHooks: false, + useRegistryChat: false, + }, +) + +console.log('run status:', result.completion.status) +console.log('analysts: ', result.execution_plan.analysts.length) +for (const finding of result.findings) { + const span = finding.evidence_refs[0] + console.log(`${finding.severity.padEnd(6)} ${finding.claim} [${span?.uri}]`) +} diff --git a/examples/evaluate-a-change/README.md b/examples/evaluate-a-change/README.md new file mode 100644 index 00000000..18bff8d4 --- /dev/null +++ b/examples/evaluate-a-change/README.md @@ -0,0 +1,44 @@ +# Score One Change On The Same Cases + +## When to use this + +Use this example when you changed a prompt, a skill, or a configuration value, and you must know whether the change helped. +It is the smallest complete path through the package: cases in, scores out. +Start here before any optimizer. + +## How to run it + +```sh +pnpm tsx examples/evaluate-a-change/index.ts +``` + +No API key is required. +The agent and the judge are local functions. + +## What it does + +1. `defineAgentEval()` receives three cases, an agent, one judge, and a starting surface. +2. `evaluate()` runs every case on the starting surface and scores each result. +3. A second `evaluate({ surface })` call runs the same cases on the changed surface. +4. Each call returns the score distribution under `aggregates.byJudge`. + +The output is: + +```text +baseline: { 'ticket-id': { mean: 0, stdev: 0, ci95: [ 0, 0 ], n: 3 } } +candidate: { 'ticket-id': { mean: 1, stdev: 0, ci95: [ 1, 1 ], n: 3 } } +``` + +## Why it is built this way + +The surface is the only value that changes between the two calls. +The cases, the agent, and the judge stay identical, so the score difference measures the change and nothing else. + +`expectUsage: 'off'` is set because this agent makes no paid model calls. +The default is `'assert'`, which fails a run whose cells report no cost receipt. +Keep the default whenever real model calls happen: it is the check that stops an unmeasured run from reading as a free one. + +## Next + +- Give the same object a candidate generator and a release rule: [`selfimprove-quickstart`](../selfimprove-quickstart/). +- Inspect the per-case grid before you spend: [`plan-before-you-spend`](../plan-before-you-spend/). diff --git a/examples/evaluate-a-change/index.ts b/examples/evaluate-a-change/index.ts new file mode 100644 index 00000000..01a81b20 --- /dev/null +++ b/examples/evaluate-a-change/index.ts @@ -0,0 +1,44 @@ +/** + * The smallest complete evaluation: one surface, one judge, two scores. + * + * Run with: pnpm tsx examples/evaluate-a-change/index.ts + * + * Everything here is offline. Replace `agent` with your product call and + * `judge` with your real scoring function to point this at production. + */ + +import { defineAgentEval } from '../../src/contract' + +interface SupportCase { + id: string + kind: 'support' +} + +const evalKit = defineAgentEval({ + scenarios: [ + { id: 'refund', kind: 'support' }, + { id: 'shipping', kind: 'support' }, + { id: 'cancel', kind: 'support' }, + ], + agent: async (prompt, scenario) => + String(prompt).includes('ticket') ? `Ticket ${scenario.id}: on it.` : 'On it.', + judge: { + name: 'ticket-id', + dimensions: [{ key: 'present', description: 'The answer includes the ticket id' }], + score: ({ artifact, scenario }) => { + const present = artifact.includes(scenario.id) ? 1 : 0 + return { dimensions: { present }, composite: present, notes: '' } + }, + }, + baselineSurface: 'Answer politely.', + // This agent makes no paid calls, so no usage receipt can exist. + expectUsage: 'off', +}) + +const baseline = await evalKit.evaluate() +const candidate = await evalKit.evaluate({ + surface: 'Answer politely and cite the ticket id.', +}) + +console.log('baseline: ', baseline.aggregates.byJudge) +console.log('candidate:', candidate.aggregates.byJudge) diff --git a/examples/plan-before-you-spend/README.md b/examples/plan-before-you-spend/README.md new file mode 100644 index 00000000..4f5ac339 --- /dev/null +++ b/examples/plan-before-you-spend/README.md @@ -0,0 +1,55 @@ +# See The Grid Before You Pay For It + +## When to use this + +Use this example when a run costs real money and you must know what it will execute first. +A campaign runs one cell per case per replicate. +`planCampaignRun()` reports the state of every cell without dispatching anything. + +Use it also when a run failed partway and you must decide what to rerun. + +## How to run it + +```sh +pnpm tsx examples/plan-before-you-spend/index.ts +``` + +No API key is required. +The example writes its cache into a temporary directory. + +## What it does + +1. `planCampaignRun()` prints the cell schedule before any work starts. + Every cell reports `run`, because nothing is cached yet. +2. `runCampaign()` executes the same grid and caches each completed cell. +3. `planCampaignRun()` runs again. Every cell now reports `cached`. + +A cell reports one of three states: + +| Status | Meaning | +|---|---| +| `run` | The cell must execute. It has no valid cached result. | +| `cached` | A valid cached result exists. The cell will not execute again. | +| `blocked` | A cached file exists but is unreadable or has untrustworthy cost data. | + +Set `rerunInvalidCachedCells: true` to turn every blocked cell into a `run` cell and keep the valid cached ones. +Set `resumable: false` only when you intend to rerun the whole grid. + +## Why it is built this way + +`runCampaign()` refuses to start when a cached file is unreadable or lacks trustworthy cost data. +That check covers the whole schedule before concurrent work begins, so one bad cache file cannot waste the paid calls of earlier cells. + +`abortOnCellError: true` stops the campaign on the first failed cell. +The failed cell writes `//failure-receipt.json` first. +That file holds the original error, the cell result, the exact call ids, and the settled agent-plus-judge cost and token totals. +Active sibling cells are cancelled and are allowed to record their own receipts before the campaign rejects. +Leave the option unset to record the error and continue the remaining cases. + +Two settings must match between the two calls: `costLedger` and `costTags`. +Both calls read the same receipts, so different values make the plan describe a different run. + +## Next + +- Run the same grid across models and profiles: [`docs/eval-surface-map.md`](../../docs/eval-surface-map.md). +- Let a candidate generator drive the grid: [`selfimprove-quickstart`](../selfimprove-quickstart/). diff --git a/examples/plan-before-you-spend/index.ts b/examples/plan-before-you-spend/index.ts new file mode 100644 index 00000000..865ec802 --- /dev/null +++ b/examples/plan-before-you-spend/index.ts @@ -0,0 +1,62 @@ +/** + * Inspect the per-case grid before a campaign spends anything. + * + * Run with: pnpm tsx examples/plan-before-you-spend/index.ts + * + * A campaign runs one cell per case per replicate. `planCampaignRun()` reports + * which cells are reusable, which must run, and which are blocked, without + * dispatching. `runCampaign()` then executes the same grid. + */ + +import { mkdtempSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { planCampaignRun, runCampaign } from '../../src/campaign' +import type { JudgeConfig, Scenario } from '../../src/contract' + +interface ExtractionCase extends Scenario { + input: string + expected: string +} + +const scenarios: ExtractionCase[] = [ + { id: 'invoice', kind: 'extract', input: 'Invoice 4417 is due', expected: '4417' }, + { id: 'order', kind: 'extract', input: 'Order 9032 shipped', expected: '9032' }, + { id: 'ticket', kind: 'extract', input: 'Ticket 1188 reopened', expected: '1188' }, +] + +/** The agent under test. Replace with your product call. */ +async function dispatch(scenario: ExtractionCase): Promise { + return scenario.input.match(/\d+/)?.[0] ?? '' +} + +const judge: JudgeConfig = { + name: 'exact-id', + dimensions: [{ key: 'exact', description: 'The extracted id matches the expected id' }], + score: ({ artifact, scenario }) => { + const exact = artifact === scenario.expected ? 1 : 0 + return { dimensions: { exact }, composite: exact, notes: '' } + }, +} + +const runDir = mkdtempSync(join(tmpdir(), 'agent-eval-plan-')) + +// Nothing has run yet, so every cell is runnable and none is reusable. +const plan = planCampaignRun({ scenarios, dispatch, judges: [judge], runDir }) +console.table(plan.cells) + +const first = await runCampaign({ + scenarios, + dispatch, + judges: [judge], + runDir, + // Stop the whole campaign on the first failed cell instead of paying for + // the rest of the grid. The failed cell writes its receipt first. + abortOnCellError: true, + expectUsage: 'off', +}) +console.log('cells run:', first.cells.length) + +// The same plan after a complete run: every cell is now reusable. +const second = planCampaignRun({ scenarios, dispatch, judges: [judge], runDir }) +console.table(second.cells) diff --git a/examples/sealed-experiment/README.md b/examples/sealed-experiment/README.md new file mode 100644 index 00000000..bbcd39bd --- /dev/null +++ b/examples/sealed-experiment/README.md @@ -0,0 +1,63 @@ +# Register The Rules, Then Execute Only Those Rules + +## When to use this + +Use this example when the result must convince someone who does not trust you. +An experiment that chooses its threshold after seeing the numbers proves nothing. +This front door removes that possibility: the rules are data, the data is hashed, and the execution surface accepts no rule that is not in the hash. + +Use it before you spend on an A/B comparison, not after. + +## How to run it + +```sh +pnpm tsx examples/sealed-experiment/index.ts +``` + +No API key is required. + +## What it does + +1. `ExperimentSpec` states the arms, the outcome, the admission funnel, the estimand, the interval, and the decision table. +2. `sealExperiment()` hashes the whole rule tree and returns a digest. +3. `verifySealedExperiment()` proves the tree still matches its digest. +4. `openSealedExperiment()` returns the executors bound to that seal. +5. `admit()` runs the funnel, `estimate()` computes the contrast, `interval()` brackets it, and `decide()` reads the registered table. + +The output is: + +```text +seal digest: 514cf827e89aad8dfea580f15e03fb90c4572f7fc07be8fec2d6fd265f600878 +seal verifies: true +population: support cases the baseline failed +input: 8 +stage entering excluded remaining +------------------- -------- -------- --------- +has-a-known-outcome 8 1 7 +baseline-failed-it 7 1 6 +surviving: 6 (input 8 = surviving 6 + excluded 2) +risk difference: 0.8333333333333334 +95% interval: [ 0.6666666666666666, 1 ] +verdict: citation-helps +``` + +## Why it is built this way + +`estimate()`, `interval()`, and `decide()` take a registered name plus evidence. +None of them takes an alpha, a threshold, a metric, or a stopping rule. +A rule that is not in the seal cannot run, and a rule that is in the seal cannot run differently. + +The funnel is the denominator chain. +Every stage reports how many rows entered, how many it removed, and how many remain, and the total must reconcile. +A result whose denominator changed between stages is visible in the table rather than lost. + +The outcome field `passed` is a boolean. +It reads as 1 or 0, so the paired mean difference is the risk difference and no caller has to re-encode the rows. + +The interval resamples pair differences, not raw pass flags. +The registered quantity is a contrast, so its uncertainty must be the uncertainty of that contrast. + +## Next + +- Read the doctrine and the refusals: [`docs/experiment.md`](../../docs/experiment.md). +- Certify a result that has no answer key: [`verify-without-an-answer-key`](../verify-without-an-answer-key/). diff --git a/examples/sealed-experiment/index.ts b/examples/sealed-experiment/index.ts new file mode 100644 index 00000000..bc5a7949 --- /dev/null +++ b/examples/sealed-experiment/index.ts @@ -0,0 +1,141 @@ +/** + * Register an experiment's rules as data, seal them, then execute only what + * the seal contains. + * + * Run with: pnpm tsx examples/sealed-experiment/index.ts + * + * The execution surface takes a sealed rule plus evidence rows. It has no + * parameter for a threshold, a metric, or a stopping rule, so the rule that + * ran cannot differ from the rule that was registered. + */ + +import { + type EvidenceRecord, + renderFunnelTable, + type ExperimentSpec, + openSealedExperiment, + sealExperiment, + verifySealedExperiment, +} from '../../src/experiment' + +const spec: ExperimentSpec = { + id: 'cite-the-ticket-20260815', + hypothesis: 'Asking the agent to cite the ticket id raises the pass rate on failed cases.', + arms: [ + { id: 'baseline', role: 'control', policyDigest: 'support-reply@v3' }, + { id: 'cite-ticket', role: 'treatment', policyDigest: 'support-reply@v3' }, + ], + // A binary outcome: each row either passed its own suite or did not. + outcome: { kind: 'binary', source: 'injected-suite', pass: 'exit-0' }, + admission: { + population: 'support cases the baseline failed', + stages: [ + { + id: 'has-a-known-outcome', + keep: { kind: 'compare', field: 'outcomeKnown', op: 'eq', value: true }, + }, + { + id: 'baseline-failed-it', + keep: { kind: 'compare', field: 'baselinePassed', op: 'eq', value: false }, + }, + ], + }, + estimands: { + pairedContrast: { + kind: 'paired-mean-diff', + armField: 'arm', + treatment: 'cite-ticket', + control: 'baseline', + pairBy: 'caseId', + // The outcome field is a boolean. It reads as 1 or 0, so the mean of + // the paired differences is the risk difference. + value: 'passed', + missing: 'zero-diff', + }, + }, + intervals: { + pairedContrast95: { + kind: 'cluster-bootstrap', + clusterBy: 'queue', + resamples: 2_000, + seed: 20260815, + level: 0.95, + method: 'percentile', + }, + }, + decision: { + kind: 'table', + branches: [ + { + when: { kind: 'interval-excludes-zero', interval: 'pairedContrast95', sign: 'positive' }, + verdict: 'citation-helps', + report: ['pairedContrast', 'pairedContrast95'], + }, + { + when: { kind: 'interval-excludes-zero', interval: 'pairedContrast95', sign: 'negative' }, + verdict: 'citation-hurts', + report: ['pairedContrast', 'pairedContrast95'], + }, + { + when: { kind: 'interval-includes-zero', interval: 'pairedContrast95' }, + verdict: 'no-effect-resolved-at-this-n', + report: ['pairedContrast', 'pairedContrast95'], + }, + ], + }, + seed: 20260815, +} + +const sealed = await sealExperiment(spec, { sealedAt: '2026-08-15T00:00:00Z' }) +console.log('seal digest: ', sealed.digest) +console.log('seal verifies:', await verifySealedExperiment(sealed)) + +const registered = await openSealedExperiment(sealed) + +// Eight recorded cases in two queues. Two of them do not belong in the +// population, and the funnel is where that is stated. +const cases = [ + { caseId: 'c0', queue: 'billing', outcomeKnown: true, baselinePassed: false }, + { caseId: 'c1', queue: 'billing', outcomeKnown: true, baselinePassed: false }, + { caseId: 'c2', queue: 'billing', outcomeKnown: true, baselinePassed: false }, + { caseId: 'c3', queue: 'shipping', outcomeKnown: true, baselinePassed: false }, + { caseId: 'c4', queue: 'shipping', outcomeKnown: true, baselinePassed: false }, + { caseId: 'c5', queue: 'shipping', outcomeKnown: true, baselinePassed: false }, + { caseId: 'c6', queue: 'billing', outcomeKnown: false, baselinePassed: false }, + { caseId: 'c7', queue: 'shipping', outcomeKnown: true, baselinePassed: true }, +] + +// The funnel is the denominator chain: every stage reports what it removed. +const admission = registered.admit(cases as EvidenceRecord[]) +console.log(renderFunnelTable(admission.funnel)) + +/** The treatment recovered every admitted case except c2. */ +const recovered = (caseId: string): boolean => caseId !== 'c2' + +const outcomeRows: EvidenceRecord[] = admission.survivors.flatMap((row) => [ + { ...row, arm: 'baseline', passed: false }, + { ...row, arm: 'cite-ticket', passed: recovered(String(row.caseId)) }, +]) + +const estimate = registered.estimate('pairedContrast', outcomeRows) +console.log('risk difference:', estimate.value) + +// The bootstrap resamples whole queues of PAIR DIFFERENCES, not raw pass +// flags: the registered quantity is a contrast, so its interval is too. +const differenceRows: EvidenceRecord[] = admission.survivors.map((row) => ({ + ...row, + diff: (recovered(String(row.caseId)) ? 1 : 0) - 0, +})) +const interval = registered.interval('pairedContrast95', { + kind: 'rows', + rows: differenceRows, + value: 'diff', +}) +console.log('95% interval: ', [interval.lower, interval.upper]) + +const outcome = registered.decide({ + intervals: { pairedContrast95: { lower: interval.lower, upper: interval.upper } }, + quantities: {}, + obligationsMet: {}, +}) +console.log('verdict: ', outcome.verdict) diff --git a/examples/verify-without-an-answer-key/README.md b/examples/verify-without-an-answer-key/README.md new file mode 100644 index 00000000..886efca1 --- /dev/null +++ b/examples/verify-without-an-answer-key/README.md @@ -0,0 +1,56 @@ +# Certify A Result That Has No Answer Key + +## When to use this + +Use this example when the work has no held-out test suite. +An unsolved problem has none by definition, and a novel result cannot be graded against a key that does not exist. + +Use it also when you must state how strong a certificate is. +"Certified" is not one bit: a proof kernel and an LLM judge can both return `{ valid: true, score: 1 }`, and they mean very different things. + +## How to run it + +```sh +pnpm tsx examples/verify-without-an-answer-key/index.ts +``` + +No API key is required. +The checker in this file is a stand-in, not a proof assistant. + +## What it does + +1. `VERIFICATION_STRATEGIES` reports each family member with its determinism class and its documented failure mode. +2. `defineEquivalenceCheck()` registers a blind two-arm design over one claim. +3. `runEquivalenceCheck()` runs an injected checker over the two committed statements. +4. Two arms that agree produce a `proved` obligation. Two arms that disagree produce a refutation and a separating witness. + +The output is: + +```text +test deterministic assumes an answer key; certifies nothing outside suite coverage, and a stubbed integration reports green +proof-kernel deterministic the formalization gap: the kernel certifies the formal statement, never that it matches the informal claim +agreement probabilistic the shared blind spot: derivers with common corpora or priors agree for the same wrong reason +agreeing arms: proved +checker: example-alpha-equivalence +divergent arms: refuted-with-separating-witness +witness: the two statements disagree outside bound-variable names +``` + +## Why it is built this way + +This package ships the taxonomy, the record types, and the refusals. +It ships no checker. +A checker is an injected boundary that returns a typed outcome, so you bind your own proof assistant, invariant harness, replication runner, or judge. +The binding carries its own identity and its own determinism claim, which is what makes a certificate re-runnable by someone else. + +Both blindness flags must be `true`. +An arm that saw the other arm's statement, or saw the outcome, derived nothing independently. +The module refuses to record such a check rather than store an invalid one that looks valid. + +A refutation is a successful check. +It reports the formalization gap — the specific way a proof kernel can certify a wrong result — with the witness that shows where the two statements part. + +## Next + +- Read the full family, the port shape, and the worked pilot: [`docs/verification-strategies.md`](../../docs/verification-strategies.md). +- See where every verifier lands its result: [`docs/verdicts.md`](../../docs/verdicts.md). diff --git a/examples/verify-without-an-answer-key/index.ts b/examples/verify-without-an-answer-key/index.ts new file mode 100644 index 00000000..3817d63f --- /dev/null +++ b/examples/verify-without-an-answer-key/index.ts @@ -0,0 +1,94 @@ +/** + * Certify a result that has no answer key. + * + * Run with: pnpm tsx examples/verify-without-an-answer-key/index.ts + * + * A held-out test suite is one member of a family of ten verification + * strategies, not the family itself. Each member has a documented way it can + * certify a wrong result. This example reads those failure modes, then runs + * the blind two-arm statement-equivalence protocol against an injected + * checker. + */ + +import { + type EquivalenceArm, + type EquivalenceChecker, + defineEquivalenceCheck, + runEquivalenceCheck, + VERIFICATION_STRATEGIES, +} from '../../src/index' + +for (const member of ['test', 'proof-kernel', 'agreement'] as const) { + const profile = VERIFICATION_STRATEGIES[member] + console.log(`${member.padEnd(13)} ${profile.determinism.padEnd(14)} ${profile.failureMode}`) +} + +// The formal claim two people are asked to state, independently. +const definition = defineEquivalenceCheck({ + source: 'proof-kernel', + artifact: 'arXiv:1507.05650 inequality (4.6)', + arms: 2, + blind: true, +}) + +const arms: [EquivalenceArm, EquivalenceArm] = [ + { + armId: 'from-the-paper', + statement: 'theorem bcww (x y : Real) : x * y <= (x ^ 2 + y ^ 2) / 2', + derivedFrom: 'the published LaTeX source', + blindness: { toOtherArms: true, toOutcome: true }, + }, + { + armId: 'from-the-artifacts', + statement: 'theorem bcww (a b : Real) : a * b <= (a ^ 2 + b ^ 2) / 2', + derivedFrom: "the campaign's produced artifacts", + blindness: { toOtherArms: true, toOutcome: true }, + }, +] + +/** + * The package ships the port, never the checker. A real binding runs a proof + * assistant here and returns its kernel's answer. This stand-in accepts the + * two statements when they differ only by bound-variable names. + */ +const checker: EquivalenceChecker = { + strategy: 'proof-kernel', + identity: { name: 'example-alpha-equivalence', version: '0.0.0' }, + determinism: 'deterministic', + async check({ statements }) { + const normalize = (s: string) => s.replace(/\b[a-z]\b/g, '_') + const equivalent = normalize(statements[0]) === normalize(statements[1]) + if (!equivalent) { + return { + succeeded: true, + value: { + status: 'refuted-with-separating-witness', + separatingWitness: 'the two statements disagree outside bound-variable names', + evidenceDigest: `sha256:${'1'.repeat(64)}`, + }, + } + } + return { + succeeded: true, + value: { + status: 'proved', + evidenceDigest: `sha256:${'0'.repeat(64)}`, + }, + } + }, +} + +const agreed = await runEquivalenceCheck(definition, arms, checker) +console.log('agreeing arms: ', agreed.obligation.status) +console.log('checker: ', agreed.obligation.checker?.name) + +// The interesting outcome. Arm B stated a strict inequality, so the two arms +// never verified the same claim. A refutation is a successful check: it is +// the formalization gap made visible, with a witness in hand. +const divergent: [EquivalenceArm, EquivalenceArm] = [ + arms[0], + { ...arms[1], statement: 'theorem bcww (a b : Real) : a * b < (a ^ 2 + b ^ 2) / 2' }, +] +const refuted = await runEquivalenceCheck(definition, divergent, checker) +console.log('divergent arms:', refuted.obligation.status) +console.log('witness: ', refuted.obligation.separatingWitness) diff --git a/scripts/verify-package-exports.mjs b/scripts/verify-package-exports.mjs index 2c27432b..aaeda106 100644 --- a/scripts/verify-package-exports.mjs +++ b/scripts/verify-package-exports.mjs @@ -145,7 +145,7 @@ try { symlinkSync(packageDir, join(appDir, 'node_modules', '@tangle-network', 'agent-eval'), 'dir') verifyDistTypeRuntimeAgreement(packageJson, appDir) const readme = readFileSync(join(repoRoot, 'README.md'), 'utf8') - const quickstart = readme.match(/## Evaluate An Agent[\s\S]*?```ts\n([\s\S]*?)\n```/)?.[1] + const quickstart = readme.match(/## Quickstart[\s\S]*?```ts\n([\s\S]*?)\n```/)?.[1] if (!quickstart) throw new Error('README quickstart TypeScript block was not found') writeFileSync(join(appDir, 'quickstart.ts'), `${quickstart}\n`) writeFileSync(join(appDir, 'package.json'), JSON.stringify({ type: 'module' })) diff --git a/src/experiment/ast.ts b/src/experiment/ast.ts index 385c261a..1c3a18df 100644 --- a/src/experiment/ast.ts +++ b/src/experiment/ast.ts @@ -249,6 +249,10 @@ export type Estimand = treatment: string control: string pairBy: string + /** + * Field path to the per-row outcome. A boolean reads as 1 or 0, so a + * binary pass/fail outcome gives the risk difference directly. + */ value: string /** A pair one arm did not answer contributes a difference of exactly zero. */ missing: 'zero-diff' @@ -291,6 +295,24 @@ function evaluateSetExpr( return out } +/** + * Read one registered outcome field as a number. + * + * A binary outcome reaches evidence as `true` or `false`. Its mean is the pass + * rate and the mean of its paired differences is the risk difference, so + * `true` reads as 1 and `false` as 0 — the same quantity a caller would + * otherwise encode by hand, and the same reading in every interpreter here. + * Every other type, and a non-finite number, is a measurement defect: it + * rejects instead of poisoning the mean with `NaN` or a coerced zero. + */ +function readNumericOutcome(raw: unknown, context: string, field: string, where: string): number { + if (typeof raw === 'boolean') return raw ? 1 : 0 + if (typeof raw === 'number' && Number.isFinite(raw)) return raw + throw new ValidationError( + `${context}: value field '${field}' is not a finite number or a boolean on ${where}`, + ) +} + /** Compute an estimand over evidence rows. Pure; reads only registered fields. */ export function computeEstimand( estimand: Estimand, @@ -321,12 +343,12 @@ export function computeEstimand( const arm = String(readField(row, estimand.armField)) if (arm !== estimand.treatment && arm !== estimand.control) continue const pair = String(readField(row, estimand.pairBy)) - const value = readField(row, estimand.value) - if (typeof value !== 'number') { - throw new ValidationError( - `computeEstimand: paired-mean-diff value field '${estimand.value}' is not a number on pair '${pair}'`, - ) - } + const value = readNumericOutcome( + readField(row, estimand.value), + 'computeEstimand paired-mean-diff', + estimand.value, + `pair '${pair}'`, + ) const slot = byPair.get(pair) ?? {} if (arm === estimand.treatment) slot.treatment = value else slot.control = value @@ -408,12 +430,12 @@ export function computeInterval( const clusters = new Map() for (const row of evidence.rows) { const cluster = String(readField(row, spec.clusterBy)) - const value = readField(row, evidence.value) - if (typeof value !== 'number') { - throw new ValidationError( - `computeInterval: value field '${evidence.value}' is not a number in cluster '${cluster}'`, - ) - } + const value = readNumericOutcome( + readField(row, evidence.value), + 'computeInterval cluster-bootstrap', + evidence.value, + `cluster '${cluster}'`, + ) const bucket = clusters.get(cluster) if (bucket) bucket.push(value) else clusters.set(cluster, [value]) diff --git a/tests/experiment/paired-binary-outcome.test.ts b/tests/experiment/paired-binary-outcome.test.ts new file mode 100644 index 00000000..98184ca3 --- /dev/null +++ b/tests/experiment/paired-binary-outcome.test.ts @@ -0,0 +1,236 @@ +/** + * A binary outcome, carried end to end through a sealed experiment. + * + * The spec below is the shape `scripts/tb-gated-stop-ab.ts` registers: a + * `binary` outcome, a `paired-mean-diff` over the boolean `passed` field, a + * task-clustered bootstrap, and the four-branch decision table those two feed. + * The suite runs that path on evidence rows whose `passed` field is a boolean, + * because that is the type an injected test suite reports. + */ + +import { describe, expect, it } from 'vitest' +import { ValidationError } from '../../src/errors' +import { + computeEstimand, + type EvidenceRecord, + type ExperimentSpec, + openSealedExperiment, + sealExperiment, +} from '../../src/experiment/index' + +const gatedStopSpec: ExperimentSpec = { + id: 'gated-stop-binary-outcome', + hypothesis: + 'A gated continuation recovers more failed rows than a blind continuation on the same budget.', + arms: [ + { id: 'blind-continue', role: 'control', policyDigest: 'pinned-continuation@v2' }, + { id: 'gated-continue', role: 'treatment', policyDigest: 'pinned-continuation@v2' }, + ], + outcome: { + kind: 'binary', + source: 'injected-suite', + digestVerified: true, + pass: 'reward-file-contains-1', + droppedRollouts: 'forbidden', + }, + estimands: { + pairedContrast: { + kind: 'paired-mean-diff', + armField: 'arm', + treatment: 'gated-continue', + control: 'blind-continue', + pairBy: 'rowId', + value: 'passed', + missing: 'zero-diff', + }, + }, + intervals: { + pairedContrast95: { + kind: 'cluster-bootstrap', + clusterBy: 'taskName', + resamples: 2_000, + seed: 20260814, + level: 0.95, + method: 'percentile', + }, + }, + decision: { + kind: 'table', + branches: [ + { + when: { + kind: 'all', + of: [ + { kind: 'interval-excludes-zero', interval: 'pairedContrast95', sign: 'positive' }, + { kind: 'obligation-met', obligation: 'matched-realized-tokens' }, + ], + }, + verdict: 'gated-stop-survives', + report: ['pairedContrast', 'pairedContrast95'], + }, + { + when: { + kind: 'all', + of: [ + { kind: 'interval-excludes-zero', interval: 'pairedContrast95', sign: 'negative' }, + { kind: 'obligation-met', obligation: 'matched-realized-tokens' }, + ], + }, + verdict: 'gated-stop-dies', + report: ['pairedContrast', 'pairedContrast95'], + }, + { + when: { kind: 'interval-includes-zero', interval: 'pairedContrast95' }, + verdict: 'no-effect-resolved-at-this-n', + report: ['pairedContrast', 'pairedContrast95'], + }, + { + when: { + kind: 'not', + of: { kind: 'obligation-met', obligation: 'matched-realized-tokens' }, + }, + verdict: 'contrast-refused-unmatched-budget', + report: ['pairedContrast95'], + }, + ], + }, + obligations: [ + { + id: 'matched-realized-tokens', + appliesToVerdicts: ['gated-stop-survives', 'gated-stop-dies'], + control: 'realized prompt and completion tokens agree within 5 % between arms', + }, + ], + seedDerivation: { from: ['seed', 'rowId', 'rolloutIndex'] }, + seed: 20260814, +} + +/** Eight tasks, two rows each: 16 pairs, clustered by task. */ +const TASKS = ['t0', 't1', 't2', 't3', 't4', 't5', 't6', 't7'] as const + +/** Treatment passes the first row of every task, and the second row of t0..t2. */ +function treatmentPassed(task: string, row: number): boolean { + return row === 0 || task === 't0' || task === 't1' || task === 't2' +} + +/** Control passes only the second row of t0 and t1. */ +function controlPassed(task: string, row: number): boolean { + return row === 1 && (task === 't0' || task === 't1') +} + +function booleanRows(): EvidenceRecord[] { + const rows: EvidenceRecord[] = [] + for (const taskName of TASKS) { + for (const row of [0, 1]) { + const rowId = `${taskName}-r${row}` + rows.push({ rowId, taskName, arm: 'gated-continue', passed: treatmentPassed(taskName, row) }) + rows.push({ rowId, taskName, arm: 'blind-continue', passed: controlPassed(taskName, row) }) + } + } + return rows +} + +/** The same evidence with the outcome hand-encoded as 1 and 0. */ +function encodedRows(): EvidenceRecord[] { + return booleanRows().map((row) => ({ ...row, passed: row.passed === true ? 1 : 0 })) +} + +/** One difference per pair, the row shape the clustered bootstrap resamples. */ +function pairDifferenceRows(): EvidenceRecord[] { + const rows: EvidenceRecord[] = [] + for (const taskName of TASKS) { + for (const row of [0, 1]) { + const treatment = treatmentPassed(taskName, row) ? 1 : 0 + const control = controlPassed(taskName, row) ? 1 : 0 + rows.push({ rowId: `${taskName}-r${row}`, taskName, diff: treatment - control }) + } + } + return rows +} + +describe('paired-mean-diff over a binary outcome', () => { + it('reads boolean pass/fail as the risk difference', () => { + const result = computeEstimand(gatedStopSpec.estimands!.pairedContrast!, booleanRows()) + // 9 of 16 pairs improve; none regress. + expect(result.numerator).toBe(9) + expect(result.denominator).toBe(16) + expect(result.value).toBeCloseTo(9 / 16, 12) + }) + + it('gives the identical number whether the caller encodes the outcome or not', () => { + const fromBooleans = computeEstimand(gatedStopSpec.estimands!.pairedContrast!, booleanRows()) + const fromNumbers = computeEstimand(gatedStopSpec.estimands!.pairedContrast!, encodedRows()) + expect(fromBooleans).toEqual(fromNumbers) + }) + + it('keeps zero-diff semantics when one arm never answered a pair', () => { + const rows = booleanRows().filter( + (row) => !(row.rowId === 't0-r0' && row.arm === 'blind-continue'), + ) + // t0-r0 kept its passing treatment side, so the sum is unchanged. + const result = computeEstimand(gatedStopSpec.estimands!.pairedContrast!, rows) + expect(result.numerator).toBe(9) + expect(result.denominator).toBe(16) + }) + + it.each([ + ['a string', 'true'], + ['null', null], + ['NaN', Number.NaN], + ['Infinity', Number.POSITIVE_INFINITY], + ['an absent field', undefined], + ])('rejects %s in the outcome field', (_label, value) => { + const rows = booleanRows().map((row) => + row.rowId === 't3-r1' && row.arm === 'gated-continue' ? { ...row, passed: value } : row, + ) + expect(() => computeEstimand(gatedStopSpec.estimands!.pairedContrast!, rows)).toThrow( + ValidationError, + ) + expect(() => computeEstimand(gatedStopSpec.estimands!.pairedContrast!, rows)).toThrow( + /is not a finite number or a boolean on pair 't3-r1'/, + ) + }) +}) + +describe('the sealed gated-stop path, from boolean rows to a verdict', () => { + it('estimates, brackets, and decides without the caller re-encoding anything', async () => { + const sealed = await sealExperiment(gatedStopSpec, { sealedAt: '2026-08-14T00:00:00Z' }) + const registered = await openSealedExperiment(sealed) + + const estimate = registered.estimate('pairedContrast', booleanRows()) + expect(estimate.value).toBeCloseTo(9 / 16, 12) + + const interval = registered.interval('pairedContrast95', { + kind: 'rows', + rows: pairDifferenceRows(), + value: 'diff', + }) + // Every task cluster carries at least one improved pair and no regression, + // so no resample of whole clusters can reach zero. + expect(interval.lower).toBeGreaterThan(0) + expect(interval.upper).toBeLessThanOrEqual(1) + expect(interval.level).toBe(0.95) + + const outcome = registered.decide({ + intervals: { pairedContrast95: { lower: interval.lower, upper: interval.upper } }, + quantities: {}, + obligationsMet: { 'matched-realized-tokens': true }, + }) + expect(outcome.verdict).toBe('gated-stop-survives') + expect(outcome.report).toEqual(['pairedContrast', 'pairedContrast95']) + }) + + it('brackets a boolean pass rate directly, with the same reading', async () => { + const sealed = await sealExperiment(gatedStopSpec, { sealedAt: '2026-08-14T00:00:00Z' }) + const registered = await openSealedExperiment(sealed) + const treatmentRows = booleanRows().filter((row) => row.arm === 'gated-continue') + const interval = registered.interval('pairedContrast95', { + kind: 'rows', + rows: treatmentRows, + value: 'passed', + }) + // 11 of 16 treatment rows pass, so the bootstrap sits inside (0, 1). + expect(interval.lower).toBeGreaterThan(0) + expect(interval.upper).toBeLessThanOrEqual(1) + }) +})