@tangle-network/agent-eval

Measure agent behavior, compare changes on the same cases, and improve prompts or skills without showing the final test cases to the optimizer.

The evaluation path runs in your TypeScript process. Model calls happen only through the clients and agents you configure.

New to the package? Read concepts first — it takes five minutes and defines every word used here.

Install

pnpm add @tangle-network/agent-eval

Quickstart

This example is offline and complete. Copy it, run it, then replace the agent and the judge with your product code.

import { defineAgentEval } from '@tangle-network/agent-eval/contract'

interface SupportCase {
  id: string
  kind: 'support'
}

const evalKit = defineAgentEval<SupportCase, string>({
  scenarios: [
    { id: 'refund', kind: 'support' },
    { id: 'shipping', kind: 'support' },
    { id: 'cancel', kind: 'support' },
  ],
  agent: async (prompt, scenario) =>
    String(prompt).includes('ticket') ? `Ticket ${scenario.id}: on it.` : 'On it.',
  judge: {
    name: 'ticket-id',
    dimensions: [{ key: 'present', description: 'The answer includes the ticket id' }],
    score: ({ artifact, scenario }) => {
      const present = artifact.includes(scenario.id) ? 1 : 0
      return { dimensions: { present }, composite: present, notes: '' }
    },
  },
  baselineSurface: 'Answer politely.',
  expectUsage: 'off',
})

console.log((await evalKit.evaluate()).aggregates.byJudge)
console.log(
  (await evalKit.evaluate({ surface: 'Answer politely and cite the ticket id.' })).aggregates
    .byJudge,
)

Each call runs every case, records what the agent produced, applies the same judge, and returns score distributions.

Three words carry this example. A case is one task the agent must do. A surface is the value being changed: a prompt, a skill, or a serialized configuration. A judge is a function that scores one produced result.

expectUsage: 'off' is set because this agent makes no paid calls. The default, 'assert', fails a run whose cells report no cost receipt. Keep the default whenever real model calls happen.

Runnable copy: examples/evaluate-a-change.

Which Front Door

Every row is a function you call. Each links to a runnable example.

When to call it	What you give it	What you get back
defineAgentEval() — you changed a surface and must know whether it helped	cases, an agent, a judge, a starting surface	evaluate() for scores, improve() for a search plus a release decision
selfImprove() — you want candidate generation, scoring, and a release decision in one call	cases, an agent, a judge, a starting surface	a report, a winner surface, and a gateDecision
analyzeRuns() — the runs already happened and no agent needs to run again	RunRecord[]	an InsightReport: distributions, paired lift, judge agreement, cost, failure clusters
fromFeedbackTable() (example) / fromOtelSpans() (example) — your data is in a table or an OTel collector, not in RunRecord shape	source rows or spans	RunRecord[] ready for analyzeRuns()
planCampaignRun() / runCampaign() — you need direct control of the case grid, or you must see it before paying for it	cases, a dispatch function, judges, a run directory	a per-cell schedule, then a campaign result with cached cells
loadEvalFixtureScenarios() — agents should add cases as folders on disk	evals/<name>/PROMPT.md plus checks	Scenario[] for runCampaign()
compareOptimizationMethods() — two search methods must be compared at equal budget	methods, a starting surface, train, selection, and final cases	per-method final lift, intervals, pairwise contrasts, and cost
gepaOptimizationMethod() / skillOptOptimizationMethod() — official GEPA or Microsoft SkillOpt should own the search	an objective, a recipe or trainer, an optimizer budget	an optimization method for the comparison above
externalTextOptimizationMethod() — another package owns text search and you keep the scoring	the package identity, limits, and a run callback	the same, with the final cases never exposed
SurfaceProposer — candidate generation belongs to your product	a propose() function	candidates the campaign executes, scores, and gates
runProfileMatrix() — the same cases must run across models or profiles	axes of models and profiles, cases	one row per cell, with an explicit unknown model rather than an invented one
sealExperiment() / openSealedExperiment() — the result must convince someone who does not trust you	arms, an admission funnel, an estimand, an interval, a decision table	a hashed rule tree, and executors that can run no other rule
runEquivalenceCheck() / VERIFICATION_STRATEGIES — the work has no held-out test suite	a claim, two blind arms, an injected checker	a certification that names who vouched and how it can fail
AnalystRegistry.runExact() — a batch of runs failed and you need cited findings	recorded evidence, a declared analyst list	findings with evidence references, an execution plan, and a receipt
runAnalystBenchmark() — an analyst's accuracy must be measured, not assumed	labeled issues and exact span locations	scored findings, trace reads, model calls, tokens, cost, and runtime
deltaRepair() — a finding must be graded by executing the repair it proposes	a trajectory, an analyst finding, a sandbox	the repair's measured effect against a no-fix control
replayVerify() — you must know whether a recorded failure still reproduces	a recorded shell trajectory and its pinned image	a re-execution verdict and the divergences found
analyzeSupervisorRun() — a recursive or supervised run directory must be read	a run directory	counts that stay missing when a measurement is missing, never zero
buildRlDataset() — scored runs should become training data	run records and preferences	reward, preference, and supervised rows

Configure Model Calls

Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters all take the same ChatClient.

import { createChatClient } from '@tangle-network/agent-eval'

const chat = createChatClient({
  transport: 'router',
  apiKey: process.env.TANGLE_API_KEY!,
  defaultModel: 'openai/gpt-4.1',
  maximumAttempts: 3,
})

Use direct-provider for an OpenAI-compatible endpoint, cli-bridge for a local subscription, sandbox-sdk for Sandbox, or custom to adapt another SDK. A custom adapter must return a ChatResponse and declare maximumAttempts before a capped cost account can dispatch it.

The official GEPA and SkillOpt optimizers run through a Python bridge. Install commands, version pins, and the reason for each pin: GEPA, SkillOpt, and DSPy.

Entry Points

Import	Use
@tangle-network/agent-eval/contract	Define an evaluation, run it, improve it, and analyze existing runs.
@tangle-network/agent-eval/campaign	Campaigns, optimization methods, comparisons, storage, and release rules.
@tangle-network/agent-eval/experiment	Experiments as sealed objects: registered rules, funnels, estimands, refusals.
@tangle-network/agent-eval/analyst	Built-in and custom trace analysts, labeled comparison, costs, and reports.
@tangle-network/agent-eval/trace-repair	Grade one analyst finding by executing the repair it proposes.
@tangle-network/agent-eval/trajectory-replay	Re-execute a recorded shell trajectory and check whether its failure reproduces.
@tangle-network/agent-eval/traces	Store, replay, and inspect structured traces.
@tangle-network/agent-eval/reporting	Statistical comparisons and report rendering.
@tangle-network/agent-eval/supervisor-run	Read recursive run directories without collapsing missing measurements to zero.
@tangle-network/agent-eval/profile-cell	Create and validate portable agent-profile identities.
@tangle-network/agent-eval/ledger-core	Append-only hash-chained journal with idempotent append and chain verification.
@tangle-network/agent-eval/benchmarks	Benchmark adapters and retrieval metrics.
@tangle-network/agent-eval/rl	Export rewards, preferences, and training rows.
@tangle-network/agent-eval/wire	HTTP and RPC schemas for other languages.

Use the root import for common primitives. Use a subpath when you want an explicit capability boundary.

Documentation

Question	Read
What do these words mean?	docs/concepts.md
Why does this package exist, and where is it going?	docs/charter.md
Which run* function do I want?	docs/eval-surface-map.md
How do I choose a candidate-generation method?	docs/campaign-proposers.md
What is in an InsightReport?	docs/insight-report.md
How do I register an experiment as a sealed object?	docs/experiment.md
How is something certified without an answer key?	docs/verification-strategies.md
Where does every verifier land its result?	docs/verdicts.md
How do I score a string from another language?	docs/wire-protocol.md

The example index lists every runnable example.

Development

pnpm install
pnpm typecheck
pnpm typecheck:examples
pnpm test
pnpm build

Python compatibility tests use the locked dependencies:

cd clients/python
uv sync --frozen --extra dev --group gepa-release
AGENT_EVAL_EXPECT_GEPA_RELEASE=1 \
  uv run --frozen --extra dev --group gepa-release \
  pytest tests/test_gepa_release_compatibility.py tests/test_gepa_bridge.py

uv sync --frozen --extra dev --group skillopt-source --group gepa-source
uv run --frozen pytest

uv sync --frozen --extra dev --extra dspy
uv run --frozen pytest tests/test_dspy_metric.py

License

MIT.