🤖 AgentEval

Open source agent evaluation framework — score any AI agent across providers with standardized, reproducible benchmarks.

---

Why AgentEval?

Every team building AI agents needs to answer one question:

How good is my agent, actually?

There are frameworks for orchestration (LangChain), observability (Langfuse), and tracking (MLflow) — but no standardized, provider-agnostic way to score an agent's performance. AgentEval fills that gap:

• One command → comparable scores across OpenAI, Anthropic, Ollama, and local models
• Standardized task suites for the capabilities that matter: codegen, QA, reasoning, summarization, tool-use
• Real metrics: accuracy, pass@k, latency, cost per run
• CI-ready: fail builds when agent quality drops below a threshold

pip install agenteval
agenteval run --provider openai --model gpt-4o-mini --suite all

# AgentEval Report

- Provider: openai (gpt-4o-mini)
- Suite: all
- Tasks: 8/10 passed
- Accuracy: 80.0%
- Avg latency: 420.3 ms
- Total cost: $0.000412

---

Features

Feature	Description
Multi-provider	OpenAI-compatible APIs, Anthropic, Gemini, Groq, Ollama, local models, mock
Task suites	codegen, qa, reasoning, summarization, tool-use (+ custom)
pass@k	Run a task N times, pass if any sample succeeds
Cost tracking	Per-run USD cost for every provider
CLI + SDK	Python CLI and TypeScript SDK (typed, tested)
CI integration	Non-zero exit below threshold; JSON output
Reusable workflow	uses: rexblade58/agenteval/.github/workflows/eval.yml@main
Docker image	Multi-arch GHCR image for CLI + dashboard — see docs
Zero heavy deps	httpx + rich only — no framework lock-in

---

Status

Actively developed. Current state of the repo:

Area	Status
Python core (CLI, providers, evaluator, reports)	✅ Done — 9 unit tests
TypeScript SDK (types + mock runner + CLI wrapper)	✅ Done — 4 unit tests, strict TS
CI pipeline	✅ Python 3.10/3.11/3.12 + TS typecheck/test
Mock provider (keyless testing)	✅ Done
Providers: OpenAI, Anthropic, Ollama, mock	✅ Done
Gemini provider	✅ Done — --provider gemini
Groq provider (free tier)	✅ Done — --provider groq
Semantic scoring (--scoring semantic)	✅ Done
pass@k + cost + latency metrics	✅ Done
Web dashboard (agenteval serve)	✅ Done — local, zero-dep
GitHub Actions reusable workflow	✅ Done — see docs
Docker image (multi-arch GHCR)	✅ Done — see docs

See the issues for what's next, GitHub Actions integration for CI setup, and the commit history for recent progress.

Quick start

Mock (no API key)

git clone https://github.com/rexblade58/agenteval.git
cd agenteval
pip install -e packages/core
agenteval run --provider mock --suite all

OpenAI

export OPENAI_API_KEY=sk-...
agenteval run --provider openai --model gpt-4o-mini --suite codegen \
  --format json --output reports/openai-codegen.json

Reports are written in a stable JSON schema (schema_version, generated_at, metrics, per-task results) — see examples/reports/ for samples.

Anthropic

export ANTHROPIC_API_KEY=sk-ant-...
agenteval run --provider anthropic --model claude-3-5-sonnet-latest --suite qa

Google Gemini

export GEMINI_API_KEY=AIza...
agenteval run --provider gemini --model gemini-2.0-flash --suite reasoning

Groq (free tier)

export GROQ_API_KEY=gsk_...
agenteval run --provider groq --model llama-3.3-70b-versatile --suite codegen

Semantic scoring (paraphrase-tolerant)

agenteval run --provider mock --suite qa --scoring semantic

Local (Ollama)

agenteval run --provider ollama --model llama3.2 --suite reasoning

Web dashboard

agenteval run --provider mock --suite all --output reports/mock.json
agenteval serve --dir reports
# → http://127.0.0.1:8000  (accuracy/cost/latency trends, provider comparison)

TypeScript SDK

import { evaluateMock, runCli } from '@agenteval/sdk';

const report = runCli({
  provider: 'openai',
  model: 'gpt-4o-mini',
  suite: 'codegen',
});
console.log(`Accuracy: ${(report.accuracy * 100).toFixed(1)}%`);

---

Architecture

agenteval/
├── packages/
│   ├── core/               Python evaluation engine
│   │   └── agenteval/
│   │       ├── cli.py      argparse CLI
│   │       ├── providers.py  OpenAI / Anthropic / Ollama / mock
│   │       ├── evaluator.py  pass@k engine + scoring
│   │       ├── tasks.py      built-in task suites
│   │       └── report.py     JSON + Markdown reports
│   └── sdk-ts/             TypeScript SDK
├── examples/               Runnable examples + sample reports
├── docs/                   Guides
└── .github/workflows/      CI (tests, lint, publish)

---

Adding a provider

Providers implement a single method:

class MyProvider(BaseProvider):
    name = "my-provider"

    def complete(self, messages, temperature=0.7) -> ProviderResult:
        # call your API, return ProviderResult(text=..., latency_ms=..., ...)
        ...

Register it in PROVIDER_REGISTRY and it works with the CLI, SDK, and reporting — no other changes needed.

---

Roadmap

Planned work is tracked as GitHub issues — community PRs are very welcome.

• Web dashboard with historical comparisons — agenteval serve
• Semantic similarity scoring for QA tasks — --scoring semantic
• Google Gemini provider — --provider gemini
• Groq provider with free-tier models — --provider groq
• GitHub Actions reusable workflow — uses: .../eval.yml@main
• Docker image for self-hosting — ghcr.io/rexblade58/agenteval
• Human-in-the-loop task review
• Adversarial / robustness evaluation
• Agent trace evaluation (multi-step tool calls)

---

Contributing

See CONTRIBUTING.md. Bug reports, provider PRs, and new task suites are all welcome.

License

MIT © Menard Rosal