Agent Trajectory Diff

Diffs two agent trace logs by case_id and tells you three things a final-answer-only eval misses: did the grade change, did the tool-call sequence change, and did cases get silently added or removed from the eval set itself.

Built to read JSONL traces in eval-lite-style format: case_id, input, output, grade, trajectory. It is not coupled to eval-lite as a dependency. Any JSONL log using those field names works, including reshaped exports from other tools.

Why This Exists

A final answer can be right by luck. An agent that succeeds after six wrong tool calls and an agent that succeeds in two calls both show "pass" if you only grade the final output. This tool diffs the path: which tools got called, in what order, between a baseline run and a candidate run.

Usage

python cli.py baseline.jsonl candidate.jsonl
python cli.py baseline.jsonl candidate.jsonl --json diff.json

Exits with code 2 if any regressions are found, so it can be wired into CI as a quality gate:

- name: Check for agent regressions
  run: python cli.py baseline.jsonl candidate.jsonl

One-Command Demo

python cli.py examples/baseline.jsonl examples/candidate.jsonl

The pre-recorded logs include one regression, one improvement, and one same-grade path change.

What Counts as a Diff

For every case_id present in both files:

• regression: graded pass in baseline, fail in candidate
• improvement: graded fail in baseline, pass in candidate
• path_changed_same_grade: same grade, different tool-call sequence
• unchanged: same grade, same tool sequence

Cases present in only one file are reported separately as only_in_baseline or only_in_candidate.

Example Output

Trajectory diff: 3 shared case(s)
  1 regression(s)  1 improvement(s)  1 path-changed (same grade)

Regressions:
  [route_planner_01] pass -> fail

Improvements:
  [eta_estimate_03] fail -> pass
      tools: ['eta_calc'] -> ['eta_calc', 'traffic_check']

Path changed, same grade:
  [path_change_05] pass -> pass
      tools: ['carrier_lookup', 'price_compare'] -> ['price_compare', 'carrier_lookup']

Run Tests

pip install pytest
pytest tests/ -v

The test suite also uses tests/fixtures/baseline.jsonl and tests/fixtures/candidate.jsonl with deliberately known diffs.

License

MIT. See LICENSE for details.