Feat/eval optimization loop 91

[wzxsdf]

Summary

Added examples/optimization/eval_optimize_loop/: a reproducible closed-loop pipeline of Evaluate → Attribute → Optimize → Validate → Gate → Audit. Runs without an API key in fake mode (< 1s).

A lightweight orchestration layer on top of the SDK's AgentEvaluator / AgentOptimizer: extracts candidates from OptimizeResult.rounds, independently re-evaluates them on train/validation sets (optimizer self-scores are not trusted), performs per-case diff comparison, and applies gating.

Domain: Library Catalog Agent (Novel/Science/History/FAQ classification + catalog tools + JSON). Six test cases cover three essential scenarios: optimizable-success, no-op, and overfitting-regression.

Three modes: fake / trace (deterministic, no LLM) and online (real AgentOptimizer).

Motivation

Issue #91 transforms prompt optimization into an auditable release decision. Aggregate scores cannot reveal validation-set regressions, new critical failures, or root causes. Therefore, optimization and release gating are decoupled: every candidate must pass independent re-evaluation before acceptance.

Design

• Layered Attribution (pipeline/attribution.py): A rule engine attributes based on actual vs. expected trajectory differences (format / tool selection / parameters / knowledge / final response mismatch). Falls back to single-variable counterfactuals when rules miss. Counterfactuals run local pure-Python metrics (zero API cost), enabling ≥75% accuracy with ≤3 min runtime.
• Triple Overfitting Detection (pipeline/gate.py): Explicit train↑ while val↓, generalization gap (only when val is unmet), and trend divergence — plus critical-case regression. Candidates with val-down / train-up are always rejected.
• Deterministic + Auditable: fake/trace modes use pre-recorded variants and two LLM-free SDK evaluators (final_response_avg_score contains, tool_trajectory_avg_score exact). Features atomic writes + SHA256 digests; three-state cost.measurement where unknown costs are treated as failures (fail-closed).
• Gating: Configurable AND rules with three-state decisions (accept / reject / needs_review) and corresponding exit codes (0 / 2 / 1).

Validation

• pytest tests/ -q: 9 passed (three scenarios, overfitting detection, attribution coverage/accuracy, ≤3 min runtime, report fields, hidden samples, CLI exit codes, data quality).
• fake/trace → exit 0, selects robust; overfit → rejected (overfitting + critical regression); ineffective → rejected (tie).
• Attribution coverage: 100%, category accuracy: 100%.
• flake8 + yapf + git diff --check all pass.

Deliverables

• run_pipeline.py + 7 modules in pipeline/
• offline/fixtures.py
• agent/ (online LlmAgent with prompt hot-reload)
• 3 + 6 test cases + gate.json + optimizer.json
• sample_output/ (reports + audit logs)
• README.md (~400-char design note) + DESIGN.md

Closes #91

[wzxsdf]

I have read the CLA Document and I hereby sign the CLA

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@e113610). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff             @@
##             main        #179   +/-   ##
==========================================
  Coverage        ?   87.55867%           
==========================================
  Files           ?         467           
  Lines           ?       44103           
  Branches        ?           0           
==========================================
  Hits            ?       38616           
  Misses          ?        5487           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.