feat(examples): add auditable AgentOptimizer evaluation loop

.gitignore

@@ -21,6 +21,9 @@ coverage.xml
 test-ngtest-ut-trpc-agent-py.xml
 .pytest_cache
 
+# Runtime artefacts for Issue #91; stable reviewed reports live in sample_output/.
+examples/optimization/eval_optimize_loop/runs/
+
 node_modules
 package-lock.json
 pyrightconfig.json

docs/superpowers/plans/2026-07-11-issue-91-phase-1-2.md

@@ -0,0 +1,288 @@
+# Issue #91 Phase 1 and 2 Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Deliver the deterministic, no-key fake evaluation-and-optimization loop for Issue #91 and defer all live optimizer integration.
+
+**Architecture:** All behavior remains under `examples/optimization/eval_optimize_loop/`. Phase 1 implements strict pipeline models, config validation, normalization, case comparison, independent gate, and reporting. Phase 2 uses the public `AgentEvaluator.evaluate_eval_set()` for every baseline/candidate train and validation evaluation, with deterministic fake dependencies and a prompt sandbox that restores baseline prompts.
+
+**Tech Stack:** Python 3.10+, Pydantic v2, pytest/pytest-asyncio, existing `trpc_agent_sdk.evaluation` APIs.
+
+## Global Constraints
+
+- Only implement Phase 1 and Phase 2 of the user-provided `trpc_agent_issue_91_solution.md`. Do not implement `--mode live`, `AgentOptimizerBackend`, or change SDK public APIs.
+- Fake mode requires no key, environment variable, network access, or LLM judge; every result must be deterministic.
+- Candidate prompts must be restored in `PromptSandbox.__aexit__`, including evaluator exceptions.
+- A fake candidate never supplies scores: the real `AgentEvaluator` produces baseline and candidate results.
+- Include exactly three train and three validation public cases. The general candidate is accepted; noop and overfit are rejected.
+- Add only `examples/optimization/eval_optimize_loop/runs/` to `.gitignore`; commit reviewed `sample_output/` files.
+- Never place secrets in config snapshots or reports.
+
+---
+
+## File Structure
+
+- Create: `examples/optimization/eval_optimize_loop/{README.md,DESIGN.md,run_pipeline.py,optimizer.json,train.evalset.json,val.evalset.json}`
+- Create: `examples/optimization/eval_optimize_loop/agent/{__init__.py,agent.py,prompts/system.md,prompts/router.md}`
+- Create: `examples/optimization/eval_optimize_loop/pipeline/{__init__.py,models.py,config.py,normalization.py,comparator.py,gate.py,reporter.py,evaluator.py,prompt_sandbox.py,audit.py}`
+- Create: `examples/optimization/eval_optimize_loop/fake/{__init__.py,fake_agent.py,fake_judge.py,fixture_optimizer.py,candidates.json}`
+- Create: `examples/optimization/eval_optimize_loop/sample_output/{optimization_report.json,optimization_report.md}`
+- Create: `tests/examples/optimization/eval_optimize_loop/{__init__.py,test_config.py,test_normalization.py,test_comparator.py,test_gate.py,test_prompt_sandbox.py,test_fake_pipeline.py,test_report_schema.py}`
+- Modify: `.gitignore`
+
+### Task 1: Phase 1 strict data and configuration contracts
+
+**Files:**
+- Create: `pipeline/models.py`, `pipeline/config.py`, `optimizer.json`
+- Test: `tests/examples/optimization/eval_optimize_loop/test_config.py`
+
+**Interfaces:**
+- `canonical_sha256(value: object) -> str` serializes via `json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))`.
+- `load_pipeline_config(config_path: Path, *, mode: Literal["fake"]) -> PipelineConfig`.
+- Pydantic models use `ConfigDict(extra="forbid")`: `CaseSnapshot`, `CandidateRecord`, `CaseDelta`, `GateRuleResult`, `GateDecision`, `OptimizationReport`.
+
+- [ ] **Step 1: Write failing tests**
+
+```python
+def test_digest_is_order_independent() -> None:
+    assert canonical_sha256({"b": 2, "a": 1}) == canonical_sha256({"a": 1, "b": 2})
+
+def test_fake_loader_needs_no_live_key(monkeypatch, config_path) -> None:
+    monkeypatch.delenv("TRPC_AGENT_API_KEY", raising=False)
+    assert load_pipeline_config(config_path, mode="fake").pipeline.reproducibility.seed == 42
+```
+
+- [ ] **Step 2: Verify RED**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_config.py -q`
+
+Expected: fails because the configuration module is absent.
+
+- [ ] **Step 3: Implement minimum contract**
+
+```python
+def canonical_sha256(value: object) -> str:
+    encoded = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8")
+    return f"sha256:{hashlib.sha256(encoded).hexdigest()}"
+
+def load_pipeline_config(config_path: Path, *, mode: Literal["fake"]) -> PipelineConfig:
+    if mode != "fake":
+        raise ValueError("Phase 1 and 2 implement only --mode fake")
+    raw = json.loads(config_path.read_text(encoding="utf-8"))
+    return PipelineConfig(raw=raw, pipeline=PipelineSettings.model_validate(raw["pipeline"]))
+```
+
+- [ ] **Step 4: Verify GREEN and commit**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_config.py -q`
+
+Commit: `git add examples/optimization/eval_optimize_loop tests/examples/optimization/eval_optimize_loop && git commit -m "feat: add issue 91 pipeline contracts"`
+
+### Task 2: Phase 1 evaluator normalization, case deltas, independent gate, and reporter
+
+**Files:**
+- Create: `pipeline/normalization.py`, `pipeline/comparator.py`, `pipeline/gate.py`, `pipeline/reporter.py`
+- Test: `test_normalization.py`, `test_comparator.py`, `test_gate.py`, `test_report_schema.py`
+
+**Interfaces:**
+- `normalize_eval_results(results_by_eval_id, *, split, metric_weights) -> dict[str, CaseSnapshot]`
+- `compare_case(baseline, candidate, *, epsilon, critical_case_ids) -> CaseDelta`
+- `evaluate_gate(baseline, candidate, *, settings) -> GateDecision`
+- `write_reports(report, output_dir) -> tuple[Path, Path]`
+
+- [ ] **Step 1: Write failing behavior tests**
+
+```python
+def test_critical_pass_to_fail_is_hard_regression(snapshot_factory) -> None:
+    delta = compare_case(snapshot_factory("val_refund_critical", True, 1.0),
+                         snapshot_factory("val_refund_critical", False, 0.0),
+                         epsilon=1e-6, critical_case_ids={"val_refund_critical"})
+    assert (delta.transition, delta.hard_fail_added) == ("REGRESSION", True)
+
+def test_gate_rejects_overfit(baseline, overfit, gate_settings) -> None:
+    assert evaluate_gate(baseline, overfit, settings=gate_settings).accepted is False
+```
+
+- [ ] **Step 2: Verify RED**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_normalization.py tests/examples/optimization/eval_optimize_loop/test_comparator.py tests/examples/optimization/eval_optimize_loop/test_gate.py -q`
+
+Expected: fails because normalization, comparison, and gate functions are absent.
+
+- [ ] **Step 3: Implement minimum behavior**
+
+```python
+if not baseline.passed and candidate.passed:
+    transition = "NEW_PASS"
+elif baseline.passed and not candidate.passed:
+    transition = "REGRESSION"
+elif candidate.aggregate_score > baseline.aggregate_score + epsilon:
+    transition = "IMPROVED"
+elif candidate.aggregate_score < baseline.aggregate_score - epsilon:
+    transition = "DEGRADED"
+else:
+    transition = "UNCHANGED"
+```
+
+Gate acceptance requires validation score improvement, zero new hard fails, zero validation regressions, and no critical regression. Report JSON uses `model_dump(mode="json")`; Markdown lists selected candidate and every gate reason.
+
+- [ ] **Step 4: Verify GREEN and commit**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_normalization.py tests/examples/optimization/eval_optimize_loop/test_comparator.py tests/examples/optimization/eval_optimize_loop/test_gate.py tests/examples/optimization/eval_optimize_loop/test_report_schema.py -q`
+
+Commit: `git add examples/optimization/eval_optimize_loop/pipeline tests/examples/optimization/eval_optimize_loop && git commit -m "feat: add issue 91 comparison gate reports"`
+
+### Task 3: Phase 2 deterministic fake sources and six public cases
+
+**Files:**
+- Create: agent prompts/agent, fake package, both evalset JSON files, candidate fixture
+- Test: `test_fake_pipeline.py`
+
+**Interfaces:**
+- `FakeSupportAgent.call_agent(query: str) -> str` rereads registered prompt files and returns deterministic JSON.
+- `FixtureOptimizerBackend.load_candidates() -> list[CandidateRecord]` returns only prompt maps, source metadata, and zero cost.
+
+- [ ] **Step 1: Write failing fixture tests**
+
+```python
+@pytest.mark.asyncio
+async def test_fake_agent_needs_no_key(monkeypatch, fake_agent) -> None:
+    monkeypatch.delenv("TRPC_AGENT_API_KEY", raising=False)
+    assert json.loads(await fake_agent.call_agent("查询订单 A100"))["tool"] == "lookup_order"
+
+def test_fixture_candidates_are_unscored() -> None:
+    candidates = FixtureOptimizerBackend(CANDIDATES_PATH).load_candidates()
+    assert [c.candidate_id for c in candidates] == ["candidate_general_fix", "candidate_noop", "candidate_overfit"]
+    assert all(c.generation_cost_usd == 0 for c in candidates)
+```
+
+- [ ] **Step 2: Verify RED**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_fake_pipeline.py -q`
+
+Expected: fails due to missing fake components.
+
+- [ ] **Step 3: Implement deterministic fixture matrix**
+
+Create train case IDs `train_json_format`, `train_tool_argument`, `train_missing_knowledge`; validation IDs `val_json_generalization`, `val_refund_critical`, `val_stable_faq`. The fake rubric returns only 0, 0.25, 0.5, 0.75, or 1.0. `candidate_general_fix` generalizes; `candidate_noop` is unchanged; `candidate_overfit` breaks `val_refund_critical`.
+
+- [ ] **Step 4: Verify GREEN and commit**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_fake_pipeline.py -q`
+
+Commit: `git add examples/optimization/eval_optimize_loop/{agent,fake,train.evalset.json,val.evalset.json} tests/examples/optimization/eval_optimize_loop && git commit -m "feat: add issue 91 deterministic fake fixtures"`
+
+### Task 4: Phase 2 evaluator-first execution and prompt safety
+
+**Files:**
+- Create: `pipeline/evaluator.py`, `pipeline/prompt_sandbox.py`
+- Test: `test_prompt_sandbox.py`, `test_fake_pipeline.py`
+
+**Interfaces:**
+- `evaluate_split(...) -> SplitReport` invokes `AgentEvaluator.evaluate_eval_set()` then normalizes its returned `eval_results_by_eval_id`.
+- `PromptSandbox(target_prompt, candidate_prompts)` captures baseline, applies candidate prompts, and verifies restored baseline on all exits.
+
+- [ ] **Step 1: Write failing safety tests**
+
+```python
+@pytest.mark.asyncio
+async def test_sandbox_restores_after_evaluation_exception(target_prompt) -> None:
+    baseline = await target_prompt.read_all()
+    with pytest.raises(RuntimeError):
+        async with PromptSandbox(target_prompt, {"system_prompt": "changed", "router_prompt": "changed"}):
+            raise RuntimeError("evaluation failed")
+    assert await target_prompt.read_all() == baseline
+
+@pytest.mark.asyncio
+async def test_fake_pipeline_uses_agent_evaluator(monkeypatch, pipeline) -> None:
+    called = False
+    async def tracked(*args, **kwargs):
+        nonlocal called
+        called = True
+        return await original(*args, **kwargs)
+    original = AgentEvaluator.evaluate_eval_set
+    monkeypatch.setattr(AgentEvaluator, "evaluate_eval_set", tracked)
+    await pipeline.evaluate_baseline()
+    assert called is True
+```
+
+- [ ] **Step 2: Verify RED**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_prompt_sandbox.py tests/examples/optimization/eval_optimize_loop/test_fake_pipeline.py -q`
+
+Expected: fails due to missing evaluator/sandbox behavior.
+
+- [ ] **Step 3: Implement and verify restoration**
+
+```python
+async def __aexit__(self, exc_type, exc, tb) -> None:
+    await self._target_prompt.write_all(self._baseline)
+    if await self._target_prompt.read_all() != self._baseline:
+        raise SourceRestoreError("baseline prompt digest verification failed")
+```
+
+- [ ] **Step 4: Verify GREEN and commit**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_prompt_sandbox.py tests/examples/optimization/eval_optimize_loop/test_fake_pipeline.py -q`
+
+Commit: `git add examples/optimization/eval_optimize_loop/pipeline tests/examples/optimization/eval_optimize_loop && git commit -m "feat: evaluate fake candidates safely"`
+
+### Task 5: Phase 2 CLI, runtime ignore, canonical sample evidence, and final verification
+
+**Files:**
+- Create: `run_pipeline.py`, `README.md`, `DESIGN.md`, `sample_output/optimization_report.{json,md}`
+- Modify: `.gitignore`
+- Test: complete example test directory
+
+- [ ] **Step 1: Write the failing end-to-end test**
+
+```python
+@pytest.mark.asyncio
+async def test_fake_closed_loop_is_deterministic_and_restores_prompt(tmp_path, target_prompt) -> None:
+    baseline = await target_prompt.read_all()
+    report = await run_fake_pipeline(output_dir=tmp_path)
+    assert report.selected_candidate_id == "candidate_general_fix"
+    assert {c.candidate_id: c.gate.accepted for c in report.candidates} == {
+        "candidate_general_fix": True, "candidate_noop": False, "candidate_overfit": False,
+    }
+    assert await target_prompt.read_all() == baseline
+```
+
+- [ ] **Step 2: Verify RED**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop/test_fake_pipeline.py::test_fake_closed_loop_is_deterministic_and_restores_prompt -q`
+
+Expected: fails because the fake orchestration is absent.
+
+- [ ] **Step 3: Implement orchestration and ignore only generated runtime output**
+
+Append this exact rule to `.gitignore`:
+
+```gitignore
+# Runtime artefacts for Issue #91; stable reviewed reports live in sample_output/.
+examples/optimization/eval_optimize_loop/runs/
+```
+
+`run_fake_pipeline` must run baseline train+validation, then each candidate train+validation inside the sandbox, compare, gate, select the best accepted candidate, and write JSON/Markdown reports. CLI supports `--mode fake` and `--output-dir`, returns zero for either normal accept or reject, and prints report paths.
+
+- [ ] **Step 4: Generate and inspect stable evidence**
+
+Run: `python examples/optimization/eval_optimize_loop/run_pipeline.py --mode fake --output-dir examples/optimization/eval_optimize_loop/sample_output`
+
+Expected: exit 0; selected candidate `candidate_general_fix`; rejected noop and overfit; no key required.
+
+- [ ] **Step 5: Complete verification and commit**
+
+Run: `python -m pytest tests/examples/optimization/eval_optimize_loop -q; git diff --check; git check-ignore -v examples/optimization/eval_optimize_loop/runs/probe.txt`
+
+Expected: tests pass, no whitespace errors, and the ignore command identifies the new rule.
+
+Commit: `git add .gitignore examples/optimization/eval_optimize_loop tests/examples/optimization/eval_optimize_loop && git commit -m "feat: add issue 91 no-key fake optimization loop"`
+
+## Final Verification Checklist
+
+- [ ] Run fake CLI twice to fresh directories and verify deterministic report content after excluding run timestamp fields.
+- [ ] Confirm all baseline/candidate evaluations crossed `AgentEvaluator.evaluate_eval_set()`.
+- [ ] Confirm baseline prompt contents after normal completion and an injected evaluation failure.
+- [ ] Confirm no `runs/` content is staged and `sample_output/` is reviewed and intentionally tracked.
+- [ ] Do not begin live-mode implementation until this complete test suite is green and reviewed.

examples/optimization/eval_optimize_loop/DESIGN.md

@@ -0,0 +1,9 @@
+# 方案设计说明
+
+本示例在公开的 `AgentEvaluator` 与 `AgentOptimizer` 之上增加独立编排层，形成“基线评测、失败归因、候选生成、独立回归、Gate 决策、审计落盘”的完整闭环，不修改 SDK 公共 API。失败归因采用规则优先策略：先根据执行异常、实际与期望工具选择、参数差异、工具响应、JSON 格式和判定原因分类；trace 模式优先使用真实中间轨迹，期望会话只作为参照。只有规则不能解释时才保留可扩展的 judge 入口；每个结论都带有类型、证据和置信度。
+
+接受策略由 Gate 独立掌握，优化器的 round 状态或摘要不能直接决定通过。候选必须具有完整提示词映射，并在 `PromptSandbox` 中重新跑完整训练集和验证集。Gate 会拒绝评测不完整、新增 hard fail、关键回归、超出允许范围的回归、指标低于下限、验证集无增益、成本或耗时超预算以及违反平局策略的候选；未知成本会记录为警告。赢家按新增 hard fail、关键回归数、验证集通过率和得分、成本、耗时与稳定候选 ID 排序。
+
+防过拟合依靠训练集与验证集的独立全量回归，而不是读取优化器轮次指标。若训练提升而验证下降，或泛化差距超过配置阈值，Gate 会直接拒绝。优化期间固定 `update_source=False`，候选只作用于临时目标提示词目录；默认不回写，显式允许回写时还要校验基线和候选摘要。
+
+每次运行都会生成可由 Pydantic 读取的 `OptimizationReport`，并保存脱敏后的输入、环境、原始、归一化、候选与 Gate 审计产物。fake 和 trace 路径不需要 API Key、网络或 LLM judge，且输出稳定可复现；live 模式缺少必要环境变量会在启动优化器前安全退出。

examples/optimization/eval_optimize_loop/README.md

@@ -0,0 +1,27 @@
+# Issue #91: Auditable evaluation and prompt optimization loop
+
+This example demonstrates an evaluation-first prompt-optimization loop around the public `AgentOptimizer` API.  It keeps the checked-in fake and trace paths deterministic and credential-free, while live mode is an opt-in adapter with independent acceptance checks.
+
+## Run
+
+From the repository root:
+
+```text
+python examples/optimization/eval_optimize_loop/run_pipeline.py --mode fake --output-dir examples/optimization/eval_optimize_loop/sample_output
+python examples/optimization/eval_optimize_loop/run_pipeline.py --mode trace --output-dir .tmp/issue-91-trace
+python examples/optimization/eval_optimize_loop/run_pipeline.py --mode live --output-dir .tmp/issue-91-live
+```
+
+`fake` uses checked-in candidates and a local rubric; it needs no key and makes no network calls. `trace` evaluates the checked-in recorded conversation only. `live` validates `TRPC_AGENT_API_KEY`, `TRPC_AGENT_BASE_URL`, and `TRPC_AGENT_MODEL_NAME` before constructing an optimizer. It exits with code `2` if any are absent, and does not start network work in that case.
+
+Live mode writes candidates to a temporary `TargetPrompt` workspace, passes only the SDK `evaluate`/`optimize` configuration to `AgentOptimizer.optimize`, and always uses `update_source=False`. Source write-back is disabled by default; it is considered only after a Gate-approved winner and guarded by prompt-digest checks.
+
+## Outputs
+
+Each run writes `optimization_report.json`, `optimization_report.md`, and secret-free audit evidence:
+
+- `input.snapshot.json` and `environment.snapshot.json`
+- `audit/raw_reports.json` and `audit/normalized_reports.json`
+- `audit/candidate_reports.json` and `audit/gate_decisions.json`
+
+The JSON report is Pydantic-readable as `OptimizationReport` and is the source of truth for the selected candidate.

examples/optimization/eval_optimize_loop/__init__.py

@@ -0,0 +1 @@
+"""Deterministic no-key evaluation and optimization loop example."""

examples/optimization/eval_optimize_loop/agent/__init__.py

@@ -0,0 +1 @@
+"""Prompt files for the deterministic fake support agent."""

examples/optimization/eval_optimize_loop/agent/agent.py

@@ -0,0 +1,5 @@
+from __future__ import annotations
+
+from ..fake.fake_agent import FakeSupportAgent
+
+__all__ = ["FakeSupportAgent"]

examples/optimization/eval_optimize_loop/agent/prompts/router.md

@@ -0,0 +1 @@
+ROUTER_V1

examples/optimization/eval_optimize_loop/agent/prompts/system.md

@@ -0,0 +1 @@
+BASELINE

examples/optimization/eval_optimize_loop/fake/__init__.py

@@ -0,0 +1 @@
+"""No-key deterministic dependencies for the fake pipeline."""

examples/optimization/eval_optimize_loop/fake/candidates.json

@@ -0,0 +1,5 @@
+[
+  {"candidate_id": "candidate_general_fix", "prompts": {"system_prompt": "GENERAL_FIX\n", "router_prompt": "ROUTER_V1\n"}, "round_index": 1},
+  {"candidate_id": "candidate_noop", "prompts": {"system_prompt": "BASELINE\n", "router_prompt": "ROUTER_V1\n"}, "round_index": 2},
+  {"candidate_id": "candidate_overfit", "prompts": {"system_prompt": "OVERFIT\n", "router_prompt": "ROUTER_V1\n"}, "round_index": 3}
+]