feat(agent-arena): add the continuous online evaluation loop

[sishuo-yang]

Summary

• add the runnable Agent Arena continuous-evaluation workflow: Boolean thumbs feedback, human-review queue, provenance-aware golden-dataset promotion, paired experiments, and gated online evaluators
• add versioned business policies and generalizable operational-policy evaluation instead of a one-question customer-count evaluator
• make seeded product failures deterministic while keeping the candidate fix general across active-product, revenue, and conversion questions
• harden seeded-failure preflight, evaluator prompts, score/version resolution, and online rollout gates
• preserve per-question numeric precision so semantically equivalent q017 results remain correct
• update the dashboard and workshop assets for the current Agent Arena experience and captured results

Verification

• 313 Python tests passed
• Python compile check passed
• 5 dashboard tests passed
• dashboard production build passed
• final independent review found no Critical, Important, or Minor issues

Live Qwen verification

On the reused 22-question verification project:

• baseline: 16/22
• candidate: 19/22
• all three seeded policy failures changed from incorrect/FAIL to correct/PASS
• q017 remained correct
• no 1-to-0 regressions
• catalog-boundary questions remained NOT_APPLICABLE
• adversarial question and SQL-comment verdict injection remained NOT_APPLICABLE
• online rules were enabled only after experiment calibration and readiness checks

A clean project starts with 18 experiment questions from the 20-question source set because q019 and q020 are held out; promoting three reviewed failures produces 21 experiment items. The reused project contained one older unrelated approved item, hence 22.

Verification boundary

The in-app browser was unavailable for the final literal Langfuse annotation click. That UI-only action was not claimed and was not substituted with an API call. The live feedback/evaluator pipeline, promotion logic, paired experiments, readiness gates, and online replay were exercised directly.