Focus Agent Arena dashboard on workshop content

workshops/agent_arena/README.md

@@ -24,7 +24,7 @@ source of truth for benchmark results.
 
 ## What you get
 
-A single web app (`web/`, http://localhost:5174) with four tabs:
+A focused web app (`web/`, http://localhost:5174) with two tabs:
 
 - **Leaderboard** — the contest results: every model×prompt config ranked by
   accuracy, **cost-per-correct-answer** (the headline), latency, per-tier
@@ -34,14 +34,9 @@ A single web app (`web/`, http://localhost:5174) with four tabs:
   generated SQL → error → tokens → span timings). A **"View conversation"**
   button replays the agent's session **live from the LangFuse API**, and an
   **LLM-judge** column scores SQL quality.
-- **Countdown** — a live-event "stage" screen for demos: a presenter countdown
-  timer, the model families as contenders, and the current best accuracy per
-  family (read live from the latest run).
 - **Chat** — the production chatbot: ask a question against a picked
   model+prompt config and watch the SQL, cost, and latency; rate each answer
   👍/👎, which is written back to the trace as a **LangFuse score**.
-- **Architecture** — an animated React Flow diagram of the whole system.
-
 Plus the serving API (`serving/api.py`): **`POST /ask`** (run the agent live and
 return SQL/results/cost/latency, traced to LangFuse) and **`POST /feedback`**
 (attach a 👍/👎 score to a trace) — the same endpoints the Chat tab calls.
@@ -59,9 +54,6 @@ Golden dataset → benchmark harness → agent → OpenRouter
                                                   leaderboard API/UI
 ```
 
-The web app's **Architecture** tab renders the live component graph from
-`web/src/diagram/graph.js`.
-
 **The key idea — one service per job:**
 - **ClickHouse** is the application database: it holds the business data and
   executes generated and golden SQL.
@@ -185,7 +177,6 @@ Everything tunable lives in [`config.yaml`](config.yaml):
 - **`models`** — OpenRouter model id, display name, family, and per-1M-token input/output prices (used for cost).
 - **`prompts`** — the strategies `P1_zeroshot` … `P3_dialect`.
 - **`grid`** — which models × prompts to actually run (`["*"]` = all).
-- **`profiles`** — curated presets (Budget tier / Frontier / Everything) for the web UI's "Run benchmark" panel.
 - **`clickhouse.query_limits`** — the server-side caps enforced on agent SQL.
 
 Adding a model or prompt is a config edit, not a code change.

workshops/agent_arena/arena/config.py

@@ -81,12 +81,6 @@ class GridCfg(BaseModel):
     prompts: list[str]
 
 
-class ProfileCfg(BaseModel):
-    name: str
-    desc: str = ""
-    models: list[str] = []
-
-
 class Config(BaseModel):
     clickhouse: ClickHouseCfg
     openrouter: OpenRouterCfg
@@ -95,7 +89,6 @@ class Config(BaseModel):
     models: list[ModelCfg]
     prompts: list[PromptCfg]
     grid: GridCfg
-    profiles: list[ProfileCfg] = []
 
     def resolved_grid(self) -> tuple[list[str], list[str]]:
         all_models = [m.name for m in self.models]

workshops/agent_arena/config.yaml

@@ -92,16 +92,3 @@ prompts:
 grid:
   models: ["*"]
   prompts: ["*"]
-
-# Preset model selections for the web UI's "Run benchmark" panel. Each profile is
-# a one-click curated comparison; models reference the names above.
-profiles:
-  - name: "Open weight"
-    desc: "Open-weight contenders — DeepSeek, Qwen, GLM."
-    models: [deepseek-v4-flash, qwen3.7-flash, glm-4.7-flash]
-  - name: "Proprietary"
-    desc: "Closed-weight models — Claude, GPT, Gemini."
-    models: [claude-sonnet-5, gpt-5.6-luna, gemini-flash-lite]
-  - name: "Everything"
-    desc: "All six models — the open-vs-proprietary contest."
-    models: [claude-sonnet-5, gpt-5.6-luna, gemini-flash-lite, deepseek-v4-flash, qwen3.7-flash, glm-4.7-flash]

workshops/agent_arena/dashboard/app.py

@@ -151,106 +151,9 @@ def meta():
             "datasets_url": f"{_lf_base}/datasets" if _lf_base else None}
 
 
-# --- Run the benchmark from the UI (spawns the harness as a subprocess) -------
-# Local/demo use only: this lets the web app trigger `python -m eval.harness`.
-# The API process must have an OpenRouter API key in its environment.
-import subprocess
-import sys
-import threading
-from pathlib import Path
-
-_ROOT = Path(__file__).resolve().parent.parent
-_run_state = {"running": False, "run_id": None, "started_at": None,
-              "lines": [], "returncode": None}
-_run_lock = threading.Lock()
-
-
 @app.get("/api/grid-options")
 def grid_options():
-    """Prompts available to run (from config.yaml), with descriptions for hovers.
-    Models now come from the catalog endpoint /api/models."""
+    """Prompt descriptions used by leaderboard labels and tooltips."""
     return {
         "prompts": [{"name": p.name, "desc": p.desc or p.name} for p in _cfg.prompts],
     }
-
-
-@app.get("/api/profiles")
-def profiles():
-    """Preset model selections (config.yaml `profiles`) resolved to invocable ids."""
-    by_name = {m.name: m.id for m in _cfg.models}
-    return [{"name": p.name, "desc": p.desc,
-             "model_ids": [by_name[n] for n in p.models if n in by_name]}
-            for p in _cfg.profiles]
-
-
-# --- Model catalog ------------------------------------------------------------
-@app.get("/api/models")
-def models():
-    """Curated config.yaml models grouped by family, with LIVE OpenRouter prices
-    (falls back to config.yaml prices with a `degraded` reason if unreachable)."""
-    prices, degraded = {}, None
-    try:
-        from agents.llm import fetch_openrouter_prices
-        prices = fetch_openrouter_prices(_cfg.openrouter.base_url)
-    except Exception as e:  # noqa: BLE001
-        degraded = str(e)
-    fam_map = {}
-    for m in _cfg.models:
-        pin, pout = prices.get(m.id, (m.price_per_1m_in, m.price_per_1m_out))
-        fam_map.setdefault(m.family, []).append(
-            {"id": m.id, "name": m.name, "price_per_1m_in": pin,
-             "price_per_1m_out": pout, "in_default": True})
-    families = [{"family": fam, "models": ms} for fam, ms in fam_map.items()]
-    return {"provider": "openrouter", "region": "OpenRouter",
-            "families": families, "degraded": degraded}
-
-
-def _stream_harness(cmd: list[str]):
-    proc = subprocess.Popen(cmd, cwd=str(_ROOT), env=os.environ.copy(),
-                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
-                            text=True, bufsize=1)
-    for line in proc.stdout:
-        line = line.rstrip()
-        with _run_lock:
-            _run_state["lines"].append(line)
-            _run_state["lines"] = _run_state["lines"][-400:]  # keep tail
-    proc.wait()
-    with _run_lock:
-        _run_state["running"] = False
-        _run_state["returncode"] = proc.returncode
-        _results_cache.update(loaded_at=0.0, model=None)
-
-
-@app.post("/api/run")
-def start_run(body: dict):
-    with _run_lock:
-        if _run_state["running"]:
-            return {"ok": False, "error": "a run is already in progress",
-                    "run_id": _run_state["run_id"]}
-        run_id = body.get("run_id") or f"ui-{int(time.time())}"
-        cmd = [sys.executable, "-m", "eval.harness", "--run-id", run_id]
-        models = body.get("models") or []
-        if models and isinstance(models[0], dict):
-            # full specs from the live-catalog browser → temp models-file
-            import json
-            specs = [{"id": m["id"], "name": m["name"], "family": m.get("family", "other"),
-                      "price_per_1m_in": float(m.get("price_in") or 0),
-                      "price_per_1m_out": float(m.get("price_out") or 0)} for m in models]
-            (_ROOT / ".run").mkdir(exist_ok=True)
-            mf = _ROOT / ".run" / f"models-{run_id}.json"
-            mf.write_text(json.dumps(specs))
-            cmd += ["--models-file", str(mf)]
-        elif models:
-            cmd += ["--models", ",".join(models)]
-        if body.get("prompts"):
-            cmd += ["--prompts", ",".join(body["prompts"])]
-        _run_state.update(running=True, run_id=run_id, started_at=time.time(),
-                          lines=[f"$ {' '.join(cmd[2:])}"], returncode=None)
-    threading.Thread(target=_stream_harness, args=(cmd,), daemon=True).start()
-    return {"ok": True, "run_id": run_id}
-
-
-@app.get("/api/run/status")
-def run_status():
-    with _run_lock:
-        return dict(_run_state)

workshops/agent_arena/tests/test_dashboard_langfuse_results.py

@@ -209,6 +209,16 @@ def test_health_endpoint_does_not_load_langfuse(monkeypatch):
     assert dashboard.healthz() == {"status": "ok"}
 
 
+def test_dashboard_api_is_read_only_for_workshop_learners():
+    """The public dashboard must not expose its former demo run-builder API."""
+    paths = {route.path for route in dashboard.app.routes}
+    assert "/api/grid-options" in paths
+    assert "/api/models" not in paths
+    assert "/api/profiles" not in paths
+    assert "/api/run" not in paths
+    assert "/api/run/status" not in paths
+
+
 def test_reused_run_id_keeps_only_the_newest_question_result():
     """A repeated run/config/question must not inflate counts or preserve stale scores."""
     base = {"run_id": "r1", "config_id": "m__p", "model_name": "m",

workshops/agent_arena/web/README.md

@@ -1,16 +1,6 @@
 # AgentArena — Web UI
 
-A single [React](https://react.dev) ([Vite](https://vite.dev)) SPA with four tabs:
-
-- **Countdown** — a live-event "stage" screen for demos: a presenter countdown timer,
-  the model families as contenders, and the current best accuracy per family (read live
-  from the latest run).
-- **Architecture** — an animated [React Flow](https://reactflow.dev) (`@xyflow/react`)
-  data-flow diagram of the whole system. Packets travel along each edge to show
-  flow direction; edge colors group flow types, node colors show where each
-  component runs (legend top-right). Edge labels are short so they don't cover the
-  arrows — **hover an edge label** to see the full description. Nodes are
-  draggable; drag the canvas to pan, scroll to zoom.
+A focused [React](https://react.dev) ([Vite](https://vite.dev)) SPA with two tabs:
 - **Leaderboard** — the contest results, read from LangFuse Experiments through its
   Public API:
   run selector, winner cards, sortable leaderboard ranked by
@@ -28,19 +18,15 @@ A single [React](https://react.dev) ([Vite](https://vite.dev)) SPA with four tab
   config, see the generated SQL/results/cost/latency, and rate each answer
   👍/👎 — feedback is written back to the trace as a LangFuse score.
 
-Diagram layout is computed by **dagre** (layered left→right with crossing
-minimization); edges use orthogonal smoothstep routing.
-
 ## Run
 
 ```bash
 cd web
 npm install
-npm run dev            # → http://localhost:5174  (all four tabs)
+npm run dev            # → http://localhost:5174
 ```
 
-The **Leaderboard** and **Chat** tabs need the dashboard/serving APIs running
-(**Countdown** and **Architecture** need nothing):
+The **Leaderboard** and **Chat** tabs need the dashboard/serving APIs running:
 
 ```bash
 # from the repo root, in other shells:
@@ -54,15 +40,10 @@ The UI calls `http://localhost:8000` (dashboard) and `http://localhost:8100`
 CORS enabled for the SPA.
 
 ## Files
-- `src/App.jsx` — tab shell (Countdown / Architecture / Leaderboard / Chat).
+- `src/App.jsx` — tab shell (Leaderboard / Chat).
 - `src/ui.jsx` — shared presentational atoms (brand lock, icons, ClickHouse logomark).
 - `src/api.js` — dashboard/serving API bases + fetch helpers (`VITE_API_BASE`, `VITE_SERVING_BASE`).
-- `src/diagram/graph.js` — architecture model (nodes + edges, env colors). Edit here to change the diagram.
-- `src/diagram/layout.js` — dagre layered LR auto-layout.
-- `src/diagram/FlowDiagram.jsx` — React Flow canvas + legends + minimap.
-- `src/diagram/nodes/CardNode.jsx`, `src/diagram/edges/AnimatedFlowEdge.jsx` — renderers.
 - `src/leaderboard/Leaderboard.jsx` — leaderboard UI (fetches the dashboard API).
-- `src/countdown/Countdown.jsx` — the live-event stage screen.
 - `src/chat/Chat.jsx` — the production chat UI (calls the serving API's `/ask` + `/feedback`).
 
 > The leaderboard data comes from `dashboard/app.py` (a FastAPI adapter over the