[experiments] Daily Experiment Report — 2026-08-17

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-08-17

44 workflows currently declare active A/B experiments in github/gh-aw, totaling 47 experiment definitions. Of these, 23 have all variants at or above min_samples (ready for outcome-metric analysis) and 24 still need more data. 12 experiments show a statistically significant variant-count imbalance (p<0.05) in the assignment-balance check — this flags a possible randomization/weighting issue, not necessarily a winning variant, since outcome-metric data (success rate, duration) is not tracked in the git branch state and would require per-run artifact correlation beyond this report's scope.

⚡ Quick Stats

Metric	Value
Active workflows with experiments	44
Total experiment definitions	47
Ready for analysis (all variants ≥ min_samples)	23
Still collecting data (EXTEND)	24
Balance-significant imbalance (p<0.05)	12
Workflows with a tracking issue configured	20

Note on chart generation: the sandbox's Python environment had no working matplotlib/numpy installation available in this run (venv was broken and system Python is externally managed), so no bar charts could be rendered this cycle. All statistics below come directly from gh aw experiments analyze output, computed locally against fetched experiments/* git branches (the gh CLI in this sandbox is unauthenticated and does not support the --repo flag used by the tool's remote-fetch path, so branches were fetched directly via git fetch and analysed against local refs).

---

📋 Full Experiment Table (all 47 experiments across 44 workflows)

Workflow                   Experiment             n(total)    p-value  Recommendation      Variant progress toward min_samples
----------------------------------------------------------------------------------------------------------------------------------------------------------------
architectureguardian       sub_agent_strategy           38     0.5239  EXTEND             single_agent=███████░ 17/20 (85%); sub_agents=████████ 21/20 (100%)
auditworkflows             audit_decomposition          43      0.166  EXTEND             phased_sub_agents=█░░░░░░░ 26/264 (10%); single_agent=█░░░░░░░ 17/264 (6%)
blogauditor                prompt_style                  5     0.6589  EXTEND             concise=█░░░░░░░ 3/20 (15%); detailed=█░░░░░░░ 2/20 (10%)
breakingchangechecker      tone_variant                 29     0.3561  EXTEND             neutral=███████░ 17/20 (85%); urgent=█████░░░ 12/20 (60%)
dailyarchitecturediagram   detail_level                 12     0.5711  EXTEND             brief=███░░░░░ 7/20 (35%); comprehensive=██░░░░░░ 5/20 (25%)
dailycachestrategyanalyzer model_size                   72  1.308e-14  EXTEND             agent=░░░░░░░░ 1/20 (5%); gpt-5-codex=███░░░░░ 8/20 (40%); gpt-5-mini=█░░░░░░░ 2/20 (10%); gpt-5.4=████████ 31/20 (100%); gpt-5.4-mini=████████ 29/20 (100%); small-agent=░░░░░░░░ 1/20 (5%)
dailycavemanoptimizer      model_size                   74   2.95e-12  EXTEND             agent=░░░░░░░░ 1/20 (5%); claude-haiku-4.5=████████ 41/20 (100%); claude-sonnet-4.6=████████ 30/20 (100%); small-agent=█░░░░░░░ 2/20 (10%)
dailycodemetrics           output_format                92   1.35e-08  EXTEND             executive_summary=████████ 41/20 (100%); full_detail=████████ 49/20 (100%); ste=█░░░░░░░ 2/20 (10%)
dailycompilerquality       output_format                93  1.652e-08  EXTEND             concise=████████ 46/20 (100%); detailed=████████ 45/20 (100%); ste=█░░░░░░░ 2/20 (10%)
dailydochealer             model_size                   74  8.975e-12  EXTEND             agent=█░░░░░░░ 2/20 (10%); claude-haiku-4.5=████████ 37/20 (100%); claude-sonnet-4.6=████████ 34/20 (100%); small-agent=░░░░░░░░ 1/20 (5%)
dailydocupdater            model_size                   74  4.385e-12  EXTEND             agent=░░░░░░░░ 1/20 (5%); claude-haiku-4.5=████████ 31/20 (100%); claude-sonnet-4.6=████████ 40/20 (100%); small-agent=█░░░░░░░ 2/20 (10%)
dailyfact                  reasoning_depth              67    0.06341  EXTEND             multi_candidate=███████░ 26/30 (87%); single_pass=████████ 41/30 (100%)
dailyfunctionnamer         model_size                    3          0  EXTEND             (no variant data)
dailyissuesreport          output_format               104  1.365e-08  EXTEND             collapsible=████████ 47/20 (100%); inline=████████ 53/20 (100%); ste=██░░░░░░ 4/20 (20%)
dailysemgrepscan           semgrep_output_format        77    0.02318  EXTEND             bullet_list=███████░ 28/30 (93%); prose=███████░ 27/30 (90%); structured_sections=██████░░ 22/30 (73%)
dailysubagentoptimizer     timeout_setting               4     0.7814  EXTEND             default=█░░░░░░░ 2/20 (10%); relaxed=░░░░░░░░ 1/20 (5%); tight=░░░░░░░░ 1/20 (5%)
dataflowprdiscussiondataset caveman_mode                 10     0.5346  EXTEND             no=███░░░░░ 4/10 (40%); yes=█████░░░ 6/10 (60%)
deepreport                 output_format                77  4.869e-05  EXTEND             annotated_brief=████████ 19/20 (95%); executive_brief=████████ 29/20 (100%); full_briefing=████████ 27/20 (100%); ste=█░░░░░░░ 2/20 (10%)
dependabotcampaign         summary_detail               12          1  EXTEND             brief=██░░░░░░ 6/20 (30%); detailed=██░░░░░░ 6/20 (30%)
dependabotgochecker        prompt_style                 34     0.9591  EXTEND             concise=████░░░░ 11/20 (55%); detailed=█████░░░ 12/20 (60%); step_by_step=████░░░░ 11/20 (55%)
issuearborist              prompt_style                 39     0.8472  EXTEND             concise=████████ 20/20 (100%); detailed=████████ 19/20 (95%)
plan                       reasoning_depth               7     0.3694  EXTEND             baseline=██░░░░░░ 4/20 (20%); deep=█░░░░░░░ 2/20 (10%); shallow=░░░░░░░░ 1/20 (5%)
smokecopilotsubagents      sub_agent_strategy           24     0.6542  EXTEND             delegated_sequential=██░░░░░░ 9/30 (30%); inline_strict=██░░░░░░ 6/30 (20%); single_agent_control=██░░░░░░ 9/30 (30%)
weeklyblogpostwriter       prefetch_strategy            10     0.5346  EXTEND             eager=██░░░░░░ 4/20 (20%); lazy=██░░░░░░ 6/20 (30%)
agentperformanceanalyzer   prompt_compression           88       0.53  READY_FOR_ANALYSIS caveman=████████ 47/20 (100%); verbose=████████ 41/20 (100%)
agentpersonaexplorer       sub_agent_strategy           88     0.1313  READY_FOR_ANALYSIS batch=████████ 51/20 (100%); per_scenario=████████ 37/20 (100%)
awfailureinvestigator      tone_variant                307      0.224  READY_FOR_ANALYSIS assertive=████████ 99/20 (100%); clinical=████████ 116/20 (100%); narrative=████████ 92/20 (100%)
cicoach                    prompt_style                 66     0.3264  READY_FOR_ANALYSIS concise=████████ 29/20 (100%); detailed=████████ 37/20 (100%)
copilotagentanalysis       output_format                69  0.0006757  READY_FOR_ANALYSIS prose=████████ 31/30 (100%); structured=████████ 38/30 (100%)
dailyagentrxtraceoptimizer sub_agent_strategy           55     0.6878  READY_FOR_ANALYSIS single_agent=████████ 26/20 (100%); sub_agents=████████ 29/20 (100%)
dailyastrostylelitemarkdownspellcheck prompt_style                106     0.4431  READY_FOR_ANALYSIS concise=████████ 57/20 (100%); detailed=████████ 49/20 (100%)
dailycommunityattribution  prompt_style                104          1  READY_FOR_ANALYSIS concise=████████ 52/20 (100%); verbose=████████ 52/20 (100%)
dailynews                  prompt_style                 82     0.3809  READY_FOR_ANALYSIS concise=████████ 45/20 (100%); detailed=████████ 37/20 (100%)
dailysafeoutputoptimizer   log_fetch_strategy           46     0.7612  READY_FOR_ANALYSIS eager=████████ 22/20 (100%); lazy=████████ 24/20 (100%)
dailysecurityredteam       reasoning_depth              97    0.03114  READY_FOR_ANALYSIS iterative=████████ 59/30 (100%); single_pass=████████ 38/30 (100%)
gpclean                    tool_verbosity               85     0.2307  READY_FOR_ANALYSIS full_bash=████████ 37/20 (100%); minimal_toolset=████████ 48/20 (100%)
smokeantigravity           sub_agent_strategy          208     0.4952  READY_FOR_ANALYSIS single_agent=████████ 99/20 (100%); sub_agents=████████ 109/20 (100%)
smokecopilot               caveman                     320          1  READY_FOR_ANALYSIS no=████████ 160/20 (100%); yes=████████ 160/20 (100%)
smokecopilot               subagent_model              300          1  READY_FOR_ANALYSIS large=████████ 150/20 (100%); small=████████ 150/20 (100%)
smokecopilotaoaiapikey     caveman                     164          1  READY_FOR_ANALYSIS no=████████ 82/20 (100%); yes=████████ 82/20 (100%)
smokecopilotaoaiapikey     subagent_model              164          1  READY_FOR_ANALYSIS large=████████ 82/20 (100%); small=████████ 82/20 (100%)
smokecopilotaoaientra      caveman                     137     0.8922  READY_FOR_ANALYSIS no=████████ 69/20 (100%); yes=████████ 68/20 (100%)
smokecopilotaoaientra      subagent_model              137     0.8922  READY_FOR_ANALYSIS large=████████ 69/20 (100%); small=████████ 68/20 (100%)
smokegemini                sub_agent_strategy          247     0.1779  READY_FOR_ANALYSIS single_agent=████████ 113/20 (100%); sub_agents=████████ 134/20 (100%)
smokepi                    sub_agent_decomposition       76     0.3619  READY_FOR_ANALYSIS parallel_sub_agents=████████ 42/20 (100%); single_agent=████████ 34/20 (100%)
testqualitysentinel        model_size                  886     0.4255  READY_FOR_ANALYSIS claude-haiku-4.5=████████ 455/20 (100%); claude-sonnet-4.6=████████ 431/20 (100%)
typist                     tone_style                   59     0.3654  READY_FOR_ANALYSIS conversational=████████ 33/20 (100%); formal=████████ 26/20 (100%)

Significance shown is the chi-square balance test (are variant counts as expected given declared weights?), not an outcome-metric significance test — outcome data (success/duration) was out of scope for this pass since it requires per-run artifact correlation not available via the git-branch state alone.

🔴 Experiments Flagged for Balance Review (p<0.05, assignment imbalance)

These experiments show variant counts that deviate significantly from their declared weights. This does not by itself indicate a winning variant — it flags that the random-assignment mechanism may be producing skewed splits (e.g., due to weighting config, small-n effects, or non-uniform variant lists including rarely-used values like agent/small-agent).

Workflow	Experiment	p-value	Variant counts
dailyfunctionnamer	model_size	0.00e+00
dailycachestrategyanalyzer	model_size	1.31e-14	agent=1, gpt-5-codex=8, gpt-5-mini=2, gpt-5.4=31, gpt-5.4-mini=29, small-agent=1
dailycavemanoptimizer	model_size	2.95e-12	agent=1, claude-haiku-4.5=41, claude-sonnet-4.6=30, small-agent=2
dailydocupdater	model_size	4.38e-12	agent=1, claude-haiku-4.5=31, claude-sonnet-4.6=40, small-agent=2
dailydochealer	model_size	8.97e-12	agent=2, claude-haiku-4.5=37, claude-sonnet-4.6=34, small-agent=1
dailycodemetrics	output_format	1.35e-08	executive_summary=41, full_detail=49, ste=2
dailyissuesreport	output_format	1.36e-08	collapsible=47, inline=53, ste=4
dailycompilerquality	output_format	1.65e-08	concise=46, detailed=45, ste=2
deepreport	output_format	4.87e-05	annotated_brief=19, executive_brief=29, full_briefing=27, ste=2
copilotagentanalysis	output_format	6.76e-04	prose=31, structured=38
dailysemgrepscan	semgrep_output_format	2.32e-02	bullet_list=28, prose=27, structured_sections=22
dailysecurityredteam	reasoning_depth	3.11e-02	iterative=59, single_pass=38

✅ Recommendation Summary

• READY_FOR_ANALYSIS (23): all variants have reached min_samples; outcome-metric statistics (success rate, duration significance) can now be computed if per-run outcome data is collected.
• EXTEND (24): at least one variant below min_samples — keep collecting data before drawing conclusions. Several multi-model experiments (e.g. dailycachestrategyanalyzer, dailycavemanoptimizer, dailydochealer, dailydocupdater) have near-empty legacy variants (agent, small-agent) that are unlikely to ever reach min_samples and may be candidates for pruning from the variant list.
• No ABANDON recommendations were computed in this pass since guardrail pass/fail requires per-run outcome data (success rate, empty-output rate) that was not correlated to variant assignment in this run — see the scope note below.

⚠️ Scope Limitation

Per-run outcome metrics (success rate, duration, guardrail pass/fail) require correlating the last 30 completed workflow runs with their variant assignments via recent_runs or downloaded artifacts, then running Welch's t-test / two-proportion z-tests as specified in the report procedure. This pass focused on validating that experiment tracking infrastructure is healthy (branches exist, state.json is populated, gh aw experiments analyze runs cleanly) and on the balance/readiness statistics available directly from that command. A follow-up pass with GitHub MCP run-log access budget should complete the outcome-metric correlation and significance testing described in Steps 2–4 of the procedure for the 23 workflows already READY_FOR_ANALYSIS.

Analysis window: gh aw experiments analyze against all experiments/* branches (44 workflows, 47 experiments) · Balance significance threshold: p < 0.05 (two-tailed, chi-square)
Run: 32008536167

Generated by 🧪 daily-experiment-report · auto · 113.1 AIC · ⌖ 12.6 AIC · ⊞ 16.7K · ◷

• expires on Aug 20, 2026, 12:23 AM UTC-08:00