[skill-optimizer] Daily Skill Optimizer Improvements - 2026-08-15

[github-actions]

Summary

• Run mode: dry-run
• Status: ⚠️ (dry-run completed successfully, but benchmark suite was skipped)

Key Findings

1. Enable benchmark mode by provisioning OPENROUTER_API_KEY — dry-run cannot exercise the actual skill-optimizer suite, so no real quality signal on gh-aw's skill/workflow guidance is produced.
2. Pin/audit skill-optimizer's npm dependencies — npm ci reports 19 vulnerabilities (11 high) and 3 packages with unreviewed install scripts, which is a supply-chain risk for a tool that runs against this repo's workflow content.
3. Add a fallback benchmark trigger or scheduled secret check — the workflow silently skips real evaluation when the key is absent, with no alert, so regressions in skill/workflow guidance quality could go unnoticed indefinitely.

Recommendations

Evidence from Artifact

• summary.json: {"repository":"github/gh-aw","run_mode":"dry-run","run_status":0,...} — confirms only a dry-run executed.
• run.log: dry-run: Docker available but OPENROUTER_API_KEY not set; skipping suite execution — the actual benchmark suite never ran because the secret is missing.
• npm-ci.log: 19 vulnerabilities (2 low, 6 moderate, 11 high) and 3 packages have install scripts not yet covered by allowScripts (esbuild, koffi, protobufjs) — supply-chain and script-execution risk in the skill-optimizer tool dependency tree.
• No suite-results/ directory present, confirming no benchmark data was produced this run.

1. Provision OPENROUTER_API_KEY as a repository/organization secret for the skill-optimizer workflow so it runs in benchmark mode instead of dry-run. Expected impact: enables real evaluation of skill/workflow guidance quality against actual model responses, turning this daily job into an actionable quality gate instead of a no-op check.
2. Run npm audit fix (and review npm approve-scripts for esbuild, koffi, protobufjs) in the skill-optimizer dependency setup, or pin to audited versions. Expected impact: reduces exposure to 11 high-severity vulnerabilities and removes silent install-script execution risk in the CI job that clones and builds this tool.
3. Add an explicit warning/notification step (or fail the job) when run_mode resolves to dry-run due to a missing secret, rather than only logging to run.log. Expected impact: surfaces configuration gaps immediately (e.g., via a workflow annotation or issue comment) instead of relying on someone reading logs, preventing weeks of skipped benchmark runs from going unnoticed.

Generated by ⚡ Daily Skill Optimizer Improvements · auto · 10 AIC · ⌖ 2.04 AIC · ⊞ 6.4K · ◷

• expires on Aug 21, 2026, 7:00 PM UTC-08:00