[copilot-cli-research] Copilot CLI Deep Research - 2026-03-15 #21122

2026-03-15T21:25:06Z

github-actions[bot]
Bot Mar 15, 2026

📊 Executive Summary

Analysis Date: 2026-03-15 | Workflow Run: §23119478023
Scope: 172 total workflows, 80 using Copilot engine (46.5%)

This is the first comprehensive Copilot CLI deep research for this repository. The Copilot engine is the dominant choice (vs. 35 Claude, 13 Codex), but a number of high-value features remain almost completely unused. Key findings include: custom agent files are available for 10 personas but adopted by only 3 workflows; max-continuations (autopilot mode) is deployed in exactly 1 smoke test; rate-limiting is configured in just 3 workflows despite many being user-triggered; and checkout: false optimization is missing from ~50+ read-only workflows.

The repository demonstrates excellent baseline hygiene — features.copilot-requests: true at 50% adoption, structured safe-outputs in nearly all workflows, strict: true widely applied, and toolsets used thoughtfully. The opportunities below are about closing the gap between available and used.

🔴 Critical Findings

High Priority

Custom agent files going unused: 10 .agent.md files in .github/agents/ but only 3 workflows reference them via engine.agent — a largely untapped specialization mechanism
Rate limiting gaps: Only 3 of 172 workflows configure rate-limit: — user-triggered workflows are fully open to repeat invocations
checkout: false almost never used: Only 1 workflow opts out of checkout despite many being read-only (wasting ~5-15s startup per run)

Medium Priority

max-continuations underused: Only the smoke-copilot test uses autopilot mode — complex tasks like code-scanning-fixer, dead-code-remover, and repository-quality-improver would benefit from extended autonomous operation
features.copilot-requests: true at 50%: This is the recommended authentication path; the other 50% still rely on COPILOT_GITHUB_TOKEN secret

1️⃣ Current State Analysis

View Copilot CLI Capabilities Inventory

Copilot CLI Capabilities Inventory

Automatic (always applied by compiler):

--add-dir /tmp/gh-aw/ — Allows CLI to read generated prompts/scripts
--add-dir "\$\{GITHUB_WORKSPACE}" — Workspace access in AWF sandbox
--disable-builtin-mcps — External MCP servers only
--log-level all --log-dir (logsFolder) — Full session logging
--prompt "..." — Workflow prompt injection

Tool Permission Flags (compiler-generated from tools: config):

--allow-tool shell((cmd)) — Specific shell commands
--allow-all-tools — Bash wildcard triggers this
--allow-tool write — When edit: tool is present
--allow-tool web_fetch — When web-fetch: tool present
--allow-all-paths — Automatically added with edit: tool
--allow-tool github((op)) — Granular GitHub MCP tool permissions

Configurable via Frontmatter (workflow author controls):

engine.agent → --agent (id) — Custom agent persona from .github/agents/
engine.model → COPILOT_MODEL env var — LLM model selection
engine.version → Install script pin — Reproducible builds
engine.args → Custom CLI args injected before --prompt
engine.env → Custom environment variables for the Copilot process
engine.command → Override binary path entirely
max-continuations → --autopilot --max-autopilot-continues N — Extended autonomous operation
features.copilot-requests: true → S2STOKENS auth (no personal PAT needed)
features.mcp-gateway: true — Unified HTTP gateway for MCP servers
features.disable-xpia-prompt: true — Disable anti-injection system prompt
features.action-tag: "v0" — Pin compiled action references to a tag
engine.error_patterns — Custom error pattern detection in logs
sandbox.agent: awf + AWF config — Network firewall with domain allowlist
sandbox.agent.mounts — Mount host tools into AWF sandbox
plugins: — Plugin ecosystem integration (currently unused in all 172 workflows)

View Usage Statistics

Usage Statistics

Metric	Count
Total workflows	172
Copilot engine workflows	80 (46.5%)
Claude engine workflows	35 (20.3%)
Codex engine workflows	13 (7.6%)

Copilot-Specific Feature Adoption (of 80 Copilot workflows)

Feature	Used	%
`features.copilot-requests: true`	40	50%
`toolsets: [default]` (most common)	42+	52%+
AWF sandbox (`sandbox: agent: awf`)	7	8.75%
`engine.version` pinning	8	10%
`engine.model` override	7	8.75%
`engine.agent` custom agent	3	3.75%
`max-continuations`	1	1.25%
`features.mcp-gateway`	~2	~2.5%

Cross-All-Workflows Feature Adoption

Feature	Used	Total	%
`tracker-id:`	52	172	30.2%
`status-comment: true`	19	172	11%
`skip-if-match:`	11	172	6.4%
`rate-limit:`	3	172	1.7%
`checkout: false`	1	172	0.6%
`inlined-imports: true`	2	172	1.2%
`error_patterns:`	0	172	0%
`plugins:`	0	172	0%

Most Common Toolsets

Toolset	Count
`[default]`	42
`[default, discussions]`	9
`[default, actions]`	4
`[pull_requests, repos]`	4
`[all]`	3 ⚠️

2️⃣ Feature Usage Matrix

Feature Category	Available Features	Used	Not Used	Usage Rate
CLI Flags (auto)	`--add-dir`, `--disable-builtin-mcps`, `--log-level`, `--prompt`	All (compiler)	—	100%
CLI Flags (manual)	`--agent`, `--autopilot`, `--allow-tool`, `--allow-all-paths`	Most via tools config	`--agent` (3%), `--autopilot` (1%)	~60%
Engine Config	`id`, `version`, `model`, `agent`, `args`, `env`, `command`, `max-continuations`, `error_patterns`	`model` (8%), `version` (10%), `agent` (3%), `max-continuations` (1%)	`args`, `env`, `command`, `error_patterns`	~25%
MCP Servers	GitHub, Brave, Tavily, Serena, Playwright, custom	GitHub (most), Serena (17), Tavily (5), Playwright (several)	Custom via mcp-scripts (1 only)	~60%
Network Config	`network.allowed`, AWF sandbox, `network.blocked`, domain defaults	`allowed` (15+), AWF (7)	`blocked`, fine-grained per-domain	~25%
Sandbox Options	AWF, mounts, custom firewall config	AWF (7), mounts (1)	Version pinning, log-level	~10%
Features Flags	`copilot-requests`, `mcp-gateway`, `disable-xpia-prompt`, `action-tag`	`copilot-requests` (50%)	Others (~0%)	~50%
Auth Features	`copilot-requests`, `COPILOT_GITHUB_TOKEN` secret	Mixed	50% still use secret	50%

3️⃣ Missed Opportunities

🔴 View High Priority Opportunities

Opportunity 1: Custom Agent Files Are Almost Completely Unused

What: 10 agent persona files exist in .github/agents/ but only 3 Copilot workflows reference them via engine.agent.

Available agents: agentic-workflows.agent.md, ci-cleaner.agent.md, contribution-checker.agent.md, create-safe-output-type.agent.md, custom-engine-implementation.agent.md, developer.instructions.md, grumpy-reviewer.agent.md, interactive-agent-designer.agent.md, technical-doc-writer.agent.md, w3c-specification-writer.agent.md

Currently using: ci-cleaner (1 workflow), technical-doc-writer (2 workflows)

Unused agents: grumpy-reviewer, contribution-checker, agentic-workflows, developer.instructions, interactive-agent-designer, w3c-specification-writer, create-safe-output-type

Why It Matters: Custom agents give Copilot a specialized system prompt tuned for the task. Workflows doing code review (grumpy-reviewer), contribution checking (contribution-checker), or documentation (technical-doc-writer) get dramatically better results when the agent persona matches the task.

Where:

grumpy-reviewer.md → use agent: grumpy-reviewer
pr-nitpick-reviewer.md → use agent: grumpy-reviewer
contribution-check.md → use agent: contribution-checker
docs-noob-tester.md, blog-auditor.md, claude-code-user-docs-review.md → use agent: technical-doc-writer
workflow-generator.md, daily-workflow-updater.md → use agent: agentic-workflows

How to Implement:

engine:
  id: copilot
  agent: grumpy-reviewer   # references .github/agents/grumpy-reviewer.agent.md

Opportunity 2: Rate Limiting Missing from User-Triggered Workflows

What: Only 3 of 172 workflows configure rate-limit:. Many user-triggered workflows (slash commands, reactions, issue events) have no protection against repeated invocations.

Why It Matters: Without rate limiting, a single user can repeatedly trigger expensive agent runs. rate-limit: prevents abuse while keeping workflows accessible.

Where: Any workflow with on: slash_command:, on: reaction:, or user-triggered on: issues:/on: pull_request: that doesn't have rate-limit:.

Examples: archie.md, brave.md, dictation-prompt.md, mcp-inspector.md, plan.md

How to Implement:

rate-limit:
  max: 3
  window: 60          # 3 runs per user per hour
  ignored-roles: [admin, maintain, write]

🟡 View Medium Priority Opportunities

Opportunity 3: `max-continuations` Underused in Complex Analysis Workflows

What: The max-continuations field enables --autopilot --max-autopilot-continues N mode, allowing Copilot to run multiple continuation turns autonomously. Only smoke-copilot.md uses it (value: 2).

Why It Matters: Complex multi-step tasks — scanning all open issues, fixing multiple code problems, or performing deep repository analysis — benefit from Copilot being able to continue without human re-prompting.

Where:

code-scanning-fixer.md — Iterates through multiple security alerts
dead-code-remover.md — Potentially many files to process
repository-quality-improver.md — Cross-cutting improvements
daily-testify-uber-super-expert.md — Might need multiple passes
ci-doctor.md — Complex CI debugging

How to Implement:

engine:
  id: copilot
  max-continuations: 3   # Allow up to 3 autopilot continuation turns
timeout-minutes: 45      # Increase timeout proportionally

Opportunity 4: `checkout: false` for Read-Only Workflows

What: Only 1 of 172 workflows disables repository checkout. Many workflows only use GitHub API tools (no repo file access) and waste 5-15 seconds on a full git clone.

Why It Matters: Disabling checkout speeds up workflow startup and reduces token consumption. For daily/hourly workflows running hundreds of times, this compounds significantly.

Where: Any workflow that:

Only uses GitHub MCP tools (no edit:, no bash: with file operations)
Doesn't reference GITHUB_WORKSPACE content
Examples: daily-news.md, weekly-issue-summary.md, org-health-report.md, stale-repo-identifier.md, copilot-pr-merged-report.md

How to Implement:

checkout: false   # Skip repo clone — saves 5-15s per run

Opportunity 5: `features.copilot-requests: true` at Only 50% Adoption

What: The recommended authentication approach (copilot-requests: true) uses GitHub Actions token instead of a personal COPILOT_GITHUB_TOKEN secret. Currently only 40 of 80 Copilot workflows enable this.

Why It Matters: This is the preferred auth path — it avoids managing long-lived PATs, uses GitHub's built-in token, and is simpler to maintain.

Where: All Copilot workflows that don't have features.copilot-requests: true in their features: block.

How to Implement:

features:
  copilot-requests: true

Opportunity 6: `toolsets: [all]` Over-Permissioning

What: 3 workflows use toolsets: [all], which grants access to every GitHub MCP tool. This includes potentially sensitive operations beyond what the workflow needs.

Where: github-mcp-structural-analysis.md, github-mcp-tools-report.md, security-review.md

Why It Matters: Least-privilege principle — restrict GitHub MCP access to only what each workflow actually needs.

How to Implement: Replace with specific toolsets:

tools:
  github:
    toolsets: [repos, issues, pull_requests]   # Only what's needed

🟢 View Low Priority Opportunities

Opportunity 7: Custom `error_patterns` for Better Diagnostics

What: The engine.error_patterns field (defined in EngineConfig) is never used across all 172 workflows. It allows defining custom regex patterns that the log parser uses to detect and classify errors.

Why It Matters: Workflows with specific failure modes (API errors, domain-specific error messages) could get better error reporting and alerting when failures occur.

How to Implement:

engine:
  id: copilot
  error_patterns:
    - pattern: "RateLimitError: (.+)"
      level_group: 1
    - pattern: "Failed to fetch (.+): connection refused"
      level_group: 1

Opportunity 8: AWF Firewall Version Pinning for Reproducibility

What: Workflows using AWF sandbox (sandbox: agent: awf) don't pin the AWF version. The network.firewall.version and network.firewall.log-level options are unused.

Why It Matters: AWF version changes could affect network behavior. Pinning ensures reproducible security boundaries.

How to Implement:

network:
  firewall:
    version: "v0.15.0"     # Pin for reproducibility
    log-level: debug        # Enable for troubleshooting

Opportunity 9: `engine.env` for Debug Instrumentation

What: The engine.env field (passed directly to the Copilot process environment) is not used in any workflow. This could be useful for enabling debug modes or customizing behavior without modifying prompts.

Where: Workflows needing conditional debugging or custom API configurations.

How to Implement:

engine:
  id: copilot
  env:
    COPILOT_DEBUG_REQUESTS: "true"   # Enable request debugging

4️⃣ Specific Workflow Recommendations

View Workflow-Specific Recommendations

`grumpy-reviewer.md`

Current State: Uses Copilot engine for PR review
Recommended: Add engine.agent: grumpy-reviewer to use the purpose-built grumpy reviewer persona
Expected: More consistent, specialized review style matching the agent's training

`contribution-check.md`

Current State: Standard Copilot for PR contribution checking
Recommended: Add engine.agent: contribution-checker; add rate-limit: max: 2, window: 60
Expected: Better contribution checking with rate abuse protection

`code-scanning-fixer.md`

Current State: Complex multi-alert remediation workflow, timeout-minutes: 20
Recommended: Add max-continuations: 3; increase timeout-minutes: 60
Expected: Can process more alerts per run without user re-intervention

`workflow-generator.md`

Current State: Creates agentic workflows from descriptions
Recommended: Add engine.agent: agentic-workflows; add rate-limit: max: 2, window: 60
Expected: Leverages specialized workflow authoring knowledge; prevents spam

`daily-news.md`, `weekly-issue-summary.md`, `stale-repo-identifier.md`

Current State: Read-only workflows with full repository checkout
Recommended: Add checkout: false
Expected: 5-15 second startup time savings; reduced I/O costs per run

`security-review.md`

Current State: Uses toolsets: [all] and AWF sandbox
Recommended: Replace [all] with [repos, code_security]; consider rate-limit: max: 1, window: 120
Expected: Better least-privilege posture; prevents repeated expensive security scans

`docs-noob-tester.md`, `blog-auditor.md`

Current State: Documentation review workflows
Recommended: Add engine.agent: technical-doc-writer
Expected: Specialized documentation writing perspective during review

5️⃣ Trends & Insights

View Historical Trends

This is the first run of this research workflow. No previous baseline exists for trend comparison.

Baseline established (2026-03-15):

80 Copilot workflows (46.5% of 172 total)
50% copilot-requests adoption
3.75% custom agent file adoption
1.25% max-continuations adoption
0% error_patterns adoption

Future runs will compare against this baseline to track:

Adoption rate of copilot-requests: true
Growth in custom agent file usage
Rate-limit adoption for user-triggered workflows
checkout: false adoption for read-only workflows

6️⃣ Best Practice Guidelines

Based on this research, here are recommended best practices for Copilot workflows:

Always set features.copilot-requests: true — Use GitHub's built-in token auth instead of managing COPILOT_GITHUB_TOKEN secrets for simpler, more secure workflows.
Match agent files to workflow purpose — When a .github/agents/ file exists for your task type (code review, documentation, contribution checking, workflow authoring), use engine.agent: (name) for specialized prompting.
Rate-limit all user-triggered workflows — Any workflow triggered by slash commands, reactions, or direct issue/PR events should have rate-limit: max: 3, window: 60 to prevent abuse.
Use checkout: false for API-only workflows — If a workflow only uses GitHub MCP tools and doesn't access repository files, skip the checkout for faster startup.
Use specific GitHub toolsets — Prefer [repos, issues] over [default] or [all] when you know which tools the agent needs; avoid [all].
Consider max-continuations for multi-step tasks — Complex analysis or remediation workflows benefit from max-continuations: 2 or 3 with a proportionally increased timeout-minutes.

7️⃣ Action Items

Immediate Actions (this week):

Add rate-limit: to the top 5 highest-traffic user-triggered workflows (archie, brave, plan, dictation-prompt, mcp-inspector)
Enable features.copilot-requests: true in the remaining 40 Copilot workflows that lack it

Short-term (this month):

Wire up engine.agent for at least 5 more workflows matching existing agent files (grumpy-reviewer, contribution-checker, technical-doc-writer x3+)
Add checkout: false to 10+ read-only, API-only workflows
Replace toolsets: [all] with specific toolsets in the 3 workflows that use it

Long-term (this quarter):

Enable max-continuations: 2-3 for complex remediation workflows (code-scanning-fixer, dead-code-remover, repository-quality-improver)
Prototype engine.error_patterns in one high-value workflow to evaluate benefit
Evaluate the plugin ecosystem for applicable tools

View Supporting Evidence & Methodology

Research Methodology

Data Sources:

Inspected all .github/workflows/*.md frontmatter (172 files)
Read pkg/workflow/copilot_engine*.go (execution, tools, installation, core) for available feature inventory
Read pkg/workflow/engine.go for EngineConfig struct fields
Read .github/aw/github-agentic-workflows.md for configuration documentation
Read docs/src/content/docs/reference/engines.md for user-facing feature docs
Inventoried .github/agents/ for available custom agent files (10 files)

Analysis Approach:

Extracted EngineConfig struct to enumerate all configurable fields
Traced GetExecutionSteps() to identify which fields map to which CLI flags
Cross-referenced with actual workflow frontmatter via grep
Counted adoption rates for each feature
Identified patterns suggesting missed opportunities

Tools Used: grep, ls, head, Go source code analysis

References

Engine documentation: docs/src/content/docs/reference/engines.md
Copilot engine implementation: pkg/workflow/copilot_engine_execution.go
Workflow configuration reference: .github/aw/github-agentic-workflows.md
Available agent files: .github/agents/*.agent.md

References:

§23119478023

AI generated by Copilot CLI Deep Research Agent · history

expires on Mar 16, 2026, 9:25 PM UTC

2026-03-16T21:31:59Z

github-actions[bot]
Bot Mar 16, 2026
Author

This discussion has been marked as outdated by Copilot CLI Deep Research Agent.

A newer discussion is available at Discussion #21290.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[copilot-cli-research] Copilot CLI Deep Research - 2026-03-15 #21122

Uh oh!

{{title}}

Uh oh!

Copilot CLI Capabilities Inventory

Usage Statistics

Copilot-Specific Feature Adoption (of 80 Copilot workflows)

Cross-All-Workflows Feature Adoption

Most Common Toolsets

Opportunity 1: Custom Agent Files Are Almost Completely Unused

Opportunity 2: Rate Limiting Missing from User-Triggered Workflows

Opportunity 3: `max-continuations` Underused in Complex Analysis Workflows

Opportunity 4: `checkout: false` for Read-Only Workflows

Opportunity 5: `features.copilot-requests: true` at Only 50% Adoption

Opportunity 6: `toolsets: [all]` Over-Permissioning

Opportunity 7: Custom `error_patterns` for Better Diagnostics

Opportunity 8: AWF Firewall Version Pinning for Reproducibility

Opportunity 9: `engine.env` for Debug Instrumentation

`grumpy-reviewer.md`

`contribution-check.md`

`code-scanning-fixer.md`

`workflow-generator.md`

`daily-news.md`, `weekly-issue-summary.md`, `stale-repo-identifier.md`

`security-review.md`

`docs-noob-tester.md`, `blog-auditor.md`

Research Methodology

References

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

[copilot-cli-research] Copilot CLI Deep Research - 2026-03-15 #21122

Uh oh!

github-actions[bot] Bot Mar 15, 2026

📊 Executive Summary

🔴 Critical Findings

High Priority

Medium Priority

1️⃣ Current State Analysis

Copilot CLI Capabilities Inventory

Usage Statistics

Copilot-Specific Feature Adoption (of 80 Copilot workflows)

Cross-All-Workflows Feature Adoption

Most Common Toolsets

2️⃣ Feature Usage Matrix

3️⃣ Missed Opportunities

Opportunity 1: Custom Agent Files Are Almost Completely Unused

Opportunity 2: Rate Limiting Missing from User-Triggered Workflows

Opportunity 3: max-continuations Underused in Complex Analysis Workflows

Opportunity 4: checkout: false for Read-Only Workflows

Opportunity 5: features.copilot-requests: true at Only 50% Adoption

Opportunity 6: toolsets: [all] Over-Permissioning

Opportunity 7: Custom error_patterns for Better Diagnostics

Opportunity 8: AWF Firewall Version Pinning for Reproducibility

Opportunity 9: engine.env for Debug Instrumentation

4️⃣ Specific Workflow Recommendations

grumpy-reviewer.md

contribution-check.md

code-scanning-fixer.md

workflow-generator.md

daily-news.md, weekly-issue-summary.md, stale-repo-identifier.md

security-review.md

docs-noob-tester.md, blog-auditor.md

5️⃣ Trends & Insights

6️⃣ Best Practice Guidelines

7️⃣ Action Items

Research Methodology

References

Replies: 1 comment

Uh oh!

github-actions[bot] Bot Mar 16, 2026 Author

github-actions[bot]
Bot Mar 15, 2026

Opportunity 3: `max-continuations` Underused in Complex Analysis Workflows

Opportunity 4: `checkout: false` for Read-Only Workflows

Opportunity 5: `features.copilot-requests: true` at Only 50% Adoption

Opportunity 6: `toolsets: [all]` Over-Permissioning

Opportunity 7: Custom `error_patterns` for Better Diagnostics

Opportunity 9: `engine.env` for Debug Instrumentation

`grumpy-reviewer.md`

`contribution-check.md`

`code-scanning-fixer.md`

`workflow-generator.md`

`daily-news.md`, `weekly-issue-summary.md`, `stale-repo-identifier.md`

`security-review.md`

`docs-noob-tester.md`, `blog-auditor.md`

github-actions[bot]
Bot Mar 16, 2026
Author