Skip to content

agentHost: Classify cancelled model call telemetry - #329695

Open
Aaron Munger (amunger) wants to merge 3 commits into
mainfrom
amunger/model-call-cancelled-forwarding
Open

agentHost: Classify cancelled model call telemetry#329695
Aaron Munger (amunger) wants to merge 3 commits into
mainfrom
amunger/model-call-cancelled-forwarding

Conversation

@amunger

@amunger Aaron Munger (amunger) commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

What

Declare the SDK-forwarded copilotCli/model_call_cancelled event in the Agent Host telemetry forwarder.

The declaration includes:

  • bounded model-call cancellation properties and measurements;
  • SDK correlation enrichment;
  • restricted repository context supplied by the generic runtime pipeline;
  • privacy-safe model and request identifier semantics.

No Agent Host turn behavior or public protocol surface changes.

Why

Part of microsoft/vscode-internalbacklog#8247.

The paired runtime change, github/copilot-agent-runtime#15211, adds per-physical-attempt model_call_cancelled telemetry. The generic SDK callback already forwards runtime telemetry into Microsoft telemetry; this change supplies the required GDPR contract.

Existing agentHost.turnCompleted(result=cancelled) remains the cross-provider turn-cancellation signal. This event instead completes the Copilot runtime model-attempt success/error/cancelled denominator.

Data parity

Analytical signal Historical source Agent Host source Parity Validation
Cancelled physical model attempts Copilot Chat response.cancelled copilotCli/model_call_cancelled Runtime code + VS Code forwarding test; legacy also counted premature stream closure
Model/API/transport/request/timing/token/request-shape dimensions Local response payload SDK-forwarded runtime payload ✅ / ≈ Field classifications match the runtime contract
Turn cancellation response.cancelled was not a turn event agentHost.turnCompleted(result=cancelled) Separate signal Existing behavior unchanged
TTFT, bytes, Auto selection, suspend/resume Local measurements Not emitted Omitted rather than populated with placeholders

Query and row implications

  • Historical model-attempt queries should filter local rows to agent, subagent, background, and compaction request kinds.
  • New Agent Host queries should combine copilotCli/response.success, copilotCli/response.error, and copilotCli/model_call_cancelled.
  • agentHost.turnCompleted(result=cancelled) must remain a separate per-turn metric.
  • The runtime event is one row per actually dispatched cancelled physical model attempt.

Semantic shifts

Signal Shift Impact
Cancellation category Runtime exposes bounded cancellation sources rather than the legacy mixed cancellation bucket Queries can slice new rows more precisely; historical comparisons remain approximate
Source attribution Runtime uses normalized interaction and initiator fields Historical source values require translation
Request identifiers Non-UUID caller values are hashed Correlation remains privacy-safe but representation may differ

Targeted ESLint passes.

Declare the Copilot runtime model_call_cancelled event in the generic GitHub telemetry forwarder and cover forwarding with a focused unit test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings August 7, 2026 21:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the GDPR contract and forwarding coverage for cancelled Copilot CLI model-call attempts.

Changes:

  • Classifies cancellation properties, measurements, repository context, and correlation fields.
  • Adds a focused forwarding unit test.
Show a summary per file
File Description
copilotGitHubTelemetryForwarder.ts Declares cancellation telemetry classifications.
copilotGitHubTelemetryForwarder.test.ts Verifies cancellation event forwarding.

Review details

  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Balanced

Aaron Munger (amunger) and others added 2 commits August 7, 2026 15:14
The generic forwarder behavior is already covered; this PR only needs the GDPR declaration for the new runtime event.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the remaining whitespace-only diff after dropping the redundant test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@amunger
Aaron Munger (amunger) marked this pull request as ready for review August 7, 2026 22:18
@amunger
Aaron Munger (amunger) enabled auto-merge (squash) August 7, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants