Skip to content

Add lifecycle tracing spans to instance delete path#299

Merged
sjmiller609 merged 1 commit into
mainfrom
hypeship/instrument-delete-path
Jul 8, 2026
Merged

Add lifecycle tracing spans to instance delete path#299
sjmiller609 merged 1 commit into
mainfrom
hypeship/instrument-delete-path

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

DELETE /instances/{id} shows a heavy latency tail in production (p50 ~300ms, p95 ~3.4s), but its trace spans are opaque: the delete path in lib/instances/delete.go runs the same teardown sequence as stop (graceful guest shutdown, hypervisor kill, network release, data removal) without any of the lifecycle step spans that stop.go already emits, so the slow phase is invisible in traces.

This wraps each blocking phase of deleteInstance in the existing startLifecycleSpan/startLifecycleStep helpers, mirroring stop.go:

  • top-level instances.delete span with hypervisor and instance_state attributes
  • cancel_compression_job
  • graceful_guest_shutdown (marked as error on timeout, same as stop)
  • force_kill_hypervisor
  • release_network
  • delete_instance_data

No behavior changes — instrumentation only.

Test plan

  • go build ./lib/instances/ and go vet ./lib/instances/
  • go test ./lib/instances/ -run TestDeleteInstance passes
  • Full lib/instances suite not run to completion locally: Firecracker VM lifecycle tests require real hypervisor binaries/network and fail identically on unmodified main in this environment
  • After deploy, verify delete step spans appear under /instances/{id} DELETE traces

🤖 Generated with Claude Code


Note

Low Risk
Observability-only changes around existing delete flow; no logic or API changes.

Overview
Adds OpenTelemetry lifecycle instrumentation to deleteInstance so DELETE latency can be broken down in traces, matching the pattern already used on stop/create.

A top-level instances.delete span records instance_id, hypervisor, and instance_state via enrichInstancesTrace, with the function signature updated to named retErr so finishInstancesSpan runs on exit. Child step spans wrap the blocking teardown phases: cancel_compression_job, graceful_guest_shutdown (failed attempts end the step with errGracefulShutdownFailed, same as stop), force_kill_hypervisor, release_network, and delete_instance_data.

Teardown logic, timeouts, and error handling are unchanged—only tracing hooks around existing calls.

Reviewed by Cursor Bugbot for commit 38dea8a. Bugbot is set up for automated code reviews on this repo. Configure here.

The delete path runs the same teardown sequence as stop (graceful guest
shutdown, hypervisor kill, network release, data removal) but emitted no
step spans, so slow deletes showed up in traces as one opaque server
span. Wrap each blocking phase in the existing lifecycle step helpers,
mirroring the instrumentation already present in stop.go.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@sjmiller609 sjmiller609 marked this pull request as ready for review July 8, 2026 13:32
@sjmiller609 sjmiller609 requested a review from yummybomb July 8, 2026 13:33
@sjmiller609 sjmiller609 merged commit 3c0861f into main Jul 8, 2026
11 checks passed
@sjmiller609 sjmiller609 deleted the hypeship/instrument-delete-path branch July 8, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants