Sync guest realtime clock after standby restore#301
Closed
yummybomb wants to merge 4 commits into
Closed
Conversation
The guest wall clock resumes from the snapshot's saved time after a restore, so every in-guest timestamp lags by the time spent in standby until corrected. Add a SyncClock guest-agent RPC that sets CLOCK_REALTIME to the host's current time, and call it from the restore path once the VM resumes. Instances whose snapshots predate the new agent binary fall back to setting the clock via exec. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
unix.ClockSettime does not exist on darwin, which broke test-darwin compiling the guest_agent package. The shipped agent binary is linux-only; non-linux builds get an error stub. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clock sync is non-fatal and runs on every restore, so an unresponsive agent should cost seconds of restore latency, not the 120s the fatal fork-only network reconfigure path tolerates. The agent was running at snapshot time and is reachable almost immediately after resume, so the short wait covers the happy path. This also bounds how stale the timestamp baked into the exec fallback command can get. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
TestStandbyAndRestore now sleeps 10s in standby and asserts the guest clock does not lag after restore, and exercises the exec fallback by skewing the guest clock back an hour and syncing it via syncGuestClockWithExec. Without the restore-time SyncClock the guest lags by the full standby duration and the assertion fails. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After a snapshot restore, the guest wall clock resumes from the moment the snapshot was saved — Firecracker does not advance
CLOCK_REALTIMEon restore, and nothing in the guest resyncs it. Every in-guest timestamp (telemetry events, logs, recordings) therefore lags real time by however long the instance sat in standby.This adds a clock resync to the restore path:
SyncClockguest-agent RPC (lib/guest/guest.proto,lib/system/guest_agent/clock.go): setsCLOCK_REALTIMEto a host-provided wall-clock time viaclock_settime(2).lib/guest/clock.go):SyncClockInInstancewith the same wait-for-agent retry behavior asReconfigureNetworkInInstance. The host timestamp is captured per attempt so a long agent wait isn't baked into the guest clock.lib/instances/restore.go): after resume, sync the clock (gated onSkipGuestAgent). Non-fatal — a skewed clock degrades timestamps, not the VM, so failure logs a warning instead of failing the restore.Unimplemented; those getdate -u -s @<epoch>via the existing exec path, mirroring the network-reconfigure fallback.A discontinuous step (rather than slewing) is intentional: the skew accrues while the VM is paused pre-claim, so nothing in the guest observes the jump, and slewing would take hours to absorb multi-minute offsets.
Generated files (
guest.pb.go,guest_grpc.pb.go) were regenerated withmake generate-grpc(protoc 33.4, plugin versions from go.mod).Testing
go build/go vetonlib/guest,lib/instances,lib/system/guest_agent— cleango test ./lib/system/guest_agent ./lib/guest— pass (includes newSyncClockinput-validation test)make test TEST=TestStandbyAndRestorecould not complete in my environment: the test fails in setup (test image never reaches "ready" within its 60s window), before any restore code runs. The end-to-end behavior (clock actually stepped after restore, plus the exec fallback against an old-agent snapshot) still needs verification on real infrastructure.Follow-ups (not in this PR)
🤖 Generated with Claude Code