Skip to content

Fix PTT screen grounding and canonical agent status#9775

Open
Git-on-my-level wants to merge 10 commits into
BasedHardware:mainfrom
Git-on-my-level:codex/fix-realtime-screen-freshness
Open

Fix PTT screen grounding and canonical agent status#9775
Git-on-my-level wants to merge 10 commits into
BasedHardware:mainfrom
Git-on-my-level:codex/fix-realtime-screen-freshness

Conversation

@Git-on-my-level

@Git-on-my-level Git-on-my-level commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Keep every PTT screen answer bound to one locally owned, turn-scoped screenshot/report protocol.
  • Close the voice turn from a verified local report or deterministic failure, instead of waiting for optional provider narration.
  • Preserve the protocol through screenshot transport, report validation, persistence, playback ownership, errors, cancellation, and barge-in.
  • Make canonical spawn acknowledgements and deterministic screen answers use the same authoritative output takeover path.

Product invariants affected

  • INV-AGENT-*
  • INV-CHAT-1
  • INV-VOICE-1

Root cause

The screen report was accepted locally, but the reducer's generic tool-start path also set a provider-continuation obligation. The screen-report path presented a local result without discharging that obligation. When Gemini finished its provider turn with no further narration, the controller parked in awaitingResponse until the generic 20-second provider-response deadline. The live trace for turn 4645612F shows report_accepted at 00:07:35.587, provider_turn_done at 00:07:52.064, then cancellation at 00:08:12.528.

Durable guard

VoiceScreenEvidenceProtocolToken makes the screenshot/report handshake an explicit reducer-owned protocol tied to the original turn, screenshot effect, session, and epoch. Only a matching verified report may close it; enqueue failure, rejected report, provider terminal without a report, hub error, freshness expiry, cancellation, and barge-in all resolve the exact protocol fail-closed. Its deadline is the remaining five-second evidence lifetime, not the generic 20-second narration timeout. A stale deadline is rejected if its effect identity no longer matches the live state. The result remains fenced by persistence, playback, and journal completion; deterministic screen audio can preempt speculative native speech through the same output-lease ownership path used by canonical spawn acknowledgement.

Verification

  • xcrun swift test --package-path desktop/macos/Desktop --skip-build --filter 'VoiceTurnOutputOwnershipTests|VoiceTurnReducerTests|RealtimeScreenEvidenceTests|VoiceTurnReducerFuzzTests' — 99 passed.
  • xcrun swift test --package-path desktop/macos/Desktop --skip-build --filter 'VoiceTurnReducerTests|VoiceTurnCoordinatorTests|RealtimeScreenEvidenceTests|RealtimeHubSpawnAgentTests' — 130 passed.
  • bash desktop/macos/scripts/agent-logic-harness.sh --cross-surface-smoke — 242 Swift agent-contract tests and 13 Rust retrieval-contract tests passed.
  • python3 desktop/macos/scripts/check_desktop_test_quality.py — passed.
  • git diff --check — passed.
  • Independent Sol review — approved after adding the output-lease takeover regression.

Manual QA

Build and launch the existing /Applications/omi-qa-live-fix.app against development GCP. A PTT "what is on my screen" request should produce a current-screen answer or the deterministic verification failure within five seconds; it must not remain in a 20-second post-tool spinner. Barge-in remains silent unless there is an actionable failure.

@Git-on-my-level Git-on-my-level force-pushed the codex/fix-realtime-screen-freshness branch 2 times, most recently from 0211aef to 39ae598 Compare July 15, 2026 01:35
Make visual evidence receipts truthful at the local websocket enqueue boundary, fail no-transport sends through the terminal session path, and reject only direct contradictory foreground-app claims. Isolate named-bundle, harness, and local backend logs so concurrent QA runs cannot contaminate diagnostics. Keep one-shot TTS fallback from speaking a cancelled turn.\n\nVerification: focused Swift screen/transport/voice/lifecycle suites (133 tests); cargo check + fmt; omi harness, hardening smoke, and named backend log-isolation tests.
Allowlist three existing file-isolated configuration tests whose measured CPU cost consistently exceeds the local 120ms guard. The test guide prescribes exact-node allowlisting for this structural cost; the global budget remains unchanged.\n\nVerification: backend/test.sh (604 selected unit test files).
Apply the repository Flutter formatter to historical files carried by this branch so the protected pre-push formatter gate is reproducible.\n\nVerification: dart format --line-length 120 on all ten affected files.
@Git-on-my-level Git-on-my-level force-pushed the codex/fix-realtime-screen-freshness branch from 39ae598 to 6085f2d Compare July 15, 2026 02:36
@Git-on-my-level

Copy link
Copy Markdown
Collaborator Author

Monitor summary for PR #9775:

  • No code changes were needed in this run: the context's unresolved trusted review thread list is empty, and a fresh review-thread scan found no unresolved inline comments.
  • The previously reported cancelled Repo Checks were superseded by successful reruns.
  • Verification: gh pr checks 9775 --repo BasedHardware/omi reports all active required checks passing or skipped; git diff --check origin/main...HEAD passed; python3 .github/scripts/check-desktop-changelog.py --base origin/main --head HEAD passed with a desktop changelog fragment present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant