fix(deploy): reconcile Firestore indexes in runtime project

[tianmind-studio]

Summary

• reconcile generated Firestore indexes in RUNTIME_GCP_PROJECT_ID, the project used by serving Firestore clients
• apply the same target contract to manual and automatic backend deployments
• preserve fix(dev): provision Firestore indexes explicitly #9756's explicit development --provision-missing behavior in the runtime project
• make runtime-env validation reject index reconciliation against the Cloud Run deployment project
• cover the cross-project deployment/runtime case with a regression test

Root cause

The backend deployment for commit 4399e927f reconciled the corrected index from #9751 in based-hardware-dev and reported all 13 indexes READY, but the deployed backend explicitly runs with GOOGLE_CLOUD_PROJECT=based-hardware. The default Firestore client therefore serves from the runtime project, not the Cloud Run deployment project.

The acceptance smoke still returned HTTP 500 after the index was ready because the workflow certified a different Firestore project than the request path used:
https://github.com/BasedHardware/omi/actions/runs/29365024038

A second independent main deployment reproduced the same project split and HTTP 500:
https://github.com/BasedHardware/omi/actions/runs/29365288366

This keeps deployment readiness aligned with backend/deploy/runtime_env.yaml, which already models gcp_project and runtime_gcp_project as separate boundaries.

Scope and safety

• deployment workflow and static validation only
• no application data, authentication, request, model, or traffic behavior changes
• no secrets added, read, or exposed
• this PR does not itself run a deployment or mutate Firestore; an authorized workflow run performs reconciliation after merge

Product invariants affected

None. This aligns an existing datastore readiness gate with the configured serving project.

Verification

• focused runtime-env, index reconciliation, smoke, and deployment tests after rebasing onto fix(dev): provision Firestore indexes explicitly #9756: 64 passed
• backend runtime-env validation: dev and prod passed
• actionlint: both changed workflows passed
• Black and Ruff: passed before rebase; the resolved workflow-only overlap introduced no Python changes
• OpenAPI JSON, app-client compatibility, integration surface, and Dart generated models: up to date on the original revision

[tianmind-studio]

A second independent main deployment at 306f22378b reproduced the same project split and HTTP 500 after this PR was opened:

• index reconciliation command: --project based-hardware-dev
• readiness result: 13 composite indexes READY
• deployed backend runtime: GOOGLE_CLOUD_PROJECT=based-hardware
• acceptance result: HTTP 500

Run: https://github.com/BasedHardware/omi/actions/runs/29365288366

This confirms the failure is stable across two main revisions and that the readiness gate is certifying the deployment project rather than the configured serving project.

[tianmind-studio]

Rebased onto current main after #9756. The auto-dev command now composes both fixes: it targets vars.RUNTIME_GCP_PROJECT_ID and retains --provision-missing. Focused runtime-env, reconciliation, smoke, and deployment coverage is green (64 tests), with dev/prod runtime validation and actionlint also passing.

[Git-on-my-level]

Changes requested: adopt a safe two-stage shared-Firestore protocol

This PR correctly identifies the cross-project root cause: dev Cloud Run deploys in based-hardware-dev, but serving Firestore clients use based-hardware.

The current implementation is not safe to merge because the development auto-deploy and manual-development path both reconcile that runtime project with a mutating command. In the development GitHub environment, RUNTIME_GCP_PROJECT_ID=based-hardware; --provision-missing can therefore run gcloud firestore indexes composite create against production without an explicit production approval boundary.

Required approach for this PR — Phase 1 only: detect and plan

Rework the deployment paths to be read-only against the runtime Firestore project:

1. Compare the generated, source-owned manifest with the exact live runtime-project index inventory.
2. When an index is missing or non-READY, fail closed before candidate acceptance or traffic promotion.
3. Emit a deterministic, redacted machine-readable schema proposal artifact containing the target project, exact create-only manifest signatures, source commit/input hash, and a short TTL.
4. Preserve the proposal artifact when the gate fails so an agent can relay it to the human runner.
5. Do not invoke Firebase deploy or gcloud ... composite create from automatic dev or manual-development deployment paths.

Separate follow-up — Phase 2: approved apply

A separately named, source-managed production schema workflow may later apply an approved proposal. It must require explicit human approval of the exact plan hash, revalidate target/hash/TTL/live state immediately before write, create only manifest-declared missing indexes, never delete indexes or mutate application data, wait for exact READY, and then allow the dev candidate gate to retry.

This preserves automation and zero-console operations while treating shared production Firebase as the production dependency it is. It also prevents future agents from silently turning a dev deployment into a production Firestore writer.

Do not merge this PR until Phase 1 is implemented and the mutating runtime-project paths are removed.

[Git-on-my-level]

Thanks for the careful rework here. The current direction is much safer than the earlier version: the backend deploy paths now do read-only --check-only validation against RUNTIME_GCP_PROJECT_ID, branch deploys use the main Firestore manifest/control-plane script, and production index writes are separated behind a manually dispatched production workflow.

I still need to keep this blocked because CI is failing on the backend unit suite. The failure is in tests/unit/test_workflow_contracts.py::test_every_external_workflow_contract_source_triggers_backend_unit_workflow: the new production_firestore_schema_deployment workflow contract adds external sources, but .github/workflows/backend-unit-tests.yml does not include them as triggers. The missing paths are:

• .github/workflows/gcp_backend.yml
• .github/workflows/gcp_backend_auto_dev.yml
• .github/workflows/gcp_firestore_indexes_prod.yml
• firestore.indexes.json

Please wire those workflow-contract sources into the backend unit-test workflow trigger (or adjust the contract if a listed source is not intended to select backend unit coverage), then rerun the backend unit suite.

Agent-instruction impact: this PR also updates .github/AGENTS.md and backend/AGENTS.md with Firestore deployment guidance. That guidance will steer future AI coding/review agents away from mutating Firestore indexes in backend deploy workflows and toward the separate reviewer-protected production workflow. The guidance matches the safer boundary this PR is trying to establish, but because it affects deploy/review-agent behavior and production schema operations, it still warrants human maintainer review before merge.

I did not dismiss the previous automation change request because this head still has a blocking CI failure, even though the earlier mutating-runtime-project concern appears to have been addressed.

---

by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.