Maincloud: Pro+PAYG databases suspended; resumed instance served SQL/reducers/snapshots but interval schedules and subscription streaming were dead (2 incidents July 9)

[austindanson1]

Account / database: Maincloud Pro (pay-as-you-go enabled, spend limit far from reached), database puncher-prod, identity c2004458746817b5608f2a8610e33370ec44366e5535d666ce868883d349dc88 (owner will provide account details privately on request). All times UTC on 2026-07-09.

Incident 1 — suspension despite Pro + PAYG, then a sick resume (~06:00–20:35 UTC)

Databases in the account were suspended overnight (the dashboard showed the generic "Databases are paused" banner) even though the docs state Pro databases are paused "only when requested" and the account had energy balance, PAYG on, and its spend limit ~22% consumed. No incident appeared on status.spacetimedb.com.

Auto-resume on connection worked as documented (<1s), but the resumed instance was sick for ~2 hours in a way no connectivity check can see:

Channel	State after resume
HTTP API / spacetime sql	✅ works
Reducer calls	✅ work (joins committed, events logged)
Subscription initial snapshots	✅ work
Interval scheduled reducers (100ms tick)	❌ never fire — schedule row present
Live subscription row-update streaming	❌ dead or intermittent

Game impact: clients connect, call join_map successfully, then never receive their own fighter row — they hang, or play "blind" against a frozen world. Server-side we can prove fighters died at their exact spawn coordinates (to full double precision) because the input-gated clients never learned their fighter ids.

Recovery attempts that did NOT immediately revive it: deleting + re-inserting the schedule row via reducer (a fresh insert normally arms a timer), dashboard pause→start cycle, full module republish. It recovered on its own ~2h after resume (possibly the accumulated restarts). Evidence that the scheduler was genuinely stopped overnight rather than idle: the database's energy chart shows zero TeV burned overnight, though its always-on 100ms tick burns CPU around the clock; several earlier zero-energy days on the chart suggest this had happened before, silently.

Incident 2 — same day, ~21:34–21:40 UTC

maincloud.spacetimedb.com began returning 502s, then hung entirely (15s+ connect timeouts, in-flight SQL queries dying). Recovered ~21:40. At ~22:03 UTC spacetimedb.com itself served 503 Service Temporarily Unavailable (nginx) for several minutes. Neither event appears on status.spacetimedb.com (which as of this writing shows only a 5-minute Maincloud blip on July 8 — the July 8→9 suspension isn't listed either).

Asks

1. Root cause for the pause of Pro+PAYG databases (this looks like the class of platform-side pauses previously tracked in Investigate the Maincloud live cluster pauses reported by users #2756, which is closed).
2. A fix (or workaround guidance) for the sick-resume state: interval schedules not firing and update streaming dead while everything else answers. If resume re-arms timers from schedule rows, that path appears to have not run.
3. Status page coverage for these events.

We now run an external heartbeat/watchdog that detects both failure modes and can share timestamped probe logs, dashboards, and query transcripts for the whole window — happy to provide anything useful.