Skip to content

perf(cloud): broadcast signals and storage-backed replay segments #511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Harry19081 merged 4 commits into from
Jul 24, 2026
Merged

perf(cloud): broadcast signals and storage-backed replay segments #511

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3d1f085
feat(cloud): broadcast-signal transport with capability probe
Neonforge98 Jul 24, 2026
bdea98b
fix(cloud): roster broadcasts also run coarse recovery
Neonforge98 Jul 24, 2026
e972041
feat(cloud): storage-backed frozen segment transport
Neonforge98 Jul 24, 2026
d305a21
docs: descriptive names for the broadcast and storage design notes
Neonforge98 Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions docs/cloud-broadcast-and-storage-design-2026-07-24/broadcast-change-signals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Change Signals via Broadcast-from-Database (audit item H4)

Date: 2026-07-24. Basis: realtime signal-chain architecture map (agent survey
of 0001/0003 server path + full client subscription topology). Companion:
`replay-storage-offload.md`.

## Why

`postgres_changes` evaluates RLS per change × per subscriber on a single
shared WAL poller — the platform's first scalability wall (server-schema-audit
H4). The client consumes **zero row data** from any of its three
postgres_changes subscriptions (all onChange callbacks are 0-arg edges), so
the entire contract is "something in scope changed" — a perfect fit for
fire-and-forget broadcast with join-time authorization.

## Decision summary

| Question | Decision | Why |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | -------- | --------- | ------ | -------------------------------------------------------------------------------------------- |
| Topic | **Reuse the existing private org channel `presence:org:<org_id>`** | Already open on every active client, already authorized by the `realtime.messages` policies (0001:5694-5706), already carries client-side broadcast events (`org-control-changed` etc.). Server-side `realtime.send` from a definer trigger needs no new policy. Channels per client drop 4 → 2. |
| New event | `org-db-changed` with payload `{kind}` | Distinct from client-sent `org-control-changed`. `kind` ∈ `sessions | comments | projects | workItems | roster | policy`derived from`TG_TABLE_NAME` / call site — the discriminator today's signal row lacks. |
| Debounce | Broadcast fires **only when the debounced signal-row bump actually fired** (the 0003 250ms `WHERE` clause) — per (org, kind)? No: per org, same as today | Identical delivery semantics to the postgres_changes transport; the client's 60s coarse throttle and 5min control TTL stay unchanged. |
| Slice A (self-roster eviction) | **Stays postgres_changes** on `org_memberships` `user_id=eq.<uid>` | A member removed while disconnected can never be reached by join-time-auth broadcast (`is_org_member()` already false). Exactly the audit's recommendation. |
| Slice B (org signal) + B-roster (org memberships) | Replaced by broadcast events on the org channel when the backend supports it | The two org-scoped postgres_changes channels disappear; SUBSCRIBED-edge recovery bookkeeping moves to the org channel's join edge. |
| Old signal row / publication | **Kept and still bumped** (org_change_signals stays in the publication) | Old clients keep working; removal is a later cleanup migration once the fleet has upgraded. |
| Capability detection | New RPC `get_cloud_capabilities()` → `{broadcastSignals: true}`; client probes once per endpoint, PGRST202 ⇒ legacy | Same probe-and-remember pattern as the 0004 paged listings. |
| Roster nudges for members | New AFTER trigger on `org_memberships` → send `kind:'roster'` to the org topic | Replaces the org-filtered memberships channel for present members; the removed user is covered by Slice A. |
| Policy nudges | The 3 RPCs with direct in-RPC bumps (`cloud_rename_org`, `cloud_set_org_sharing_floor`, `cloud_set_member_sharing_floor`) recreated with `kind:'policy'` send | They bypass the trigger today; same bypass, now with a broadcast. |

## Server (cloud-infra 0005)

1. Helper `org2_cloud.nudge_org_signal(p_org_id uuid, p_kind text)`:
debounced signal-row upsert (unchanged 250ms suppress) and, **iff the
bump fired**, `realtime.send(jsonb_build_object('kind', p_kind),
'org-db-changed', 'presence:org:' || p_org_id, true)`.
2. `touch_org_change_signal()` trigger fn delegates to the helper with kind
mapped from `TG_TABLE_NAME` (`cloud_projects`→projects,
`cloud_work_items`→workItems, `cloud_sessions`→sessions,
`cloud_session_comments`→comments).
3. New `AFTER INSERT OR UPDATE OR DELETE` trigger on `org_memberships` →
`nudge_org_signal(org_id, 'roster')`.
4. Recreate the 3 policy RPCs replacing their inline bump with the helper
(`kind:'policy'`).
5. `get_cloud_capabilities()` → `jsonb_build_object('broadcastSignals',
true)`, grant `authenticated`.
6. Lock-order invariant preserved: the helper is still the statement-final
signal-row acquisition (canonical slot 6); `realtime.send` inserts into
`realtime.messages` (a partitioned insert, no user-table locks) after it.
7. Verify tail: single-signature checks; trigger presence; capabilities
callable.

Offline validation: brew-PG harness stubs `realtime.send` (records rows into
a scratch table) — assert one broadcast per debounce window per kind source,
none when suppressed, roster/policy events fire, legacy signal row byte-same.

## Client (ORGII)

1. `org2CloudClient` (or syncClient): `getCloudCapabilities(accessToken)`
with PGRST202→null; per-endpoint memory (`capabilitiesByEndpoint`).
2. `org2CloudRealtimeClient.joinPresence`: surface server broadcasts —
`org-db-changed` events dispatch to a new `onOrgDbChanged(kind)` option
next to the existing broadcast handlers; expose the channel's SUBSCRIBED
edge (already surfaced for presence re-track) to the caller for recovery
bookkeeping.
3. `useOrg2CloudRealtime`: when capabilities report `broadcastSignals`,
skip creating Slice B and B-roster postgres_changes channels; wire
`onOrgDbChanged`: `roster` → `bumpRosterVersion(orgId)` (+ the
rosterRealtimeConnected atom keyed to the org channel's join edge);
every other kind → `scheduleCoarseSignalRefresh()` (behavior-identical
coarse path; per-kind narrowing is a later optimization, the payload
already carries it). SUBSCRIBED-edge full/delta recovery
(`decideSubscribedEdgeRecovery`) moves onto the org channel's join edge.
Slice A unchanged.
4. Legacy backends (probe fails): topology unchanged (3 pg channels + 1
presence). Tests: transport tests reworked in
`org2CloudRealtimeClient.test.ts`; recovery/lease/scope tests untouched.

## Quota effect

Broadcast messages still bill per delivery, but: no per-change × per-client
RLS evaluation, no WAL-poller serialization, and the `kind` discriminator
unlocks future per-plane refetch narrowing (today every signal shotguns all
planes after the 60s throttle). The 250ms server debounce and 60s client
throttle keep message volume identical to today's.

## Rollout order

0005 applies live (additive, old clients unaffected) → client PR merges →
fleet upgrades → later cleanup migration drops org_change_signals from the
publication (and eventually the table, once fleet-wide).
72 changes: 72 additions & 0 deletions docs/cloud-broadcast-and-storage-design-2026-07-24/replay-storage-offload.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Replay Segment Payload Offload to Supabase Storage (audit item H5)

Date: 2026-07-24. Basis: segment-lifecycle architecture map (agent survey of
0001/0002/0003 + client push/pull paths). Companion: `broadcast-change-signals.md`.

## Decision summary

| Question | Decision | Why |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| What moves to Storage | **Frozen segments only** (`seq >= 1`) | Immutable + append-only + the bulk of bytes. No overwrite churn, no orphan risk on the hot path. |
| What stays in Postgres | **Tail (`seq = 0`) stays `payload_gz` inline** | Mutable, rewritten every push, small (recent non-terminal events). Keeps append OCC single-transaction. |
| Object encoding | raw gzip bytes (`application/gzip`) | Drops the base64 +33% leg entirely; codec already isolates the base64 step (`segmentCodec.ts`). |
| Object key | `replay/{org_id}/{session_id}/{epoch}/{seq}-{segment_hash}.gz` | Immutable-by-name; hash in key makes retry idempotent and collision-safe inside the org/session auth domain (no cross-tenant dedup by design — see SharedSessionPerformance.md side-channel note). |
| Member reads/writes | **Direct Storage REST with user JWT + storage RLS** | The whole member auth ladder (membership → retention → metadata_only → restricted-visibility grant) is already SQL (`assert_session_readable`); a `security definer` helper called from the storage policy reuses it verbatim. |
| Share-token guest reads | **Vercel signer route** (`apps/org2-cloud-web`, has `SUPABASE_SERVICE_ROLE_KEY`) | Guest tier has no `auth` role, inexpressible in storage RLS. RPC mints a short-lived grant; the route redeems it and returns signed URLs. Rare path. |
| Atomicity | **Upload blobs first, then commit manifest RPC** | RPC verifies each object exists in `storage.objects` and reads its真实 size from object metadata — server-measured bytes, no client attestation. Missing object ⇒ `ORG2_VALIDATION`, nothing committed. |
| Quota unit | `byte_size` = raw object bytes (frozen) / `octet_length(payload_gz)` (tail, unchanged) | −25% vs base64 accounting: quotas get slightly looser, matches what is actually stored. Documented, not compensated. |
| Orphan GC | SQL view `replay_orphan_objects` (objects with no matching segment row, older than 24h) + service-role sweep via Storage API (Vercel cron) | Postgres FK cascade cannot delete bucket objects; rewrite/delete-account leave objects behind by design and the sweep reaps them. |
| Rollout | Additive: `cloud_session_segments.storage_path text null`; null = legacy inline. Read RPCs return `storagePath` XOR `payloadGz` per segment. | Old clients fail closed on new-format sessions (zod `payloadGz` missing ⇒ import error, no corruption). Pre-release population is auto-updating; accepted. |

## Server (cloud-infra migration, offline-validatable parts)

1. `cloud_session_segments.storage_path text` + relax `payload_gz` to nullable
with `check ((payload_gz is not null) or (storage_path is not null))` and
`check (seq = 0 → payload_gz is not null)` (tail always inline).
2. Bucket `replay` (private) + storage RLS policies delegating to
`org2_cloud.can_read_replay_object(name)` / `can_write_replay_object(name)`
security-definer helpers (path parse → session ladder / owner+entitlement
gates + epoch/seq sanity).
3. `cloud_append_session_events` / `cloud_rewrite_session_events`: frozen
segment wire accepts `{seq, storagePath, eventCount, segmentHash}` as an
alternative to `payloadGz`; on the storage form the RPC looks the object up
in `storage.objects` (existence + `(metadata->>'size')::bigint` + key must
embed the claimed `segment_hash`), charges quota with the object size, and
stores `storage_path`. Legacy inline form byte-identical behavior.
4. `cloud_get_session_events(_page)`: rows return `storagePath` when set,
`payloadGz` otherwise. Guest grant RPC `cloud_authorize_replay_read` +
redeem RPC (service-role) for the signer route.
5. `reconcile_org_stored_bytes` unchanged (byte_size stays truthful).
6. `replay_orphan_objects` view + `gc_replay_orphans` bookkeeping (row side);
object deletion happens in the cron sweeper, not SQL.

## Client (ORGII)

- Push seam: inside `org2CloudSyncClient.appendSessionEvents`/`rewrite…` —
before the RPC, upload each frozen segment's raw gzip bytes via
`PUT /storage/v1/object/replay/{key}` (JWT), bounded concurrency (reuse
`mapSegmentsBounded` cap 4); segment wire swaps `payloadGz` →
`storagePath`. Tail keeps the base64 inline path.
- Pull seam: `decodeCloudSegments` (`org2CloudBackendAdapter.ts`) — when a
segment carries `storagePath`, `GET /storage/v1/object/replay/{key}` (JWT)
→ gunzip raw bytes; integrity check unchanged (sha256 of pre-gzip bytes vs
`segmentHash`). Share-token imports call the signer route first.
- Capability fallback mirror of the 0004 pattern: on storage-form rejection
(pre-migration backend), remember per endpoint and fall back to inline
wire. Full backward compatibility both directions.

## Explicitly out of scope

- Migrating EXISTING inline segments (backfill mover is a later service-role
batch job; inline reads stay supported indefinitely).
- TUS/resumable uploads (objects are ≤ ~256 KiB pre-gzip; far below 6 MB).
- Cross-org content dedup (side channel, rejected in SharedSessionPerformance).

## Validation plan

- Offline (brew PG + storage schema stub): migration idempotency, wire A/B
(inline form byte-identical vs 0004 DB), manifest commit vs missing object,
quota deltas with mixed inline/storage segments, orphan view correctness.
- Live: real Storage bucket + policies (user pastes migration, bucket created
via dashboard or SQL insert into storage.buckets), dual-instance push/pull,
cmd+5 + CPU/RAM per standing rule.
Loading