diff --git a/.github/workflows/feed-proxy-deploy.yml b/.github/workflows/feed-proxy-deploy.yml index 2b6ecb5d..2d5318e4 100644 --- a/.github/workflows/feed-proxy-deploy.yml +++ b/.github/workflows/feed-proxy-deploy.yml @@ -54,6 +54,43 @@ jobs: - name: Run tests run: bun test + deploy-staging: + name: Deploy to Staging + needs: [typecheck, test] + # Same cadence as the staging Worker: every push to main. Staging has its own + # Fly app (fly.staging.toml) so it never shares the prod machine or its cache, + # and config drift between the two files surfaces here before a release. + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + runs-on: ubuntu-latest + environment: staging + steps: + - name: Checkout + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + + - name: Setup flyctl + uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1 # master + + # The app is provisioned by hand (Phase 3 of docs/plans/D1_FEED_TIMELINE.md: + # app + volume + PROXY_SECRET). Until that happens this job would fail on + # every push to main, so skip it with a notice instead. + - name: Check staging app exists + id: staging_app + run: | + if flyctl status -a skyreader-feed-proxy-staging >/dev/null 2>&1; then + echo "exists=true" >> "$GITHUB_OUTPUT" + else + echo "exists=false" >> "$GITHUB_OUTPUT" + echo "::notice::skyreader-feed-proxy-staging not provisioned yet; skipping staging deploy (see docs/plans/D1_FEED_TIMELINE.md, Phase 3)." + fi + env: + FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} + + - name: Deploy + if: steps.staging_app.outputs.exists == 'true' + run: flyctl deploy --remote-only --config fly.staging.toml + env: + FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} + deploy: name: Deploy to Production needs: [typecheck, test] diff --git a/CLAUDE.md b/CLAUDE.md index ddefe3b5..8e062228 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ This is a monorepo with 6 packages: - `backend/` - Cloudflare Workers API - `frontend/` - SvelteKit PWA - `admin/` - SvelteKit admin dashboard (Cloudflare Pages) -- `feed-proxy/` - Feed caching proxy (Fly.io) +- `feed-proxy/` - Feed crawler + article extraction (Fly.io; one app per environment) - `linkblog-site/` - Standalone SvelteKit app rendering public linkblogs at `linkblogs.skyreader.app` (Cloudflare Pages) - `extension/` - Chrome extension for one-click saves with live-DOM article extraction (Manifest V3) @@ -85,6 +85,9 @@ This script runs D1 migrations, then starts the feed proxy (port 3000), backend ``` FRONTEND_URL=http://127.0.0.1:5173 FEED_PROXY_URL=http://127.0.0.1:3000 + # Shared with the proxy (dev-local.sh exports the same value as PROXY_SECRET). + # Feed ingest is fail-closed, so without this the local reader stays empty. + FEED_PROXY_SECRET=dev-proxy-secret ``` **Resetting the local database:** @@ -217,14 +220,19 @@ AT Protocol (Bluesky PDS) + Fly.io Feed Proxy + Jetstream Firehose - **Database:** D1 (SQLite) - reads the same database as the backend - **Features:** Ops panel (cron liveness, firehose lag, proxy cache health) with 30-day trend sparklines, system metrics, user management, feed health monitoring, search/sort/pagination -- **Pages:** Dashboard (ops + metrics + trends), Users (list + detail), Feeds (health + error tracking) +- **Pages:** Dashboard (ops + metrics + trends), Users (list + detail), Feeds (health + error tracking). + Feed health is the crawler's own verdict from `feeds.error_count` / `feeds.crawl_stale`, not an + inference from `last_ingest_at` — that only moves when a feed publishes, so it says nothing about + whether the feed still works. - **Deploy:** Cloudflare Pages via GitHub Actions (staging on push to main, production on release) ### Key Data Flow 1. **Auth:** Handle → DID resolution → OAuth PKCE/DPoP → session token 2. **Subscriptions:** Stored in user's PDS, cached locally in IndexedDB -3. **Feed Updates:** Frontend requests feeds → backend proxies via Fly.io feed cache +3. **Feed Updates:** Fly.io proxy crawls feeds and pushes new items into D1 (`feed_items`); the + frontend refreshes with one `GET /api/v2/timeline` query that joins subscriptions and read + state. Reads never touch Fly — see `docs/plans/D1_FEED_TIMELINE.md` 4. **Social:** Jetstream firehose → D1 shares table → frontend polls for updates ## AT Protocol Integration diff --git a/README.md b/README.md index b5f910d5..cfebcddb 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,9 @@ cd skyreader ``` FRONTEND_URL=http://127.0.0.1:5173 FEED_PROXY_URL=http://127.0.0.1:3000 + # Shared with the proxy (dev-local.sh exports the same value as PROXY_SECRET). + # Feed ingest is fail-closed, so without this the local reader stays empty. + FEED_PROXY_SECRET=dev-proxy-secret ``` ### Local Development diff --git a/admin/src/lib/components/StatusBadge.svelte b/admin/src/lib/components/StatusBadge.svelte index db8ae5c4..240108a0 100644 --- a/admin/src/lib/components/StatusBadge.svelte +++ b/admin/src/lib/components/StatusBadge.svelte @@ -1,9 +1,13 @@ -{labels[status]} +{label ?? labels[status]} diff --git a/backend/ARCHITECTURE.md b/backend/ARCHITECTURE.md index 3a5489f8..8092209c 100644 --- a/backend/ARCHITECTURE.md +++ b/backend/ARCHITECTURE.md @@ -2,7 +2,7 @@ ## Overview -The Skyreader backend is a Cloudflare Worker that serves as an API gateway between the frontend and the AT Protocol ecosystem. It handles authentication, RSS feed fetching/parsing via a Fly.io proxy, social features, saved articles, and background Jetstream polling. +The Skyreader backend is a Cloudflare Worker that serves as an API gateway between the frontend and the AT Protocol ecosystem. It handles authentication, the D1-served feed timeline (crawled and pushed by a Fly.io proxy), social features, saved articles, and background Jetstream polling. ``` ┌──────────────────────────────────────────────────────────────────────────┐ @@ -17,8 +17,8 @@ The Skyreader backend is a Cloudflare Worker that serves as an API gateway betwe │ CLOUDFLARE WORKER │ │ │ │ ┌──────────┐ ┌──────────────┐ ┌───────────────┐ ┌──────────────────┐ │ -│ │ auth.ts │ │ feeds-v2.ts │ │ social.ts │ │ subscriptions.ts │ │ -│ │ OAuth │ │ Feed proxy │ │ Social feed │ │ CRUD │ │ +│ │ auth.ts │ │ timeline.ts │ │ social.ts │ │ subscriptions.ts │ │ +│ │ OAuth │ │ D1 timeline │ │ Social feed │ │ CRUD │ │ │ └────┬─────┘ └──────┬───────┘ └───────┬───────┘ └────────┬─────────┘ │ │ │ │ │ │ │ │ ┌────┴────┐ ┌───────┴──────┐ ┌────────┴───────┐ ┌───────┴────────┐ │ @@ -51,7 +51,7 @@ The Skyreader backend is a Cloudflare Worker that serves as an API gateway betwe ↓ ↓ │ ┌───────────────────┐ ┌───────────────────┐ ┌─────────────────────────┐ │ BLUESKY PDS │ │ FLY.IO FEED PROXY │ │ JETSTREAM FIREHOSE │ -│ (user's data) │ │ (RSS caching) │ │ (AT Protocol events) │ +│ (user's data) │ │ (crawler) │ │ (AT Protocol events) │ └───────────────────┘ └───────────────────┘ └─────────────────────────┘ ``` @@ -134,15 +134,18 @@ Upsert user in D1 Redirect to frontend with auth exchange code ``` -### Feeds (`src/routes/feeds-v2.ts`) +### Feeds (`src/routes/timeline.ts`, `src/routes/ingest.ts`, `src/routes/feeds-v2.ts`) -All feed fetching is proxied through a Fly.io feed proxy service (`FEED_PROXY_URL`), authenticated with `FEED_PROXY_SECRET`. +Reads are served from D1. The Fly.io feed proxy (`FEED_PROXY_URL`) is the crawler: it pushes new and edited items into `feed_items` and pulls the set of feeds to crawl, both authenticated with the shared `FEED_PROXY_SECRET`. A client refresh is one `/api/v2/timeline` request — a single query joining subscriptions and read state. See `docs/plans/D1_FEED_TIMELINE.md`. -| Endpoint | Method | Auth | Description | -| ------------------------ | ------ | ------ | --------------------------------- | -| `/api/v2/feeds/fetch` | GET | Bearer | Fetch and parse a single RSS feed | -| `/api/v2/feeds/batch` | POST | Bearer | Batch fetch multiple feeds | -| `/api/v2/feeds/discover` | GET | Bearer | Discover feeds on a website URL | +| Endpoint | Method | Auth | Description | +| ------------------------- | ------ | ------ | -------------------------------------------- | +| `/api/v2/timeline` | GET | Bearer | The whole refresh: items since a cursor | +| `/api/internal/ingest` | POST | Secret | Crawler pushes new/edited items into D1 | +| `/api/internal/crawl-set` | GET | Secret | Crawler pulls the feeds to crawl | +| `/api/v2/feeds/fetch` | GET | Bearer | Single feed from D1, with proxy pull-through | +| `/api/v2/feeds/batch` | POST | Bearer | Legacy batch fetch (fallback path) | +| `/api/v2/feeds/discover` | GET | Bearer | Discover feeds on a website URL | ### Social (`src/routes/social.ts`) @@ -346,9 +349,8 @@ Key tables: | `auth_exchange_codes` | Short-lived auth exchange codes | | `subscriptions_cache` | RSS feed subscriptions cached from PDS | | `shares` | Aggregated share data from Jetstream (with reshare tracking) | -| `feed_metadata` | Feed caching metadata (ETags, errors, subscriber count, shard_id) | -| `feed_cache` | D1-based parsed feed cache | -| `feed_items` | Individual feed items for efficient querying | +| `feeds` | One row per crawled feed (title/site/image + `last_ingest_at`) | +| `feed_items` | The item archive the timeline serves, keyed `(feed_url, guid)` with `seq` | | `documents` | orphaned — documents moved to on-demand proxy fetch; table left in place | | `publications_cache` | orphaned — the publication cache moved to the feed proxy | | `social_read_positions_cache` | Unified social read tracking (shares + documents) | @@ -357,8 +359,10 @@ Key tables: | `rate_limits` | Per-user per-endpoint rate limiting | | `user_settings` | User feature preferences (Leaflet sync, PDS sync) | | `did_handle_cache` | Handle resolution cache | -| `sync_state` | Jetstream cursor and other sync state | +| `sync_state` | Jetstream cursor, archive generation token, crawler heartbeat | | `reshares` | Reshare tracking | +| `system_status` | Cron-written health board (cron liveness, poller lag, proxy stats) | +| `metrics_snapshots` | Hourly trend points behind the admin's sparklines (90-day retention) | --- @@ -421,8 +425,8 @@ crons = ["* * * * *"] # Every minute ### Feed Errors -- Feed proxy handles caching and error tracking -- `feed_metadata` tracks `error_count` and `fetch_error` +- The feed proxy (the crawler) owns caching, retries and per-feed error tracking; its `/stats` is where `feedsInError` comes from +- D1 keeps only ingest state: `feeds.last_ingest_at` per feed, `sync_state.crawler_heartbeat_at` for the crawler as a whole ### OAuth Errors diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index d86bc174..3819d64a 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -4,7 +4,25 @@ ## Project Overview -Skyreader backend is a Cloudflare Workers API that serves as a gateway between the frontend and the AT Protocol ecosystem. It handles authentication, RSS feed fetching via a Fly.io proxy, social features, saved articles, labels, and background Jetstream polling. +Skyreader backend is a Cloudflare Workers API that serves as a gateway between the frontend and the AT Protocol ecosystem. It handles authentication, the feed timeline, social features, saved articles, labels, and background Jetstream polling. + +**Feed reads are served from D1, not the proxy.** The Fly.io proxy is the crawler: it pushes new +and edited items into `feed_items` (`POST /api/internal/ingest`), pulls the set of feeds to +crawl (`GET /api/internal/crawl-set`), and reports which feeds are failing to crawl +(`POST /api/internal/feed-health`) — all authenticated with the shared `FEED_PROXY_SECRET` +(fail-closed when unset). A client refresh is one `GET /api/v2/timeline` — a single query joining +subscriptions and read state. Because reads never touch the crawler, a broken feed just goes quiet; +the health report is the only thing that tells a reader its feed is dead rather than idle, and its +payload is the COMPLETE unhealthy set (recovery = absence from the next report). All three internal +endpoints stamp `sync_state.crawler_heartbeat_at`, and the timeline reports `ingestActive` as the +AND of that heartbeat and `sync_state.timeline_enabled` — the operator's rollout gate. A crawler +heartbeat only says a crawler is attached; it arrives seconds into a backfill that takes hours, so +the gate is what actually admits readers to the archive (and, set back to `'0'`, is the fast +rollback that returns every client to the legacy batch path with no deploy). Either half false and +clients stay on `/batch`; the request short-circuits rather than building a page they will discard. +See `docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path for `/api/extract`, feed +discovery, standard.site documents, and social context — plus the one crawl per new subscription +(`warmFeedIntoArchive`) and the subscription-gated pull-through in `/api/v2/feeds/fetch`. ## Key Concepts @@ -43,22 +61,24 @@ See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed documentation. ### Routes -| File | Purpose | -| ----------------------------- | ----------------------------------------------------- | -| `src/routes/auth.ts` | OAuth flow (login, callback, logout, client metadata) | -| `src/routes/feeds-v2.ts` | RSS fetching via Fly.io proxy | -| `src/routes/social.ts` | Content detection for a DID (detect-content) | -| `src/routes/shares.ts` | User shares CRUD (with PDS sync) | -| `src/routes/subscriptions.ts` | Subscription CRUD (with PDS sync) | -| `src/routes/records.ts` | PDS record listing | -| `src/routes/reading.ts` | Article + document read positions (forward delta) | -| `src/routes/labels.ts` | Unified item labels (read/starred/archived/tags) | -| `src/routes/saved.ts` | Saved articles CRUD | -| `src/routes/settings.ts` | User settings | -| `src/routes/sync.ts` | PDS full sync, subscription sync, sync status | -| `src/routes/lexicons.ts` | Serve lexicon schemas at /.well-known/lexicons | -| `src/routes/health.ts` | `/api/health` (shallow) + `/api/health/deep` (gated) | -| `src/routes/telemetry.ts` | `/api/telemetry/error` — sampled client error reports | +| File | Purpose | +| ----------------------------- | ------------------------------------------------------ | +| `src/routes/auth.ts` | OAuth flow (login, callback, logout, client metadata) | +| `src/routes/timeline.ts` | `GET /api/v2/timeline` — the whole refresh, one query | +| `src/routes/ingest.ts` | Crawler endpoints: item ingest, crawl set, feed health | +| `src/routes/feeds-v2.ts` | Single-feed read (D1 + pull-through), discover, docs | +| `src/routes/social.ts` | Social feed, popular, grouped, detect-content | +| `src/routes/shares.ts` | User shares CRUD (with PDS sync) | +| `src/routes/subscriptions.ts` | Subscription CRUD (with PDS sync) | +| `src/routes/records.ts` | PDS record listing | +| `src/routes/reading.ts` | Article + document read positions (forward delta) | +| `src/routes/labels.ts` | Unified item labels (read/starred/archived/tags) | +| `src/routes/saved.ts` | Saved articles CRUD | +| `src/routes/settings.ts` | User settings | +| `src/routes/sync.ts` | PDS full sync, subscription sync, sync status | +| `src/routes/lexicons.ts` | Serve lexicon schemas at /.well-known/lexicons | +| `src/routes/health.ts` | `/api/health` (shallow) + `/api/health/deep` (gated) | +| `src/routes/telemetry.ts` | `/api/telemetry/error` — sampled client error reports | ### Services @@ -91,7 +111,7 @@ never `Sentry.captureException` directly, so the vendor stays a one-file decisio Logging: use `log.*` with a stable low-cardinality `event` slug and put the details in fields. Workers Logs indexes the fields of a logged object but treats a string -as opaque text, so `log.info('feed_fetched', { feedCount })` is queryable and an +as opaque text, so `log.info('feed_ingested', { itemCount })` is queryable and an interpolated `console.log` sentence is not. Never log a credential — there is no redaction layer on this path. @@ -102,8 +122,11 @@ and outbound feed-proxy calls. The every-minute cron also _records_: poller lag and cron liveness to `system_status`, the proxy's cache stats every 5th minute, and an hourly row in `metrics_snapshots` (pruned at 90 days). The admin renders those tables directly. -Recording failures never withhold the cron heartbeat — losing a data point is not -an outage; see the note on `runRecordingStep()`. +The snapshot's counts come from D1 (`users`, `feeds`, `feed_items`, …) and its +health numbers from the `system_status` rows — including `feeds_with_errors`, +which is the crawler's `feedsInError` and is recorded NULL, not 0, when the proxy +row is stale. Recording failures never withhold the cron heartbeat — losing a data +point is not an outage; see the note on `runRecordingStep()`. The browser reports its own errors to `/api/telemetry/error` (unauthenticated by design — an error on the login screen counts, and the route is answered before @@ -117,9 +140,9 @@ procedures: [`docs/RUNBOOK.md`](../docs/RUNBOOK.md). ### Durable Objects -| File | Purpose | -| ----------------------------------------- | ----------------------------------------------------------------- | -| `src/durable-objects/jetstream-poller.ts` | Jetstream firehose for `app.skyreader.feed.subscription` (alarms) | +| File | Purpose | +| ----------------------------------------- | ----------------------------------------------------- | +| `src/durable-objects/jetstream-poller.ts` | Long-running Jetstream firehose connection via alarms | ### Storage @@ -131,21 +154,24 @@ Key tables: - `sessions` - Server-side sessions (tokens, DPoP key, expiry) - `subscriptions_cache` - Cached feed subscriptions from PDS - `shares` - Aggregated share data from Jetstream -- `feed_metadata` - Feed caching metadata (ETags, errors, shard_id) -- `feed_cache` - Parsed feed cache -- `feed_items` - Individual feed items -- `documents` - orphaned. Nothing reads or writes it since documents moved to - on-demand proxy fetch; the table is left in place (as `shares` was) rather than - dropped -- `publications_cache` - orphaned too. The publication metadata cache it backed - now lives in the feed proxy (`feed-proxy/src/standard-site.ts`) +- `feeds` - One row per crawled feed (title/site/image + `last_ingest_at`), plus the crawler's + health verdict: `error_count`/`last_error`/`next_retry_at`/`last_fetch_at` (unix seconds), which + the timeline serves to readers as `feedHealth`, and `crawl_stale` for a feed the crawler isn't + reaching at all (operator-only; the admin alarms on it). `last_ingest_at` is publishing cadence, + NOT health — it only moves when a fetch yields a new item +- `feed_items` - The feed archive the timeline serves: every item the crawler has ever pushed, + keyed `(feed_url, guid)` with a monotonic `seq`. Never pruned in ordinary operation — see + `docs/plans/D1_FEED_TIMELINE.md` +- `documents` / `publications_cache` - orphaned after standard.site document reads moved to the + feed proxy; retained in place but no longer read or written - `item_labels_cache` - Unified labels (read/starred/archived/tags) - `saved_articles` - Saved/bookmarked articles - `social_read_positions_cache` - Legacy social read tracking (superseded; document reads now live in `item_labels_cache` as `item_type='document'`/`label='read'`) - `user_settings` - User preferences - `rate_limits` - Per-user rate limiting -- `sync_state` - Jetstream cursor and other sync state +- `sync_state` - Jetstream cursor, the archive generation token, the crawler heartbeat, and + `timeline_enabled` (the rollout gate: only an explicit `'0'` holds clients on the batch path) - `system_status` - Cron-written health board (cron liveness, poller lag, proxy stats) - `metrics_snapshots` - Hourly trend points behind the admin's sparklines (90-day retention) @@ -181,7 +207,10 @@ FEED_PROXY_URL = "https://skyreader-feed-proxy.fly.dev" SENTRY_ENVIRONMENT = "production" # "staging" in [env.staging] # Secrets (set via `wrangler secret put`): -# FEED_PROXY_SECRET - authenticates with the Fly.io feed proxy +# FEED_PROXY_SECRET - shared with the Fly.io proxy, both directions: outbound +# proxy calls and the inbound crawler endpoints +# (/api/internal/ingest, /api/internal/crawl-set), which +# are fail-closed when it is unset # SENTRY_DSN - error reporting (unset ⇒ silent no-op) # HEARTBEAT_URL - dead-man ping for the every-minute cron # HEALTH_CHECK_SECRET - gates /api/health/deep (X-Health-Secret header) diff --git a/backend/migrations/0068_feed_timeline.sql b/backend/migrations/0068_feed_timeline.sql new file mode 100644 index 00000000..620c26d1 --- /dev/null +++ b/backend/migrations/0068_feed_timeline.sql @@ -0,0 +1,57 @@ +-- D1-served feed timeline (ingest-push architecture). +-- +-- The Fly proxy stays the crawler but leaves the read path entirely: it pushes +-- new/edited items into the tables below, and a client refresh becomes ONE query +-- (feed_items JOIN subscriptions_cache LEFT JOIN item_labels_cache) instead of +-- N batched Worker->Fly hops plus chunked read-key lookups. +-- +-- Retention: D1 is the ARCHIVE. Ordinary ingest never deletes an item; the proxy's +-- own K=200 log is only the outbox (a delivery window). The single trim that +-- exists is a per-feed sanity cap (5,000) that healthy feeds never reach — see +-- routes/ingest.ts. + +-- The dormant pre-Fly trio. Zero references in backend/src; feed_cache was +-- already wiped to '{}' by 0018. Dropped so the names are free for the new shapes. +DROP TABLE IF EXISTS feed_items; +DROP TABLE IF EXISTS feed_cache; +DROP TABLE IF EXISTS feed_metadata; + +-- One row per crawled feed: metadata for the reader plus ingest observability. +CREATE TABLE feeds ( + feed_url TEXT PRIMARY KEY, + title TEXT, + site_url TEXT, + description TEXT, + image_url TEXT, + last_ingest_at INTEGER, + created_at INTEGER NOT NULL DEFAULT (unixepoch()) +); + +-- The durable item archive. Same invariants as the proxy's proven log: +-- AUTOINCREMENT seq is a never-reused monotonic cursor, an edit updates in place +-- (seq unchanged, so it is not re-delivered), content_hash is NOT NULL because a +-- NULL would make the edit predicate (`<>`) silently never match. +CREATE TABLE feed_items ( + seq INTEGER PRIMARY KEY AUTOINCREMENT, + feed_url TEXT NOT NULL, + guid TEXT NOT NULL, + item_json TEXT NOT NULL, + published_at INTEGER, + first_seen_at INTEGER NOT NULL, + content_hash TEXT NOT NULL, + UNIQUE(feed_url, guid) +); + +CREATE INDEX idx_feed_items_feed_seq ON feed_items(feed_url, seq); +CREATE INDEX idx_feed_items_feed_published_seq + ON feed_items(feed_url, published_at DESC, seq DESC); + +-- The timeline join probes subscriptions by (user_did, feed_url). +CREATE INDEX IF NOT EXISTS idx_subscriptions_cache_user_feed + ON subscriptions_cache(user_did, feed_url); + +-- Generation token for the archive (D1 recreation / Time Travel restore guard). +-- Clients store it beside their cursor and cold-start on mismatch. Any D1 restore +-- must bump this (seqs rewind while the token would otherwise stay the same). +INSERT OR IGNORE INTO sync_state (key, value) +VALUES ('items_generation', lower(hex(randomblob(16)))); diff --git a/backend/migrations/0069_metrics_snapshots_feeds_in_error.sql b/backend/migrations/0069_metrics_snapshots_feeds_in_error.sql new file mode 100644 index 00000000..86d30254 --- /dev/null +++ b/backend/migrations/0069_metrics_snapshots_feeds_in_error.sql @@ -0,0 +1,39 @@ +-- `feeds_with_errors` loses its NOT NULL, because its source changed. +-- +-- It used to be `SELECT COUNT(*) FROM feed_metadata WHERE error_count > 0` — a +-- local count that always had an answer. 0068 drops `feed_metadata`: the new +-- `feeds` table carries ingest metadata only (`last_ingest_at`), and per-feed +-- fetch errors now live where the fetching happens, in the crawler. The hourly +-- snapshot therefore reads the same number the live tile does, the proxy's +-- `feedsInError`, out of the `proxy_stats` row the cron writes every 5 minutes. +-- +-- That number can genuinely be unknown — an unreachable proxy leaves the row +-- stale and the snapshot has nothing to record. Storing 0 there would draw a +-- healthy flat line through exactly the outage the column exists to show, so +-- the column joins `firehose_lag_ms` and `proxy_fresh_pct` in being nullable. +-- +-- SQLite can't drop a NOT NULL in place, hence the rebuild. `captured_at` stays +-- the INTEGER PRIMARY KEY (= rowid), so range scans by time stay index-free. +CREATE TABLE metrics_snapshots_new ( + captured_at INTEGER PRIMARY KEY, + users INTEGER NOT NULL, + feeds INTEGER NOT NULL, + feed_items INTEGER NOT NULL, + subscriptions INTEGER NOT NULL, + saved_articles INTEGER NOT NULL, + feeds_with_errors INTEGER, + active_sessions INTEGER NOT NULL, + firehose_lag_ms INTEGER, + proxy_fresh_pct REAL +); + +INSERT INTO metrics_snapshots_new + (captured_at, users, feeds, feed_items, subscriptions, saved_articles, + feeds_with_errors, active_sessions, firehose_lag_ms, proxy_fresh_pct) +SELECT captured_at, users, feeds, feed_items, subscriptions, saved_articles, + feeds_with_errors, active_sessions, firehose_lag_ms, proxy_fresh_pct + FROM metrics_snapshots; + +DROP TABLE metrics_snapshots; + +ALTER TABLE metrics_snapshots_new RENAME TO metrics_snapshots; diff --git a/backend/migrations/0070_feed_health.sql b/backend/migrations/0070_feed_health.sql new file mode 100644 index 00000000..f108cb9b --- /dev/null +++ b/backend/migrations/0070_feed_health.sql @@ -0,0 +1,46 @@ +-- Per-feed crawl health in the archive. +-- +-- On the legacy batch path the client learned a feed was broken from the proxy's +-- own response (`status: 'error'`, errorCount, nextRetryAt). The timeline path +-- never touches the proxy, so that signal disappeared: a feed failing to crawl +-- simply delivered no items, which is indistinguishable from a quiet feed. The +-- crawler now reports health here (POST /api/internal/feed-health) and the +-- timeline hands it to the client alongside its items. +-- +-- Units: unix SECONDS, like `last_ingest_at` and the rest of the backend. The +-- proxy stores these as milliseconds and converts on send; the timeline converts +-- back to milliseconds on the way out, because that is what the client's +-- FeedStatus contract has always used. + +ALTER TABLE feeds ADD COLUMN error_count INTEGER NOT NULL DEFAULT 0; +ALTER TABLE feeds ADD COLUMN last_error TEXT; +ALTER TABLE feeds ADD COLUMN last_error_at INTEGER; +ALTER TABLE feeds ADD COLUMN next_retry_at INTEGER; +-- Last time the crawler successfully FETCHED the feed. Distinct from +-- `last_ingest_at`, which only moves when a fetch produced a new or edited item: +-- a healthy feed that hasn't published in a month still gets fetched hourly. +ALTER TABLE feeds ADD COLUMN last_fetch_at INTEGER; +-- The crawler has this feed in its crawl set but hasn't managed to fetch it in +-- hours — starved by a saturated warm loop rather than failing outright. Distinct +-- from `error_count`: nothing is erroring, the crawler simply never gets to it. +-- +-- This is the difference between an alarm and noise. `last_ingest_at` only moves +-- when a fetch yields a NEW item, so "no ingest in an hour" describes every +-- weekly blog in the archive; the admin's old stale-feed metric was therefore +-- warning permanently and meant nothing. Being in the crawl set and un-fetched is +-- a genuine fault, and it is the one the current warm-loop capacity work needs. +ALTER TABLE feeds ADD COLUMN crawl_stale INTEGER NOT NULL DEFAULT 0; + +-- The trouble set is a handful of rows out of the whole archive, so let the +-- per-user health lookup, the recovery sweep and the admin's counts start from it +-- rather than scanning `feeds`. Recovery is "currently flagged, and absent from +-- the latest report", which this index makes cheap to enumerate — the +-- alternative, a NOT IN list of every healthy feed, would blow the +-- bound-parameter limit at ~1,300. +CREATE INDEX idx_feeds_unhealthy ON feeds(feed_url) WHERE error_count > 0 OR crawl_stale = 1; + +-- Bumped whenever a health report actually changes something. Clients echo the +-- revision they last saw and the timeline only re-sends the health payload when +-- it differs, so the steady-state poll stays at one query. Only `error_count` +-- feeds into it: `crawl_stale` is an operator signal, invisible to readers. +INSERT OR IGNORE INTO sync_state (key, value) VALUES ('feed_health_rev', '0'); diff --git a/backend/migrations/0071_timeline_rollout_gate.sql b/backend/migrations/0071_timeline_rollout_gate.sql new file mode 100644 index 00000000..8f19dc38 --- /dev/null +++ b/backend/migrations/0071_timeline_rollout_gate.sql @@ -0,0 +1,30 @@ +-- Rollout gate for the D1-served timeline. +-- +-- `ingestActive` alone flips every client onto the timeline the instant the +-- crawler first checks in — which is the moment the backfill STARTS, not the +-- moment it finishes. The proxy stamps a heartbeat on its very first crawl-set +-- pull (seconds after the release), so without this gate every reader cold-starts +-- against a nearly-empty archive and then drags the entire ~200-items-per-feed +-- backfill through its incremental drain: the expensive global scan, at its worst +-- case, for as long as the backfill takes. +-- +-- This decouples the two. Enable INGEST_URL, let the archive fill, watch the +-- proxy's `ingest.pending` trend to ~0, THEN flip this to '1'. It is also the +-- only fast way back: setting it to '0' returns every client to the legacy batch +-- path on their next poll, with no Worker deploy and no waiting out the 30-minute +-- heartbeat freshness window. +-- +-- npx wrangler d1 execute skyreader --remote \ +-- --command "UPDATE sync_state SET value='1', updated_at=unixepoch() WHERE key='timeline_enabled'" +-- +-- Only an explicit '0' gates: an absent row means enabled, so a hand-built schema +-- or a future environment can't be silently held on the legacy path. +-- +-- The value depends on whether this deployment already has readers. An existing +-- one (prod, staging) has users and a backfill ahead of it, so it starts gated and +-- an operator opens it deliberately. A fresh one (local dev, e2e, CI) has no users +-- and no backlog, so it starts open and nothing has to know this key exists. +INSERT OR IGNORE INTO sync_state (key, value, updated_at) +SELECT 'timeline_enabled', + CASE WHEN EXISTS (SELECT 1 FROM users) THEN '0' ELSE '1' END, + unixepoch(); diff --git a/backend/src/index.ts b/backend/src/index.ts index 202c584f..63601262 100644 --- a/backend/src/index.ts +++ b/backend/src/index.ts @@ -16,6 +16,8 @@ import { handleV2Mentions, handleV2MentionLane, } from './routes/feeds-v2'; +import { handleIngest, handleCrawlSet, handleFeedHealth } from './routes/ingest'; +import { handleTimeline } from './routes/timeline'; import { handleDetectContent } from './routes/social'; import { handleCreateLinkblogShare, @@ -315,10 +317,28 @@ async function route( response = await handleAuthMe(request, env); break; + // Internal crawler endpoints authenticate with FEED_PROXY_SECRET in their + // handlers; they intentionally do not require a user session. + case url.pathname === '/api/internal/ingest': + response = await handleIngest(request, env); + break; + case url.pathname === '/api/internal/crawl-set': + response = await handleCrawlSet(request, env); + break; + case url.pathname === '/api/internal/feed-health': + response = await handleFeedHealth(request, env); + break; + + // The reader's whole refresh, served from the D1 archive in one query. + case url.pathname === '/api/v2/timeline': + if (!session) return unauthorizedResponse(headers); + response = await handleTimeline(request, env, session); + break; + // Feed routes (v2 via Fly.io proxy) case url.pathname === '/api/v2/feeds/fetch': if (!session) return unauthorizedResponse(headers); - response = await handleV2FeedFetch(request, env); + response = await handleV2FeedFetch(request, env, session); break; case url.pathname === '/api/v2/feeds/batch': if (!session) return unauthorizedResponse(headers); diff --git a/backend/src/observability/ops-metrics.ts b/backend/src/observability/ops-metrics.ts index 3d9c52db..a905f1f7 100644 --- a/backend/src/observability/ops-metrics.ts +++ b/backend/src/observability/ops-metrics.ts @@ -270,8 +270,9 @@ export async function recordCronRun( } /** - * One row per hour: the counts the admin shows as tiles, plus the two health - * numbers that only exist in `system_status`. Same job prunes the tail, so the + * One row per hour: the counts the admin shows as tiles, plus the three health + * numbers that only exist in `system_status` (firehose lag, proxy cache + * freshness, feeds the crawler has in error). Same job prunes the tail, so the * table can't grow without an owner. */ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise { @@ -282,11 +283,10 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise< const counts = await env.DB.batch<{ count: number }>([ env.DB.prepare('SELECT COUNT(*) AS count FROM users'), - env.DB.prepare('SELECT COUNT(*) AS count FROM feed_metadata'), + env.DB.prepare('SELECT COUNT(*) AS count FROM feeds'), env.DB.prepare('SELECT COUNT(*) AS count FROM feed_items'), env.DB.prepare('SELECT COUNT(*) AS count FROM subscriptions_cache'), env.DB.prepare('SELECT COUNT(*) AS count FROM saved_articles'), - env.DB.prepare('SELECT COUNT(*) AS count FROM feed_metadata WHERE error_count > 0'), env.DB.prepare('SELECT COUNT(*) AS count FROM sessions WHERE expires_at > ?').bind( sessionCutoff ), @@ -300,6 +300,11 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise< const usable = (row: StatusRow | null): T | null => row && now - row.updatedAt <= SNAPSHOT_MAX_AGE_MS ? row.value : null; + // Per-feed fetch errors live in the crawler, not in D1: `feeds` carries ingest + // metadata only. So the trend point reads the same number the live tile does, + // and is null — not 0 — when the proxy row is missing or stale (see 0069). + const feedsWithErrors = usable(proxy)?.feedsInError ?? null; + await env.DB.prepare( `INSERT OR REPLACE INTO metrics_snapshots (captured_at, users, feeds, feed_items, subscriptions, saved_articles, @@ -313,8 +318,8 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise< at(2), at(3), at(4), + feedsWithErrors, at(5), - at(6), usable(poller)?.lagMs ?? null, usable(proxy)?.freshPct ?? null ) @@ -328,7 +333,7 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise< capturedAt, users: at(0), feeds: at(1), - feedsWithErrors: at(5), + feedsWithErrors, prunedRows: pruned.meta?.changes ?? 0, }); } diff --git a/backend/src/routes/feeds-v2.ts b/backend/src/routes/feeds-v2.ts index 87d2eb89..6dfbbf13 100644 --- a/backend/src/routes/feeds-v2.ts +++ b/backend/src/routes/feeds-v2.ts @@ -8,6 +8,8 @@ import type { } from '../services/feed-proxy-client'; import { resolveStandardSite } from '../utils/canonical-url'; import { chunkArray, getReadKeys } from './reading'; +import { clearFeedHealth, ingestProxyFeed } from './ingest'; +import { readFeedMetadata, readFeedSlice, type FeedHealth } from './timeline'; import { getLinkblogTargets, publicationUri as linkblogPublicationUri, @@ -19,10 +21,11 @@ interface V2FeedResponse { siteUrl?: string; imageUrl?: string; items: FeedItem[]; + // Unix ms of the last ingest for this feed (freshness, for the client's UI) — + // no longer a live upstream fetch time, since reads never touch the proxy. fetchedAt: number; - cursor?: number; - generation?: string; - hasMore?: boolean; + // Present only when the crawler currently considers this feed broken. + health?: FeedHealth; } interface V2BatchFeedResult { @@ -52,22 +55,59 @@ interface V2BatchResponse { readCursor?: number; } +// Newest-N a single-feed fetch delivers from the D1 archive. +const SINGLE_FEED_LIMIT = 30; +const SINGLE_FEED_MAX_LIMIT = 200; + +/** + * Does this user hold a subscription to this feed? The gate on every write into + * the shared archive that a user request can trigger. Parked (`active = 0`) subs + * count — the user owns the feed either way, and re-activating it shouldn't need + * a different code path. + */ +async function callerSubscribes(env: Env, userDid: string, feedUrl: string): Promise { + const row = await env.DB.prepare( + `SELECT 1 AS ok FROM subscriptions_cache WHERE user_did = ? AND feed_url = ? LIMIT 1` + ) + .bind(userDid, feedUrl) + .first<{ ok: number }>(); + return !!row; +} + /** * GET /api/v2/feeds/fetch * - * Fetch a single feed via Fly.io proxy with GUID-based incremental sync. + * One feed's newest slice, served from the D1 archive with read state joined in + * (the timeline's per-feed sibling). This is the "new subscription gap" path: + * a feed the user just subscribed to contributes nothing to the global timeline + * cursor (its items sit below it), so the client fetches it directly here. + * + * If D1 has nothing for the feed (nobody was subscribed, so the crawler never + * pushed it), we PULL THROUGH: fetch it from the proxy once, ingest the result, + * then serve from D1. Steady state never touches Fly. + * + * The pull-through only runs for a feed the CALLER actually subscribes to. The + * archive is shared and (by design) never pruned, so an open ingest surface would + * let any authenticated user write arbitrary feeds into it forever; requiring a + * subscription bounds writes to each user's own feed list. Subscriptions are + * written to `subscriptions_cache` synchronously by POST /api/subscriptions + * before the client fetches, so the add-feed path is unaffected. * * Query params: * - url: Feed URL (required) - * - since_guids: Comma-separated GUIDs the client already has (optional) - * - limit: Max items to return (optional, default 100) + * - limit: Max items to return (optional, default 30) + * - refresh: `1` to force the pull-through even when the archive already has the + * feed — the "retry this feed" action, the one path that still asks the + * crawler for a fresh fetch on demand. + * - since_guids: accepted and ignored (legacy); the client dedupes by GUID. */ -export async function handleV2FeedFetch(request: Request, env: Env): Promise { +export async function handleV2FeedFetch( + request: Request, + env: Env, + session: Session +): Promise { const url = new URL(request.url); const feedUrl = url.searchParams.get('url'); - const sinceGuidsParam = url.searchParams.get('since_guids'); - const sinceSeqParam = url.searchParams.get('since_seq'); - const generationParam = url.searchParams.get('generation'); const limitParam = url.searchParams.get('limit'); if (!feedUrl) { @@ -87,28 +127,62 @@ export async function handleV2FeedFetch(request: Request, env: Env): Promise 0 + ? { + errorCount: metadata.error_count, + error: metadata.last_error ?? undefined, + lastErrorAt: metadata.last_error_at ? metadata.last_error_at * 1000 : undefined, + nextRetryAt: metadata.next_retry_at ? metadata.next_retry_at * 1000 : undefined, + lastFetchedAt: metadata.last_fetch_at ? metadata.last_fetch_at * 1000 : undefined, + } + : undefined, }; return new Response(JSON.stringify(response), { @@ -790,26 +864,31 @@ export interface WarmCacheResult { } /** - * Warm up the proxy cache for a single feed. - * Just fetches via proxy to ensure it's cached - no D1 storage. + * Fetch a newly subscribed feed through the proxy and INGEST it into the archive. + * + * This used to only warm the proxy's cache and throw the parse away. Now that D1 + * is the read path, the same one fetch we were already paying for at subscribe + * time populates the archive — so the client's first per-feed read is a plain D1 + * query instead of another synchronous crawl through the pull-through. */ -export async function warmProxyCache(env: Env, feedUrl: string): Promise { +export async function warmFeedIntoArchive(env: Env, feedUrl: string): Promise { try { const client = new FeedProxyClient(env); const feed = await client.fetchFeed(feedUrl); + await ingestProxyFeed(env, feedUrl, feed); return { success: true, itemCount: feed.items.length }; } catch (error) { const errorMessage = error instanceof Error ? error.message : 'Failed to fetch feed'; - console.error(`[warmProxyCache] Error fetching ${feedUrl}:`, errorMessage); + console.error(`[warmFeedIntoArchive] Error fetching ${feedUrl}:`, errorMessage); return { success: false, error: errorMessage }; } } /** - * Warm up the proxy cache for multiple feeds in a single batch request. - * Just fetches via proxy to ensure they're cached - no D1 storage. + * The same, for several feeds in one proxy batch request (bulk / OPML import). + * A feed whose ingest fails is reported as an error but never fails the others. */ -export async function warmProxyCacheBatch( +export async function warmFeedsIntoArchive( env: Env, feedUrls: string[] ): Promise> { @@ -834,15 +913,20 @@ export async function warmProxyCacheBatch( } else if (feedResult.status === 'error') { results[feedUrl] = { success: false, error: feedResult.error }; } else { - results[feedUrl] = { - success: true, - itemCount: feedResult.items.length, - }; + try { + await ingestProxyFeed(env, feedUrl, feedResult); + results[feedUrl] = { success: true, itemCount: feedResult.items.length }; + } catch (error) { + results[feedUrl] = { + success: false, + error: error instanceof Error ? error.message : 'Ingest failed', + }; + } } } } catch (error) { const errorMessage = error instanceof Error ? error.message : 'Batch fetch failed'; - console.error(`[warmProxyCacheBatch] Batch error:`, errorMessage); + console.error(`[warmFeedsIntoArchive] Batch error:`, errorMessage); for (const feedUrl of feedUrls) { results[feedUrl] = { success: false, error: errorMessage }; diff --git a/backend/src/routes/ingest.ts b/backend/src/routes/ingest.ts new file mode 100644 index 00000000..28ccc783 --- /dev/null +++ b/backend/src/routes/ingest.ts @@ -0,0 +1,621 @@ +import type { Env, FeedItem } from '../types'; + +/** + * Internal (proxy → Worker) endpoints for the D1-served feed timeline. + * + * The Fly proxy is the crawler; it pushes deltas from its durable log here and + * pulls the set of feeds it should crawl. Neither endpoint has a user session — + * both are authenticated with the shared `FEED_PROXY_SECRET` (the same value the + * Worker sends outbound as `X-Proxy-Secret`), so no new secret exists to manage. + */ + +// Per-feed sanity cap. D1 is an archive: ordinary ingest never prunes. This trim +// exists only to bound a pathological feed (GUID churn re-minting ids every +// fetch, calendar feeds reposting their whole window). A daily-cadence feed takes +// ~14 years to reach it, so a feed at the cap is a bug signal, not steady state. +export const SANITY_CAP = 5000; + +// Stored-content cap per item. Unbounded retention makes this mandatory rather +// than optional: D1 has a hard 10 GB database ceiling, so storage grows with +// ingest velocity × item size × time. Oversized bodies are dropped at ingest +// (summary/title/url/image kept) and `contentTruncated` is set, which the reader +// acts on: ArticleCard auto-extracts the full text via /api/extract when a +// truncated article is opened, so the body the user sees is still the whole +// article (see frontend/src/lib/components/ArticleCard.svelte). +export const MAX_ITEM_CONTENT_BYTES = 8 * 1024; + +// How long a crawler heartbeat stays "fresh". The proxy pulls the crawl set every +// 5 minutes whenever INGEST_URL is set, so a stamp older than this means this +// environment has no crawler pushing into its D1 — the timeline says so and +// clients stay on the legacy batch path instead of serving an empty archive. +export const CRAWLER_HEARTBEAT_KEY = 'crawler_heartbeat_at'; +export const CRAWLER_HEARTBEAT_FRESH_SECONDS = 30 * 60; + +// The rollout gate (migration 0071). A crawler heartbeat says a crawler is +// ATTACHED; it says nothing about whether the archive it fills is complete, and +// the first heartbeat lands seconds into a backfill that takes hours. This key is +// what actually moves readers onto the timeline, so the two can be sequenced: +// enable ingest, let the archive fill, then open the gate. Setting it back to '0' +// is also the fast rollback — every client is on the batch path at its next poll. +// +// Only an explicit '0' gates. An absent row means enabled, so a hand-built schema +// (the unit-test harness) or a new environment is never silently held back. +export const TIMELINE_ENABLED_KEY = 'timeline_enabled'; + +// Revision token for the set of feeds the crawler currently considers broken. +// The timeline sends the per-feed health payload only when the client's echoed +// revision differs from this, so a steady-state poll costs no extra query. +export const FEED_HEALTH_REV_KEY = 'feed_health_rev'; + +// Bounds on one ingest call. The pusher chunks to ~100 items; these are abuse +// guards, not tuning knobs. +const MAX_INGEST_ITEMS = 1000; +const MAX_INGEST_BODY_BYTES = 8 * 1024 * 1024; + +// D1 caps statements per batch; keep well under it. +const INGEST_BATCH_SIZE = 50; + +/** + * Subscriptions that aren't RSS feeds (standard.site documents, collections) + * never belong to the crawl set or the timeline — their `feed_url` is an at:// + * URI, and the client skips every `atproto.*` source on the feed path too. + */ +export function rssSubscriptionPredicate(alias = ''): string { + const column = alias ? `${alias}.source_type` : 'source_type'; + return `(${column} IS NULL OR ${column} NOT LIKE 'atproto.%')`; +} + +export interface IngestFeed { + feedUrl: string; + title?: string | null; + siteUrl?: string | null; + description?: string | null; + imageUrl?: string | null; +} + +export interface IngestItem { + feedUrl: string; + guid: string; + item: FeedItem; + publishedAt?: number | null; + firstSeenAt: number; + contentHash: string; +} + +/** + * Timing-safe secret comparison. Fails closed: an unset `FEED_PROXY_SECRET` + * rejects every request rather than turning the ingest endpoint into an open + * write surface (local dev sets the pair explicitly — see scripts/dev-local.sh). + */ +export function isAuthorizedProxyRequest(request: Request, env: Env): boolean { + const expected = env.FEED_PROXY_SECRET; + if (!expected) return false; + const provided = request.headers.get('X-Proxy-Secret'); + if (!provided || provided.length !== expected.length) return false; + let diff = 0; + for (let i = 0; i < expected.length; i++) { + diff |= provided.charCodeAt(i) ^ expected.charCodeAt(i); + } + return diff === 0; +} + +function unauthorized(): Response { + return new Response(JSON.stringify({ error: 'Unauthorized' }), { + status: 401, + headers: { 'Content-Type': 'application/json' }, + }); +} + +function badRequest(message: string, status = 400): Response { + return new Response(JSON.stringify({ error: message }), { + status, + headers: { 'Content-Type': 'application/json' }, + }); +} + +/** + * Apply the stored-content cap. Returns the item to persist: unchanged when it + * fits, otherwise the same item with `content` dropped and `contentTruncated` + * set. `content_hash` is computed by the proxy over the FULL item, so truncation + * never confuses edit detection. + */ +export function capItemContent(item: FeedItem): FeedItem { + const content = item.content; + if (!content) return item; + // Byte length, not code units — the cap is about stored bytes. + const bytes = new TextEncoder().encode(content).length; + if (bytes <= MAX_ITEM_CONTENT_BYTES) return item; + const { content: _dropped, ...rest } = item; + return { ...rest, contentTruncated: true }; +} + +/** + * Stable hash of an item's mutable content — byte-for-byte the same function the + * proxy applies before pushing (`itemContentHash` in feed-proxy/src/app.ts), so + * an item ingested by the subscribe-time pull-through and later pushed by the + * crawler hashes identically and doesn't register as a spurious edit. + */ +export async function computeContentHash(item: FeedItem): Promise { + const payload = `${item.title}|${item.url}|${item.content ?? ''}|${item.summary ?? ''}`; + const digest = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(payload)); + return [...new Uint8Array(digest)] + .map((b) => b.toString(16).padStart(2, '0')) + .join('') + .slice(0, 16); +} + +/** + * The write itself, shared by the pushed-batch endpoint and the subscribe-time + * pull-through in feeds-v2.ts. Idempotent: a re-pushed item with the same content + * hash is a no-op, a changed one updates in place (seq unchanged → not + * re-delivered to clients that already saw it). That makes the proxy's + * at-least-once delivery safe. + * + * Throws on a D1 write failure so callers can surface a 5xx (which leaves the + * pusher's outbox state untouched, so it retries). + */ +export async function ingestBatch( + env: Env, + feeds: IngestFeed[], + items: IngestItem[] +): Promise<{ inserted: number; updated: number }> { + const now = Math.floor(Date.now() / 1000); + const statements: D1PreparedStatement[] = []; + + for (const feed of feeds) { + if (!feed?.feedUrl) continue; + statements.push( + env.DB.prepare( + `INSERT INTO feeds (feed_url, title, site_url, description, image_url, last_ingest_at) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(feed_url) DO UPDATE SET + title = COALESCE(excluded.title, feeds.title), + site_url = COALESCE(excluded.site_url, feeds.site_url), + description = COALESCE(excluded.description, feeds.description), + image_url = COALESCE(excluded.image_url, feeds.image_url), + last_ingest_at = excluded.last_ingest_at` + ).bind( + feed.feedUrl, + feed.title ?? null, + feed.siteUrl ?? null, + feed.description ?? null, + feed.imageUrl ?? null, + now + ) + ); + } + + // Max seq before this batch: everything above it in the RETURNING rows below is + // a fresh insert, everything at or below is an edit-in-place. Purely for the + // response's counters — no correctness depends on it. + const before = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{ + max_seq: number | null; + }>(); + const maxSeqBefore = before?.max_seq ?? 0; + + const touchedFeeds = new Set(); + for (const entry of items) { + if (!entry?.feedUrl || !entry.guid || !entry.item || !entry.contentHash) continue; + touchedFeeds.add(entry.feedUrl); + statements.push( + env.DB.prepare( + `INSERT INTO feed_items (feed_url, guid, item_json, published_at, first_seen_at, content_hash) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(feed_url, guid) DO UPDATE SET + item_json = excluded.item_json, + content_hash = excluded.content_hash + WHERE feed_items.content_hash <> excluded.content_hash + RETURNING seq` + ).bind( + entry.feedUrl, + entry.guid, + JSON.stringify(capItemContent(entry.item)), + entry.publishedAt ?? null, + entry.firstSeenAt || Date.now(), + entry.contentHash + ) + ); + } + + let inserted = 0; + let updated = 0; + for (let i = 0; i < statements.length; i += INGEST_BATCH_SIZE) { + const results = await env.DB.batch<{ seq: number }>(statements.slice(i, i + INGEST_BATCH_SIZE)); + for (const result of results) { + for (const row of result.results ?? []) { + if (row.seq > maxSeqBefore) inserted++; + else updated++; + } + } + } + + await trimFeedsToSanityCap(env, [...touchedFeeds]); + + return { inserted, updated }; +} + +/** + * The ONLY pruning that happens at ingest. Under the cap the OFFSET subquery + * yields no row, so `seq <= NULL` matches nothing and the DELETE is a no-op + * costing a bounded index scan — which is the expected case for every healthy + * feed. `cap` is a parameter so tests can exercise the trim without writing + * 5,000 rows. + */ +export async function trimFeedsToSanityCap( + env: Env, + feedUrls: string[], + cap = SANITY_CAP +): Promise { + if (feedUrls.length === 0) return; + const trims = feedUrls.map((feedUrl) => + env.DB.prepare( + `DELETE FROM feed_items + WHERE feed_url = ?1 + AND seq <= (SELECT seq FROM feed_items WHERE feed_url = ?1 + ORDER BY seq DESC LIMIT 1 OFFSET ?2)` + ).bind(feedUrl, cap) + ); + for (let i = 0; i < trims.length; i += INGEST_BATCH_SIZE) { + await env.DB.batch(trims.slice(i, i + INGEST_BATCH_SIZE)); + } +} + +/** + * Ingest a feed fetched straight from the proxy (the subscribe-time warm/ingest + * and the pull-through in feeds-v2.ts). Hashes match what the crawler will push + * later, so the first real push over these rows is a no-op rather than a phantom + * edit. + * + * Order matters: a proxy feed is newest-first, and `seq` is assigned in insert + * order, so we walk the array BACKWARDS — oldest first — exactly as the proxy's + * own `writeFeedItems` does. Ingesting forward would give the newest item the + * lowest seq, and since a re-push of an unchanged item is a no-op, that + * inversion would never heal: every later per-feed cold start (`ORDER BY seq + * DESC`) would serve the feed's OLDEST items, and the sanity-cap trim (deletes + * the lowest seqs) would delete its newest. + */ +export async function ingestProxyFeed( + env: Env, + feedUrl: string, + feed: { + title?: string; + description?: string; + siteUrl?: string; + imageUrl?: string; + items: FeedItem[]; + } +): Promise { + const nowMs = Date.now(); + const oldestFirst = [...feed.items].reverse(); + const items: IngestItem[] = await Promise.all( + oldestFirst.map(async (item) => { + const publishedMs = new Date(item.publishedAt).getTime(); + return { + feedUrl, + guid: item.guid, + item, + publishedAt: Number.isNaN(publishedMs) ? null : publishedMs, + firstSeenAt: nowMs, + contentHash: await computeContentHash(item), + }; + }) + ); + + await ingestBatch( + env, + [ + { + feedUrl, + title: feed.title ?? null, + siteUrl: feed.siteUrl ?? null, + description: feed.description ?? null, + imageUrl: feed.imageUrl ?? null, + }, + ], + items + ); +} + +/** + * Record that this environment's crawler just talked to us. Both internal + * endpoints stamp it, so the signal survives a quiet period with no new items + * (the crawl-set pull runs every 5 minutes regardless of what the feeds do). + * + * This is what makes "is ingest live here?" server-authoritative instead of + * inferred from an empty archive: a Worker whose proxy has no INGEST_URL never + * gets a stamp, so `/api/v2/timeline` reports `ingestActive: false` and clients + * keep using the legacy batch path rather than committing a cursor against an + * archive nothing is filling. + */ +export async function stampCrawlerHeartbeat(env: Env): Promise { + try { + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(CRAWLER_HEARTBEAT_KEY, String(Math.floor(Date.now() / 1000))) + .run(); + } catch (error) { + // Observability only — never fail an ingest because the stamp didn't land. + console.error('[ingest] Failed to stamp crawler heartbeat:', error); + } +} + +/** + * Per-feed crawl health, as the crawler reports it. Unix SECONDS on this wire + * (the proxy keeps milliseconds internally and converts on send); the timeline + * converts back to milliseconds for the client. + */ +export interface FeedHealthReport { + feedUrl: string; + errorCount: number; + lastError?: string | null; + lastErrorAt?: number | null; + nextRetryAt?: number | null; + lastFetchAt?: number | null; + // In the crawl set but going unfetched — starved, not failing. Independent of + // `errorCount`, which can be 0 while this is true. + crawlStale?: boolean; +} + +// A report carries only the unhealthy feeds, so this is far above any plausible +// real number — an abuse guard, not a tuning knob. +const MAX_HEALTH_FEEDS = 2000; +// Error strings are rendered in a popover and matched against substrings; the +// crawler's messages are short, so anything longer is a runaway. +const MAX_HEALTH_ERROR_CHARS = 500; +// Feed URLs per recovery statement. Well under D1's per-statement bind limit, +// and the recovered set is normally a handful anyway. +const HEALTH_CLEAR_CHUNK = 50; + +/** + * A cheap fingerprint of the whole unhealthy set, used as the `feed_health_rev` + * token. Clients echo the revision they last saw and the timeline re-sends the + * health payload only when it differs, which keeps a steady-state poll at the + * one query the architecture promises. + * + * Aggregates rather than a real hash: the partial index makes this a scan of the + * handful of broken feeds, not of the archive. Two genuinely different sets can + * alias only if they share a count, an error-count sum, and both timestamps — and + * the cost of that is one poll showing yesterday's error, self-healing on the + * next change. + */ +async function readFeedHealthRev(env: Env): Promise { + const row = await env.DB.prepare( + `SELECT COUNT(*) AS n, COALESCE(SUM(error_count), 0) AS errors, + COALESCE(MAX(last_error_at), 0) AS newest, COALESCE(MAX(next_retry_at), 0) AS retry + FROM feeds WHERE error_count > 0` + ).first<{ n: number; errors: number; newest: number; retry: number }>(); + return `${row?.n ?? 0}:${row?.errors ?? 0}:${row?.newest ?? 0}:${row?.retry ?? 0}`; +} + +/** + * Clear one feed's error state because we just fetched it successfully. + * + * The pull-through in `/api/v2/feeds/fetch` is the user's explicit "retry this + * feed" action, and it is proof the feed works — but the crawler's next health + * report is up to five minutes away, so without this the reader would keep + * showing the error the user just cleared. Refreshes the revision so every other + * client picks the recovery up too. + */ +export async function clearFeedHealth(env: Env, feedUrl: string): Promise { + try { + const result = await env.DB.prepare( + `UPDATE feeds + SET error_count = 0, last_error = NULL, last_error_at = NULL, next_retry_at = NULL, + crawl_stale = 0, last_fetch_at = ? + WHERE feed_url = ? AND (error_count > 0 OR crawl_stale = 1)` + ) + .bind(Math.floor(Date.now() / 1000), feedUrl) + .run(); + if (!result.meta?.changes) return; + + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(FEED_HEALTH_REV_KEY, await readFeedHealthRev(env)) + .run(); + } catch (error) { + // Health is observability, not the read itself — never fail a fetch over it. + console.error('[ingest] Failed to clear feed health:', error); + } +} + +/** + * POST /api/internal/feed-health + * + * The crawler's periodic report of every feed it currently considers broken. + * + * This is the timeline path's replacement for what the legacy batch response + * carried inline: on `/api/v2/feeds/batch` a failing feed came back with + * `status: 'error'` plus its error, count and retry time, which is what fed the + * reader's per-feed error badge and popover. Reads no longer touch the proxy, and + * a feed that fails to crawl pushes no items, so without this a broken feed is + * indistinguishable from a quiet one and the user gets no signal at all. + * + * The payload is the COMPLETE unhealthy set, not a delta: recovery is inferred + * from absence (see the sweep below), so a feed that starts working again clears + * without the crawler having to say anything about it. + */ +export async function handleFeedHealth(request: Request, env: Env): Promise { + if (request.method !== 'POST') return badRequest('Method not allowed', 405); + if (!isAuthorizedProxyRequest(request, env)) return unauthorized(); + + const declaredLength = Number(request.headers.get('Content-Length') ?? '0'); + if (declaredLength > MAX_INGEST_BODY_BYTES) return badRequest('Payload too large', 413); + + let body: { feeds?: FeedHealthReport[] }; + try { + body = await request.json(); + } catch { + return badRequest('Invalid JSON body'); + } + + // A report entry means "something is wrong with this feed" — it is erroring, + // or the crawler isn't reaching it, or both. An entry claiming neither is + // noise and would keep the feed flagged forever. + const reports = (Array.isArray(body.feeds) ? body.feeds : []).filter( + (f) => f?.feedUrl && ((Number.isFinite(f.errorCount) && f.errorCount > 0) || f.crawlStale) + ); + if (reports.length > MAX_HEALTH_FEEDS) return badRequest('Too many feeds'); + + try { + const before = await readFeedHealthRev(env); + + // Which feeds are flagged right now. Read BEFORE the upserts so the + // difference against this report is exactly "was in trouble, isn't any more". + // Small by construction (the partial index covers only flagged feeds), so the + // chunked IN list below can never approach the bound-parameter limit. + const flagged = await env.DB.prepare( + `SELECT feed_url FROM feeds WHERE error_count > 0 OR crawl_stale = 1` + ).all<{ feed_url: string }>(); + + const stillTroubled = new Set(reports.map((r) => r.feedUrl)); + const recovered = flagged.results + .map((r) => r.feed_url) + .filter((url) => !stillTroubled.has(url)); + + // A feed can be broken from its very first crawl, in which case it has never + // been ingested and has no `feeds` row at all — so this inserts rather than + // assuming one exists. Such a row carries health only (NULL title/site_url), + // which the metadata backfill already skips. + const upserts = reports.map((report) => + env.DB.prepare( + `INSERT INTO feeds (feed_url, error_count, last_error, last_error_at, next_retry_at, + last_fetch_at, crawl_stale) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) + ON CONFLICT(feed_url) DO UPDATE SET + error_count = excluded.error_count, + last_error = excluded.last_error, + last_error_at = excluded.last_error_at, + next_retry_at = excluded.next_retry_at, + last_fetch_at = COALESCE(excluded.last_fetch_at, feeds.last_fetch_at), + crawl_stale = excluded.crawl_stale` + ).bind( + report.feedUrl, + Number.isFinite(report.errorCount) ? Math.max(report.errorCount, 0) : 0, + report.lastError ? report.lastError.slice(0, MAX_HEALTH_ERROR_CHARS) : null, + report.lastErrorAt ?? null, + report.nextRetryAt ?? null, + report.lastFetchAt ?? null, + report.crawlStale ? 1 : 0 + ) + ); + for (let i = 0; i < upserts.length; i += INGEST_BATCH_SIZE) { + await env.DB.batch(upserts.slice(i, i + INGEST_BATCH_SIZE)); + } + + // Recovery: a feed the crawler stopped listing is fine again. Absence is the + // entire signal, which is why the report has to be the complete set. + // Chunked well under D1's per-statement bind limit — the same limit a + // subscription-sized IN list walked into once already. + for (let i = 0; i < recovered.length; i += HEALTH_CLEAR_CHUNK) { + const chunk = recovered.slice(i, i + HEALTH_CLEAR_CHUNK); + await env.DB.prepare( + `UPDATE feeds + SET error_count = 0, last_error = NULL, last_error_at = NULL, next_retry_at = NULL, + crawl_stale = 0 + WHERE feed_url IN (${chunk.map(() => '?').join(',')})` + ) + .bind(...chunk) + .run(); + } + + const after = await readFeedHealthRev(env); + if (after !== before) { + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(FEED_HEALTH_REV_KEY, after) + .run(); + } + + await stampCrawlerHeartbeat(env); + return new Response(JSON.stringify({ ok: true, unhealthy: reports.length, rev: after }), { + headers: { 'Content-Type': 'application/json' }, + }); + } catch (error) { + console.error('[ingest] Feed-health write error:', error); + return new Response(JSON.stringify({ error: 'Feed health update failed' }), { + status: 500, + headers: { 'Content-Type': 'application/json' }, + }); + } +} + +/** + * POST /api/internal/ingest + * + * Upsert a batch of feed metadata + items pushed by the crawler. Any 5xx leaves + * the proxy's outbox state untouched — the idempotent upsert above makes + * at-least-once delivery safe. + */ +export async function handleIngest(request: Request, env: Env): Promise { + if (request.method !== 'POST') return badRequest('Method not allowed', 405); + if (!isAuthorizedProxyRequest(request, env)) return unauthorized(); + + const declaredLength = Number(request.headers.get('Content-Length') ?? '0'); + if (declaredLength > MAX_INGEST_BODY_BYTES) return badRequest('Payload too large', 413); + + let body: { feeds?: IngestFeed[]; items?: IngestItem[] }; + try { + body = await request.json(); + } catch { + return badRequest('Invalid JSON body'); + } + + const feeds = Array.isArray(body.feeds) ? body.feeds : []; + const items = Array.isArray(body.items) ? body.items : []; + if (items.length > MAX_INGEST_ITEMS) return badRequest('Too many items'); + + try { + const { inserted, updated } = await ingestBatch(env, feeds, items); + await stampCrawlerHeartbeat(env); + return new Response(JSON.stringify({ ok: true, inserted, updated }), { + headers: { 'Content-Type': 'application/json' }, + }); + } catch (error) { + console.error('[ingest] D1 WRITE ERROR:', error); + return new Response(JSON.stringify({ error: 'Ingest failed' }), { + status: 500, + headers: { 'Content-Type': 'application/json' }, + }); + } +} + +/** + * GET /api/internal/crawl-set + * + * The feeds this environment wants crawled. Replaces the proxy's request-driven + * warmth: once clients stop reading through Fly, nothing stamps + * `last_requested_at`, so every feed would silently age out of the warm loop. + * The proxy polls this and stamps the rows itself. + */ +export async function handleCrawlSet(request: Request, env: Env): Promise { + if (request.method !== 'GET') return badRequest('Method not allowed', 405); + if (!isAuthorizedProxyRequest(request, env)) return unauthorized(); + + // The crawl-set pull is the crawler's liveness signal (it runs every 5 minutes + // whether or not any feed produced an item). + await stampCrawlerHeartbeat(env); + + const rows = await env.DB.prepare( + `SELECT feed_url, COUNT(*) AS subscribers + FROM subscriptions_cache + WHERE active = 1 + AND feed_url IS NOT NULL AND feed_url <> '' + AND ${rssSubscriptionPredicate()} + GROUP BY feed_url` + ).all<{ feed_url: string; subscribers: number }>(); + + const feeds = rows.results.map((row) => ({ + feedUrl: row.feed_url, + subscribers: row.subscribers, + })); + + return new Response(JSON.stringify({ feeds, count: feeds.length }), { + headers: { 'Content-Type': 'application/json' }, + }); +} diff --git a/backend/src/routes/subscriptions.ts b/backend/src/routes/subscriptions.ts index 709d04a6..7d97aa07 100644 --- a/backend/src/routes/subscriptions.ts +++ b/backend/src/routes/subscriptions.ts @@ -1,6 +1,6 @@ import type { Env, Session } from '../types'; import { getSessionFromRequest } from '../services/oauth'; -import { warmProxyCache, warmProxyCacheBatch } from './feeds-v2'; +import { warmFeedIntoArchive, warmFeedsIntoArchive } from './feeds-v2'; import { getUserSettings } from './settings'; import { pushSubscriptionToPds, deleteSubscriptionFromPds } from '../services/subscription-sync'; import { @@ -806,13 +806,14 @@ export async function handleCreateSubscription( ) .run(); - // Warm up proxy cache for RSS subscriptions only + // Crawl the feed once and ingest it into the archive, so the client's first + // read of this brand-new subscription is a plain D1 query (RSS only). if (!isAtProto && feedUrl) { - const cacheResult = await warmProxyCache(env, feedUrl); + const cacheResult = await warmFeedIntoArchive(env, feedUrl); if (cacheResult.success) { - console.log(`Warmed proxy cache: ${feedUrl} (${cacheResult.itemCount} items)`); + console.log(`Ingested new feed: ${feedUrl} (${cacheResult.itemCount} items)`); } else { - console.error(`Failed to warm cache for ${feedUrl}: ${cacheResult.error}`); + console.error(`Failed to ingest ${feedUrl}: ${cacheResult.error}`); } } @@ -1242,16 +1243,17 @@ export async function handleBulkCreateSubscriptions( await env.DB.batch(batchStatements); } - // Warm up proxy cache for new subscriptions (batch is efficient) + // Crawl + ingest the first few new subscriptions (batch is efficient) so the + // client's backfill reads them straight from the archive. const MAX_FEEDS_TO_WARM = 10; const feedsToWarmNow = feedsToFetch.slice(0, MAX_FEEDS_TO_WARM); if (feedsToWarmNow.length > 0) { - const cacheResults = await warmProxyCacheBatch(env, feedsToWarmNow); + const cacheResults = await warmFeedsIntoArchive(env, feedsToWarmNow); for (const [feedUrl, result] of Object.entries(cacheResults)) { if (result.success) { - console.log(`Warmed proxy cache: ${feedUrl} (${result.itemCount} items)`); + console.log(`Ingested new feed: ${feedUrl} (${result.itemCount} items)`); } else { - console.error(`Failed to warm cache for ${feedUrl}: ${result.error}`); + console.error(`Failed to ingest ${feedUrl}: ${result.error}`); } } } diff --git a/backend/src/routes/timeline.ts b/backend/src/routes/timeline.ts new file mode 100644 index 00000000..cbeefa04 --- /dev/null +++ b/backend/src/routes/timeline.ts @@ -0,0 +1,506 @@ +import type { Env, FeedItem, Session } from '../types'; +import { + CRAWLER_HEARTBEAT_KEY, + CRAWLER_HEARTBEAT_FRESH_SECONDS, + FEED_HEALTH_REV_KEY, + TIMELINE_ENABLED_KEY, + rssSubscriptionPredicate, +} from './ingest'; + +/** + * GET /api/v2/timeline — the whole feed refresh, in one request. + * + * Replaces the `1 + ceil((N-8)/50)` batched `POST /api/v2/feeds/batch` calls + * (each a Worker → Fly hop plus `ceil(GUIDs/88)` chunked read-key queries). The + * crawler pushes items into D1; this serves them with subscriptions AND read + * state resolved in the same query, so `getReadKeys` never runs on the feed path. + * + * Named `/timeline` rather than `/sync` to avoid colliding with the existing + * PDS-subscription `/api/sync/*` routes. + */ + +// Max items in one page. Also the cap on how much a single response can weigh — +// items carry (capped) content bodies. +const MAX_LIMIT = 200; +const DEFAULT_LIMIT = 200; + +// Cold start delivers a per-feed newest slice, not a global one: a global +// `ORDER BY seq DESC LIMIT n` would let one chatty feed starve every other. +const COLD_START_PER_FEED = 30; +// Statements per D1 batch on the cold-start path. +const COLD_START_CHUNK = 25; +// A cold start touches every subscribed feed; bound the work (and log if hit — +// no silent truncation). +const COLD_START_MAX_FEEDS = 500; +// Rows one cold-start PAGE may accumulate. A cold start walks feeds in a stable +// order and stops once it passes this budget, handing back `nextColdOffset` for +// the next page — so a 150-feed reader (or everyone at once after a generation +// bump) can't ask the Worker to buffer thousands of content-bearing items in a +// single response. Checked per chunk, so a page can overshoot by at most +// COLD_START_CHUNK × COLD_START_PER_FEED. +const COLD_START_MAX_ITEMS = 750; + +export interface TimelineItem extends FeedItem { + seq: number; + feedUrl: string; + read: boolean; +} + +interface ItemRow { + seq: number; + feed_url: string; + item_json: string; + read: number; +} + +// The per-user read probe. An EXISTS (rather than a LEFT JOIN) keeps the result +// one row per item even if a user somehow holds duplicate label rows, while +// probing the same (user_did, item_key) index a join would. +const READ_FLAG_SQL = `EXISTS ( + SELECT 1 FROM item_labels_cache il + WHERE il.user_did = ?1 AND il.item_key = fi.guid + AND il.item_type = 'article' AND il.label = 'read' AND il.deleted_at IS NULL + ) AS read`; + +function toTimelineItems(rows: ItemRow[]): TimelineItem[] { + const items: TimelineItem[] = []; + for (const row of rows) { + try { + const item = JSON.parse(row.item_json) as FeedItem; + items.push({ ...item, seq: row.seq, feedUrl: row.feed_url, read: row.read === 1 }); + } catch { + // A corrupt row must not poison the whole page; skip it. The cursor still + // advances past it, so it can't wedge the client's drain loop. + console.error(`[timeline] Unparseable item_json at seq ${row.seq}`); + } + } + return items; +} + +/** + * Newest slice of one feed, read-annotated — the single-feed read path + * (`GET /api/v2/feeds/fetch`), which now serves D1 like everything else. + */ +export async function readFeedSlice( + env: Env, + userDid: string, + feedUrl: string, + limit: number +): Promise { + const rows = await env.DB.prepare( + `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} + FROM feed_items fi + WHERE fi.feed_url = ?2 + ORDER BY fi.published_at DESC, fi.seq DESC + LIMIT ?3` + ) + .bind(userDid, feedUrl, limit) + .all(); + return toTimelineItems(rows.results); +} + +export interface FeedMetadataRow { + title: string | null; + site_url: string | null; + description: string | null; + image_url: string | null; + last_ingest_at: number | null; + // Crawl health (unix seconds), so a single-feed read can report a broken feed + // even while it serves the archived items the feed still has. + error_count: number; + last_error: string | null; + last_error_at: number | null; + next_retry_at: number | null; + last_fetch_at: number | null; +} + +export async function readFeedMetadata(env: Env, feedUrl: string): Promise { + return env.DB.prepare( + `SELECT title, site_url, description, image_url, last_ingest_at, + error_count, last_error, last_error_at, next_retry_at, last_fetch_at + FROM feeds WHERE feed_url = ?` + ) + .bind(feedUrl) + .first(); +} + +export interface ArchiveState { + generation: string; + // True when this environment's crawler has checked in recently. False means + // nothing is filling this D1 (no INGEST_URL on the paired proxy, or the proxy + // is down), so the client must not treat an empty/partial archive as the truth. + crawlerFresh: boolean; + // The operator's rollout gate (`timeline_enabled`, migration 0071). A fresh + // heartbeat only proves a crawler is attached — it arrives seconds into a + // backfill that takes hours — so this is the switch that actually moves readers + // onto the timeline, and the switch that moves them back. + timelineEnabled: boolean; + // Revision of the unhealthy-feed set. A client that echoes this back unchanged + // already holds current health and is sent no health payload. + healthRev: string; +} + +/** + * Generation token, crawler liveness, the rollout gate and the feed-health + * revision in one `sync_state` read (the timeline needs all four on every + * request, and they are four rows of the same small table). + */ +export async function readArchiveState(env: Env): Promise { + const rows = await env.DB.prepare( + `SELECT key, value FROM sync_state WHERE key IN ('items_generation', ?, ?, ?)` + ) + .bind(CRAWLER_HEARTBEAT_KEY, FEED_HEALTH_REV_KEY, TIMELINE_ENABLED_KEY) + .all<{ key: string; value: string }>(); + + let generation = ''; + let heartbeat = 0; + let healthRev = ''; + // Absent means enabled: only an explicit '0' holds clients on the batch path, + // so an environment that never learned about this key behaves as it did before. + let timelineEnabled = true; + for (const row of rows.results) { + if (row.key === 'items_generation') generation = row.value; + else if (row.key === CRAWLER_HEARTBEAT_KEY) heartbeat = parseInt(row.value, 10) || 0; + else if (row.key === FEED_HEALTH_REV_KEY) healthRev = row.value; + else if (row.key === TIMELINE_ENABLED_KEY) timelineEnabled = row.value !== '0'; + } + + const age = Math.floor(Date.now() / 1000) - heartbeat; + return { + generation, + crawlerFresh: heartbeat > 0 && age <= CRAWLER_HEARTBEAT_FRESH_SECONDS, + timelineEnabled, + healthRev, + }; +} + +/** + * Per-feed crawl health for the caller's subscriptions — the timeline path's + * replacement for the per-feed `status: 'error'` the legacy batch response + * carried inline. + * + * Only broken feeds are returned: the client treats absence as healthy, which is + * what makes recovery work without sending a row per subscription. Driven from + * the (small, partially indexed) unhealthy set rather than from the caller's + * subscriptions, because on a healthy system that set is empty and this costs + * nothing. + * + * Timestamps go out in MILLISECONDS — D1 stores seconds like the rest of the + * backend, but the client's FeedStatus contract has always been ms. + */ +export async function subscribedFeedHealth( + env: Env, + userDid: string +): Promise> { + const rows = await env.DB.prepare( + `SELECT f.feed_url, f.error_count, f.last_error, f.last_error_at, f.next_retry_at, f.last_fetch_at + FROM feeds f + WHERE f.error_count > 0 + AND EXISTS ( + SELECT 1 FROM subscriptions_cache sc + WHERE sc.user_did = ? AND sc.feed_url = f.feed_url AND sc.active = 1 + AND ${rssSubscriptionPredicate('sc')} + )` + ) + .bind(userDid) + .all<{ + feed_url: string; + error_count: number; + last_error: string | null; + last_error_at: number | null; + next_retry_at: number | null; + last_fetch_at: number | null; + }>(); + + const health: Record = {}; + for (const row of rows.results) { + health[row.feed_url] = { + errorCount: row.error_count, + error: row.last_error ?? undefined, + lastErrorAt: row.last_error_at ? row.last_error_at * 1000 : undefined, + nextRetryAt: row.next_retry_at ? row.next_retry_at * 1000 : undefined, + lastFetchedAt: row.last_fetch_at ? row.last_fetch_at * 1000 : undefined, + }; + } + return health; +} + +/** One broken feed as the timeline reports it. Timestamps are unix ms. */ +export interface FeedHealth { + errorCount: number; + error?: string; + lastErrorAt?: number; + nextRetryAt?: number; + lastFetchedAt?: number; +} + +/** The archive's current head; 0 when nothing has ever been ingested. */ +async function archiveHead(env: Env): Promise { + const row = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{ + max_seq: number | null; + }>(); + return row?.max_seq ?? 0; +} + +/** + * Subscribed RSS feed URLs in a STABLE order — the cold start pages through this + * list by index, so the ordering has to be the same from one page to the next. + */ +async function subscribedFeedUrls(env: Env, userDid: string): Promise { + const rows = await env.DB.prepare( + `SELECT DISTINCT feed_url FROM subscriptions_cache + WHERE user_did = ? AND active = 1 + AND feed_url IS NOT NULL AND feed_url <> '' + AND ${rssSubscriptionPredicate()} + ORDER BY feed_url` + ) + .bind(userDid) + .all<{ feed_url: string }>(); + return rows.results.map((r) => r.feed_url); +} + +/** + * Feed-level metadata for the feeds the caller subscribes to. Small (tens of + * rows) and only sent alongside a non-empty page, so a steady-state poll that + * returns nothing costs exactly one query. + */ +async function subscribedFeedMetadata( + env: Env, + userDid: string +): Promise> { + const rows = await env.DB.prepare( + `SELECT f.feed_url, f.title, f.site_url, f.image_url + FROM feeds f + JOIN (SELECT DISTINCT feed_url FROM subscriptions_cache + WHERE user_did = ? AND active = 1 + AND ${rssSubscriptionPredicate()}) sc + ON sc.feed_url = f.feed_url` + ) + .bind(userDid) + .all<{ + feed_url: string; + title: string | null; + site_url: string | null; + image_url: string | null; + }>(); + + const feeds: Record = {}; + for (const row of rows.results) { + feeds[row.feed_url] = { + title: row.title ?? undefined, + siteUrl: row.site_url ?? undefined, + imageUrl: row.image_url ?? undefined, + }; + } + return feeds; +} + +export async function handleTimeline( + request: Request, + env: Env, + session: Session +): Promise { + if (request.method !== 'GET') { + return new Response(JSON.stringify({ error: 'Method not allowed' }), { + status: 405, + headers: { 'Content-Type': 'application/json' }, + }); + } + + const url = new URL(request.url); + const sinceSeqParam = url.searchParams.get('since_seq'); + const generationParam = url.searchParams.get('generation'); + const limitParam = url.searchParams.get('limit'); + const coldOffsetParam = url.searchParams.get('cold_offset'); + // The feed-health revision this client already holds. Absent (a fresh page + // load, whose status store is empty) means "send it". + const healthRevParam = url.searchParams.get('health_rev'); + + const parsedLimit = limitParam ? parseInt(limitParam, 10) : DEFAULT_LIMIT; + const limit = Number.isInteger(parsedLimit) + ? Math.min(Math.max(parsedLimit, 1), MAX_LIMIT) + : DEFAULT_LIMIT; + + const parsedSince = sinceSeqParam !== null ? parseInt(sinceSeqParam, 10) : NaN; + const sinceSeq = Number.isInteger(parsedSince) && parsedSince >= 0 ? parsedSince : undefined; + + // Continuation index into the caller's (stably ordered) subscribed-feed list. + const parsedColdOffset = coldOffsetParam !== null ? parseInt(coldOffsetParam, 10) : NaN; + const coldOffset = + Number.isInteger(parsedColdOffset) && parsedColdOffset > 0 ? parsedColdOffset : 0; + + const { generation, crawlerFresh, timelineEnabled, healthRev } = await readArchiveState(env); + // Two independent conditions, one wire field. The client's contract is + // unchanged — `ingestActive: false` has always meant "stay on the batch path" — + // and it does not need to know WHY, only the operator does. + const ingestActive = timelineEnabled && crawlerFresh; + + // Server time (unix seconds) at annotation. The client seeds its forward + // read-delta cursor from this, exactly as /batch does today, so the delta + // starts from bootstrap with no client/server clock skew. + const readCursor = Math.floor(Date.now() / 1000); + + // A client told `ingestActive: false` discards the page and refetches through + // /batch, so building one is pure waste — and during a gated rollout that waste + // is every reader, every poll, for as long as the gate stays shut. Answer with + // the state and nothing else. `coldStart: true` keeps the older + // empty-cold-start heuristic pointing the same way as the flag, so a client + // reading either signal reaches the same conclusion. + if (!ingestActive) { + return json({ + items: [], + cursor: 0, + generation, + ingestActive, + hasMore: false, + readCursor, + coldStart: true, + healthRev, + }); + } + + // Send health when the client's copy is stale. A cold start always gets it: + // it is the one page that delivers already-archived items for a feed that may + // have broken since, and its blanket "these feeds delivered, so they're fine" + // pass would otherwise clear a live error. + const healthStale = healthRevParam !== healthRev; + + const incremental = + sinceSeq !== undefined && + coldOffset === 0 && + generationParam === generation && + generation !== ''; + + try { + if (incremental) { + // Drain oldest-unseen first so a backlog larger than one page is paged + // across polls, never skipped. limit+1 probes hasMore without a second query. + // + // Scaling note: this walks the `feed_items` rowid range above the cursor and + // probes the subscription set per row, so the cost tracks GLOBAL ingest above + // the cursor rather than the caller's own new items. That is the accepted + // fan-out-on-read trade at ~1,300 feeds; if D1 row-reads ever become the + // constraint, the fix is to bound the scan with per-feed `(feed_url, seq)` + // seeks (idx_feed_items_feed_seq already supports them), not to materialize + // per-user timelines. + const rows = await env.DB.prepare( + `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} + FROM feed_items fi + WHERE fi.seq > ?2 + AND EXISTS ( + SELECT 1 FROM subscriptions_cache sc + WHERE sc.user_did = ?1 AND sc.feed_url = fi.feed_url AND sc.active = 1 + AND ${rssSubscriptionPredicate('sc')} + ) + ORDER BY fi.seq ASC + LIMIT ?3` + ) + .bind(session.did, sinceSeq, limit + 1) + .all(); + + const hasMore = rows.results.length > limit; + const page = hasMore ? rows.results.slice(0, limit) : rows.results; + + // Rewound-archive guard, the D1 twin of the proxy's snapshot-restore check. + // A Time Travel restore (or a rebuild from export) rewinds `feed_items` seqs + // while `items_generation` comes back unchanged, leaving every client cursor + // above the head: `seq > ?` then returns nothing on every poll, forever. A + // cursor can never legitimately exceed the head, so treat that as a cold + // start instead of silently starving the client. Only costs a MAX(seq) on + // an otherwise empty page — never on a page that carried items. + if (page.length === 0 && sinceSeq > (await archiveHead(env))) { + console.warn( + `[timeline] Cursor ${sinceSeq} is above the archive head; cold-starting (archive rewound?).` + ); + } else { + // Cursor comes from the returned rows, never a separate MAX(seq): the + // latter races ingest and would skip everything written in between. + const cursor = page.length > 0 ? page[page.length - 1].seq : sinceSeq; + const items = toTimelineItems(page); + + return json({ + items, + cursor, + generation, + ingestActive, + hasMore, + readCursor, + coldStart: false, + feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, + healthRev, + feedHealth: healthStale ? await subscribedFeedHealth(env, session.did) : undefined, + }); + } + } + + // Cold start: no cursor, a generation mismatch (D1 recreated / restored), a + // rewound archive, or a continuation page of one already in progress. + const allFeedUrls = await subscribedFeedUrls(env, session.did); + const feedUrls = allFeedUrls.slice(0, COLD_START_MAX_FEEDS); + if (allFeedUrls.length > feedUrls.length) { + console.warn( + `[timeline] Cold start covering ${feedUrls.length} of ${allFeedUrls.length} feeds for ${session.did}; the rest fill in as new items arrive.` + ); + } + + // The cold-start cursor is the archive head read BEFORE the per-feed slices, + // and the client keeps the first page's value across a paged cold start. + // Reading it first is what makes it safe: an item ingested while we page gets + // a seq above this head, so it arrives on the first incremental poll instead + // of being skipped. (Re-delivering a handful of items is harmless — the merge + // dedupes by GUID.) The incremental path still derives its cursor from the + // rows it returned. + const cursor = await archiveHead(env); + + // Walk feeds from the continuation point until the page budget is spent. The + // client re-requests with `cold_offset=nextColdOffset` until hasMore is false. + // A subscription added or removed mid-cold-start shifts the offsets by one, so + // a feed can be missed by this bootstrap; the client's per-feed backfill picks + // up any subscription that ends up with no articles. + const rows: ItemRow[] = []; + let nextIndex = Math.min(coldOffset, feedUrls.length); + while (nextIndex < feedUrls.length && rows.length < COLD_START_MAX_ITEMS) { + const chunk = feedUrls.slice(nextIndex, nextIndex + COLD_START_CHUNK); + const statements = chunk.map((feedUrl) => + env.DB.prepare( + `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL} + FROM feed_items fi + WHERE fi.feed_url = ?2 + ORDER BY fi.published_at DESC, fi.seq DESC + LIMIT ?3` + ).bind(session.did, feedUrl, COLD_START_PER_FEED) + ); + const results = await env.DB.batch(statements); + for (const result of results) rows.push(...(result.results ?? [])); + nextIndex += chunk.length; + } + const hasMore = nextIndex < feedUrls.length; + + const items = toTimelineItems(rows); + return json({ + items, + cursor, + generation, + ingestActive, + hasMore, + nextColdOffset: hasMore ? nextIndex : undefined, + readCursor, + coldStart: true, + feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined, + healthRev, + feedHealth: await subscribedFeedHealth(env, session.did), + }); + } catch (error) { + console.error('[timeline] Query error:', error); + return new Response(JSON.stringify({ error: 'Failed to load timeline' }), { + status: 500, + headers: { 'Content-Type': 'application/json' }, + }); + } +} + +function json(body: unknown): Response { + return new Response(JSON.stringify(body), { + headers: { 'Content-Type': 'application/json' }, + }); +} diff --git a/backend/src/services/rate-limit.ts b/backend/src/services/rate-limit.ts index e8119789..38a3241b 100644 --- a/backend/src/services/rate-limit.ts +++ b/backend/src/services/rate-limit.ts @@ -65,6 +65,14 @@ const RATE_LIMITS: Record = { '/api/leaflet/subscriptions': STANDARD_LIMIT, // Light operations (cached data, simple queries) + // The whole feed refresh is one D1 query per page now, and a returning reader + // drains several pages in a burst — a light limit, not a standard one. + '/api/v2/timeline': LIGHT_LIMIT, + // Single-feed reads are D1 queries now (the archive), and an OPML import fires + // one per imported feed to backfill it. The one expensive case — the + // pull-through crawl — only happens for a feed the caller subscribes to and + // only when the archive has nothing for it yet. + '/api/v2/feeds/fetch': LIGHT_LIMIT, '/api/feeds/cached': LIGHT_LIMIT, '/api/feeds/batch': LIGHT_LIMIT, '/api/feeds/status': LIGHT_LIMIT, diff --git a/backend/src/types.ts b/backend/src/types.ts index 06521e1d..c08571e2 100644 --- a/backend/src/types.ts +++ b/backend/src/types.ts @@ -56,6 +56,9 @@ export interface FeedItem { // Stamped by the authed batch fetch handler (feeds-v2.ts) from a per-user read // join. Not a stored feed field — only present on annotated responses. read?: boolean; + // Set at ingest when `content` exceeded the stored-content cap and was dropped + // (routes/ingest.ts). The reader falls back to /extract for the full text. + contentTruncated?: boolean; } export interface ParsedFeed { diff --git a/backend/test/feed-timeline.spec.ts b/backend/test/feed-timeline.spec.ts new file mode 100644 index 00000000..236fd4d7 --- /dev/null +++ b/backend/test/feed-timeline.spec.ts @@ -0,0 +1,894 @@ +import { env } from 'cloudflare:test'; +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { + handleIngest, + handleCrawlSet, + handleFeedHealth, + clearFeedHealth, + trimFeedsToSanityCap, + ingestProxyFeed, + CRAWLER_HEARTBEAT_KEY, + FEED_HEALTH_REV_KEY, + TIMELINE_ENABLED_KEY, +} from '../src/routes/ingest'; +import { handleTimeline, readFeedSlice } from '../src/routes/timeline'; +import type { Env, FeedItem, Session } from '../src/types'; + +const TEST_DID = 'did:plc:timeline123'; +const OTHER_DID = 'did:plc:timelineother'; +const SECRET = 'test-proxy-secret'; + +const FEED_A = 'https://example.com/a.xml'; +const FEED_B = 'https://example.com/b.xml'; + +const SESSION: Session = { + did: TEST_DID, + handle: 'timeline.bsky.social', + pdsUrl: 'https://test.pds.example', + accessToken: 'token', + refreshToken: 'refresh', + dpopPrivateKey: '{}', + expiresAt: Date.now() + 3600000, +}; + +function item(guid: string, overrides: Partial = {}): FeedItem { + return { + guid, + url: `https://example.com/${guid}`, + title: `Title ${guid}`, + publishedAt: '2026-01-01T00:00:00.000Z', + ...overrides, + }; +} + +function ingestRequest( + body: unknown, + secret: string | null = SECRET, + extraHeaders: Record = {} +): Request { + const headers: Record = { + 'Content-Type': 'application/json', + ...extraHeaders, + }; + if (secret !== null) headers['X-Proxy-Secret'] = secret; + return new Request('https://api.example/api/internal/ingest', { + method: 'POST', + headers, + body: JSON.stringify(body), + }); +} + +async function ingest( + feedUrl: string, + items: Array<{ item: FeedItem; contentHash: string }> +): Promise<{ inserted: number; updated: number }> { + const res = await handleIngest( + ingestRequest({ + feeds: [{ feedUrl, title: 'Feed A' }], + items: items.map((entry) => ({ + feedUrl, + guid: entry.item.guid, + item: entry.item, + publishedAt: Date.parse(entry.item.publishedAt), + firstSeenAt: Date.now(), + contentHash: entry.contentHash, + })), + }), + env + ); + expect(res.status).toBe(200); + return (await res.json()) as { inserted: number; updated: number }; +} + +async function addSubscription( + did: string, + feedUrl: string, + opts: { active?: number; sourceType?: string | null } = {} +) { + await env.DB.prepare( + `INSERT INTO subscriptions_cache (user_did, record_uri, feed_url, title, source_type, active) + VALUES (?, ?, ?, ?, ?, ?)` + ) + .bind( + did, + `at://${did}/app.skyreader.feed.subscription/${Math.random().toString(36).slice(2)}`, + feedUrl, + 'Feed', + opts.sourceType ?? null, + opts.active ?? 1 + ) + .run(); +} + +async function timeline(params: Record = {}) { + const search = new URLSearchParams(params).toString(); + const res = await handleTimeline( + new Request(`https://api.example/api/v2/timeline${search ? `?${search}` : ''}`), + env, + SESSION + ); + expect(res.status).toBe(200); + return (await res.json()) as { + items: Array<{ seq: number; feedUrl: string; guid: string; read: boolean; content?: string }>; + cursor: number; + generation: string; + hasMore: boolean; + nextColdOffset?: number; + ingestActive: boolean; + readCursor: number; + coldStart: boolean; + feeds?: Record; + healthRev?: string; + feedHealth?: Record< + string, + { + errorCount: number; + error?: string; + lastErrorAt?: number; + nextRetryAt?: number; + lastFetchedAt?: number; + } + >; + }; +} + +async function clearCrawlerHeartbeat() { + await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(CRAWLER_HEARTBEAT_KEY).run(); +} + +async function setTimelineGate(value: string) { + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at` + ) + .bind(TIMELINE_ENABLED_KEY, value) + .run(); +} + +async function clearTimelineGate() { + await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(TIMELINE_ENABLED_KEY).run(); +} + +async function reportHealth(feeds: unknown[], secret: string | null = SECRET) { + const headers: Record = { 'Content-Type': 'application/json' }; + if (secret !== null) headers['X-Proxy-Secret'] = secret; + return handleFeedHealth( + new Request('https://api.example/api/internal/feed-health', { + method: 'POST', + headers, + body: JSON.stringify({ feeds }), + }), + env + ); +} + +function brokenFeed(feedUrl: string, overrides: Record = {}) { + const nowSeconds = Math.floor(Date.now() / 1000); + return { + feedUrl, + errorCount: 3, + lastError: 'Failed to fetch (HTTP 404)', + lastErrorAt: nowSeconds, + nextRetryAt: nowSeconds + 600, + lastFetchAt: nowSeconds - 7200, + ...overrides, + }; +} + +describe('feed timeline (D1 ingest + serve)', () => { + let savedSecret: string | undefined; + + beforeEach(async () => { + savedSecret = (env as Env).FEED_PROXY_SECRET; + (env as Env).FEED_PROXY_SECRET = SECRET; + for (const did of [TEST_DID, OTHER_DID]) { + await env.DB.prepare( + `INSERT OR IGNORE INTO users (did, handle, pds_url, tier, created_at) + VALUES (?, ?, 'https://test.pds.example', 'free', unixepoch())` + ) + .bind(did, `${did}.test`) + .run(); + } + }); + + afterEach(async () => { + (env as Env).FEED_PROXY_SECRET = savedSecret as string; + await env.DB.prepare('DELETE FROM feed_items').run(); + await env.DB.prepare('DELETE FROM feeds').run(); + await env.DB.prepare('DELETE FROM subscriptions_cache').run(); + await env.DB.prepare('DELETE FROM item_labels_cache').run(); + await clearCrawlerHeartbeat(); + await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(FEED_HEALTH_REV_KEY).run(); + // Absent is the open position, so this restores the default the rest of the + // suite runs under. + await clearTimelineGate(); + }); + + describe('ingest auth', () => { + it('rejects a request with no secret', async () => { + const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, null), env); + expect(res.status).toBe(401); + }); + + it('rejects a mismatched secret', async () => { + const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, 'nope'), env); + expect(res.status).toBe(401); + }); + + it('fails closed when the server secret is unset', async () => { + (env as Env).FEED_PROXY_SECRET = '' as string; + const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, SECRET), env); + expect(res.status).toBe(401); + }); + + it('rejects an oversized body by Content-Length', async () => { + const res = await handleIngest( + ingestRequest({ feeds: [], items: [] }, SECRET, { + 'Content-Length': String(64 * 1024 * 1024), + }), + env + ); + expect(res.status).toBe(413); + }); + }); + + describe('ingest writes', () => { + it('inserts items and upserts feed metadata', async () => { + const result = await ingest(FEED_A, [ + { item: item('g1'), contentHash: 'h1' }, + { item: item('g2'), contentHash: 'h2' }, + ]); + expect(result.inserted).toBe(2); + expect(result.updated).toBe(0); + + const feed = await env.DB.prepare('SELECT * FROM feeds WHERE feed_url = ?') + .bind(FEED_A) + .first<{ title: string; last_ingest_at: number }>(); + expect(feed?.title).toBe('Feed A'); + expect(feed?.last_ingest_at).toBeGreaterThan(0); + }); + + it('is idempotent: re-pushing the same batch creates no duplicates', async () => { + await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]); + const before = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?') + .bind('g1') + .first<{ seq: number }>(); + + const second = await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]); + expect(second.inserted).toBe(0); + expect(second.updated).toBe(0); + + const rows = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?') + .bind('g1') + .all<{ seq: number }>(); + expect(rows.results.length).toBe(1); + expect(rows.results[0].seq).toBe(before?.seq); + }); + + it('edits in place: a changed hash rewrites the row without a new seq', async () => { + await ingest(FEED_A, [{ item: item('g1', { title: 'Old' }), contentHash: 'h1' }]); + const before = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?') + .bind('g1') + .first<{ seq: number }>(); + + const result = await ingest(FEED_A, [ + { item: item('g1', { title: 'New' }), contentHash: 'h2' }, + ]); + expect(result.updated).toBe(1); + expect(result.inserted).toBe(0); + + const after = await env.DB.prepare('SELECT seq, item_json FROM feed_items WHERE guid = ?') + .bind('g1') + .first<{ seq: number; item_json: string }>(); + expect(after?.seq).toBe(before?.seq); + expect(JSON.parse(after!.item_json).title).toBe('New'); + }); + + it('never prunes: a feed accumulates past the proxy’s 200-item window', async () => { + // Five pushes of 50 distinct items each — well past the proxy's K = 200 + // outbox window. D1 is the archive; ordinary ingest deletes nothing. + for (let batch = 0; batch < 5; batch++) { + await ingest( + FEED_A, + Array.from({ length: 50 }, (_, i) => ({ + item: item(`b${batch}-i${i}`), + contentHash: `h${batch}-${i}`, + })) + ); + } + const count = await env.DB.prepare('SELECT COUNT(*) AS c FROM feed_items WHERE feed_url = ?') + .bind(FEED_A) + .first<{ c: number }>(); + expect(count?.c).toBe(250); + }); + + it('sanity cap trims oldest-first, and only above the cap', async () => { + await ingest( + FEED_A, + Array.from({ length: 5 }, (_, i) => ({ item: item(`s${i}`), contentHash: `h${i}` })) + ); + + // Under the cap: a no-op. + await trimFeedsToSanityCap(env, [FEED_A], 10); + let rows = await env.DB.prepare('SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq') + .bind(FEED_A) + .all<{ guid: string }>(); + expect(rows.results.length).toBe(5); + + // Above the cap: the oldest go, newest survive. + await trimFeedsToSanityCap(env, [FEED_A], 3); + rows = await env.DB.prepare('SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq') + .bind(FEED_A) + .all<{ guid: string }>(); + expect(rows.results.map((r) => r.guid)).toEqual(['s2', 's3', 's4']); + }); + + it('caps stored content and marks it truncated, leaving small items alone', async () => { + const big = 'x'.repeat(9000); + await ingest(FEED_A, [ + { item: item('big', { content: big, summary: 'kept' }), contentHash: 'hbig' }, + { item: item('small', { content: 'tiny' }), contentHash: 'hsmall' }, + ]); + + const bigRow = await env.DB.prepare('SELECT item_json FROM feed_items WHERE guid = ?') + .bind('big') + .first<{ item_json: string }>(); + const parsedBig = JSON.parse(bigRow!.item_json); + expect(parsedBig.content).toBeUndefined(); + expect(parsedBig.contentTruncated).toBe(true); + expect(parsedBig.summary).toBe('kept'); + + const smallRow = await env.DB.prepare('SELECT item_json FROM feed_items WHERE guid = ?') + .bind('small') + .first<{ item_json: string }>(); + const parsedSmall = JSON.parse(smallRow!.item_json); + expect(parsedSmall.content).toBe('tiny'); + expect(parsedSmall.contentTruncated).toBeUndefined(); + }); + }); + + describe('pull-through ingest (proxy feed → archive)', () => { + it('assigns seq oldest→newest, so the newest entries are what a per-feed read serves', async () => { + // A proxy feed is newest-first. Ingesting it forward would give the newest + // item the lowest seq, and every later `ORDER BY seq DESC` read would then + // serve the feed's OLDEST items. + const newestFirst = Array.from({ length: 5 }, (_, i) => item(`p${5 - i}`)); + await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: newestFirst }); + + const rows = await env.DB.prepare( + 'SELECT guid, seq FROM feed_items WHERE feed_url = ? ORDER BY seq ASC' + ) + .bind(FEED_A) + .all<{ guid: string; seq: number }>(); + expect(rows.results.map((r) => r.guid)).toEqual(['p1', 'p2', 'p3', 'p4', 'p5']); + + const slice = await readFeedSlice(env, TEST_DID, FEED_A, 2); + expect(slice.map((i) => i.guid)).toEqual(['p5', 'p4']); + }); + + it('hashes identically to a later crawler push, so the push is a no-op', async () => { + await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: [item('same')] }); + const before = await env.DB.prepare('SELECT seq, content_hash FROM feed_items WHERE guid = ?') + .bind('same') + .first<{ seq: number; content_hash: string }>(); + + await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: [item('same')] }); + const rows = await env.DB.prepare('SELECT seq, content_hash FROM feed_items WHERE guid = ?') + .bind('same') + .all<{ seq: number; content_hash: string }>(); + expect(rows.results.length).toBe(1); + expect(rows.results[0].seq).toBe(before?.seq); + expect(rows.results[0].content_hash).toBe(before?.content_hash); + }); + + it('serves newest publications when pull-through and crawler ingest interleave', async () => { + // Subscribe-time pull-through can write the newest proxy window before + // the crawler's initial backlog reaches this feed. The older backlog then + // has higher seq values, so per-feed reads must not use seq as recency. + await ingestProxyFeed(env, FEED_A, { + title: 'Pulled', + items: [ + item('newest', { publishedAt: '2026-03-01T00:00:00.000Z' }), + item('newer', { publishedAt: '2026-02-01T00:00:00.000Z' }), + ], + }); + await ingest(FEED_A, [ + { + item: item('oldest', { publishedAt: '2025-12-01T00:00:00.000Z' }), + contentHash: 'oldest-hash', + }, + { + item: item('older', { publishedAt: '2026-01-01T00:00:00.000Z' }), + contentHash: 'older-hash', + }, + ]); + + const slice = await readFeedSlice(env, TEST_DID, FEED_A, 2); + expect(slice.map((i) => i.guid)).toEqual(['newest', 'newer']); + + await addSubscription(TEST_DID, FEED_A); + const page = await timeline(); + expect(page.items.map((i) => i.guid)).toEqual(['newest', 'newer', 'older', 'oldest']); + }); + }); + + describe('crawl set', () => { + it('requires the shared secret', async () => { + const res = await handleCrawlSet( + new Request('https://api.example/api/internal/crawl-set'), + env + ); + expect(res.status).toBe(401); + }); + + it('returns active RSS feeds with subscriber counts', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(OTHER_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B, { active: 0 }); + await addSubscription(TEST_DID, 'at://did:plc:x/pub', { sourceType: 'atproto.documents' }); + + const res = await handleCrawlSet( + new Request('https://api.example/api/internal/crawl-set', { + headers: { 'X-Proxy-Secret': SECRET }, + }), + env + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { + feeds: Array<{ feedUrl: string; subscribers: number }>; + }; + expect(body.feeds).toEqual([{ feedUrl: FEED_A, subscribers: 2 }]); + }); + }); + + describe('timeline serving', () => { + it('cold-starts with a per-feed newest slice and inline read flags', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await ingest(FEED_A, [ + { item: item('a1'), contentHash: 'h1' }, + { item: item('a2'), contentHash: 'h2' }, + ]); + await ingest(FEED_B, [{ item: item('b1'), contentHash: 'h3' }]); + + await env.DB.prepare( + `INSERT INTO item_labels_cache (user_did, item_key, item_type, label, created_at, updated_at) + VALUES (?, 'a1', 'article', 'read', unixepoch(), unixepoch())` + ) + .bind(TEST_DID) + .run(); + + const body = await timeline(); + expect(body.coldStart).toBe(true); + expect(body.items.map((i) => i.guid).sort()).toEqual(['a1', 'a2', 'b1']); + const byGuid = Object.fromEntries(body.items.map((i) => [i.guid, i.read])); + expect(byGuid['a1']).toBe(true); + expect(byGuid['a2']).toBe(false); + expect(body.cursor).toBeGreaterThan(0); + expect(body.hasMore).toBe(false); + expect(typeof body.readCursor).toBe('number'); + expect(body.feeds?.[FEED_A]?.title).toBe('Feed A'); + }); + + it('excludes parked, unsubscribed and atproto sources', async () => { + await addSubscription(TEST_DID, FEED_A, { active: 0 }); + await addSubscription(OTHER_DID, FEED_B); + await ingest(FEED_A, [{ item: item('a1'), contentHash: 'h1' }]); + await ingest(FEED_B, [{ item: item('b1'), contentHash: 'h2' }]); + + const cold = await timeline(); + expect(cold.items).toEqual([]); + + // ...and the incremental path applies the same filter. + const incremental = await timeline({ since_seq: '0', generation: cold.generation }); + expect(incremental.items).toEqual([]); + }); + + it('drains incrementally from the cursor, paging with hasMore', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [ + { item: item('i1'), contentHash: 'h1' }, + { item: item('i2'), contentHash: 'h2' }, + { item: item('i3'), contentHash: 'h3' }, + ]); + + const generation = (await timeline()).generation; + const first = await timeline({ since_seq: '0', generation, limit: '2' }); + expect(first.coldStart).toBe(false); + expect(first.items.map((i) => i.guid)).toEqual(['i1', 'i2']); + expect(first.hasMore).toBe(true); + + const second = await timeline({ + since_seq: String(first.cursor), + generation, + limit: '2', + }); + expect(second.items.map((i) => i.guid)).toEqual(['i3']); + expect(second.hasMore).toBe(false); + + // Steady state: nothing new, cursor held, one empty page. + const third = await timeline({ since_seq: String(second.cursor), generation, limit: '2' }); + expect(third.items).toEqual([]); + expect(third.cursor).toBe(second.cursor); + expect(third.hasMore).toBe(false); + expect(third.feeds).toBeUndefined(); + }); + + it('does not re-deliver an edited item (seq unchanged)', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('e1', { title: 'Old' }), contentHash: 'h1' }]); + const cold = await timeline(); + + await ingest(FEED_A, [{ item: item('e1', { title: 'New' }), contentHash: 'h2' }]); + const after = await timeline({ + since_seq: String(cold.cursor), + generation: cold.generation, + }); + expect(after.items).toEqual([]); + }); + + it('cold-starts again on a generation mismatch', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]); + + const body = await timeline({ since_seq: '999999', generation: 'stale-generation' }); + expect(body.coldStart).toBe(true); + expect(body.items.map((i) => i.guid)).toEqual(['g1']); + }); + + it('starts an empty account at the archive head, not at zero', async () => { + await addSubscription(TEST_DID, FEED_B); + await ingest(FEED_A, [{ item: item('other'), contentHash: 'h1' }]); + + const body = await timeline(); + expect(body.items).toEqual([]); + expect(body.cursor).toBeGreaterThan(0); + }); + + it('cold-starts a client whose cursor sits above the head (rewound archive)', async () => { + // The generation survives a Time Travel restore while the seqs rewind, so a + // cursor above the head is the only symptom — and `seq > cursor` would + // otherwise return nothing on every poll, forever. + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('r1'), contentHash: 'h1' }]); + const cold = await timeline(); + + const stalled = await timeline({ + since_seq: String(cold.cursor + 5000), + generation: cold.generation, + }); + expect(stalled.coldStart).toBe(true); + expect(stalled.items.map((i) => i.guid)).toEqual(['r1']); + expect(stalled.cursor).toBeLessThan(cold.cursor + 5000); + }); + + it('pages a large cold start and continues from nextColdOffset', async () => { + // 26 feeds × 30 items is past the per-page item budget, so the first page + // stops early and hands back a continuation index. + const feedUrls = Array.from({ length: 26 }, (_, i) => `https://example.com/paged${i}.xml`); + for (const feedUrl of feedUrls) { + await addSubscription(TEST_DID, feedUrl); + await ingest( + feedUrl, + Array.from({ length: 30 }, (_, i) => ({ + item: item(`${feedUrl}#${i}`), + contentHash: `${feedUrl}-${i}`, + })) + ); + } + + const first = await timeline(); + expect(first.coldStart).toBe(true); + expect(first.hasMore).toBe(true); + expect(first.nextColdOffset).toBeGreaterThan(0); + expect(first.items.length).toBeLessThan(26 * 30); + + const second = await timeline({ cold_offset: String(first.nextColdOffset) }); + expect(second.coldStart).toBe(true); + expect(second.hasMore).toBe(false); + expect(second.items.length).toBeGreaterThan(0); + + // Every feed is covered across the two pages, each with its newest slice. + const seen = new Set([...first.items, ...second.items].map((i) => i.feedUrl)); + expect(seen.size).toBe(26); + expect(first.items.length + second.items.length).toBe(26 * 30); + }); + }); + + describe('crawler liveness (ingestActive)', () => { + it('is false until the crawler checks in', async () => { + await addSubscription(TEST_DID, FEED_A); + await clearCrawlerHeartbeat(); + const body = await timeline(); + expect(body.ingestActive).toBe(false); + }); + + it('is true after an ingest push', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('h1'), contentHash: 'h1' }]); + expect((await timeline()).ingestActive).toBe(true); + }); + + it('is true after a crawl-set pull, even with nothing ingested', async () => { + await addSubscription(TEST_DID, FEED_A); + await clearCrawlerHeartbeat(); + const res = await handleCrawlSet( + new Request('https://api.example/api/internal/crawl-set', { + headers: { 'X-Proxy-Secret': SECRET }, + }), + env + ); + expect(res.status).toBe(200); + + const body = await timeline(); + expect(body.ingestActive).toBe(true); + expect(body.items).toEqual([]); + }); + + it('goes stale when the last heartbeat is old', async () => { + await addSubscription(TEST_DID, FEED_A); + await env.DB.prepare( + `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch()) + ON CONFLICT(key) DO UPDATE SET value = excluded.value` + ) + .bind(CRAWLER_HEARTBEAT_KEY, String(Math.floor(Date.now() / 1000) - 4 * 3600)) + .run(); + + expect((await timeline()).ingestActive).toBe(false); + }); + }); + + describe('rollout gate (timeline_enabled)', () => { + it('holds clients on the batch path while the gate is shut, however live the crawler', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g1'), contentHash: 'g1' }]); + expect((await timeline()).ingestActive).toBe(true); + + await setTimelineGate('0'); + const gated = await timeline(); + expect(gated.ingestActive).toBe(false); + // The whole point of the short-circuit: a page the client is about to throw + // away is never built, so the gated window costs one sync_state read. + expect(gated.items).toEqual([]); + // Both fallback signals agree, so a client reading either one stays put. + expect(gated.coldStart).toBe(true); + expect(gated.hasMore).toBe(false); + }); + + it('reopens without a deploy, serving the archive that filled while it was shut', async () => { + await addSubscription(TEST_DID, FEED_A); + await setTimelineGate('0'); + // Ingest keeps running while the gate is shut — that is the sequencing the + // gate exists to allow: fill the archive first, admit readers second. + await ingest(FEED_A, [{ item: item('g2'), contentHash: 'g2' }]); + expect((await timeline()).items).toEqual([]); + + await setTimelineGate('1'); + const open = await timeline(); + expect(open.ingestActive).toBe(true); + expect(open.items.map((i) => i.guid)).toEqual(['g2']); + }); + + it('is open when the row is absent, so an environment that never set it is unaffected', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g3'), contentHash: 'g3' }]); + await clearTimelineGate(); + expect((await timeline()).ingestActive).toBe(true); + }); + + it('still requires a live crawler when open', async () => { + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('g4'), contentHash: 'g4' }]); + await setTimelineGate('1'); + await clearCrawlerHeartbeat(); + expect((await timeline()).ingestActive).toBe(false); + }); + }); + + describe('feed health', () => { + it('requires the shared secret', async () => { + expect((await reportHealth([brokenFeed(FEED_A)], null)).status).toBe(401); + expect((await reportHealth([brokenFeed(FEED_A)], 'nope')).status).toBe(401); + }); + + it('stamps the crawler heartbeat like the other internal endpoints', async () => { + await addSubscription(TEST_DID, FEED_A); + await clearCrawlerHeartbeat(); + expect((await reportHealth([])).status).toBe(200); + expect((await timeline()).ingestActive).toBe(true); + }); + + it('serves a broken feed to its subscribers, in milliseconds', async () => { + await addSubscription(TEST_DID, FEED_A); + const report = brokenFeed(FEED_A); + await reportHealth([report]); + + const body = await timeline(); + expect(body.feedHealth?.[FEED_A]).toEqual({ + errorCount: 3, + error: 'Failed to fetch (HTTP 404)', + lastErrorAt: report.lastErrorAt * 1000, + nextRetryAt: report.nextRetryAt * 1000, + lastFetchedAt: report.lastFetchAt * 1000, + }); + }); + + it('records a feed that has never ingested a single item', async () => { + // Broken from its first crawl: no items were ever pushed, so there is no + // `feeds` row for the health report to update. + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_B)]); + expect((await timeline()).feedHealth?.[FEED_B]?.errorCount).toBe(3); + }); + + it('does not leak another user’s broken feeds', async () => { + await addSubscription(OTHER_DID, FEED_B); + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]); + + const body = await timeline(); + expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_A]); + }); + + it('clears a feed that recovers, by absence from the next report', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A)]); + expect((await timeline()).feedHealth?.[FEED_A]).toBeDefined(); + + // The crawler no longer lists it — that is the whole recovery signal. + await reportHealth([]); + expect((await timeline()).feedHealth?.[FEED_A]).toBeUndefined(); + }); + + it('recovers one feed without disturbing another that is still broken', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]); + await reportHealth([brokenFeed(FEED_B)]); + + const body = await timeline(); + expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_B]); + }); + + it('omits the payload when the client already holds the current revision', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A)]); + + const first = await timeline(); + expect(first.feedHealth).toBeDefined(); + expect(first.healthRev).toBeTruthy(); + + // Steady state: the client echoes the revision back and pays nothing. + const second = await timeline({ + since_seq: String(first.cursor), + generation: first.generation, + health_rev: first.healthRev!, + }); + expect(second.feedHealth).toBeUndefined(); + expect(second.healthRev).toBe(first.healthRev); + }); + + it('re-sends the payload once the unhealthy set changes', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_A)]); + const first = await timeline(); + + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B, { errorCount: 1 })]); + const second = await timeline({ + since_seq: String(first.cursor), + generation: first.generation, + health_rev: first.healthRev!, + }); + + expect(second.healthRev).not.toBe(first.healthRev); + expect(Object.keys(second.feedHealth ?? {}).sort()).toEqual([FEED_A, FEED_B].sort()); + }); + + it('always sends health on a cold start, whatever revision the client claims', async () => { + // A cold start replays already-archived items, including from a feed that + // has broken since — so its blanket "these delivered, they're fine" pass + // must be corrected in the same response. + await addSubscription(TEST_DID, FEED_A); + await ingest(FEED_A, [{ item: item('c1'), contentHash: 'c1' }]); + await reportHealth([brokenFeed(FEED_A)]); + + const rev = (await timeline()).healthRev!; + const cold = await timeline({ health_rev: rev }); + expect(cold.coldStart).toBe(true); + expect(cold.items).toHaveLength(1); + expect(cold.feedHealth?.[FEED_A]).toBeDefined(); + }); + + it('clearFeedHealth clears one feed and moves the revision', async () => { + await addSubscription(TEST_DID, FEED_A); + await addSubscription(TEST_DID, FEED_B); + await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]); + const before = (await timeline()).healthRev; + + await clearFeedHealth(env as Env, FEED_A); + + const body = await timeline(); + expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_B]); + expect(body.healthRev).not.toBe(before); + }); + + it('flags a starved feed without telling readers anything', async () => { + // `crawl_stale` is an operator signal: the crawler isn't reaching the feed, + // but nothing is erroring, so the reader has no error to show. + await addSubscription(TEST_DID, FEED_A); + await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: true }]); + + const body = await timeline(); + expect(body.feedHealth?.[FEED_A]).toBeUndefined(); + + const row = await env.DB.prepare( + 'SELECT error_count, crawl_stale FROM feeds WHERE feed_url = ?' + ) + .bind(FEED_A) + .first<{ error_count: number; crawl_stale: number }>(); + expect(row).toMatchObject({ error_count: 0, crawl_stale: 1 }); + }); + + it('clears a starved flag once the crawler catches up', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: true }]); + await reportHealth([]); + + const row = await env.DB.prepare('SELECT crawl_stale FROM feeds WHERE feed_url = ?') + .bind(FEED_A) + .first<{ crawl_stale: number }>(); + expect(row?.crawl_stale).toBe(0); + }); + + it('carries both flags for a feed that is erroring and starved', async () => { + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A, { crawlStale: true })]); + + const row = await env.DB.prepare( + 'SELECT error_count, crawl_stale FROM feeds WHERE feed_url = ?' + ) + .bind(FEED_A) + .first<{ error_count: number; crawl_stale: number }>(); + expect(row).toMatchObject({ error_count: 3, crawl_stale: 1 }); + // Readers still see the error, which is the part they can act on. + expect((await timeline()).feedHealth?.[FEED_A]?.errorCount).toBe(3); + }); + + it('ignores an entry that claims no fault at all', async () => { + // Otherwise a healthy feed listed by mistake would stay flagged forever. + await addSubscription(TEST_DID, FEED_A); + await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: false }]); + + const row = await env.DB.prepare('SELECT feed_url FROM feeds WHERE feed_url = ?') + .bind(FEED_A) + .first(); + expect(row).toBeNull(); + }); + + it('leaves the revision alone when only the starved flag moves', async () => { + // The reader payload holds erroring feeds only, so a crawl-capacity change + // must not make every client re-download it. + await addSubscription(TEST_DID, FEED_A); + await reportHealth([brokenFeed(FEED_A)]); + const before = (await timeline()).healthRev; + + await addSubscription(TEST_DID, FEED_B); + await reportHealth([ + brokenFeed(FEED_A), + { feedUrl: FEED_B, errorCount: 0, crawlStale: true }, + ]); + expect((await timeline()).healthRev).toBe(before); + }); + + it('leaves the revision alone when a report changes nothing', async () => { + await addSubscription(TEST_DID, FEED_A); + const report = brokenFeed(FEED_A); + await reportHealth([report]); + const first = (await timeline()).healthRev; + + await reportHealth([report]); + expect((await timeline()).healthRev).toBe(first); + }); + }); +}); diff --git a/backend/test/feeds-v2-fetch.spec.ts b/backend/test/feeds-v2-fetch.spec.ts new file mode 100644 index 00000000..1cb9f9c4 --- /dev/null +++ b/backend/test/feeds-v2-fetch.spec.ts @@ -0,0 +1,142 @@ +import { env } from 'cloudflare:test'; +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { handleV2FeedFetch } from '../src/routes/feeds-v2'; +import type { Env, Session } from '../src/types'; + +/** + * `GET /api/v2/feeds/fetch` reads the D1 archive, and pulls a feed through the + * crawler when the archive holds nothing for it. That pull-through is the one + * user-triggered write into a shared, never-pruned archive, so it is gated on the + * caller's own subscription. + */ + +const TEST_DID = 'did:plc:fetchtester'; +const FEED_URL = 'https://example.com/fetch-feed.xml'; + +const SESSION: Session = { + did: TEST_DID, + handle: 'fetch.bsky.social', + pdsUrl: 'https://test.pds.example', + accessToken: 'token', + refreshToken: 'refresh', + dpopPrivateKey: '{}', + expiresAt: Date.now() + 3600000, +}; + +function fetchRequest(params: Record): Request { + const search = new URLSearchParams(params).toString(); + return new Request(`https://api.example/api/v2/feeds/fetch?${search}`); +} + +function proxyFeed(guids: string[]): Response { + return new Response( + JSON.stringify({ + feed: { + title: 'Pulled Feed', + siteUrl: 'https://example.com', + // The proxy serves newest-first. + items: guids.map((guid) => ({ + guid, + url: `https://example.com/${guid}`, + title: `Title ${guid}`, + publishedAt: '2026-01-01T00:00:00.000Z', + })), + }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } } + ); +} + +async function subscribe(feedUrl: string) { + await env.DB.prepare( + `INSERT OR IGNORE INTO users (did, handle, pds_url, tier, created_at) + VALUES (?, 'fetch.test', 'https://test.pds.example', 'free', unixepoch())` + ) + .bind(TEST_DID) + .run(); + await env.DB.prepare( + `INSERT INTO subscriptions_cache (user_did, record_uri, feed_url, title, active) + VALUES (?, ?, ?, 'Feed', 1)` + ) + .bind(TEST_DID, `at://${TEST_DID}/app.skyreader.feed.subscription/${Date.now()}`, feedUrl) + .run(); +} + +describe('handleV2FeedFetch (archive + gated pull-through)', () => { + let originalFetch: typeof fetch; + let proxyCalls: string[]; + + beforeEach(() => { + originalFetch = globalThis.fetch; + proxyCalls = []; + (env as Env).FEED_PROXY_URL = 'https://proxy.example'; + (env as Env).FEED_PROXY_SECRET = 'test-secret'; + globalThis.fetch = vi.fn(async (input: RequestInfo | URL) => { + proxyCalls.push(typeof input === 'string' ? input : input.toString()); + return proxyFeed(['newest', 'middle', 'oldest']); + }) as unknown as typeof fetch; + }); + + afterEach(async () => { + globalThis.fetch = originalFetch; + vi.restoreAllMocks(); + await env.DB.prepare('DELETE FROM feed_items').run(); + await env.DB.prepare('DELETE FROM feeds').run(); + await env.DB.prepare('DELETE FROM subscriptions_cache').run(); + }); + + it('pulls a subscribed feed through the crawler and serves it newest-first', async () => { + await subscribe(FEED_URL); + + const res = await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + expect(res.status).toBe(200); + const body = (await res.json()) as { title: string; items: Array<{ guid: string }> }; + + expect(proxyCalls.length).toBe(1); + expect(body.title).toBe('Pulled Feed'); + expect(body.items.map((i) => i.guid)).toEqual(['newest', 'middle', 'oldest']); + + // Ingested oldest→newest, so the archive's seq order matches feed order. + const rows = await env.DB.prepare( + 'SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq ASC' + ) + .bind(FEED_URL) + .all<{ guid: string }>(); + expect(rows.results.map((r) => r.guid)).toEqual(['oldest', 'middle', 'newest']); + }); + + it('does not write the shared archive for a feed the caller does not subscribe to', async () => { + const res = await handleV2FeedFetch( + fetchRequest({ url: FEED_URL, refresh: '1' }), + env, + SESSION + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: unknown[] }; + + expect(proxyCalls).toEqual([]); + expect(body.items).toEqual([]); + const count = await env.DB.prepare('SELECT COUNT(*) AS c FROM feed_items').first<{ + c: number; + }>(); + expect(count?.c).toBe(0); + }); + + it('serves the archive without a crawl once the feed is ingested', async () => { + await subscribe(FEED_URL); + await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + expect(proxyCalls.length).toBe(1); + + const res = await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + const body = (await res.json()) as { items: Array<{ guid: string }> }; + expect(proxyCalls.length).toBe(1); + expect(body.items.map((i) => i.guid)).toEqual(['newest', 'middle', 'oldest']); + }); + + it('re-crawls a subscribed feed on refresh=1', async () => { + await subscribe(FEED_URL); + await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION); + await handleV2FeedFetch(fetchRequest({ url: FEED_URL, refresh: '1' }), env, SESSION); + expect(proxyCalls.length).toBe(2); + }); +}); diff --git a/backend/test/ops-metrics.spec.ts b/backend/test/ops-metrics.spec.ts index 4ab44bda..84cc0363 100644 --- a/backend/test/ops-metrics.spec.ts +++ b/backend/test/ops-metrics.spec.ts @@ -49,6 +49,7 @@ const pollerStatusBody = (lagMs: number | null) => ({ beforeEach(async () => { await env.DB.prepare('DELETE FROM system_status').run(); await env.DB.prepare('DELETE FROM metrics_snapshots').run(); + await env.DB.prepare('DELETE FROM feeds').run(); }); afterEach(() => vi.restoreAllMocks()); @@ -226,17 +227,19 @@ describe('writeMetricsSnapshot', () => { await writeSystemStatus( env as Env, 'proxy_stats', - { freshPct: 88.5 } as ProxyStatsValue, + { freshPct: 88.5, feedsInError: 7 } as ProxyStatsValue, now - 60_000 ); await writeMetricsSnapshot(env as Env, now); const row = await env.DB.prepare( - 'SELECT firehose_lag_ms, proxy_fresh_pct FROM metrics_snapshots' - ).first<{ firehose_lag_ms: number; proxy_fresh_pct: number }>(); + 'SELECT firehose_lag_ms, proxy_fresh_pct, feeds_with_errors FROM metrics_snapshots' + ).first<{ firehose_lag_ms: number; proxy_fresh_pct: number; feeds_with_errors: number }>(); expect(row?.firehose_lag_ms).toBe(42_000); expect(row?.proxy_fresh_pct).toBe(88.5); + // Per-feed errors come from the crawler now — `feeds` has no error column. + expect(row?.feeds_with_errors).toBe(7); }); it('records null rather than a stale value nobody refreshed', async () => { @@ -250,10 +253,31 @@ describe('writeMetricsSnapshot', () => { await writeMetricsSnapshot(env as Env, now); - const row = await env.DB.prepare('SELECT firehose_lag_ms FROM metrics_snapshots').first<{ - firehose_lag_ms: number | null; - }>(); + const row = await env.DB.prepare( + 'SELECT firehose_lag_ms, feeds_with_errors FROM metrics_snapshots' + ).first<{ firehose_lag_ms: number | null; feeds_with_errors: number | null }>(); expect(row?.firehose_lag_ms).toBeNull(); + // No proxy row at all: unknown, not "zero feeds are erroring". + expect(row?.feeds_with_errors).toBeNull(); + }); + + it('counts the feeds the crawler ingests into, not the dropped feed_metadata', async () => { + const now = 300 * HOUR_MS; + await env.DB.batch([ + env.DB.prepare( + "INSERT INTO feeds (feed_url, title, last_ingest_at) VALUES ('https://a.example/f', 'A', 1)" + ), + env.DB.prepare( + "INSERT INTO feeds (feed_url, title, last_ingest_at) VALUES ('https://b.example/f', 'B', 2)" + ), + ]); + + await writeMetricsSnapshot(env as Env, now); + + const row = await env.DB.prepare('SELECT feeds FROM metrics_snapshots').first<{ + feeds: number; + }>(); + expect(row?.feeds).toBe(2); }); it('prunes points past the retention window', async () => { diff --git a/backend/wrangler.toml b/backend/wrangler.toml index 659679a1..395cacaf 100644 --- a/backend/wrangler.toml +++ b/backend/wrangler.toml @@ -84,6 +84,17 @@ vars = { FRONTEND_URL = "http://127.0.0.1:5173" } # Staging environment [env.staging] name = "skyreader-api-staging" +# Staging is MEANT to talk to its own Fly proxy (skyreader-feed-proxy-staging), +# which pushes only into the staging D1 — prod and staging as two fully isolated +# pairs, no cross-links, distinct PROXY_SECRET/FEED_PROXY_SECRET values. +# +# The URL below still points at the prod proxy on purpose: staging deploys on +# every push to main, so flipping it before the app exists would take /extract, +# discovery, standard.site documents and social context down on staging. Flip it +# to https://skyreader-feed-proxy-staging.fly.dev in the SAME deploy that rotates +# FEED_PROXY_SECRET, once Phase 3 of docs/plans/D1_FEED_TIMELINE.md has actually +# provisioned the app. Until then staging clients stay on the legacy batch path +# by themselves (the timeline reports ingestActive: false). vars = { FRONTEND_URL = "https://staging.skyreader.app", ALLOWED_ORIGINS = "https://staging.skyreader.app,https://staging-linkblogs.skyreader.app", FEED_PROXY_URL = "https://skyreader-feed-proxy.fly.dev", LINKBLOG_PUBLIC_URL = "https://staging-linkblogs.skyreader.app", SENTRY_ENVIRONMENT = "staging" } routes = [{ pattern = "api-staging.skyreader.app", custom_domain = true }] diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index cec9ff6b..bbf7863a 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -23,7 +23,7 @@ it rather than learning to ignore it. | Backend Worker | Structured logs (Workers Logs) | One JSON object per event, keyed by `requestId`. | | Backend cron (every minute) | Heartbeat ping (`HEARTBEAT_URL`) | Dead-man's switch; also guards the firehose. | | Feed proxy | `GET /health` | Status, version, cached feed count. No auth. | -| Feed proxy | `GET /stats` | Cache freshness + per-feed error counts. Secret. | +| Feed proxy | `GET /stats` | Cache freshness, per-feed errors, ingest backlog. | | Feed proxy | Sentry (`@sentry/bun`) | Route escapes, warmer failures. | | Feed proxy warmer | Heartbeat ping (`WARM_HEARTBEAT_URL`) | Dead-man's switch for the warm loop. | | Backend cron (every minute) | `system_status` rows (D1) | Cron liveness, poller lag, proxy cache stats. | @@ -306,17 +306,17 @@ poll cycle — a `requestId`, plus `route` and `did` when known. Query patterns that pay for themselves: -| Question | Filter | -| -------------------------------- | ------------------------------------------------- | -| Everything about one failure | `requestId = ` | -| Error rate on one endpoint | `event = request AND route = /api/v2/feeds/batch` | -| Slow requests | `event = request AND durationMs > 3000` | -| Is the cron doing its work? | `event = cron_run` | -| Is the firehose keeping up? | `event = jetstream_poll` → `subscriptionsLagMs` | -| What failed inside a cron run? | `event = cron_phase_failed` → `phase` | -| Why is the ops panel stale? | `event = ops_metrics_failed` → `step` | -| Did the hourly trend point land? | `event = metrics_snapshot` → `prunedRows` | -| Are browsers throwing? | `event = client_error` → `kind`, `appVersion` | +| Question | Filter | +| -------------------------------- | ----------------------------------------------- | +| Everything about one failure | `requestId = ` | +| Error rate on one endpoint | `event = request AND route = /api/v2/timeline` | +| Slow requests | `event = request AND durationMs > 3000` | +| Is the cron doing its work? | `event = cron_run` | +| Is the firehose keeping up? | `event = jetstream_poll` → `subscriptionsLagMs` | +| What failed inside a cron run? | `event = cron_phase_failed` → `phase` | +| Why is the ops panel stale? | `event = ops_metrics_failed` → `step` | +| Did the hourly trend point land? | `event = metrics_snapshot` → `prunedRows` | +| Are browsers throwing? | `event = client_error` → `kind`, `appVersion` | The id is also returned to callers as the `X-Request-Id` response header (exposed via CORS), so a user-reported failure can be traced if they can quote it. The @@ -371,6 +371,9 @@ days, pruned by the job that writes it; the panel shows the most recent 30. Cadence, if a number looks older than expected: poller status every minute, proxy stats every 5th minute, snapshot once an hour on the hour. +The **Feeds** tiles in the metrics section below the ops panel cover the ingest +side; §4d says what they mean and how to check the pieces they can't see. + --- ## 4c. Client signal @@ -409,6 +412,85 @@ shape of most PWA weirdness, and a reload fixes it. --- +## 4d. Ingest health + +Feed reads are served from D1, so what a reader sees is only as fresh as the +crawler's **push**, not the crawler's cache. Proxy cache freshness (§4b) is the +first hop of two; this section is the second. **None of it is wired to an alert +yet** — the signals are on the admin and on the proxy, and this is the list to +walk when the reader looks stale but every tile above is green. + +| Signal | Where | Healthy | How to check | +| --------------------------- | ----------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------- | +| Crawler talking to us | `sync_state.crawler_heartbeat_at` (D1) | stamped within ~5 min | `npx wrangler d1 execute skyreader --remote --command "SELECT * FROM sync_state WHERE key = 'crawler_heartbeat_at'"` | +| Clients reading the archive | `GET /api/v2/timeline` → `ingestActive` | `true` in an ingesting environment | Any authenticated timeline response | +| Rollout gate | `sync_state.timeline_enabled` (D1) | `'1'` once rolled out | `npx wrangler d1 execute skyreader --remote --command "SELECT * FROM sync_state WHERE key = 'timeline_enabled'"` | +| Push backlog (outbox) | proxy `GET /stats` → `ingest.pending` | near zero in steady state | `curl -H "X-Proxy-Secret: $SECRET" https://skyreader-feed-proxy.fly.dev/stats` | +| Push failing | Sentry `source: ingest-push`, proxy logs | silent | `fly logs -a skyreader-feed-proxy` → `Ingest push failed` | +| Feeds gone quiet | Admin → Feeds, "Subscribed Feeds Not Ingesting" | small and stable | The dashboard's Feeds metrics | + +Reading these correctly: + +- **`crawler_heartbeat_at` is per-environment and stamped by _both_ internal + endpoints** (`/api/internal/ingest` and the 5-minutely `/api/internal/crawl-set` + pull), so it keeps ticking through a quiet period with no new items. A missing + stamp means the proxy isn't talking to this Worker at all — usually `INGEST_URL` + unset or pointed at the wrong environment, which is also the deliberate state + before Phase 3. The consequence is not an outage: `ingestActive` goes false and + clients fall back to the legacy batch path. +- **`ingest.pending` is the outbox depth**, and the push loop drains 100 items + every 15s (~400/min). A few hundred is churn. A number that holds in the + thousands across two readings means the push is erroring, not busy — check + Sentry before touching the machine. The exception is the **first** enablement of + `INGEST_URL` in an environment: the whole item log is dirty at once and the + backfill legitimately takes hours to drain. +- **`feeds.last_ingest_at` is the last time that feed produced _new or changed_ + items**, not the last time it was crawled — nothing stamps it on a fetch that + found nothing. A feed that publishes weekly therefore looks "not ingesting" for + a week, so read the admin's count as a trend (a jump means the crawl set or the + push stopped) rather than as a per-feed verdict. + +Fix, once you know which one it is: a wedged warm loop or push loop is +`fly machine restart ` (never `fly scale count` — see §4's proxy-warmer entry); +a heartbeat that never arrives is configuration, not a restart. + +### The timeline rollout gate + +`ingestActive` is the AND of two things: a fresh crawler heartbeat and +`sync_state.timeline_enabled`. They are separate because the heartbeat lands +_seconds_ into the first backfill of an environment and that backfill takes hours +— without the gate, every reader switches to the timeline at the moment the +archive is emptiest and then drags the whole backfill through the incremental +drain (the expensive global scan, at its worst case, for the duration). + +Only an explicit `'0'` gates. Migration 0071 sets it for a database that already +has users, so prod and staging start shut and a fresh environment (local dev, +e2e, CI) starts open. + +```bash +# Open it — after ingest.pending has trended to ~0. +npx wrangler d1 execute skyreader --remote --command \ + "UPDATE sync_state SET value='1', updated_at=unixepoch() WHERE key='timeline_enabled'" + +# Shut it — every client is back on the legacy batch path at its next poll. +npx wrangler d1 execute skyreader --remote --command \ + "UPDATE sync_state SET value='0', updated_at=unixepoch() WHERE key='timeline_enabled'" +``` + +Shutting it is the **fast rollback for the read path**, and the first thing to +reach for if the timeline misbehaves after a rollout: no Worker deploy, no waiting +out the 30-minute heartbeat freshness window, and the crawler keeps filling the +archive the whole time. It is not a fix for a bad Worker deploy generally — only +for "readers should not be on the timeline right now". + +One caveat when shutting it: clients hold a committed `timelineCursor` and stop +advancing their per-subscription `feedCursors` while on the timeline, so the +batch path re-drains from wherever those cursors were left. The proxy's K=200 +window bounds that, and the merge dedupes by GUID, so the cost is one heavier +sync, not duplicates. + +--- + ## 5. Post-deploy smoke checks Every deploy workflow except the admin's ends with `scripts/smoke-check.mjs`: @@ -509,7 +591,7 @@ admire, and not promises to anyone outside the project. Their only job is to mak | ------------------ | ------------------------------------ | ----------------------------------- | ------- | | API availability | 99.5% | Uptime check on `/api/health` | 30 days | | Firehose freshness | lag < 5 min, p95 of hourly snapshots | `metrics_snapshots.firehose_lag_ms` | 30 days | -| Feed freshness | ≥95% fresh, p05 of hourly snapshots | `metrics_snapshots.proxy_fresh_pct` | 30 days | +| Crawl freshness | ≥95% fresh, p05 of hourly snapshots | `metrics_snapshots.proxy_fresh_pct` | 30 days | | Cron liveness | gap < 5 min | Heartbeat check history | 30 days | 99.5% is ~3.6 hours a month. That is a deliberately loose target for a @@ -517,6 +599,15 @@ single-operator project on a singleton proxy: it says "a couple of short outages month is survivable, a daily one is not." Tighten it only if you're also willing to change the architecture it's measuring. +**Crawl freshness is the first of two hops, not the whole of feed freshness.** +`proxy_fresh_pct` says the crawler's own cache is current; since reads moved to D1, +what a reader actually sees also depends on the push that carries those items into +`feed_items` (§4d). The second hop has no SLO here because nothing records it +hourly — the outbox depth lives on the proxy's `/stats` and is read by hand. Making +it one means recording `ingest.pending` into `metrics_snapshots` beside the numbers +above; until then, say "crawl freshness" and mean it, rather than claiming a +reader-facing number this table can't compute. + Both freshness rows aggregate the **hourly** trend points, not the live tile: the tile is a point-in-time reading and "95% fresh right now" says nothing about a month. p95 for lag and p05 for freshness are the same statement pointed in @@ -534,7 +625,7 @@ npx wrangler d1 execute skyreader --remote --command " LIMIT 1 OFFSET (SELECT CAST(COUNT(*) * 95 / 100 AS INT) FROM metrics_snapshots WHERE captured_at > (unixepoch() - 30*86400) * 1000 AND firehose_lag_ms IS NOT NULL)" -# Proxy cache freshness, p05 (the 5th-worst-percent hour). Passes if >= 95. +# Crawl freshness (proxy cache), p05 (the 5th-worst-percent hour). Passes if >= 95. npx wrangler d1 execute skyreader --remote --command " SELECT proxy_fresh_pct FROM metrics_snapshots WHERE captured_at > (unixepoch() - 30*86400) * 1000 AND proxy_fresh_pct IS NOT NULL diff --git a/docs/plans/D1_FEED_TIMELINE.md b/docs/plans/D1_FEED_TIMELINE.md new file mode 100644 index 00000000..7d6217ec --- /dev/null +++ b/docs/plans/D1_FEED_TIMELINE.md @@ -0,0 +1,267 @@ +# D1-Served Feed Timeline (ingest-push architecture) + +> Supersedes, in part, `RETENTION_SYNC_PLAN.md`. That plan made the Fly proxy a durable item +> log with a monotonic cursor and it stays exactly as built — but its **serving** role moves +> to D1. The proxy is now the **crawler and outbox**; D1 is the **archive and read path**. + +## What changed + +A refresh used to be `1 + ceil((N−8)/50)` sequential `POST /api/v2/feeds/batch` calls, each +hopping Worker → Fly proxy and then paying `ceil(GUIDs/88)` sequential D1 queries to annotate +read state (`getReadKeys`). Now: + +``` + ┌────────────── crawl set (pull, every ~5 min) ──────────────┐ + ▼ │ + Fly proxy (crawler only) Cloudflare Worker + D1 (serving store) + ┌──────────────────────────┐ push deltas ┌──────────────────────────────────────┐ + │ warm loop / etag / │ ──────────────▶ │ POST /api/internal/ingest │ + │ backoff / UA fallback │ (at-least-once,│ upsert feeds + feed_items │ + │ feed_items = outbox │ seq order, │ (archive: no pruning; sanity cap) │ + │ (bounded, K = 200) │ ONE target) │ │ + └──────────────────────────┘ │ GET /api/v2/timeline?since_seq=&… │ + │ ONE query: feed_items ⋈ subs ⋈ read│ + Client ◀───────────────── one paged request ─┴──────────────────────────────────────┘ + + (× 2: one such pair for prod, one for staging — no cross-links) +``` + +- **Reads never touch Fly.** If the Fly box is down, refresh still works from D1 (freshness + stalls, serving doesn't). +- **D1 is the archive; the proxy log is a window.** The proxy keeps its K = 200 recent items per + feed as a delivery buffer; D1 keeps everything it has ever ingested. Prod D1 is therefore the + system of record for feed history — its Time Travel / export story matters. +- The proxy keeps `/extract`, `/discover`, `/documents`, `/social-context`, `/mentions`, and the + single-feed `/feed` (now only for subscribe-time pull-through). + +## Shipped in this change + +**Backend** (`backend/`) + +- Migration `0068_feed_timeline.sql`: drops the dormant pre-Fly trio (`feed_items`, `feed_cache`, + `feed_metadata` — zero references in `src/`), creates `feeds` + `feed_items` (AUTOINCREMENT + `seq`, `UNIQUE(feed_url, guid)`, `content_hash NOT NULL`), the `(user_did, feed_url)` + subscription index, and mints `sync_state.items_generation`. +- `routes/ingest.ts`: `POST /api/internal/ingest` and `GET /api/internal/crawl-set`, both + authenticated by a constant-time compare against `FEED_PROXY_SECRET` and **fail-closed** when + it is unset. Idempotent upsert (edit-in-place keeps the seq), per-item content cap + (`MAX_ITEM_CONTENT_BYTES = 8 KB`, drops `content` and sets `contentTruncated`), and the per-feed + `SANITY_CAP = 5000` trim — the only pruning that ever runs. Both endpoints stamp a **crawler + heartbeat** (`sync_state.crawler_heartbeat_at`); the crawl-set pull runs every 5 minutes, so the + stamp is fresh even when no feed produced an item. +- `routes/timeline.ts`: `GET /api/v2/timeline?since_seq=&generation=&limit=&cold_offset=` — + incremental drain (cursor derived from returned rows, `hasMore` via `limit+1`) and a **paged** + per-feed newest-30 cold start (feeds walked in a stable order, `COLD_START_MAX_ITEMS` per page, + continuation via `nextColdOffset`). Read state is an `EXISTS` probe in the same query; + `getReadKeys` is never called on the feed path. Every response carries `ingestActive` — the AND of + a fresh crawler heartbeat (this deployment has a crawler filling D1) and the `timeline_enabled` + rollout gate (an operator has admitted readers to it; see Phase 4). False on either half means + clients stay on the legacy batch path, and the request short-circuits rather than building a page + they will discard. A cursor above the archive head cold-starts (rewound-archive guard). +- `routes/ingest.ts` also serves `POST /api/internal/feed-health`: the crawler's periodic report of + every feed it currently considers broken, which the timeline hands to readers. On the batch path + a failing feed came back with `status: 'error'` inline; reads no longer touch the proxy, and a + broken feed pushes no items, so without this a dead feed and a quiet feed look identical to the + client. The payload is the **complete trouble set, not a delta** — recovery is inferred from a + feed's absence, so nothing has to announce that it started working. `feeds` carries the state + (`error_count`, `last_error`, `last_error_at`, `next_retry_at`, `last_fetch_at`, `crawl_stale`, + migration `0070`), and a report inserts rather than assumes a row, because a feed broken from its + first crawl has never ingested an item. Two distinct faults ride the same report: `error_count` + (the fetch fails — readers see this) and `crawl_stale` (the feed is in the crawl set but going + unfetched for hours — an operator signal only, invisible to readers and deliberately excluded from + `feed_health_rev` so crawl-capacity churn can't make every client re-download the payload). +- `GET /api/v2/feeds/fetch` re-backed with D1 + **pull-through**: if a feed isn't in the archive + yet (first subscriber), fetch it from the proxy once, ingest it, then serve. The pull-through is + gated on the caller's own subscription, so the shared never-pruned archive can't be written with + arbitrary feeds. Subscribe time already crawls + ingests the feed (`warmFeedIntoArchive`, which + replaced the old warm-and-discard), so the pull-through is normally not needed at all. The + response carries the feed's `health` when it has any, and a successful pull-through clears it + (`clearFeedHealth`) — that path _is_ the user's "retry this feed" action, so it must show a result + now rather than after the crawler's next report. + +**Proxy** (`feed-proxy/`) + +- `ingest-push.ts`: the durable log _is_ the outbox. `push_state(seq, pushed_hash)` marks what has + been delivered; a row is dirty when it's missing there or the hashes differ (race-free against a + mid-flight edit). Seq-ordered batches of 100 every 15 s, capped exponential backoff on failure. +- Crawl-set pull every 5 min: registers each feed's `cache` row and stamps `last_requested_at`, so + the existing warm loop / active window / eviction machinery keeps working now that read traffic + no longer stamps anything. +- A feed-health report rides the same 5-minute timer, sent right **after** the crawl-set pull so a + feed registered this cycle is already in the set. `selectFeedHealth` reads the crawl-set `cache` + rows that are either erroring (`error_count > 0`) or unfetched for `CRAWL_STALE_MS` (2 h — the + warm loop works on a minutes-long cadence, so hours means the feed is losing its turn every tick), + converting its millisecond timestamps to seconds once, at this boundary. +- `INGEST_URL` unset ⇒ both loops disabled. `push_state` cascades on the K = 200 cap trim, the + redelivery delete, and `cleanupCache` eviction. `/stats` reports `ingest.{items,pushed,pending}`. + +**Frontend** (`frontend/`) + +- `feedFetcher.fetchAllFeeds` prefers one `GET /api/v2/timeline` (plus drain pages) with a single + global cursor in Dexie `metadata` (`timelineCursor`). Pure helpers live in `timelineSync.ts`. +- Falls back to the legacy `/api/v2/feeds/batch` path when the timeline 404s (old or rolled-back + backend) **or** whenever the server reports `ingestActive: false`. The cursor is never committed + in that case. (An `ingestActive`-less backend keeps the old empty-cold-start heuristic.) +- A paged cold start commits only the FIRST page's cursor, and only after its last page merges, so + an interrupted bootstrap starts over instead of skipping feeds it never delivered. +- Subscriptions that arrive from another device sit below the global cursor, so each is backfilled + once through the per-feed endpoint (`backfillMissingSubscriptions`, ≤ 10 per sync, attempts + recorded in Dexie `metadata.timelineBackfilledFeeds`). +- OPML import backfills via the per-feed endpoint: a freshly imported feed's items sit below the + global cursor, so only the single-feed path can deliver them. The requests are paced (3 at a + time, 1 s apart) and no longer force a crawl, so a 250-feed import stays inside the rate limit. +- An article whose body was dropped at ingest (`contentTruncated`) is extracted automatically when + its card opens, so the reader shows the whole article rather than an RSS summary. +- Per-feed error badges come from the response's `feedHealth` (`reconcileFeedHealth` → + `feedStatusStore.applyHealthSnapshot`). Absence from that map is what CLEARS an error, so the + reconcile runs after the "these feeds delivered items, so they're fine" pass and overrides it — a + cold start replays archived items from feeds that may have broken since. To keep the steady-state + poll at one query, the payload is only sent when the client's echoed `health_rev` is stale, plus + unconditionally on a cold start (whose status store may be empty). + +**Admin** (`admin/`) — feed health is re-pointed at `feeds`/`feed_items`: crawled feeds, archived +item count, estimated archive size with a 6 GB alert, and a churn detector for feeds nearing the +sanity cap. Health itself is the **crawler's verdict**, not an inference: + +- **Subscribed Feeds Erroring** (`error_count > 0`) and **Subscribed Feeds Not Being Crawled** + (`crawl_stale = 1`) replace the single "Subscribed Feeds Not Ingesting" tile. That tile keyed off + `last_ingest_at`, which only moves when a fetch yields a NEW item — so it counted every feed that + simply hadn't published in an hour, warned permanently, and told an operator nothing. +- The Feeds page filters on All / Erroring / Not Crawled / OK, sorts by `error_count`, and shows the + crawler's actual message, failure count, retry time and last good fetch on the row. `last_ingest_at` + stays, relabelled "Last Item", as what it really is: publishing cadence. +- The existing "Proxy Feeds in Error" tile and its trend series are untouched. They count the + proxy's whole cache, orphaned feeds included, and stay sourced from `proxy_stats` so the series + keeps matching its tile; the new tiles are subscriber-scoped and read D1. + +## Operator steps (not code) + +### Phase 0 — measure (before enabling prod ingest) + +Snapshot prod so the growth math becomes a projection: proxy `feed_items` count and `item_json` +size distribution (drives backfill time and confirms the 8 KB content cap), per-feed new-item +velocity over a sample window (items/day, and it flags existing GUID-churn feeds), and current +refresh request counts from logs (the baseline the win is measured against). + +### Phase 3 — provision the staging Fly proxy, then soak + +```bash +# 1. Create the app (same org as skyreader-feed-proxy) +fly apps create skyreader-feed-proxy-staging + +# 2. Create its volume (same region as primary_region; 1 GB is ample for staging) +fly volumes create proxy_data -a skyreader-feed-proxy-staging -r sjc -s 1 + +# 3. Mint a DISTINCT staging secret (don't reuse prod's — smaller blast radius) +openssl rand -hex 32 # → +fly secrets set PROXY_SECRET= -a skyreader-feed-proxy-staging + +# 4. Set the same value on the staging Worker (it is both the outbound proxy auth +# and the inbound ingest auth) +cd backend && npx wrangler secret put FEED_PROXY_SECRET --env staging + +# 5. First deploy (CI takes over afterwards, on every push to main) +cd feed-proxy && fly deploy --remote-only --config fly.staging.toml +``` + +Sequencing matters: today's staging Worker authenticates to the **prod** proxy with prod's secret, +so provision + deploy the staging proxy first, then flip `FEED_PROXY_URL` in +`backend/wrangler.toml` (`[env.staging]`, still pointing at the prod proxy on purpose) and rotate +`FEED_PROXY_SECRET` together in one staging Worker deploy. The CI staging Fly job skips itself with +a notice until the app exists, so nothing breaks in the meantime. + +Verify: `fly status -a skyreader-feed-proxy-staging` shows exactly **one** machine (the singleton +invariant applies to this app too — never `fly scale count`); `/stats` answers and its +`ingest.pending` trends to ~0; staging D1 `feed_items` counts rise monotonically; the prod proxy's +logs show no staging-origin traffic. + +**Then enable prod:** uncomment `INGEST_URL` in `feed-proxy/fly.toml` and cut a release. Prod's +backfill drains through the normal pusher loop (≤ 200 × active feeds at 100 items / 15 s). Until +that happens, prod clients keep using the legacy batch path automatically: with no crawler pushing, +nothing stamps `crawler_heartbeat_at`, so `/api/v2/timeline` answers `ingestActive: false` and no +client commits a cursor. That signal is server-side on purpose — subscribe-time ingest and the +pull-through both write to the archive, so "the archive is empty for this user" would stop being +true long before the crawler existed. + +### Phase 4 — open the gate + +The heartbeat means a crawler is attached; it does **not** mean the initial archive backfill is +complete, and the first stamp lands _seconds_ after the release (the proxy pulls the crawl set +immediately at boot). On its own it would therefore switch every reader onto the timeline at the +moment the archive is emptiest, and each of them would then drag the entire backfill through the +incremental drain — the fan-out-on-read scan at its worst case, for the hours the drain takes, +surfacing back-catalogue items as unread along the way. + +`sync_state.timeline_enabled` (migration `0071`) separates the two. `ingestActive` is the AND of a +fresh heartbeat and this flag, so: + +1. Enable `INGEST_URL`; the crawler fills the archive while every client stays on the batch path. +2. Watch `/stats` until `ingest.pending` trends to ~0. +3. Open the gate — one `UPDATE sync_state`, no deploy (commands in + [`RUNBOOK.md` §4d](../RUNBOOK.md)). Clients switch on their next poll. + +Only an explicit `'0'` gates; an absent row means enabled, so a hand-built schema or a future +environment is never silently held back. The migration writes `'0'` for a database that already has +users, so prod and staging start shut while local dev, e2e and CI start open. + +Setting it back to `'0'` is the **fast rollback for the read path** — every client returns to the +batch path at its next poll, with no Worker deploy and no waiting out the 30-minute heartbeat +freshness window, and the crawler keeps ingesting throughout. A gated timeline request +short-circuits: it answers with the state and an empty page rather than building one the client is +about to discard, so the gated window costs one `sync_state` read per poll. + +### Phase 5 — cleanup (a later release, once no legacy traffic remains) + +Remove `/api/v2/feeds/batch`'s proxy passthrough + its `getReadKeys` call (keep it for documents), +the proxy's `POST /feeds` batch read endpoint, the `since_guids`/`since_seq` client plumbing, Dexie +`feedCursors`, `liveDb.getRecentGuids`, and `fetchAllFeedsViaBatch`. Optionally add an hourly cron +step deleting `feeds`/`feed_items` rows whose feed has had **zero active subscribers for > 90 days** +— the one deliberate deletion path, and skippable if even orphans should stay. + +## Invariants worth keeping + +- **The incremental cursor comes from returned rows, never `MAX(seq)`** — the latter races ingest + and silently skips rows. A cold start is the one exception, and only because it reads the head + BEFORE its per-feed slices: anything ingested while it pages lands above that head and arrives on + the next poll. +- **The heartbeat and the gate answer different questions.** "Is a crawler attached?" is not "is + the archive ready for readers?" — the first stamp arrives seconds into a backfill that takes + hours. Anything that collapses the two back into one signal reintroduces a rollout where every + client switches at the emptiest moment and then drains the entire backfill. +- **Any D1 restore bumps `items_generation`** (one `UPDATE sync_state`): Time Travel rewinds seqs + while the token would otherwise stay the same. The timeline also self-heals a cursor that sits + above the head by cold-starting that client, so a forgotten bump degrades to one extra cold start + rather than a silent, permanent stall. +- **Per-feed recency comes from `published_at`, then `seq`.** A proxy feed is newest-first, so each + ingest call still writes oldest→newest. But subscribe-time pull-through and the crawler backlog + can interleave, making archive `seq` different from publication order; per-feed slices must remain + correct in that case. Incremental delivery continues to use `seq` as its cursor. +- **Writes to the archive need a subscription.** Ordinary ingest deletes nothing, so every + user-triggered write path (subscribe-time ingest, the pull-through) is gated on the caller's own + subscription list. +- **The pusher sends `cache.url`**, the registered URL, never a post-redirect one: the timeline + joins on that exact string. +- **Ordinary ingest deletes nothing.** A feed at the sanity cap is a bug signal (GUID churn), not + steady state — investigate the feed rather than letting it rotate. +- **A health report is the whole trouble set, never a delta.** Recovery is inferred from absence, + so a partial report silently marks feeds healthy. The recovery sweep is a set difference against + the currently flagged feeds (small, partially indexed) rather than a `NOT IN` list of every + healthy feed, which at ~1,300 feeds is the bound-parameter wall `/batch` already hit once. +- **`last_ingest_at` is publishing cadence, not health.** It only moves when a fetch produces a new + or edited item, so a healthy monthly newsletter looks identical to a feed that 404s. Anything + asking "is this feed alive?" has to read `error_count` / `crawl_stale`, which is exactly what the + admin's old stale-ingest alarm got wrong. +- **`nextRetryAt` is milliseconds everywhere the client sees it.** The crawler computes it as + `Date.now() + backoff`; D1 stores seconds like the rest of the backend and converts back on the + way out. Rescaling it a second time is what put every retry ~50,000 years out and made + `canFetch` retire a feed permanently after one transient error. + +## Known scaling knob + +The incremental drain scans the `feed_items` rowid range above the caller's cursor and probes the +subscription set per row, so its cost tracks **global** ingest above the cursor rather than the +caller's own new items: a 5-feed reader returning after a week pays for everything the whole system +ingested that week. That is the accepted fan-out-on-read trade at ~1,330 feeds. If D1 row-reads or +CPU ever become the constraint, bound the scan with per-feed `(feed_url, seq)` seeks against the +subscription set (`idx_feed_items_feed_seq` already supports them) rather than materializing +per-user timelines. diff --git a/docs/plans/RETENTION_SYNC_PLAN.md b/docs/plans/RETENTION_SYNC_PLAN.md index a867484b..d961f86c 100644 --- a/docs/plans/RETENTION_SYNC_PLAN.md +++ b/docs/plans/RETENTION_SYNC_PLAN.md @@ -1,5 +1,11 @@ # Durable Item Retention: Catch Everything Since Last Visit +> **Superseded in part by `D1_FEED_TIMELINE.md`.** Everything below shipped and still runs — but +> the durable log's *serving* role has moved to D1. The proxy log is now the crawler's **outbox** +> (a bounded K = 200 delivery window pushed into D1); the **archive** clients read from is +> `feed_items` in D1, which never prunes, and the cursor clients hold is a single global one +> against that table rather than one per feed. + > Supersedes `BATCH_CURSOR_PLAN.md`. That plan optimized the *request payload* (500 GUIDs → > a cursor) but kept the proxy's replace-the-blob storage model, which structurally cannot > retain items beyond the source feed's live window. This plan changes the storage model so diff --git a/e2e/seed.ts b/e2e/seed.ts index 8f2f77cc..13d72c5f 100644 --- a/e2e/seed.ts +++ b/e2e/seed.ts @@ -136,6 +136,65 @@ export async function seedSavedArticle( return rkey; } +export interface SeedFeedItemOpts { + guid: string; + title: string; + url?: string; + publishedAt?: string; + summary?: string; +} + +/** + * Seed the server-side archive the timeline serves from: a `feeds` row plus its + * `feed_items`. Feed-scoped rather than user-scoped (the archive is shared by + * every subscriber), so it's cleaned up with `cleanupFeedItems`. + */ +export async function seedFeedItems( + feedUrl: string, + items: SeedFeedItemOpts[], + opts: { title?: string; siteUrl?: string } = {} +): Promise { + const nowMs = Date.now(); + const nowSeconds = Math.floor(nowMs / 1000); + + const statements = [ + `INSERT OR REPLACE INTO feeds (feed_url, title, site_url, last_ingest_at, created_at) VALUES (${sqlString(feedUrl)}, ${sqlNullableString(opts.title ?? null)}, ${sqlNullableString(opts.siteUrl ?? null)}, ${nowSeconds}, ${nowSeconds})`, + // Stand in for the crawler's check-in. Without a fresh heartbeat the timeline + // reports `ingestActive: false` and the client (correctly) stays on the legacy + // batch path, which is not what these tests are exercising. + `INSERT INTO sync_state (key, value, updated_at) VALUES ('crawler_heartbeat_at', ${sqlString(String(nowSeconds))}, ${nowSeconds}) ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`, + // The other half of the same gate: a heartbeat says a crawler is attached, + // this says readers may use what it wrote. Migration 0071 leaves it open on a + // fresh database, but a test DB that predates the migration with users in it + // starts closed — so set it rather than depend on which case this one is. + `INSERT INTO sync_state (key, value, updated_at) VALUES ('timeline_enabled', '1', ${nowSeconds}) ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`, + ]; + + items.forEach((item, index) => { + const publishedAt = item.publishedAt ?? new Date(nowMs - index * 60_000).toISOString(); + const itemJson = JSON.stringify({ + guid: item.guid, + url: item.url ?? `https://example.com/${item.guid}`, + title: item.title, + summary: item.summary ?? `Summary for ${item.title}`, + publishedAt, + }); + statements.push( + `INSERT OR REPLACE INTO feed_items (feed_url, guid, item_json, published_at, first_seen_at, content_hash) VALUES (${sqlString(feedUrl)}, ${sqlString(item.guid)}, ${sqlString(itemJson)}, ${Date.parse(publishedAt)}, ${nowMs}, ${sqlString(`hash-${item.guid}`)})` + ); + }); + + await execD1(statements); +} + +export async function cleanupFeedItems(feedUrl: string): Promise { + await execD1([ + `DELETE FROM feed_items WHERE feed_url = ${sqlString(feedUrl)}`, + `DELETE FROM feeds WHERE feed_url = ${sqlString(feedUrl)}`, + `DELETE FROM sync_state WHERE key = 'crawler_heartbeat_at'`, + ]); +} + export interface SeedItemLabelOpts { itemKey: string; itemType: string; diff --git a/e2e/timeline.spec.ts b/e2e/timeline.spec.ts new file mode 100644 index 00000000..124efbe1 --- /dev/null +++ b/e2e/timeline.spec.ts @@ -0,0 +1,105 @@ +import { test, expect } from './fixtures'; +import { seedSubscription, seedFeedItems, seedItemLabel, cleanupFeedItems } from './seed'; + +/** + * The D1-served timeline: a refresh is ONE request that already carries read + * state, instead of a per-feed batch fan-out against the Fly proxy. + */ +test.describe('Timeline refresh', () => { + const FEED_URL = 'https://example.com/timeline-feed.xml'; + const FEED_TITLE = 'Timeline Feed'; + + test.afterEach(async () => { + await cleanupFeedItems(FEED_URL); + }); + + test('renders archived items from a single timeline request, with read state', async ({ + authedPage, + testUser, + }) => { + await seedSubscription(testUser, { feedUrl: FEED_URL, title: FEED_TITLE }); + await seedFeedItems( + FEED_URL, + [ + { guid: 'timeline-item-1', title: 'Archived Article One' }, + { guid: 'timeline-item-2', title: 'Archived Article Two' }, + ], + { title: FEED_TITLE, siteUrl: 'https://example.com' } + ); + // Read on another device: the timeline join must stamp it inline. + await seedItemLabel(testUser, { + itemKey: 'timeline-item-1', + itemType: 'article', + label: 'read', + }); + + const timelineRequests: string[] = []; + const batchRequests: string[] = []; + authedPage.on('request', (request) => { + const url = request.url(); + if (url.includes('/api/v2/timeline')) timelineRequests.push(url); + if (url.includes('/api/v2/feeds/batch')) batchRequests.push(url); + }); + + const timelineResponsePromise = authedPage.waitForResponse((response) => + response.url().includes('/api/v2/timeline') + ); + await authedPage.reload(); + const timelineResponse = await timelineResponsePromise; + expect(timelineResponse.ok()).toBe(true); + const timelineBody = (await timelineResponse.json()) as { + items: Array<{ title?: string }>; + }; + expect(timelineBody.items.map((item) => item.title)).toEqual( + expect.arrayContaining(['Archived Article One', 'Archived Article Two']) + ); + + // Both articles land in IndexedDB. Assert the durable client merge directly: + // a read article may be hidden by the reader's current filter, which made the + // old visibility assertion depend on incidental UI state. + await expect + .poll(async () => { + return authedPage.evaluate(async () => { + const request = indexedDB.open('skyreader'); + const database: IDBDatabase = await new Promise((resolve, reject) => { + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }); + const store = database.transaction('articles', 'readonly').objectStore('articles'); + const all: Array<{ title: string }> = await new Promise((resolve, reject) => { + const req = store.getAll(); + req.onsuccess = () => resolve(req.result); + req.onerror = () => reject(req.error); + }); + return all.map((row) => row.title); + }); + }) + .toEqual(expect.arrayContaining(['Archived Article One', 'Archived Article Two'])); + + // The refresh is a single timeline call (at most one extra drain page), and + // the legacy per-feed fan-out never runs — the point of the architecture. + expect(timelineRequests.length).toBeGreaterThanOrEqual(1); + expect(timelineRequests.length).toBeLessThanOrEqual(2); + expect(batchRequests.length).toBe(0); + + // Read state arrived with the articles (no separate read fetch). + const readGuids = await authedPage.evaluate(async () => { + const request = indexedDB.open('skyreader'); + const database: IDBDatabase = await new Promise((resolve, reject) => { + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }); + const store = database.transaction('itemLabels', 'readonly').objectStore('itemLabels'); + const all: Array<{ itemKey: string; label: string }> = await new Promise( + (resolve, reject) => { + const req = store.getAll(); + req.onsuccess = () => resolve(req.result); + req.onerror = () => reject(req.error); + } + ); + return all.filter((row) => row.label === 'read').map((row) => row.itemKey); + }); + expect(readGuids).toContain('timeline-item-1'); + expect(readGuids).not.toContain('timeline-item-2'); + }); +}); diff --git a/feed-proxy/README.md b/feed-proxy/README.md index 583f32c2..bc6668a8 100644 --- a/feed-proxy/README.md +++ b/feed-proxy/README.md @@ -245,21 +245,6 @@ Cache statistics (requires authentication). "fresh": 80, "stale": 50, "inFlight": 2, - "extract": { "inUse": 1, "queued": 0 }, - "constellation": { - "breakerOpen": false, - "consecutiveFailures": 0, - "inUse": 2, - "queued": 0, - "requests": 8421, - "resets": 37, - "retries": 37, - "retriesRecovered": 36, - "failures": 1, - "shed": 0, - "breakerOpens": 0, - "shortCircuited": 0 - }, "cacheTtlSeconds": 900, "staleTtlSeconds": 3600, "errors": { @@ -303,27 +288,48 @@ curl "/feed?url=...&since_guids=old-guid&limit=50" ## Configuration -| Environment Variable | Default | Description | -| -------------------- | -------- | ----------------------------------------- | -| `PROXY_SECRET` | (none) | Shared secret for `X-Proxy-Secret` header | -| `DATA_DIR` | `./data` | SQLite database location | -| `CACHE_TTL_SECONDS` | `900` | Fresh cache duration (15 min) | -| `STALE_TTL_SECONDS` | `3600` | Stale cache max age (1 hour) | -| `PORT` | `3000` | HTTP server port | -| `SENTRY_DSN` | (none) | Error reporting; unset ⇒ no-op | -| `WARM_HEARTBEAT_URL` | (none) | Dead-man ping after each warm tick | -| `GIT_COMMIT_SHA` | `dev` | Build stamp, reported by `/health` | - -Observability setup, alert thresholds, and incident procedures live in -[`docs/RUNBOOK.md`](../docs/RUNBOOK.md). - -Constellation (mentions, social context, linkblog registry): - -| Environment Variable | Default | Description | -| --------------------------- | ------- | ------------------------------------------------- | -| `CONSTELLATION_CONCURRENCY` | `3` | Concurrent requests allowed against Constellation | -| `CONSTELLATION_QUEUE_MAX` | `200` | Callers that may queue before requests are shed | -| `WARM_MENTIONS` | `true` | Set `false` to stop pre-warming mentions entirely | +| Environment Variable | Default | Description | +| ---------------------------- | -------- | ------------------------------------------------------- | +| `PROXY_SECRET` | (none) | Shared secret for `X-Proxy-Secret` (inbound + outbound) | +| `DATA_DIR` | `./data` | SQLite database location | +| `CACHE_TTL_SECONDS` | `900` | Fresh cache duration (15 min) | +| `STALE_TTL_SECONDS` | `3600` | Stale cache max age (1 hour) | +| `PORT` | `3000` | HTTP server port | +| `INGEST_URL` | (none) | Paired Worker base URL. **Unset ⇒ ingest disabled** | +| `INGEST_INTERVAL_SECONDS` | `15` | Push cycle | +| `INGEST_BATCH_SIZE` | `100` | Items per push request | +| `CRAWL_SET_INTERVAL_SECONDS` | `300` | How often to pull the crawl set | +| `SENTRY_DSN` | (none) | Error reporting; unset ⇒ no-op | +| `WARM_HEARTBEAT_URL` | (none) | Dead-man ping after each successful warm tick | +| `GIT_COMMIT_SHA` | `dev` | Build stamp, reported by `/health` | + +Observability setup and incident procedures live in [`docs/RUNBOOK.md`](../docs/RUNBOOK.md). + +## Ingest push (crawler mode) + +With `INGEST_URL` set, this proxy stops being a read path for the reader and becomes the crawler +for exactly ONE Worker + D1 pair (prod proxy → prod Worker, staging proxy → staging Worker): + +- **Push:** the durable item log _is_ the outbox. `push_state(seq, pushed_hash)` records what has + reached D1; a row is dirty when it's missing there or its `content_hash` has changed since. Dirty + rows drain in seq order to `POST {INGEST_URL}/api/internal/ingest`; delivery is at-least-once and + the Worker's upsert is idempotent. Failures back off and retry the same rows. +- **Pull:** every `CRAWL_SET_INTERVAL_SECONDS` the proxy fetches + `GET {INGEST_URL}/api/internal/crawl-set` and stamps `last_requested_at` on each feed, which is + what keeps the warm loop working now that reads no longer touch this box. +- **Health:** immediately after each crawl-set pull, the proxy posts every crawl-set feed with + something wrong with it to `POST {INGEST_URL}/api/internal/feed-health` — erroring + (`error_count > 0`), starved (not fetched in `CRAWL_STALE_MS`, 2 h), or both. The erroring ones + are how a reader learns its feed is dead rather than idle, since reads no longer come through here + and a failing feed is otherwise indistinguishable from one that hasn't published; the starved ones + are the admin's alarm that this box can't keep up with its crawl set. The payload is the + **complete** trouble set: the Worker infers recovery from a feed's absence, so a partial report + silently marks feeds healthy. Timestamps are converted from this box's milliseconds to seconds + here, once. +- The per-feed cap (`FEED_ITEMS_CAP = 200`) bounds the **outbox**, not the archive: D1 retains + everything it has ingested. `push_state` cascades with every `feed_items` delete. + +See `docs/plans/D1_FEED_TIMELINE.md`. ## Cache Behavior @@ -374,37 +380,6 @@ When a feed is in backoff: 3. After backoff expires, a single fetch is attempted 4. On success, error tracking resets to zero -### Constellation Client - -Every Constellation caller (mentions, social context, mention lanes, the linkblog -registry) shares one client with three defenses, since they all hit one -small community-run host: - -1. **Circuit breaker** — 5 consecutive failing calls (timeout, network, 5xx/429) - open it for 30 s; calls short-circuit to `null` instead of each eating the - 10 s timeout. A clean 4xx is a healthy "no data" and does not count. The check - runs again after a call gets its concurrency permit, so callers that were - queued when the breaker opened short-circuit too. -2. **Concurrency cap** — `CONSTELLATION_CONCURRENCY` requests in flight, the rest - queued up to `CONSTELLATION_QUEUE_MAX` and then shed to `null`. Shedding is our - own backpressure, so it does **not** count toward the breaker. Keep this low: - it governs how many sockets we hold open, and resets scale with socket churn - (measured reset rate over a fixed request count — 1 concurrent: 2%, 2: 0%, - 4: 8%, 6: 8%, 12: 14%). -3. **One retry on connection resets** — `ECONNRESET` / "socket connection was - closed unexpectedly" is retried once after ~250 ms. Timeouts are not retried. - A reset-then-reset call counts as exactly one breaker failure. What resets is - connection _setup_, not idle keep-alive reuse: a warm pooled socket served 47 - of 48 sequential requests cleanly, while forcing a fresh connection per request - (`keepalive: false` / `Connection: close`) reset 60-76% of the time. So fewer, - longer-lived sockets is the fix; disabling keep-alive makes it far worse, and - some residual failure remains because a retry also has to open a socket. - -Everything degrades to `null` (never throws); mentions and social adornments -simply render empty. `GET /stats` reports `constellation`: breaker state, gate -occupancy, and `resets` / `retriesRecovered` / `failures` / `shed` counters — -`retriesRecovered` ≫ `failures` means resets are being absorbed. - ### Error Response in Bulk Endpoint The `/feeds` endpoint includes error information even when returning cached data: @@ -448,6 +423,11 @@ The `/stats` endpoint includes error statistics: ### Fly.io +Two apps, one per environment — `skyreader-feed-proxy` (prod, `fly.toml`) and +`skyreader-feed-proxy-staging` (staging, `fly.staging.toml`). Each pushes into its own Worker's D1 +and holds its own `PROXY_SECRET`; keep the two config files in sync apart from `app` and +`INGEST_URL`. Each app runs exactly ONE machine (see the singleton invariant in `fly.toml`). + ```bash # Create app fly apps create skyreader-feed-proxy @@ -455,7 +435,7 @@ fly apps create skyreader-feed-proxy # Set secret fly secrets set PROXY_SECRET=your-secret-here -# Deploy +# Deploy (staging: add --config fly.staging.toml) fly deploy ``` diff --git a/feed-proxy/fly.staging.toml b/feed-proxy/fly.staging.toml new file mode 100644 index 00000000..c234a55d --- /dev/null +++ b/feed-proxy/fly.staging.toml @@ -0,0 +1,55 @@ +app = "skyreader-feed-proxy-staging" +primary_region = "sjc" + +# KEEP IN SYNC WITH fly.toml. These two files must differ only in `app` and the +# `INGEST_URL` in [env]; everything else (mounts, http_service, vm, tuning) is +# deliberately identical so staging is a faithful soak of what prod will run. +# Staging deploys on every push to main, prod on release — so drift shows up here +# first, before a release ships it. +# +# SINGLETON INVARIANT — run exactly ONE machine. This app is not horizontally +# scalable as written: the cache is a SQLite DB on the per-machine `proxy_data` +# volume (volumes can't be shared between machines), and the Jetstream firehose, +# the self-warming loop, the ingest pusher, and the in-memory request-coalescing +# maps are all in-process singletons. A second machine would get its own empty +# volume + its own firehose + its own warm loop, with no shared coalescing — i.e. +# a split-brain cache, duplicate upstream fetches, duplicate pushes, and responses +# that differ by which machine you hit. Scale VERTICALLY via the [vm] block below, +# never by `fly scale count`. Keep only one machine provisioned; auto_start/stop +# below toggle that one machine, they do not create new ones. + +[build] + +[mounts] + source = "proxy_data" + destination = "/data" + +[http_service] + internal_port = 3000 + force_https = true + auto_stop_machines = true + auto_start_machines = true + # Keep one machine alive: the self-warming loop and the ingest pusher run + # in-process on timers and can only work while a machine is up. + min_machines_running = 1 + +[vm] + cpu_kind = "shared" + cpus = 2 + memory_mb = 1024 + +[env] + DATA_DIR = "/data" + # This proxy pushes its item log into the STAGING Worker's D1 and pulls the + # staging crawl set from it. One proxy, one Worker — never both environments. + INGEST_URL = "https://api-staging.skyreader.app" + CACHE_TTL_SECONDS = "300" + STALE_TTL_SECONDS = "3600" + WARM_ENABLED = "true" + WARM_INTERVAL_SECONDS = "60" + WARM_ACTIVE_WINDOW_SECONDS = "1209600" + WARM_BATCH_CAP = "500" + WARM_CONCURRENCY = "16" + WARM_MENTIONS = "false" + EXTRACT_CONCURRENCY = "4" + EXTRACT_QUEUE_MAX = "20" diff --git a/feed-proxy/fly.toml b/feed-proxy/fly.toml index 8db17795..f3cc6663 100644 --- a/feed-proxy/fly.toml +++ b/feed-proxy/fly.toml @@ -1,6 +1,10 @@ app = "skyreader-feed-proxy" primary_region = "sjc" +# KEEP IN SYNC WITH fly.staging.toml. The two files must differ only in `app` and +# the `INGEST_URL` in [env]; everything else is deliberately identical so staging +# soaks what prod will run. +# # SINGLETON INVARIANT — run exactly ONE machine. This app is not horizontally # scalable as written: the cache is a SQLite DB on the per-machine `proxy_data` # volume (volumes can't be shared between machines), and the Jetstream firehose, @@ -35,6 +39,11 @@ primary_region = "sjc" [env] DATA_DIR = "/data" + # Ingest push into the prod Worker's D1. Commented out until the staging pair + # has soaked (see docs/plans/D1_FEED_TIMELINE.md, Phase 3): with it unset this + # machine crawls exactly as before and pushes nothing. Enabling it is this one + # line plus a release — prod's backfill then drains through the normal pusher. + # INGEST_URL = "https://api.skyreader.app" # Fresh window. The self-warming loop refreshes active feeds before they reach # this age, so user requests stay HITs even at a tight 5-minute freshness. CACHE_TTL_SECONDS = "300" diff --git a/feed-proxy/src/app.ts b/feed-proxy/src/app.ts index ce5a31fd..7ea87db0 100644 --- a/feed-proxy/src/app.ts +++ b/feed-proxy/src/app.ts @@ -64,6 +64,9 @@ export interface AppConfig { // Defaults to "unhealthy / nothing subscribed" so behavior is unchanged when // the firehose isn't wired in. getFirehoseStatus?: () => FirehoseStatus; + // Whether this proxy pushes its item log into a paired Worker's D1 (INGEST_URL + // is set). Reported by /stats; the loops themselves live in index.ts. + ingestEnabled?: boolean; } export interface CacheRow { @@ -607,12 +610,20 @@ export function writeFeedItems( // delete) instead of an implicit commit per statement — a single fsync rather // than ~100 every warm-refresh per feed. Single-writer SQLite makes it safe. const deleteForRedelivery = db.query('DELETE FROM feed_items WHERE url_hash = ? AND guid = ?'); + // Cascade for the two deletes below. push_state is keyed by seq, so a seq that + // disappears from feed_items would otherwise leave an orphan row forever. + const dropPushStateForItem = db.query( + 'DELETE FROM push_state WHERE seq IN (SELECT seq FROM feed_items WHERE url_hash = ? AND guid = ?)' + ); const writeBatch = db.transaction(() => { if (redeliver) { // Keep retained history that has aged out of the source feed, but remove // every item in this parse so it receives a fresh seq and reaches clients // that already consumed its pre-upgrade representation. - for (const item of items) deleteForRedelivery.run(urlHash, item.guid); + for (const item of items) { + dropPushStateForItem.run(urlHash, item.guid); + deleteForRedelivery.run(urlHash, item.guid); + } } for (let i = items.length - 1; i >= 0; i--) { @@ -631,6 +642,20 @@ export function writeFeedItems( // Cheaper-than-anti-join cap: compute the (K+1)-th-newest seq once, range-delete // below it. The OFFSET subquery yields nothing when the feed has <= K rows, so // `seq <= NULL` matches nothing — a no-op. + // + // NOTE: this cap bounds the *outbox*, not the archive. Items trimmed here that + // already reached D1 stay there permanently (D1 does not prune); items trimmed + // before ever being pushed simply drop out of the dirty set — the same K-window + // bound the proxy has always had. + db.run( + `DELETE FROM push_state + WHERE seq IN ( + SELECT seq FROM feed_items + WHERE url_hash = ? + AND seq <= (SELECT seq FROM feed_items WHERE url_hash = ? ORDER BY seq DESC LIMIT 1 OFFSET ?) + )`, + [urlHash, urlHash, FEED_ITEMS_CAP] + ); db.run( `DELETE FROM feed_items WHERE url_hash = ? @@ -835,6 +860,18 @@ export function initDatabase(db: Database): void { db.run(`CREATE INDEX IF NOT EXISTS idx_feed_items_feed_seq ON feed_items(url_hash, seq)`); db.run(`CREATE INDEX IF NOT EXISTS idx_feed_items_first_seen ON feed_items(first_seen_at)`); + // Ingest delivery state (ingest-push.ts): which seqs, at which content hash, + // have reached the paired Worker's D1. A row is dirty when it's missing here or + // the hashes differ — race-free against a concurrent edit. Created + // unconditionally, even with pushing disabled, so the cascade deletes that keep + // this table bounded (writeFeedItems, cleanupCache) always have a table to hit. + db.run(` + CREATE TABLE IF NOT EXISTS push_state ( + seq INTEGER PRIMARY KEY, + pushed_hash TEXT NOT NULL + ) + `); + // Extracted article content (Defuddle output), keyed by source URL. Separate // from the feed cache: article content is effectively immutable per URL, so it // has its own long TTL and is not touched by the feed self-warming loop. @@ -1855,11 +1892,37 @@ export function createApp(db: Database, config: AppConfig) { >('SELECT COUNT(*) as count FROM cache WHERE next_retry_at > ? AND error_count >= 5') .get(now + 6 * 24 * 60 * 60 * 1000); // More than 6 days means it's a permanent error + // Ingest push (see ingest-push.ts). `pending` is the outbox backlog — the + // number that should hover near zero once a push cycle keeps up. + const itemCount = db + .query<{ count: number }, []>('SELECT COUNT(*) as count FROM feed_items') + .get(); + const pushedCount = db + .query<{ count: number }, []>('SELECT COUNT(*) as count FROM push_state') + .get(); + // Inlined rather than imported from ingest-push.ts, which imports this + // module (hashUrl/itemContentHash) — keep the dependency one-directional. + const pendingCount = db + .query<{ count: number }, []>( + `SELECT COUNT(*) AS count + FROM feed_items fi + JOIN cache c ON c.url_hash = fi.url_hash + LEFT JOIN push_state ps ON ps.seq = fi.seq + WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '')` + ) + .get(); + return c.json({ total: total?.count || 0, fresh: fresh?.count || 0, stale: stale?.count || 0, inFlight: inFlight.size, + ingest: { + enabled: config.ingestEnabled ?? false, + items: itemCount?.count || 0, + pushed: pushedCount?.count || 0, + pending: pendingCount?.count || 0, + }, extract: { inUse: extractSemaphore.inUse, queued: extractSemaphore.queued }, // Constellation health from *our* side: breaker state, gate occupancy, and // the retry/shed counters. `retriesRecovered` ≫ `failures` means the @@ -2772,6 +2835,13 @@ export function cleanupCache(db: Database): number { // cache row goes (the delete keys off cache.fetched_at). Once the feed is cold // the retained items are unreachable anyway, and this bounds feed_items growth // to the active working set. + db.run( + `DELETE FROM push_state WHERE seq IN ( + SELECT seq FROM feed_items + WHERE url_hash IN (SELECT url_hash FROM cache WHERE fetched_at < ?) + )`, + [threshold] + ); db.run( 'DELETE FROM feed_items WHERE url_hash IN (SELECT url_hash FROM cache WHERE fetched_at < ?)', [threshold] diff --git a/feed-proxy/src/index.ts b/feed-proxy/src/index.ts index e233b570..1c8d398f 100644 --- a/feed-proxy/src/index.ts +++ b/feed-proxy/src/index.ts @@ -6,6 +6,7 @@ import { mkdirSync } from 'fs'; import { createApp, initDatabase, cleanupCache } from './app'; import { DocumentFirehose } from './jetstream'; import { pingHeartbeat } from './heartbeat'; +import { pushDirtyItems, pullCrawlSet, reportFeedHealth, type IngestConfig } from './ingest-push'; // Config const PROXY_SECRET = process.env.PROXY_SECRET; @@ -51,6 +52,20 @@ const WARM_MENTIONS_ENABLED = (process.env.WARM_MENTIONS ?? 'true') !== 'false'; // local dev, so the ping is a no-op there. const WARM_HEARTBEAT_URL = process.env.WARM_HEARTBEAT_URL; +// This proxy crawls for one Worker/D1 pair. Leaving INGEST_URL unset disables +// both directions, which keeps local development and staged rollout safe. +const INGEST_URL = process.env.INGEST_URL?.replace(/\/$/, '') || ''; +const INGEST_ENABLED = INGEST_URL.length > 0; +const INGEST_INTERVAL_MS = parseInt(process.env.INGEST_INTERVAL_SECONDS || '15', 10) * 1000; +const INGEST_BATCH_SIZE = parseInt(process.env.INGEST_BATCH_SIZE || '100', 10); +const CRAWL_SET_INTERVAL_MS = parseInt(process.env.CRAWL_SET_INTERVAL_SECONDS || '300', 10) * 1000; +const PUSH_BACKOFF_BASE_MS = 30 * 1000; +const PUSH_BACKOFF_MAX_MS = 10 * 60 * 1000; + +function pushBackoff(failures: number): number { + return Math.min(PUSH_BACKOFF_BASE_MS * 2 ** (failures - 1), PUSH_BACKOFF_MAX_MS); +} + // /extract is the heaviest request (fetch + Defuddle DOM build). Cap concurrent // extractions so a burst of distinct heavy articles can't OOM the 512MB machine; // excess callers queue, then are shed with a 503 once the queue fills. @@ -96,6 +111,7 @@ const { app, warmStaleFeeds, warmStaleDocuments } = createApp(db, { extractConcurrency: EXTRACT_CONCURRENCY, extractQueueMax: EXTRACT_QUEUE_MAX, getFirehoseStatus: () => firehose?.status() ?? { healthy: false, isSubscribed: () => false }, + ingestEnabled: INGEST_ENABLED, }); // Document firehose: push-based freshness for standard.site documents. @@ -158,6 +174,70 @@ if (WARM_ENABLED) { console.log('[Proxy] Warmer: disabled'); } +// Push the durable item log into D1 and pull the registered crawl set back. +if (INGEST_ENABLED) { + const ingestConfig: IngestConfig = { + ingestUrl: INGEST_URL, + secret: PROXY_SECRET, + batchSize: INGEST_BATCH_SIZE, + }; + let pushRunning = false; + let pushFailures = 0; + let pushBlockedUntil = 0; + setInterval(() => { + if (pushRunning || Date.now() < pushBlockedUntil) return; + pushRunning = true; + pushDirtyItems(db, ingestConfig) + .then((result) => { + if (result.error) { + pushFailures++; + pushBlockedUntil = Date.now() + pushBackoff(pushFailures); + console.error(`[Proxy] Ingest push failed (${pushFailures}): ${result.error}`); + } else { + pushFailures = 0; + if (result.pushed > 0) console.log(`[Proxy] Ingest pushed ${result.pushed} item(s)`); + } + }) + .catch((error) => { + console.error('[Proxy] Ingest push error:', error); + reportError(error, { tags: { source: 'ingest-push' } }); + }) + .finally(() => { + pushRunning = false; + }); + }, INGEST_INTERVAL_MS); + + let crawlSetRunning = false; + const refreshCrawlSet = () => { + if (crawlSetRunning) return; + crawlSetRunning = true; + pullCrawlSet(db, ingestConfig) + .then((result) => { + if (result.error) console.error(`[Proxy] Crawl-set pull failed: ${result.error}`); + else console.log(`[Proxy] Crawl set: ${result.registered} feed(s) registered`); + // Report health AFTER the pull, so a feed registered for the first time + // this cycle is already in the crawl set and its errors are reportable. + // Reads no longer pass through here, so this is the only way a broken + // feed reaches the reader's error badge. + return reportFeedHealth(db, ingestConfig).then((health) => { + if (health.error) console.error(`[Proxy] Feed-health report failed: ${health.error}`); + else console.log(`[Proxy] Feed health: ${health.reported} feed(s) in error`); + }); + }) + .catch((error) => { + console.error('[Proxy] Crawl-set pull error:', error); + reportError(error, { tags: { source: 'crawl-set' } }); + }) + .finally(() => { + crawlSetRunning = false; + }); + }; + refreshCrawlSet(); + setInterval(refreshCrawlSet, CRAWL_SET_INTERVAL_MS); +} else { + console.log('[Proxy] Ingest push: disabled (INGEST_URL unset)'); +} + // Flush the firehose cursor + close its socket cleanly on shutdown so we resume // where we left off instead of replaying. for (const signal of ['SIGTERM', 'SIGINT'] as const) { diff --git a/feed-proxy/src/ingest-push.test.ts b/feed-proxy/src/ingest-push.test.ts new file mode 100644 index 00000000..c3ca58e3 --- /dev/null +++ b/feed-proxy/src/ingest-push.test.ts @@ -0,0 +1,461 @@ +import { describe, expect, it, beforeEach, afterEach, spyOn } from 'bun:test'; +import { Database } from 'bun:sqlite'; +import { + initDatabase, + hashUrl, + writeFeedItems, + cleanupCache, + itemContentHash, + FEED_ITEMS_CAP, +} from './app'; +import { + pushDirtyItems, + pullCrawlSet, + registerCrawlFeeds, + reportFeedHealth, + CRAWL_STALE_MS, + selectDirtyRows, + selectFeedHealth, + countDirtyRows, + type IngestConfig, +} from './ingest-push'; +import type { FeedItem } from './types'; + +const FEED_URL = 'https://example.com/feed.xml'; +const URL_HASH = hashUrl(FEED_URL); + +const CONFIG: IngestConfig = { + ingestUrl: 'https://api.example', + secret: 'test-secret', + batchSize: 2, +}; + +function item(guid: string, overrides: Partial = {}): FeedItem { + return { + guid, + url: `https://example.com/${guid}`, + title: `Title ${guid}`, + publishedAt: '2026-01-01T00:00:00.000Z', + ...overrides, + }; +} + +function seedCache(db: Database, url = FEED_URL, feedTitle = 'Test Blog'): void { + db.run( + `INSERT INTO cache (url_hash, url, parsed_json, parser_version, cached_at, fetched_at, last_requested_at) + VALUES (?, ?, ?, 1, ?, ?, ?)`, + [ + hashUrl(url), + url, + JSON.stringify({ + title: feedTitle, + siteUrl: 'https://example.com', + description: 'A test blog', + items: [], + }), + Date.now(), + Date.now(), + Date.now(), + ] + ); +} + +interface CapturedRequest { + url: string; + headers: Record; + body: { + feeds: Array<{ feedUrl: string; title?: string | null; siteUrl?: string | null }>; + items: Array<{ feedUrl: string; guid: string; contentHash: string; item: FeedItem }>; + }; +} + +// Capture what the pusher sends and control the Worker's reply. +function mockIngestEndpoint(status = 200): { calls: CapturedRequest[]; restore: () => void } { + const calls: CapturedRequest[] = []; + const spy = spyOn(globalThis, 'fetch').mockImplementation((async ( + input: string, + init?: RequestInit + ) => { + calls.push({ + url: String(input), + headers: (init?.headers ?? {}) as Record, + body: init?.body ? JSON.parse(String(init.body)) : { feeds: [], items: [] }, + }); + return new Response(JSON.stringify({ ok: status === 200 }), { status }); + }) as unknown as typeof fetch); + return { calls, restore: () => spy.mockRestore() }; +} + +describe('ingest push', () => { + let db: Database; + + beforeEach(() => { + db = new Database(':memory:'); + initDatabase(db); + seedCache(db); + }); + + afterEach(() => { + db.close(); + }); + + it('pushes dirty rows in seq order and acks them', async () => { + writeFeedItems(db, URL_HASH, [item('g2'), item('g1')], Date.now()); + expect(countDirtyRows(db)).toBe(2); + + const endpoint = mockIngestEndpoint(); + const result = await pushDirtyItems(db, CONFIG); + endpoint.restore(); + + expect(result.pushed).toBe(2); + expect(endpoint.calls.length).toBe(1); + expect(endpoint.calls[0].url).toBe('https://api.example/api/internal/ingest'); + expect(endpoint.calls[0].headers['X-Proxy-Secret']).toBe('test-secret'); + // Items are newest-first in the feed, written oldest-first, so seq order is g1, g2. + expect(endpoint.calls[0].body.items.map((i) => i.guid)).toEqual(['g1', 'g2']); + // Always the registered URL, never a post-redirect one. + expect(endpoint.calls[0].body.items.every((i) => i.feedUrl === FEED_URL)).toBe(true); + expect(endpoint.calls[0].body.feeds[0]).toMatchObject({ + feedUrl: FEED_URL, + title: 'Test Blog', + siteUrl: 'https://example.com', + }); + + expect(countDirtyRows(db)).toBe(0); + }); + + it('pages a backlog larger than one batch', async () => { + writeFeedItems(db, URL_HASH, [item('g3'), item('g2'), item('g1')], Date.now()); + + const endpoint = mockIngestEndpoint(); + const first = await pushDirtyItems(db, CONFIG); + expect(first.pushed).toBe(2); + expect(first.hasMore).toBe(true); + + const second = await pushDirtyItems(db, CONFIG); + endpoint.restore(); + expect(second.pushed).toBe(1); + expect(second.hasMore).toBe(false); + expect(countDirtyRows(db)).toBe(0); + }); + + it('leaves state untouched on a 5xx so the same rows retry', async () => { + writeFeedItems(db, URL_HASH, [item('g1')], Date.now()); + + const failing = mockIngestEndpoint(500); + const result = await pushDirtyItems(db, CONFIG); + failing.restore(); + expect(result.pushed).toBe(0); + expect(result.error).toContain('500'); + expect(countDirtyRows(db)).toBe(1); + + const ok = mockIngestEndpoint(); + const retry = await pushDirtyItems(db, CONFIG); + ok.restore(); + expect(retry.pushed).toBe(1); + expect(countDirtyRows(db)).toBe(0); + }); + + it('re-qualifies an item edited after it was pushed', async () => { + writeFeedItems(db, URL_HASH, [item('g1', { title: 'Old' })], Date.now()); + const first = mockIngestEndpoint(); + await pushDirtyItems(db, CONFIG); + first.restore(); + expect(countDirtyRows(db)).toBe(0); + + // An edit rewrites content_hash in place (seq unchanged) → dirty again. + writeFeedItems(db, URL_HASH, [item('g1', { title: 'New' })], Date.now()); + expect(countDirtyRows(db)).toBe(1); + + const second = mockIngestEndpoint(); + const result = await pushDirtyItems(db, CONFIG); + second.restore(); + expect(result.pushed).toBe(1); + expect(second.calls[0].body.items[0].item.title).toBe('New'); + }); + + it('acks the hash it pushed, so an edit mid-flight is not lost', async () => { + writeFeedItems(db, URL_HASH, [item('g1', { title: 'Old' })], Date.now()); + const rows = selectDirtyRows(db, 10); + expect(rows.length).toBe(1); + + const endpoint = spyOn(globalThis, 'fetch').mockImplementation((async () => { + // The warm loop rewrites the item while the request is in flight. + writeFeedItems(db, URL_HASH, [item('g1', { title: 'Newer' })], Date.now()); + return new Response('{}', { status: 200 }); + }) as unknown as typeof fetch); + await pushDirtyItems(db, CONFIG); + endpoint.mockRestore(); + + // The acked hash is the one we sent, which no longer matches — still dirty. + expect(countDirtyRows(db)).toBe(1); + }); + + it('supplies a hash for legacy rows that have none', async () => { + const parsed = item('legacy'); + db.run( + `INSERT INTO feed_items (url_hash, guid, item_json, published_at, first_seen_at, content_hash) + VALUES (?, ?, ?, ?, ?, NULL)`, + [URL_HASH, 'legacy', JSON.stringify(parsed), Date.now(), Date.now()] + ); + + const endpoint = mockIngestEndpoint(); + await pushDirtyItems(db, CONFIG); + endpoint.restore(); + + expect(endpoint.calls[0].body.items[0].contentHash).toBe(itemContentHash(parsed)); + // Acked as '' (matching COALESCE), so it doesn't re-push every cycle. + expect(countDirtyRows(db)).toBe(0); + }); + + it('cascades push_state when the per-feed cap trims items', async () => { + // Fill past the cap so the oldest rows are trimmed on the next write. + const initial = Array.from({ length: FEED_ITEMS_CAP }, (_, i) => item(`old-${i}`)); + writeFeedItems(db, URL_HASH, initial, Date.now()); + const endpoint = mockIngestEndpoint(); + let guard = 0; + while (countDirtyRows(db) > 0 && guard++ < 200) { + await pushDirtyItems(db, { ...CONFIG, batchSize: 100 }); + } + endpoint.restore(); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe( + FEED_ITEMS_CAP + ); + + writeFeedItems(db, URL_HASH, [item('fresh-1'), item('fresh-2')], Date.now()); + + const itemCount = db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM feed_items').get()?.c; + const stateCount = db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c; + expect(itemCount).toBe(FEED_ITEMS_CAP); + // Two trimmed rows dropped their delivery state with them (2 fresh rows are + // not yet pushed, so push_state holds cap - 2). + expect(stateCount).toBe(FEED_ITEMS_CAP - 2); + }); + + it('cascades push_state when an idle feed is evicted', async () => { + writeFeedItems(db, URL_HASH, [item('g1')], Date.now()); + const endpoint = mockIngestEndpoint(); + await pushDirtyItems(db, CONFIG); + endpoint.restore(); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe(1); + + // Age the feed past the 7-day eviction window. + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [ + Date.now() - 8 * 24 * 60 * 60 * 1000, + URL_HASH, + ]); + cleanupCache(db); + + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM feed_items').get()?.c).toBe(0); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe(0); + }); +}); + +describe('crawl-set registration', () => { + let db: Database; + + beforeEach(() => { + db = new Database(':memory:'); + initDatabase(db); + }); + + afterEach(() => { + db.close(); + }); + + it('creates a cache row for an unknown feed, due for the warm loop', () => { + const now = Date.now(); + expect(registerCrawlFeeds(db, [FEED_URL], now)).toBe(1); + + const row = db + .query< + { url: string; fetched_at: number; parser_version: number; last_requested_at: number }, + [string] + >('SELECT url, fetched_at, parser_version, last_requested_at FROM cache WHERE url_hash = ?') + .get(URL_HASH); + expect(row?.url).toBe(FEED_URL); + expect(row?.last_requested_at).toBe(now); + // fetched_at 0 + a stale parser version make it immediately warm-eligible. + expect(row?.fetched_at).toBe(0); + expect(row?.parser_version).toBe(0); + }); + + it('keeps a known feed warm without any read traffic', () => { + seedCache(db); + db.run('UPDATE cache SET last_requested_at = ? WHERE url_hash = ?', [1000, URL_HASH]); + + const now = Date.now(); + registerCrawlFeeds(db, [FEED_URL], now); + + const row = db + .query< + { last_requested_at: number; parsed_json: string; parser_version: number }, + [string] + >('SELECT last_requested_at, parsed_json, parser_version FROM cache WHERE url_hash = ?') + .get(URL_HASH); + expect(row?.last_requested_at).toBe(now); + // The existing cached parse is untouched. + expect(row?.parser_version).toBe(1); + expect(JSON.parse(row!.parsed_json).title).toBe('Test Blog'); + }); + + it('pulls the crawl set from the paired Worker', async () => { + const spy = spyOn(globalThis, 'fetch').mockImplementation((async ( + input: string, + init?: RequestInit + ) => { + expect(String(input)).toBe('https://api.example/api/internal/crawl-set'); + expect((init?.headers as Record)['X-Proxy-Secret']).toBe('test-secret'); + return new Response( + JSON.stringify({ + feeds: [ + { feedUrl: FEED_URL, subscribers: 3 }, + { feedUrl: 'https://other.example/f.xml' }, + ], + }) + ); + }) as unknown as typeof fetch); + + const result = await pullCrawlSet(db, CONFIG); + spy.mockRestore(); + + expect(result.registered).toBe(2); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM cache').get()?.c).toBe(2); + }); + + it('reports a failed pull without touching the cache', async () => { + const spy = spyOn(globalThis, 'fetch').mockImplementation((async () => { + return new Response('nope', { status: 503 }); + }) as unknown as typeof fetch); + + const result = await pullCrawlSet(db, CONFIG); + spy.mockRestore(); + + expect(result.registered).toBe(0); + expect(result.error).toContain('503'); + expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM cache').get()?.c).toBe(0); + }); +}); + +describe('feed health reporting', () => { + let db: Database; + + beforeEach(() => { + db = new Database(':memory:'); + initDatabase(db); + seedCache(db); + }); + + afterEach(() => { + db.close(); + }); + + function breakFeed( + url: string, + opts: { errorCount?: number; error?: string; at?: number; retryAt?: number } = {} + ): void { + const at = opts.at ?? Date.now(); + db.run( + 'UPDATE cache SET error_count = ?, last_error = ?, last_error_at = ?, next_retry_at = ? WHERE url_hash = ?', + [ + opts.errorCount ?? 3, + opts.error ?? 'Failed to fetch (HTTP 404)', + at, + opts.retryAt ?? at + 600_000, + hashUrl(url), + ] + ); + } + + it('reports only broken feeds, converting milliseconds to seconds', () => { + const at = 1_770_000_000_000; + breakFeed(FEED_URL, { errorCount: 4, at, retryAt: at + 600_000 }); + + const health = selectFeedHealth(db); + expect(health).toHaveLength(1); + expect(health[0]).toMatchObject({ + feedUrl: FEED_URL, + errorCount: 4, + lastError: 'Failed to fetch (HTTP 404)', + lastErrorAt: Math.floor(at / 1000), + nextRetryAt: Math.floor((at + 600_000) / 1000), + }); + }); + + it('leaves healthy feeds out entirely — absence is the recovery signal', () => { + seedCache(db, 'https://other.example/feed.xml', 'Other'); + breakFeed(FEED_URL); + + expect(selectFeedHealth(db).map((f) => f.feedUrl)).toEqual([FEED_URL]); + }); + + it('flags a crawl-set feed the warm loop has not fetched in hours', () => { + // Nothing is erroring; the feed is simply losing its turn every tick, which + // is what a capped warm batch does and what the admin needs to see. + const now = Date.now(); + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [ + now - CRAWL_STALE_MS - 60_000, + URL_HASH, + ]); + + const health = selectFeedHealth(db, now); + expect(health).toHaveLength(1); + expect(health[0]).toMatchObject({ feedUrl: FEED_URL, errorCount: 0, crawlStale: true }); + }); + + it('does not flag a feed fetched within the window', () => { + const now = Date.now(); + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [now - 60_000, URL_HASH]); + expect(selectFeedHealth(db, now)).toHaveLength(0); + }); + + it('reports a feed that is both erroring and starved', () => { + const now = Date.now(); + breakFeed(FEED_URL); + db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [ + now - CRAWL_STALE_MS - 60_000, + URL_HASH, + ]); + + const health = selectFeedHealth(db, now); + expect(health[0]).toMatchObject({ errorCount: 3, crawlStale: true }); + }); + + it('skips feeds that have dropped out of the crawl set', () => { + // last_requested_at NULL = evicted / never registered, so nobody is + // subscribed and its errors are not the reader's problem. + breakFeed(FEED_URL); + db.run('UPDATE cache SET last_requested_at = NULL WHERE url_hash = ?', [URL_HASH]); + + expect(selectFeedHealth(db)).toHaveLength(0); + }); + + it('posts the set to the Worker, and posts an empty set too', async () => { + const endpoint = mockIngestEndpoint(); + await reportFeedHealth(db, CONFIG); + + expect(endpoint.calls[0].url).toBe('https://api.example/api/internal/feed-health'); + expect(endpoint.calls[0].headers['X-Proxy-Secret']).toBe('test-secret'); + // Everything is healthy: the empty list is what clears the Worker's flags. + expect((endpoint.calls[0].body as unknown as { feeds: unknown[] }).feeds).toEqual([]); + + breakFeed(FEED_URL); + const result = await reportFeedHealth(db, CONFIG); + endpoint.restore(); + + expect(result.reported).toBe(1); + expect( + (endpoint.calls[1].body as unknown as { feeds: Array<{ feedUrl: string }> }).feeds[0].feedUrl + ).toBe(FEED_URL); + }); + + it('surfaces a rejected report instead of pretending it landed', async () => { + breakFeed(FEED_URL); + const endpoint = mockIngestEndpoint(503); + const result = await reportFeedHealth(db, CONFIG); + endpoint.restore(); + + expect(result.reported).toBe(0); + expect(result.error).toContain('503'); + }); +}); diff --git a/feed-proxy/src/ingest-push.ts b/feed-proxy/src/ingest-push.ts new file mode 100644 index 00000000..64260a1e --- /dev/null +++ b/feed-proxy/src/ingest-push.ts @@ -0,0 +1,348 @@ +import type { Database } from 'bun:sqlite'; +import type { FeedItem } from './types'; +import { hashUrl, itemContentHash } from './app'; + +/** + * Ingest push: the proxy stops being a read path and becomes a crawler that + * pushes deltas into its paired Worker's D1. + * + * The durable item log IS the outbox — there is no separate queue. A row is + * *dirty* when `push_state` has no row for its seq, or the hash we last pushed + * differs from the row's current `content_hash`. That comparison is race-free + * against concurrent edits: if `writeFeedItems` rewrites an item between select + * and ack, the hashes no longer match and the row simply re-qualifies. + * + * One proxy pushes to exactly ONE Worker (prod proxy → prod Worker, staging + * proxy → staging Worker), so delivery state is keyed by seq alone. With + * `INGEST_URL` unset both loops are disabled — the safe default for local dev + * and the gate for a staged rollout. + */ + +export interface IngestConfig { + // Base URL of the paired Worker, e.g. https://api.skyreader.app + ingestUrl: string; + // Shared secret, sent as X-Proxy-Secret (the same PROXY_SECRET the Worker uses + // to authenticate to us — one secret per environment, no new names). + secret?: string; + // Items per push request. The Worker's own cap is well above this. + batchSize?: number; + timeoutMs?: number; +} + +const DEFAULT_BATCH_SIZE = 100; +const DEFAULT_TIMEOUT_MS = 30_000; + +interface DirtyRow { + seq: number; + url_hash: string; + guid: string; + item_json: string; + published_at: number | null; + first_seen_at: number; + content_hash: string; + feed_url: string; +} + +interface FeedMetaRow { + url_hash: string; + url: string; + title: string | null; + site_url: string | null; + description: string | null; + image_url: string | null; +} + +export interface PushResult { + pushed: number; + // True when the log still holds dirty rows beyond this batch. + hasMore: boolean; + error?: string; +} + +export function selectDirtyRows(db: Database, limit: number): DirtyRow[] { + return db + .query( + `SELECT fi.seq, fi.url_hash, fi.guid, fi.item_json, fi.published_at, fi.first_seen_at, + COALESCE(fi.content_hash, '') AS content_hash, c.url AS feed_url + FROM feed_items fi + JOIN cache c ON c.url_hash = fi.url_hash + LEFT JOIN push_state ps ON ps.seq = fi.seq + WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '') + ORDER BY fi.seq ASC + LIMIT ?` + ) + .all(limit); +} + +export function countDirtyRows(db: Database): number { + return ( + db + .query<{ count: number }, []>( + `SELECT COUNT(*) AS count + FROM feed_items fi + JOIN cache c ON c.url_hash = fi.url_hash + LEFT JOIN push_state ps ON ps.seq = fi.seq + WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '')` + ) + .get()?.count ?? 0 + ); +} + +function feedMetadata(db: Database, urlHashes: string[]): FeedMetaRow[] { + if (urlHashes.length === 0) return []; + const placeholders = urlHashes.map(() => '?').join(','); + return db + .query( + `SELECT url_hash, url, + json_extract(parsed_json, '$.title') AS title, + json_extract(parsed_json, '$.siteUrl') AS site_url, + json_extract(parsed_json, '$.description') AS description, + json_extract(parsed_json, '$.imageUrl') AS image_url + FROM cache + WHERE url_hash IN (${placeholders})` + ) + .all(...urlHashes); +} + +function ackPushed(db: Database, rows: DirtyRow[]): void { + const upsert = db.query( + `INSERT INTO push_state (seq, pushed_hash) VALUES (?, ?) + ON CONFLICT(seq) DO UPDATE SET pushed_hash = excluded.pushed_hash` + ); + // Ack the hash we actually PUSHED, not the row's hash right now: if the item + // was edited mid-flight the two differ and the row stays dirty for the next + // cycle, which is exactly what we want. + db.transaction(() => { + for (const row of rows) upsert.run(row.seq, row.content_hash); + })(); +} + +/** + * Drain one batch of dirty rows to the paired Worker. Rows go in seq order, so + * within-feed order in D1 matches proxy first-seen order — all the cursor + * semantics need. On any failure nothing is acked and the same rows retry. + */ +export async function pushDirtyItems(db: Database, config: IngestConfig): Promise { + const batchSize = config.batchSize ?? DEFAULT_BATCH_SIZE; + const rows = selectDirtyRows(db, batchSize); + if (rows.length === 0) return { pushed: 0, hasMore: false }; + + const urlHashes = [...new Set(rows.map((r) => r.url_hash))]; + const feeds = feedMetadata(db, urlHashes).map((meta) => ({ + feedUrl: meta.url, + title: meta.title, + siteUrl: meta.site_url, + description: meta.description, + imageUrl: meta.image_url, + })); + + const items = rows.map((row) => { + const item = JSON.parse(row.item_json) as FeedItem; + return { + // ALWAYS the registered/requested URL (cache.url), never a post-redirect + // one: D1 joins subscriptions on this exact string. + feedUrl: row.feed_url, + guid: row.guid, + item, + publishedAt: row.published_at, + firstSeenAt: row.first_seen_at, + // A pre-hash legacy row still needs a hash for D1's NOT NULL column. + contentHash: row.content_hash || itemContentHash(item), + }; + }); + + const headers: Record = { 'Content-Type': 'application/json' }; + if (config.secret) headers['X-Proxy-Secret'] = config.secret; + + try { + const response = await fetch(`${config.ingestUrl}/api/internal/ingest`, { + method: 'POST', + headers, + body: JSON.stringify({ feeds, items }), + signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS), + }); + + if (!response.ok) { + const body = await response.text().catch(() => ''); + return { + pushed: 0, + hasMore: true, + error: `HTTP ${response.status}${body ? `: ${body.slice(0, 200)}` : ''}`, + }; + } + + ackPushed(db, rows); + return { pushed: rows.length, hasMore: rows.length >= batchSize }; + } catch (error) { + return { + pushed: 0, + hasMore: true, + error: error instanceof Error ? error.message : String(error), + }; + } +} + +/** + * Register a feed in the crawl set: create its cache row if missing and stamp + * `last_requested_at`. That single stamp is the whole trick — the existing warm + * loop, active window and eviction machinery then work unchanged, with the + * Worker's subscription table (rather than read traffic) driving warmth. + * + * A newly created row starts with parser_version 0 and fetched_at 0 so the warm + * loop treats it as due immediately. + */ +export function registerCrawlFeeds(db: Database, feedUrls: string[], now: number): number { + if (feedUrls.length === 0) return 0; + const upsert = db.query( + `INSERT INTO cache (url_hash, url, parsed_json, parser_version, parser_upgrade_attempted_version, + cached_at, fetched_at, error_count, last_requested_at) + VALUES (?, ?, '{"title":"","items":[],"fetchedAt":0}', 0, 0, 0, 0, 0, ?) + ON CONFLICT(url_hash) DO UPDATE SET last_requested_at = excluded.last_requested_at` + ); + let registered = 0; + db.transaction(() => { + for (const url of feedUrls) { + if (!url) continue; + upsert.run(hashUrl(url), url, now); + registered++; + } + })(); + return registered; +} + +export interface CrawlSetResult { + registered: number; + error?: string; +} + +/** + * One broken feed, as reported to the Worker. Timestamps are unix SECONDS on the + * wire — the cache stores milliseconds (everything here compares against + * `Date.now()`), and the Worker's `feeds` table is in seconds like the rest of + * the backend, so the conversion happens once, here. + */ +export interface FeedHealthReport { + feedUrl: string; + errorCount: number; + lastError: string | null; + lastErrorAt: number | null; + nextRetryAt: number | null; + lastFetchAt: number | null; + // In the crawl set but not fetched in CRAWL_STALE_MS — starved by a saturated + // warm loop rather than failing. `errorCount` can be 0 while this is true. + crawlStale: boolean; +} + +// How long a crawl-set feed may go unfetched before it counts as starved. The +// warm loop works on a minutes-long cadence (a 300s cache TTL refreshed at +// ~180s), so hours without a fetch means this feed is losing its turn every +// tick — the failure mode a capped warm batch produces. +export const CRAWL_STALE_MS = 2 * 60 * 60 * 1000; + +function toSeconds(ms: number | null | undefined): number | null { + return ms ? Math.floor(ms / 1000) : null; +} + +/** + * Every crawl-set feed with something wrong with it: failing to fetch, starved + * of fetches, or both. + * + * Deliberately the whole trouble set rather than a delta: the Worker infers + * recovery from a feed's ABSENCE here, so a feed that starts working again needs + * no message of its own. Only rows still in the crawl set count — a feed evicted + * by `cleanupCache` is nobody's problem any more. + */ +export function selectFeedHealth(db: Database, now = Date.now()): FeedHealthReport[] { + const staleBefore = now - CRAWL_STALE_MS; + return db + .query< + { + url: string; + error_count: number; + last_error: string | null; + last_error_at: number | null; + next_retry_at: number | null; + fetched_at: number | null; + }, + [number] + >( + `SELECT url, error_count, last_error, last_error_at, next_retry_at, fetched_at + FROM cache + WHERE last_requested_at IS NOT NULL + AND (error_count > 0 OR COALESCE(fetched_at, 0) < ?)` + ) + .all(staleBefore) + .map((row) => ({ + feedUrl: row.url, + errorCount: row.error_count, + lastError: row.last_error, + lastErrorAt: toSeconds(row.last_error_at), + nextRetryAt: toSeconds(row.next_retry_at), + // 0 means never fetched — an error placeholder from a first-crawl failure, + // or a row the crawl set just registered. Not a timestamp. + lastFetchAt: toSeconds(row.fetched_at), + crawlStale: (row.fetched_at ?? 0) < staleBefore, + })); +} + +export interface FeedHealthResult { + reported: number; + error?: string; +} + +/** + * Push the trouble set to the paired Worker: the erroring feeds it serves to + * readers, and the starved ones the admin alarms on. An empty list is a + * meaningful report — it is how "everything recovered" is communicated — so this + * always posts. + */ +export async function reportFeedHealth( + db: Database, + config: IngestConfig +): Promise { + const feeds = selectFeedHealth(db); + const headers: Record = { 'Content-Type': 'application/json' }; + if (config.secret) headers['X-Proxy-Secret'] = config.secret; + + try { + const response = await fetch(`${config.ingestUrl}/api/internal/feed-health`, { + method: 'POST', + headers, + body: JSON.stringify({ feeds }), + signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS), + }); + if (!response.ok) return { reported: 0, error: `HTTP ${response.status}` }; + return { reported: feeds.length }; + } catch (error) { + return { + reported: 0, + error: error instanceof Error ? error.message : String(error), + }; + } +} + +/** + * Pull the crawl set from the paired Worker and stamp every feed in it. + */ +export async function pullCrawlSet(db: Database, config: IngestConfig): Promise { + const headers: Record = {}; + if (config.secret) headers['X-Proxy-Secret'] = config.secret; + + try { + const response = await fetch(`${config.ingestUrl}/api/internal/crawl-set`, { + headers, + signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS), + }); + if (!response.ok) { + return { registered: 0, error: `HTTP ${response.status}` }; + } + const body = (await response.json()) as { feeds?: Array<{ feedUrl: string }> }; + const urls = (body.feeds ?? []).map((f) => f.feedUrl).filter(Boolean); + return { registered: registerCrawlFeeds(db, urls, Date.now()) }; + } catch (error) { + return { + registered: 0, + error: error instanceof Error ? error.message : String(error), + }; + } +} diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 888e67d3..7fabdb0e 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -39,25 +39,42 @@ All stores use Svelte 5 runes (`.svelte.ts` files): ### Services -| Service | Purpose | -| --------------- | -------------------------------------------------- | -| `api.ts` | HTTP client for backend API | -| `db.ts` | Dexie (IndexedDB) schema for offline storage | -| `sync-queue.ts` | Queue operations when offline, process when online | -| `realtime.ts` | WebSocket connection management | -| `telemetry.ts` | Sampled client error reports → our backend | - -### Error reporting - -`hooks.client.ts` wires three channels — SvelteKit's `handleError`, `window.onerror` -and `unhandledrejection` — into `reportClientError()`, which posts to the backend's -`/api/telemetry/error`. No Sentry SDK ships in the bundle and no third party is -contacted; the backend decides what reaches the error tracker. - -It is sampled at 10%, capped per page load, silent in dev and offline, and sends -no query strings. The exception is `preload_recovery_failed` — the stale-chunk -guard tripping twice, i.e. a deploy that bricked the PWA — which always sends. -See [`docs/RUNBOOK.md`](../docs/RUNBOOK.md) §4c. +| Service | Purpose | +| ----------------- | -------------------------------------------------- | +| `api.ts` | HTTP client for backend API | +| `db.ts` | Dexie (IndexedDB) schema for offline storage | +| `feedFetcher.ts` | Feed refresh (timeline sync + legacy batch path) | +| `timelineSync.ts` | Pure helpers for the timeline sync (unit-tested) | +| `sync-queue.ts` | Queue operations when offline, process when online | +| `realtime.ts` | WebSocket connection management | +| `telemetry.ts` | Sampled client error reports to the backend | + +### Feed refresh + +A refresh is **one** `GET /api/v2/timeline` request (plus drain pages while `hasMore`), served from +the backend's D1 archive with read state already stamped on each item. The client holds a single +global cursor in Dexie `metadata` (`timelineCursor` = `{cursor, generation}`), committed only after +a successful merge; a `generation` change cold-starts. A cold start is paged (`cold_offset`), and +commits the first page's cursor only once the last page has merged. New subscriptions are +backfilled through `fetchSingleFeed` → `GET /api/v2/feeds/fetch`, since their items sit below the +global cursor — including ones that arrive from another device (`backfillMissingSubscriptions`, +once per feed, ≤ 10 per sync). + +Per-feed error state (the sidebar badge and the Manage Sources popover) comes from the response's +`feedHealth`: the set of feeds the crawler currently considers broken. Reads are served from the +archive, so "this feed returned nothing" means nothing — a dead feed and a quiet feed look +identical, and only the server's health verdict distinguishes them. **Absence from that map is what +clears an error**, so `applyHealthSnapshot` runs after the per-feed `markReady` pass and overrides +it. The payload is sent on every cold start and whenever the echoed `health_rev` is stale, so a +steady-state poll pays nothing for it. + +The legacy per-feed `/api/v2/feeds/batch` path (`fetchAllFeedsViaBatch`, with per-subscription +`feedCursors`) is kept for one release as a fallback: it runs when the timeline 404s and whenever +the server reports `ingestActive: false` (this environment's crawler isn't pushing into D1). See +`docs/plans/D1_FEED_TIMELINE.md`. + +Client errors flow through `/api/telemetry/error`; no Sentry SDK ships in the frontend bundle. +See [`docs/RUNBOOK.md`](../docs/RUNBOOK.md) for sampling and recovery details. ### Key Routes diff --git a/frontend/src/lib/components/ArticleCard.svelte b/frontend/src/lib/components/ArticleCard.svelte index 8a32766e..d8331afd 100644 --- a/frontend/src/lib/components/ArticleCard.svelte +++ b/frontend/src/lib/components/ArticleCard.svelte @@ -1,4 +1,5 @@