diff --git a/.github/workflows/feed-proxy-deploy.yml b/.github/workflows/feed-proxy-deploy.yml
index 2b6ecb5d..2d5318e4 100644
--- a/.github/workflows/feed-proxy-deploy.yml
+++ b/.github/workflows/feed-proxy-deploy.yml
@@ -54,6 +54,43 @@ jobs:
- name: Run tests
run: bun test
+ deploy-staging:
+ name: Deploy to Staging
+ needs: [typecheck, test]
+ # Same cadence as the staging Worker: every push to main. Staging has its own
+ # Fly app (fly.staging.toml) so it never shares the prod machine or its cache,
+ # and config drift between the two files surfaces here before a release.
+ if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+ runs-on: ubuntu-latest
+ environment: staging
+ steps:
+ - name: Checkout
+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+
+ - name: Setup flyctl
+ uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1 # master
+
+ # The app is provisioned by hand (Phase 3 of docs/plans/D1_FEED_TIMELINE.md:
+ # app + volume + PROXY_SECRET). Until that happens this job would fail on
+ # every push to main, so skip it with a notice instead.
+ - name: Check staging app exists
+ id: staging_app
+ run: |
+ if flyctl status -a skyreader-feed-proxy-staging >/dev/null 2>&1; then
+ echo "exists=true" >> "$GITHUB_OUTPUT"
+ else
+ echo "exists=false" >> "$GITHUB_OUTPUT"
+ echo "::notice::skyreader-feed-proxy-staging not provisioned yet; skipping staging deploy (see docs/plans/D1_FEED_TIMELINE.md, Phase 3)."
+ fi
+ env:
+ FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
+
+ - name: Deploy
+ if: steps.staging_app.outputs.exists == 'true'
+ run: flyctl deploy --remote-only --config fly.staging.toml
+ env:
+ FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
+
deploy:
name: Deploy to Production
needs: [typecheck, test]
diff --git a/CLAUDE.md b/CLAUDE.md
index ddefe3b5..8e062228 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -9,7 +9,7 @@ This is a monorepo with 6 packages:
- `backend/` - Cloudflare Workers API
- `frontend/` - SvelteKit PWA
- `admin/` - SvelteKit admin dashboard (Cloudflare Pages)
-- `feed-proxy/` - Feed caching proxy (Fly.io)
+- `feed-proxy/` - Feed crawler + article extraction (Fly.io; one app per environment)
- `linkblog-site/` - Standalone SvelteKit app rendering public linkblogs at `linkblogs.skyreader.app` (Cloudflare Pages)
- `extension/` - Chrome extension for one-click saves with live-DOM article extraction (Manifest V3)
@@ -85,6 +85,9 @@ This script runs D1 migrations, then starts the feed proxy (port 3000), backend
```
FRONTEND_URL=http://127.0.0.1:5173
FEED_PROXY_URL=http://127.0.0.1:3000
+ # Shared with the proxy (dev-local.sh exports the same value as PROXY_SECRET).
+ # Feed ingest is fail-closed, so without this the local reader stays empty.
+ FEED_PROXY_SECRET=dev-proxy-secret
```
**Resetting the local database:**
@@ -217,14 +220,19 @@ AT Protocol (Bluesky PDS) + Fly.io Feed Proxy + Jetstream Firehose
- **Database:** D1 (SQLite) - reads the same database as the backend
- **Features:** Ops panel (cron liveness, firehose lag, proxy cache health) with 30-day trend
sparklines, system metrics, user management, feed health monitoring, search/sort/pagination
-- **Pages:** Dashboard (ops + metrics + trends), Users (list + detail), Feeds (health + error tracking)
+- **Pages:** Dashboard (ops + metrics + trends), Users (list + detail), Feeds (health + error tracking).
+ Feed health is the crawler's own verdict from `feeds.error_count` / `feeds.crawl_stale`, not an
+ inference from `last_ingest_at` — that only moves when a feed publishes, so it says nothing about
+ whether the feed still works.
- **Deploy:** Cloudflare Pages via GitHub Actions (staging on push to main, production on release)
### Key Data Flow
1. **Auth:** Handle → DID resolution → OAuth PKCE/DPoP → session token
2. **Subscriptions:** Stored in user's PDS, cached locally in IndexedDB
-3. **Feed Updates:** Frontend requests feeds → backend proxies via Fly.io feed cache
+3. **Feed Updates:** Fly.io proxy crawls feeds and pushes new items into D1 (`feed_items`); the
+ frontend refreshes with one `GET /api/v2/timeline` query that joins subscriptions and read
+ state. Reads never touch Fly — see `docs/plans/D1_FEED_TIMELINE.md`
4. **Social:** Jetstream firehose → D1 shares table → frontend polls for updates
## AT Protocol Integration
diff --git a/README.md b/README.md
index b5f910d5..cfebcddb 100644
--- a/README.md
+++ b/README.md
@@ -40,6 +40,9 @@ cd skyreader
```
FRONTEND_URL=http://127.0.0.1:5173
FEED_PROXY_URL=http://127.0.0.1:3000
+ # Shared with the proxy (dev-local.sh exports the same value as PROXY_SECRET).
+ # Feed ingest is fail-closed, so without this the local reader stays empty.
+ FEED_PROXY_SECRET=dev-proxy-secret
```
### Local Development
diff --git a/admin/src/lib/components/StatusBadge.svelte b/admin/src/lib/components/StatusBadge.svelte
index db8ae5c4..240108a0 100644
--- a/admin/src/lib/components/StatusBadge.svelte
+++ b/admin/src/lib/components/StatusBadge.svelte
@@ -1,9 +1,13 @@
-{labels[status]}
+{label ?? labels[status]}
diff --git a/backend/ARCHITECTURE.md b/backend/ARCHITECTURE.md
index 3a5489f8..8092209c 100644
--- a/backend/ARCHITECTURE.md
+++ b/backend/ARCHITECTURE.md
@@ -2,7 +2,7 @@
## Overview
-The Skyreader backend is a Cloudflare Worker that serves as an API gateway between the frontend and the AT Protocol ecosystem. It handles authentication, RSS feed fetching/parsing via a Fly.io proxy, social features, saved articles, and background Jetstream polling.
+The Skyreader backend is a Cloudflare Worker that serves as an API gateway between the frontend and the AT Protocol ecosystem. It handles authentication, the D1-served feed timeline (crawled and pushed by a Fly.io proxy), social features, saved articles, and background Jetstream polling.
```
┌──────────────────────────────────────────────────────────────────────────┐
@@ -17,8 +17,8 @@ The Skyreader backend is a Cloudflare Worker that serves as an API gateway betwe
│ CLOUDFLARE WORKER │
│ │
│ ┌──────────┐ ┌──────────────┐ ┌───────────────┐ ┌──────────────────┐ │
-│ │ auth.ts │ │ feeds-v2.ts │ │ social.ts │ │ subscriptions.ts │ │
-│ │ OAuth │ │ Feed proxy │ │ Social feed │ │ CRUD │ │
+│ │ auth.ts │ │ timeline.ts │ │ social.ts │ │ subscriptions.ts │ │
+│ │ OAuth │ │ D1 timeline │ │ Social feed │ │ CRUD │ │
│ └────┬─────┘ └──────┬───────┘ └───────┬───────┘ └────────┬─────────┘ │
│ │ │ │ │ │
│ ┌────┴────┐ ┌───────┴──────┐ ┌────────┴───────┐ ┌───────┴────────┐ │
@@ -51,7 +51,7 @@ The Skyreader backend is a Cloudflare Worker that serves as an API gateway betwe
↓ ↓ │
┌───────────────────┐ ┌───────────────────┐ ┌─────────────────────────┐
│ BLUESKY PDS │ │ FLY.IO FEED PROXY │ │ JETSTREAM FIREHOSE │
-│ (user's data) │ │ (RSS caching) │ │ (AT Protocol events) │
+│ (user's data) │ │ (crawler) │ │ (AT Protocol events) │
└───────────────────┘ └───────────────────┘ └─────────────────────────┘
```
@@ -134,15 +134,18 @@ Upsert user in D1
Redirect to frontend with auth exchange code
```
-### Feeds (`src/routes/feeds-v2.ts`)
+### Feeds (`src/routes/timeline.ts`, `src/routes/ingest.ts`, `src/routes/feeds-v2.ts`)
-All feed fetching is proxied through a Fly.io feed proxy service (`FEED_PROXY_URL`), authenticated with `FEED_PROXY_SECRET`.
+Reads are served from D1. The Fly.io feed proxy (`FEED_PROXY_URL`) is the crawler: it pushes new and edited items into `feed_items` and pulls the set of feeds to crawl, both authenticated with the shared `FEED_PROXY_SECRET`. A client refresh is one `/api/v2/timeline` request — a single query joining subscriptions and read state. See `docs/plans/D1_FEED_TIMELINE.md`.
-| Endpoint | Method | Auth | Description |
-| ------------------------ | ------ | ------ | --------------------------------- |
-| `/api/v2/feeds/fetch` | GET | Bearer | Fetch and parse a single RSS feed |
-| `/api/v2/feeds/batch` | POST | Bearer | Batch fetch multiple feeds |
-| `/api/v2/feeds/discover` | GET | Bearer | Discover feeds on a website URL |
+| Endpoint | Method | Auth | Description |
+| ------------------------- | ------ | ------ | -------------------------------------------- |
+| `/api/v2/timeline` | GET | Bearer | The whole refresh: items since a cursor |
+| `/api/internal/ingest` | POST | Secret | Crawler pushes new/edited items into D1 |
+| `/api/internal/crawl-set` | GET | Secret | Crawler pulls the feeds to crawl |
+| `/api/v2/feeds/fetch` | GET | Bearer | Single feed from D1, with proxy pull-through |
+| `/api/v2/feeds/batch` | POST | Bearer | Legacy batch fetch (fallback path) |
+| `/api/v2/feeds/discover` | GET | Bearer | Discover feeds on a website URL |
### Social (`src/routes/social.ts`)
@@ -346,9 +349,8 @@ Key tables:
| `auth_exchange_codes` | Short-lived auth exchange codes |
| `subscriptions_cache` | RSS feed subscriptions cached from PDS |
| `shares` | Aggregated share data from Jetstream (with reshare tracking) |
-| `feed_metadata` | Feed caching metadata (ETags, errors, subscriber count, shard_id) |
-| `feed_cache` | D1-based parsed feed cache |
-| `feed_items` | Individual feed items for efficient querying |
+| `feeds` | One row per crawled feed (title/site/image + `last_ingest_at`) |
+| `feed_items` | The item archive the timeline serves, keyed `(feed_url, guid)` with `seq` |
| `documents` | orphaned — documents moved to on-demand proxy fetch; table left in place |
| `publications_cache` | orphaned — the publication cache moved to the feed proxy |
| `social_read_positions_cache` | Unified social read tracking (shares + documents) |
@@ -357,8 +359,10 @@ Key tables:
| `rate_limits` | Per-user per-endpoint rate limiting |
| `user_settings` | User feature preferences (Leaflet sync, PDS sync) |
| `did_handle_cache` | Handle resolution cache |
-| `sync_state` | Jetstream cursor and other sync state |
+| `sync_state` | Jetstream cursor, archive generation token, crawler heartbeat |
| `reshares` | Reshare tracking |
+| `system_status` | Cron-written health board (cron liveness, poller lag, proxy stats) |
+| `metrics_snapshots` | Hourly trend points behind the admin's sparklines (90-day retention) |
---
@@ -421,8 +425,8 @@ crons = ["* * * * *"] # Every minute
### Feed Errors
-- Feed proxy handles caching and error tracking
-- `feed_metadata` tracks `error_count` and `fetch_error`
+- The feed proxy (the crawler) owns caching, retries and per-feed error tracking; its `/stats` is where `feedsInError` comes from
+- D1 keeps only ingest state: `feeds.last_ingest_at` per feed, `sync_state.crawler_heartbeat_at` for the crawler as a whole
### OAuth Errors
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md
index d86bc174..3819d64a 100644
--- a/backend/CLAUDE.md
+++ b/backend/CLAUDE.md
@@ -4,7 +4,25 @@
## Project Overview
-Skyreader backend is a Cloudflare Workers API that serves as a gateway between the frontend and the AT Protocol ecosystem. It handles authentication, RSS feed fetching via a Fly.io proxy, social features, saved articles, labels, and background Jetstream polling.
+Skyreader backend is a Cloudflare Workers API that serves as a gateway between the frontend and the AT Protocol ecosystem. It handles authentication, the feed timeline, social features, saved articles, labels, and background Jetstream polling.
+
+**Feed reads are served from D1, not the proxy.** The Fly.io proxy is the crawler: it pushes new
+and edited items into `feed_items` (`POST /api/internal/ingest`), pulls the set of feeds to
+crawl (`GET /api/internal/crawl-set`), and reports which feeds are failing to crawl
+(`POST /api/internal/feed-health`) — all authenticated with the shared `FEED_PROXY_SECRET`
+(fail-closed when unset). A client refresh is one `GET /api/v2/timeline` — a single query joining
+subscriptions and read state. Because reads never touch the crawler, a broken feed just goes quiet;
+the health report is the only thing that tells a reader its feed is dead rather than idle, and its
+payload is the COMPLETE unhealthy set (recovery = absence from the next report). All three internal
+endpoints stamp `sync_state.crawler_heartbeat_at`, and the timeline reports `ingestActive` as the
+AND of that heartbeat and `sync_state.timeline_enabled` — the operator's rollout gate. A crawler
+heartbeat only says a crawler is attached; it arrives seconds into a backfill that takes hours, so
+the gate is what actually admits readers to the archive (and, set back to `'0'`, is the fast
+rollback that returns every client to the legacy batch path with no deploy). Either half false and
+clients stay on `/batch`; the request short-circuits rather than building a page they will discard.
+See `docs/plans/D1_FEED_TIMELINE.md`. The proxy is still on the path for `/api/extract`, feed
+discovery, standard.site documents, and social context — plus the one crawl per new subscription
+(`warmFeedIntoArchive`) and the subscription-gated pull-through in `/api/v2/feeds/fetch`.
## Key Concepts
@@ -43,22 +61,24 @@ See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed documentation.
### Routes
-| File | Purpose |
-| ----------------------------- | ----------------------------------------------------- |
-| `src/routes/auth.ts` | OAuth flow (login, callback, logout, client metadata) |
-| `src/routes/feeds-v2.ts` | RSS fetching via Fly.io proxy |
-| `src/routes/social.ts` | Content detection for a DID (detect-content) |
-| `src/routes/shares.ts` | User shares CRUD (with PDS sync) |
-| `src/routes/subscriptions.ts` | Subscription CRUD (with PDS sync) |
-| `src/routes/records.ts` | PDS record listing |
-| `src/routes/reading.ts` | Article + document read positions (forward delta) |
-| `src/routes/labels.ts` | Unified item labels (read/starred/archived/tags) |
-| `src/routes/saved.ts` | Saved articles CRUD |
-| `src/routes/settings.ts` | User settings |
-| `src/routes/sync.ts` | PDS full sync, subscription sync, sync status |
-| `src/routes/lexicons.ts` | Serve lexicon schemas at /.well-known/lexicons |
-| `src/routes/health.ts` | `/api/health` (shallow) + `/api/health/deep` (gated) |
-| `src/routes/telemetry.ts` | `/api/telemetry/error` — sampled client error reports |
+| File | Purpose |
+| ----------------------------- | ------------------------------------------------------ |
+| `src/routes/auth.ts` | OAuth flow (login, callback, logout, client metadata) |
+| `src/routes/timeline.ts` | `GET /api/v2/timeline` — the whole refresh, one query |
+| `src/routes/ingest.ts` | Crawler endpoints: item ingest, crawl set, feed health |
+| `src/routes/feeds-v2.ts` | Single-feed read (D1 + pull-through), discover, docs |
+| `src/routes/social.ts` | Social feed, popular, grouped, detect-content |
+| `src/routes/shares.ts` | User shares CRUD (with PDS sync) |
+| `src/routes/subscriptions.ts` | Subscription CRUD (with PDS sync) |
+| `src/routes/records.ts` | PDS record listing |
+| `src/routes/reading.ts` | Article + document read positions (forward delta) |
+| `src/routes/labels.ts` | Unified item labels (read/starred/archived/tags) |
+| `src/routes/saved.ts` | Saved articles CRUD |
+| `src/routes/settings.ts` | User settings |
+| `src/routes/sync.ts` | PDS full sync, subscription sync, sync status |
+| `src/routes/lexicons.ts` | Serve lexicon schemas at /.well-known/lexicons |
+| `src/routes/health.ts` | `/api/health` (shallow) + `/api/health/deep` (gated) |
+| `src/routes/telemetry.ts` | `/api/telemetry/error` — sampled client error reports |
### Services
@@ -91,7 +111,7 @@ never `Sentry.captureException` directly, so the vendor stays a one-file decisio
Logging: use `log.*` with a stable low-cardinality `event` slug and put the details
in fields. Workers Logs indexes the fields of a logged object but treats a string
-as opaque text, so `log.info('feed_fetched', { feedCount })` is queryable and an
+as opaque text, so `log.info('feed_ingested', { itemCount })` is queryable and an
interpolated `console.log` sentence is not. Never log a credential — there is no
redaction layer on this path.
@@ -102,8 +122,11 @@ and outbound feed-proxy calls.
The every-minute cron also _records_: poller lag and cron liveness to
`system_status`, the proxy's cache stats every 5th minute, and an hourly row in
`metrics_snapshots` (pruned at 90 days). The admin renders those tables directly.
-Recording failures never withhold the cron heartbeat — losing a data point is not
-an outage; see the note on `runRecordingStep()`.
+The snapshot's counts come from D1 (`users`, `feeds`, `feed_items`, …) and its
+health numbers from the `system_status` rows — including `feeds_with_errors`,
+which is the crawler's `feedsInError` and is recorded NULL, not 0, when the proxy
+row is stale. Recording failures never withhold the cron heartbeat — losing a data
+point is not an outage; see the note on `runRecordingStep()`.
The browser reports its own errors to `/api/telemetry/error` (unauthenticated by
design — an error on the login screen counts, and the route is answered before
@@ -117,9 +140,9 @@ procedures: [`docs/RUNBOOK.md`](../docs/RUNBOOK.md).
### Durable Objects
-| File | Purpose |
-| ----------------------------------------- | ----------------------------------------------------------------- |
-| `src/durable-objects/jetstream-poller.ts` | Jetstream firehose for `app.skyreader.feed.subscription` (alarms) |
+| File | Purpose |
+| ----------------------------------------- | ----------------------------------------------------- |
+| `src/durable-objects/jetstream-poller.ts` | Long-running Jetstream firehose connection via alarms |
### Storage
@@ -131,21 +154,24 @@ Key tables:
- `sessions` - Server-side sessions (tokens, DPoP key, expiry)
- `subscriptions_cache` - Cached feed subscriptions from PDS
- `shares` - Aggregated share data from Jetstream
-- `feed_metadata` - Feed caching metadata (ETags, errors, shard_id)
-- `feed_cache` - Parsed feed cache
-- `feed_items` - Individual feed items
-- `documents` - orphaned. Nothing reads or writes it since documents moved to
- on-demand proxy fetch; the table is left in place (as `shares` was) rather than
- dropped
-- `publications_cache` - orphaned too. The publication metadata cache it backed
- now lives in the feed proxy (`feed-proxy/src/standard-site.ts`)
+- `feeds` - One row per crawled feed (title/site/image + `last_ingest_at`), plus the crawler's
+ health verdict: `error_count`/`last_error`/`next_retry_at`/`last_fetch_at` (unix seconds), which
+ the timeline serves to readers as `feedHealth`, and `crawl_stale` for a feed the crawler isn't
+ reaching at all (operator-only; the admin alarms on it). `last_ingest_at` is publishing cadence,
+ NOT health — it only moves when a fetch yields a new item
+- `feed_items` - The feed archive the timeline serves: every item the crawler has ever pushed,
+ keyed `(feed_url, guid)` with a monotonic `seq`. Never pruned in ordinary operation — see
+ `docs/plans/D1_FEED_TIMELINE.md`
+- `documents` / `publications_cache` - orphaned after standard.site document reads moved to the
+ feed proxy; retained in place but no longer read or written
- `item_labels_cache` - Unified labels (read/starred/archived/tags)
- `saved_articles` - Saved/bookmarked articles
- `social_read_positions_cache` - Legacy social read tracking (superseded; document
reads now live in `item_labels_cache` as `item_type='document'`/`label='read'`)
- `user_settings` - User preferences
- `rate_limits` - Per-user rate limiting
-- `sync_state` - Jetstream cursor and other sync state
+- `sync_state` - Jetstream cursor, the archive generation token, the crawler heartbeat, and
+ `timeline_enabled` (the rollout gate: only an explicit `'0'` holds clients on the batch path)
- `system_status` - Cron-written health board (cron liveness, poller lag, proxy stats)
- `metrics_snapshots` - Hourly trend points behind the admin's sparklines (90-day retention)
@@ -181,7 +207,10 @@ FEED_PROXY_URL = "https://skyreader-feed-proxy.fly.dev"
SENTRY_ENVIRONMENT = "production" # "staging" in [env.staging]
# Secrets (set via `wrangler secret put`):
-# FEED_PROXY_SECRET - authenticates with the Fly.io feed proxy
+# FEED_PROXY_SECRET - shared with the Fly.io proxy, both directions: outbound
+# proxy calls and the inbound crawler endpoints
+# (/api/internal/ingest, /api/internal/crawl-set), which
+# are fail-closed when it is unset
# SENTRY_DSN - error reporting (unset ⇒ silent no-op)
# HEARTBEAT_URL - dead-man ping for the every-minute cron
# HEALTH_CHECK_SECRET - gates /api/health/deep (X-Health-Secret header)
diff --git a/backend/migrations/0068_feed_timeline.sql b/backend/migrations/0068_feed_timeline.sql
new file mode 100644
index 00000000..620c26d1
--- /dev/null
+++ b/backend/migrations/0068_feed_timeline.sql
@@ -0,0 +1,57 @@
+-- D1-served feed timeline (ingest-push architecture).
+--
+-- The Fly proxy stays the crawler but leaves the read path entirely: it pushes
+-- new/edited items into the tables below, and a client refresh becomes ONE query
+-- (feed_items JOIN subscriptions_cache LEFT JOIN item_labels_cache) instead of
+-- N batched Worker->Fly hops plus chunked read-key lookups.
+--
+-- Retention: D1 is the ARCHIVE. Ordinary ingest never deletes an item; the proxy's
+-- own K=200 log is only the outbox (a delivery window). The single trim that
+-- exists is a per-feed sanity cap (5,000) that healthy feeds never reach — see
+-- routes/ingest.ts.
+
+-- The dormant pre-Fly trio. Zero references in backend/src; feed_cache was
+-- already wiped to '{}' by 0018. Dropped so the names are free for the new shapes.
+DROP TABLE IF EXISTS feed_items;
+DROP TABLE IF EXISTS feed_cache;
+DROP TABLE IF EXISTS feed_metadata;
+
+-- One row per crawled feed: metadata for the reader plus ingest observability.
+CREATE TABLE feeds (
+ feed_url TEXT PRIMARY KEY,
+ title TEXT,
+ site_url TEXT,
+ description TEXT,
+ image_url TEXT,
+ last_ingest_at INTEGER,
+ created_at INTEGER NOT NULL DEFAULT (unixepoch())
+);
+
+-- The durable item archive. Same invariants as the proxy's proven log:
+-- AUTOINCREMENT seq is a never-reused monotonic cursor, an edit updates in place
+-- (seq unchanged, so it is not re-delivered), content_hash is NOT NULL because a
+-- NULL would make the edit predicate (`<>`) silently never match.
+CREATE TABLE feed_items (
+ seq INTEGER PRIMARY KEY AUTOINCREMENT,
+ feed_url TEXT NOT NULL,
+ guid TEXT NOT NULL,
+ item_json TEXT NOT NULL,
+ published_at INTEGER,
+ first_seen_at INTEGER NOT NULL,
+ content_hash TEXT NOT NULL,
+ UNIQUE(feed_url, guid)
+);
+
+CREATE INDEX idx_feed_items_feed_seq ON feed_items(feed_url, seq);
+CREATE INDEX idx_feed_items_feed_published_seq
+ ON feed_items(feed_url, published_at DESC, seq DESC);
+
+-- The timeline join probes subscriptions by (user_did, feed_url).
+CREATE INDEX IF NOT EXISTS idx_subscriptions_cache_user_feed
+ ON subscriptions_cache(user_did, feed_url);
+
+-- Generation token for the archive (D1 recreation / Time Travel restore guard).
+-- Clients store it beside their cursor and cold-start on mismatch. Any D1 restore
+-- must bump this (seqs rewind while the token would otherwise stay the same).
+INSERT OR IGNORE INTO sync_state (key, value)
+VALUES ('items_generation', lower(hex(randomblob(16))));
diff --git a/backend/migrations/0069_metrics_snapshots_feeds_in_error.sql b/backend/migrations/0069_metrics_snapshots_feeds_in_error.sql
new file mode 100644
index 00000000..86d30254
--- /dev/null
+++ b/backend/migrations/0069_metrics_snapshots_feeds_in_error.sql
@@ -0,0 +1,39 @@
+-- `feeds_with_errors` loses its NOT NULL, because its source changed.
+--
+-- It used to be `SELECT COUNT(*) FROM feed_metadata WHERE error_count > 0` — a
+-- local count that always had an answer. 0068 drops `feed_metadata`: the new
+-- `feeds` table carries ingest metadata only (`last_ingest_at`), and per-feed
+-- fetch errors now live where the fetching happens, in the crawler. The hourly
+-- snapshot therefore reads the same number the live tile does, the proxy's
+-- `feedsInError`, out of the `proxy_stats` row the cron writes every 5 minutes.
+--
+-- That number can genuinely be unknown — an unreachable proxy leaves the row
+-- stale and the snapshot has nothing to record. Storing 0 there would draw a
+-- healthy flat line through exactly the outage the column exists to show, so
+-- the column joins `firehose_lag_ms` and `proxy_fresh_pct` in being nullable.
+--
+-- SQLite can't drop a NOT NULL in place, hence the rebuild. `captured_at` stays
+-- the INTEGER PRIMARY KEY (= rowid), so range scans by time stay index-free.
+CREATE TABLE metrics_snapshots_new (
+ captured_at INTEGER PRIMARY KEY,
+ users INTEGER NOT NULL,
+ feeds INTEGER NOT NULL,
+ feed_items INTEGER NOT NULL,
+ subscriptions INTEGER NOT NULL,
+ saved_articles INTEGER NOT NULL,
+ feeds_with_errors INTEGER,
+ active_sessions INTEGER NOT NULL,
+ firehose_lag_ms INTEGER,
+ proxy_fresh_pct REAL
+);
+
+INSERT INTO metrics_snapshots_new
+ (captured_at, users, feeds, feed_items, subscriptions, saved_articles,
+ feeds_with_errors, active_sessions, firehose_lag_ms, proxy_fresh_pct)
+SELECT captured_at, users, feeds, feed_items, subscriptions, saved_articles,
+ feeds_with_errors, active_sessions, firehose_lag_ms, proxy_fresh_pct
+ FROM metrics_snapshots;
+
+DROP TABLE metrics_snapshots;
+
+ALTER TABLE metrics_snapshots_new RENAME TO metrics_snapshots;
diff --git a/backend/migrations/0070_feed_health.sql b/backend/migrations/0070_feed_health.sql
new file mode 100644
index 00000000..f108cb9b
--- /dev/null
+++ b/backend/migrations/0070_feed_health.sql
@@ -0,0 +1,46 @@
+-- Per-feed crawl health in the archive.
+--
+-- On the legacy batch path the client learned a feed was broken from the proxy's
+-- own response (`status: 'error'`, errorCount, nextRetryAt). The timeline path
+-- never touches the proxy, so that signal disappeared: a feed failing to crawl
+-- simply delivered no items, which is indistinguishable from a quiet feed. The
+-- crawler now reports health here (POST /api/internal/feed-health) and the
+-- timeline hands it to the client alongside its items.
+--
+-- Units: unix SECONDS, like `last_ingest_at` and the rest of the backend. The
+-- proxy stores these as milliseconds and converts on send; the timeline converts
+-- back to milliseconds on the way out, because that is what the client's
+-- FeedStatus contract has always used.
+
+ALTER TABLE feeds ADD COLUMN error_count INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE feeds ADD COLUMN last_error TEXT;
+ALTER TABLE feeds ADD COLUMN last_error_at INTEGER;
+ALTER TABLE feeds ADD COLUMN next_retry_at INTEGER;
+-- Last time the crawler successfully FETCHED the feed. Distinct from
+-- `last_ingest_at`, which only moves when a fetch produced a new or edited item:
+-- a healthy feed that hasn't published in a month still gets fetched hourly.
+ALTER TABLE feeds ADD COLUMN last_fetch_at INTEGER;
+-- The crawler has this feed in its crawl set but hasn't managed to fetch it in
+-- hours — starved by a saturated warm loop rather than failing outright. Distinct
+-- from `error_count`: nothing is erroring, the crawler simply never gets to it.
+--
+-- This is the difference between an alarm and noise. `last_ingest_at` only moves
+-- when a fetch yields a NEW item, so "no ingest in an hour" describes every
+-- weekly blog in the archive; the admin's old stale-feed metric was therefore
+-- warning permanently and meant nothing. Being in the crawl set and un-fetched is
+-- a genuine fault, and it is the one the current warm-loop capacity work needs.
+ALTER TABLE feeds ADD COLUMN crawl_stale INTEGER NOT NULL DEFAULT 0;
+
+-- The trouble set is a handful of rows out of the whole archive, so let the
+-- per-user health lookup, the recovery sweep and the admin's counts start from it
+-- rather than scanning `feeds`. Recovery is "currently flagged, and absent from
+-- the latest report", which this index makes cheap to enumerate — the
+-- alternative, a NOT IN list of every healthy feed, would blow the
+-- bound-parameter limit at ~1,300.
+CREATE INDEX idx_feeds_unhealthy ON feeds(feed_url) WHERE error_count > 0 OR crawl_stale = 1;
+
+-- Bumped whenever a health report actually changes something. Clients echo the
+-- revision they last saw and the timeline only re-sends the health payload when
+-- it differs, so the steady-state poll stays at one query. Only `error_count`
+-- feeds into it: `crawl_stale` is an operator signal, invisible to readers.
+INSERT OR IGNORE INTO sync_state (key, value) VALUES ('feed_health_rev', '0');
diff --git a/backend/migrations/0071_timeline_rollout_gate.sql b/backend/migrations/0071_timeline_rollout_gate.sql
new file mode 100644
index 00000000..8f19dc38
--- /dev/null
+++ b/backend/migrations/0071_timeline_rollout_gate.sql
@@ -0,0 +1,30 @@
+-- Rollout gate for the D1-served timeline.
+--
+-- `ingestActive` alone flips every client onto the timeline the instant the
+-- crawler first checks in — which is the moment the backfill STARTS, not the
+-- moment it finishes. The proxy stamps a heartbeat on its very first crawl-set
+-- pull (seconds after the release), so without this gate every reader cold-starts
+-- against a nearly-empty archive and then drags the entire ~200-items-per-feed
+-- backfill through its incremental drain: the expensive global scan, at its worst
+-- case, for as long as the backfill takes.
+--
+-- This decouples the two. Enable INGEST_URL, let the archive fill, watch the
+-- proxy's `ingest.pending` trend to ~0, THEN flip this to '1'. It is also the
+-- only fast way back: setting it to '0' returns every client to the legacy batch
+-- path on their next poll, with no Worker deploy and no waiting out the 30-minute
+-- heartbeat freshness window.
+--
+-- npx wrangler d1 execute skyreader --remote \
+-- --command "UPDATE sync_state SET value='1', updated_at=unixepoch() WHERE key='timeline_enabled'"
+--
+-- Only an explicit '0' gates: an absent row means enabled, so a hand-built schema
+-- or a future environment can't be silently held on the legacy path.
+--
+-- The value depends on whether this deployment already has readers. An existing
+-- one (prod, staging) has users and a backfill ahead of it, so it starts gated and
+-- an operator opens it deliberately. A fresh one (local dev, e2e, CI) has no users
+-- and no backlog, so it starts open and nothing has to know this key exists.
+INSERT OR IGNORE INTO sync_state (key, value, updated_at)
+SELECT 'timeline_enabled',
+ CASE WHEN EXISTS (SELECT 1 FROM users) THEN '0' ELSE '1' END,
+ unixepoch();
diff --git a/backend/src/index.ts b/backend/src/index.ts
index 202c584f..63601262 100644
--- a/backend/src/index.ts
+++ b/backend/src/index.ts
@@ -16,6 +16,8 @@ import {
handleV2Mentions,
handleV2MentionLane,
} from './routes/feeds-v2';
+import { handleIngest, handleCrawlSet, handleFeedHealth } from './routes/ingest';
+import { handleTimeline } from './routes/timeline';
import { handleDetectContent } from './routes/social';
import {
handleCreateLinkblogShare,
@@ -315,10 +317,28 @@ async function route(
response = await handleAuthMe(request, env);
break;
+ // Internal crawler endpoints authenticate with FEED_PROXY_SECRET in their
+ // handlers; they intentionally do not require a user session.
+ case url.pathname === '/api/internal/ingest':
+ response = await handleIngest(request, env);
+ break;
+ case url.pathname === '/api/internal/crawl-set':
+ response = await handleCrawlSet(request, env);
+ break;
+ case url.pathname === '/api/internal/feed-health':
+ response = await handleFeedHealth(request, env);
+ break;
+
+ // The reader's whole refresh, served from the D1 archive in one query.
+ case url.pathname === '/api/v2/timeline':
+ if (!session) return unauthorizedResponse(headers);
+ response = await handleTimeline(request, env, session);
+ break;
+
// Feed routes (v2 via Fly.io proxy)
case url.pathname === '/api/v2/feeds/fetch':
if (!session) return unauthorizedResponse(headers);
- response = await handleV2FeedFetch(request, env);
+ response = await handleV2FeedFetch(request, env, session);
break;
case url.pathname === '/api/v2/feeds/batch':
if (!session) return unauthorizedResponse(headers);
diff --git a/backend/src/observability/ops-metrics.ts b/backend/src/observability/ops-metrics.ts
index 3d9c52db..a905f1f7 100644
--- a/backend/src/observability/ops-metrics.ts
+++ b/backend/src/observability/ops-metrics.ts
@@ -270,8 +270,9 @@ export async function recordCronRun(
}
/**
- * One row per hour: the counts the admin shows as tiles, plus the two health
- * numbers that only exist in `system_status`. Same job prunes the tail, so the
+ * One row per hour: the counts the admin shows as tiles, plus the three health
+ * numbers that only exist in `system_status` (firehose lag, proxy cache
+ * freshness, feeds the crawler has in error). Same job prunes the tail, so the
* table can't grow without an owner.
*/
export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise {
@@ -282,11 +283,10 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise<
const counts = await env.DB.batch<{ count: number }>([
env.DB.prepare('SELECT COUNT(*) AS count FROM users'),
- env.DB.prepare('SELECT COUNT(*) AS count FROM feed_metadata'),
+ env.DB.prepare('SELECT COUNT(*) AS count FROM feeds'),
env.DB.prepare('SELECT COUNT(*) AS count FROM feed_items'),
env.DB.prepare('SELECT COUNT(*) AS count FROM subscriptions_cache'),
env.DB.prepare('SELECT COUNT(*) AS count FROM saved_articles'),
- env.DB.prepare('SELECT COUNT(*) AS count FROM feed_metadata WHERE error_count > 0'),
env.DB.prepare('SELECT COUNT(*) AS count FROM sessions WHERE expires_at > ?').bind(
sessionCutoff
),
@@ -300,6 +300,11 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise<
const usable = (row: StatusRow | null): T | null =>
row && now - row.updatedAt <= SNAPSHOT_MAX_AGE_MS ? row.value : null;
+ // Per-feed fetch errors live in the crawler, not in D1: `feeds` carries ingest
+ // metadata only. So the trend point reads the same number the live tile does,
+ // and is null — not 0 — when the proxy row is missing or stale (see 0069).
+ const feedsWithErrors = usable(proxy)?.feedsInError ?? null;
+
await env.DB.prepare(
`INSERT OR REPLACE INTO metrics_snapshots
(captured_at, users, feeds, feed_items, subscriptions, saved_articles,
@@ -313,8 +318,8 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise<
at(2),
at(3),
at(4),
+ feedsWithErrors,
at(5),
- at(6),
usable(poller)?.lagMs ?? null,
usable(proxy)?.freshPct ?? null
)
@@ -328,7 +333,7 @@ export async function writeMetricsSnapshot(env: Env, now = Date.now()): Promise<
capturedAt,
users: at(0),
feeds: at(1),
- feedsWithErrors: at(5),
+ feedsWithErrors,
prunedRows: pruned.meta?.changes ?? 0,
});
}
diff --git a/backend/src/routes/feeds-v2.ts b/backend/src/routes/feeds-v2.ts
index 87d2eb89..6dfbbf13 100644
--- a/backend/src/routes/feeds-v2.ts
+++ b/backend/src/routes/feeds-v2.ts
@@ -8,6 +8,8 @@ import type {
} from '../services/feed-proxy-client';
import { resolveStandardSite } from '../utils/canonical-url';
import { chunkArray, getReadKeys } from './reading';
+import { clearFeedHealth, ingestProxyFeed } from './ingest';
+import { readFeedMetadata, readFeedSlice, type FeedHealth } from './timeline';
import {
getLinkblogTargets,
publicationUri as linkblogPublicationUri,
@@ -19,10 +21,11 @@ interface V2FeedResponse {
siteUrl?: string;
imageUrl?: string;
items: FeedItem[];
+ // Unix ms of the last ingest for this feed (freshness, for the client's UI) —
+ // no longer a live upstream fetch time, since reads never touch the proxy.
fetchedAt: number;
- cursor?: number;
- generation?: string;
- hasMore?: boolean;
+ // Present only when the crawler currently considers this feed broken.
+ health?: FeedHealth;
}
interface V2BatchFeedResult {
@@ -52,22 +55,59 @@ interface V2BatchResponse {
readCursor?: number;
}
+// Newest-N a single-feed fetch delivers from the D1 archive.
+const SINGLE_FEED_LIMIT = 30;
+const SINGLE_FEED_MAX_LIMIT = 200;
+
+/**
+ * Does this user hold a subscription to this feed? The gate on every write into
+ * the shared archive that a user request can trigger. Parked (`active = 0`) subs
+ * count — the user owns the feed either way, and re-activating it shouldn't need
+ * a different code path.
+ */
+async function callerSubscribes(env: Env, userDid: string, feedUrl: string): Promise {
+ const row = await env.DB.prepare(
+ `SELECT 1 AS ok FROM subscriptions_cache WHERE user_did = ? AND feed_url = ? LIMIT 1`
+ )
+ .bind(userDid, feedUrl)
+ .first<{ ok: number }>();
+ return !!row;
+}
+
/**
* GET /api/v2/feeds/fetch
*
- * Fetch a single feed via Fly.io proxy with GUID-based incremental sync.
+ * One feed's newest slice, served from the D1 archive with read state joined in
+ * (the timeline's per-feed sibling). This is the "new subscription gap" path:
+ * a feed the user just subscribed to contributes nothing to the global timeline
+ * cursor (its items sit below it), so the client fetches it directly here.
+ *
+ * If D1 has nothing for the feed (nobody was subscribed, so the crawler never
+ * pushed it), we PULL THROUGH: fetch it from the proxy once, ingest the result,
+ * then serve from D1. Steady state never touches Fly.
+ *
+ * The pull-through only runs for a feed the CALLER actually subscribes to. The
+ * archive is shared and (by design) never pruned, so an open ingest surface would
+ * let any authenticated user write arbitrary feeds into it forever; requiring a
+ * subscription bounds writes to each user's own feed list. Subscriptions are
+ * written to `subscriptions_cache` synchronously by POST /api/subscriptions
+ * before the client fetches, so the add-feed path is unaffected.
*
* Query params:
* - url: Feed URL (required)
- * - since_guids: Comma-separated GUIDs the client already has (optional)
- * - limit: Max items to return (optional, default 100)
+ * - limit: Max items to return (optional, default 30)
+ * - refresh: `1` to force the pull-through even when the archive already has the
+ * feed — the "retry this feed" action, the one path that still asks the
+ * crawler for a fresh fetch on demand.
+ * - since_guids: accepted and ignored (legacy); the client dedupes by GUID.
*/
-export async function handleV2FeedFetch(request: Request, env: Env): Promise {
+export async function handleV2FeedFetch(
+ request: Request,
+ env: Env,
+ session: Session
+): Promise {
const url = new URL(request.url);
const feedUrl = url.searchParams.get('url');
- const sinceGuidsParam = url.searchParams.get('since_guids');
- const sinceSeqParam = url.searchParams.get('since_seq');
- const generationParam = url.searchParams.get('generation');
const limitParam = url.searchParams.get('limit');
if (!feedUrl) {
@@ -87,28 +127,62 @@ export async function handleV2FeedFetch(request: Request, env: Env): Promise 0
+ ? {
+ errorCount: metadata.error_count,
+ error: metadata.last_error ?? undefined,
+ lastErrorAt: metadata.last_error_at ? metadata.last_error_at * 1000 : undefined,
+ nextRetryAt: metadata.next_retry_at ? metadata.next_retry_at * 1000 : undefined,
+ lastFetchedAt: metadata.last_fetch_at ? metadata.last_fetch_at * 1000 : undefined,
+ }
+ : undefined,
};
return new Response(JSON.stringify(response), {
@@ -790,26 +864,31 @@ export interface WarmCacheResult {
}
/**
- * Warm up the proxy cache for a single feed.
- * Just fetches via proxy to ensure it's cached - no D1 storage.
+ * Fetch a newly subscribed feed through the proxy and INGEST it into the archive.
+ *
+ * This used to only warm the proxy's cache and throw the parse away. Now that D1
+ * is the read path, the same one fetch we were already paying for at subscribe
+ * time populates the archive — so the client's first per-feed read is a plain D1
+ * query instead of another synchronous crawl through the pull-through.
*/
-export async function warmProxyCache(env: Env, feedUrl: string): Promise {
+export async function warmFeedIntoArchive(env: Env, feedUrl: string): Promise {
try {
const client = new FeedProxyClient(env);
const feed = await client.fetchFeed(feedUrl);
+ await ingestProxyFeed(env, feedUrl, feed);
return { success: true, itemCount: feed.items.length };
} catch (error) {
const errorMessage = error instanceof Error ? error.message : 'Failed to fetch feed';
- console.error(`[warmProxyCache] Error fetching ${feedUrl}:`, errorMessage);
+ console.error(`[warmFeedIntoArchive] Error fetching ${feedUrl}:`, errorMessage);
return { success: false, error: errorMessage };
}
}
/**
- * Warm up the proxy cache for multiple feeds in a single batch request.
- * Just fetches via proxy to ensure they're cached - no D1 storage.
+ * The same, for several feeds in one proxy batch request (bulk / OPML import).
+ * A feed whose ingest fails is reported as an error but never fails the others.
*/
-export async function warmProxyCacheBatch(
+export async function warmFeedsIntoArchive(
env: Env,
feedUrls: string[]
): Promise> {
@@ -834,15 +913,20 @@ export async function warmProxyCacheBatch(
} else if (feedResult.status === 'error') {
results[feedUrl] = { success: false, error: feedResult.error };
} else {
- results[feedUrl] = {
- success: true,
- itemCount: feedResult.items.length,
- };
+ try {
+ await ingestProxyFeed(env, feedUrl, feedResult);
+ results[feedUrl] = { success: true, itemCount: feedResult.items.length };
+ } catch (error) {
+ results[feedUrl] = {
+ success: false,
+ error: error instanceof Error ? error.message : 'Ingest failed',
+ };
+ }
}
}
} catch (error) {
const errorMessage = error instanceof Error ? error.message : 'Batch fetch failed';
- console.error(`[warmProxyCacheBatch] Batch error:`, errorMessage);
+ console.error(`[warmFeedsIntoArchive] Batch error:`, errorMessage);
for (const feedUrl of feedUrls) {
results[feedUrl] = { success: false, error: errorMessage };
diff --git a/backend/src/routes/ingest.ts b/backend/src/routes/ingest.ts
new file mode 100644
index 00000000..28ccc783
--- /dev/null
+++ b/backend/src/routes/ingest.ts
@@ -0,0 +1,621 @@
+import type { Env, FeedItem } from '../types';
+
+/**
+ * Internal (proxy → Worker) endpoints for the D1-served feed timeline.
+ *
+ * The Fly proxy is the crawler; it pushes deltas from its durable log here and
+ * pulls the set of feeds it should crawl. Neither endpoint has a user session —
+ * both are authenticated with the shared `FEED_PROXY_SECRET` (the same value the
+ * Worker sends outbound as `X-Proxy-Secret`), so no new secret exists to manage.
+ */
+
+// Per-feed sanity cap. D1 is an archive: ordinary ingest never prunes. This trim
+// exists only to bound a pathological feed (GUID churn re-minting ids every
+// fetch, calendar feeds reposting their whole window). A daily-cadence feed takes
+// ~14 years to reach it, so a feed at the cap is a bug signal, not steady state.
+export const SANITY_CAP = 5000;
+
+// Stored-content cap per item. Unbounded retention makes this mandatory rather
+// than optional: D1 has a hard 10 GB database ceiling, so storage grows with
+// ingest velocity × item size × time. Oversized bodies are dropped at ingest
+// (summary/title/url/image kept) and `contentTruncated` is set, which the reader
+// acts on: ArticleCard auto-extracts the full text via /api/extract when a
+// truncated article is opened, so the body the user sees is still the whole
+// article (see frontend/src/lib/components/ArticleCard.svelte).
+export const MAX_ITEM_CONTENT_BYTES = 8 * 1024;
+
+// How long a crawler heartbeat stays "fresh". The proxy pulls the crawl set every
+// 5 minutes whenever INGEST_URL is set, so a stamp older than this means this
+// environment has no crawler pushing into its D1 — the timeline says so and
+// clients stay on the legacy batch path instead of serving an empty archive.
+export const CRAWLER_HEARTBEAT_KEY = 'crawler_heartbeat_at';
+export const CRAWLER_HEARTBEAT_FRESH_SECONDS = 30 * 60;
+
+// The rollout gate (migration 0071). A crawler heartbeat says a crawler is
+// ATTACHED; it says nothing about whether the archive it fills is complete, and
+// the first heartbeat lands seconds into a backfill that takes hours. This key is
+// what actually moves readers onto the timeline, so the two can be sequenced:
+// enable ingest, let the archive fill, then open the gate. Setting it back to '0'
+// is also the fast rollback — every client is on the batch path at its next poll.
+//
+// Only an explicit '0' gates. An absent row means enabled, so a hand-built schema
+// (the unit-test harness) or a new environment is never silently held back.
+export const TIMELINE_ENABLED_KEY = 'timeline_enabled';
+
+// Revision token for the set of feeds the crawler currently considers broken.
+// The timeline sends the per-feed health payload only when the client's echoed
+// revision differs from this, so a steady-state poll costs no extra query.
+export const FEED_HEALTH_REV_KEY = 'feed_health_rev';
+
+// Bounds on one ingest call. The pusher chunks to ~100 items; these are abuse
+// guards, not tuning knobs.
+const MAX_INGEST_ITEMS = 1000;
+const MAX_INGEST_BODY_BYTES = 8 * 1024 * 1024;
+
+// D1 caps statements per batch; keep well under it.
+const INGEST_BATCH_SIZE = 50;
+
+/**
+ * Subscriptions that aren't RSS feeds (standard.site documents, collections)
+ * never belong to the crawl set or the timeline — their `feed_url` is an at://
+ * URI, and the client skips every `atproto.*` source on the feed path too.
+ */
+export function rssSubscriptionPredicate(alias = ''): string {
+ const column = alias ? `${alias}.source_type` : 'source_type';
+ return `(${column} IS NULL OR ${column} NOT LIKE 'atproto.%')`;
+}
+
+export interface IngestFeed {
+ feedUrl: string;
+ title?: string | null;
+ siteUrl?: string | null;
+ description?: string | null;
+ imageUrl?: string | null;
+}
+
+export interface IngestItem {
+ feedUrl: string;
+ guid: string;
+ item: FeedItem;
+ publishedAt?: number | null;
+ firstSeenAt: number;
+ contentHash: string;
+}
+
+/**
+ * Timing-safe secret comparison. Fails closed: an unset `FEED_PROXY_SECRET`
+ * rejects every request rather than turning the ingest endpoint into an open
+ * write surface (local dev sets the pair explicitly — see scripts/dev-local.sh).
+ */
+export function isAuthorizedProxyRequest(request: Request, env: Env): boolean {
+ const expected = env.FEED_PROXY_SECRET;
+ if (!expected) return false;
+ const provided = request.headers.get('X-Proxy-Secret');
+ if (!provided || provided.length !== expected.length) return false;
+ let diff = 0;
+ for (let i = 0; i < expected.length; i++) {
+ diff |= provided.charCodeAt(i) ^ expected.charCodeAt(i);
+ }
+ return diff === 0;
+}
+
+function unauthorized(): Response {
+ return new Response(JSON.stringify({ error: 'Unauthorized' }), {
+ status: 401,
+ headers: { 'Content-Type': 'application/json' },
+ });
+}
+
+function badRequest(message: string, status = 400): Response {
+ return new Response(JSON.stringify({ error: message }), {
+ status,
+ headers: { 'Content-Type': 'application/json' },
+ });
+}
+
+/**
+ * Apply the stored-content cap. Returns the item to persist: unchanged when it
+ * fits, otherwise the same item with `content` dropped and `contentTruncated`
+ * set. `content_hash` is computed by the proxy over the FULL item, so truncation
+ * never confuses edit detection.
+ */
+export function capItemContent(item: FeedItem): FeedItem {
+ const content = item.content;
+ if (!content) return item;
+ // Byte length, not code units — the cap is about stored bytes.
+ const bytes = new TextEncoder().encode(content).length;
+ if (bytes <= MAX_ITEM_CONTENT_BYTES) return item;
+ const { content: _dropped, ...rest } = item;
+ return { ...rest, contentTruncated: true };
+}
+
+/**
+ * Stable hash of an item's mutable content — byte-for-byte the same function the
+ * proxy applies before pushing (`itemContentHash` in feed-proxy/src/app.ts), so
+ * an item ingested by the subscribe-time pull-through and later pushed by the
+ * crawler hashes identically and doesn't register as a spurious edit.
+ */
+export async function computeContentHash(item: FeedItem): Promise {
+ const payload = `${item.title}|${item.url}|${item.content ?? ''}|${item.summary ?? ''}`;
+ const digest = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(payload));
+ return [...new Uint8Array(digest)]
+ .map((b) => b.toString(16).padStart(2, '0'))
+ .join('')
+ .slice(0, 16);
+}
+
+/**
+ * The write itself, shared by the pushed-batch endpoint and the subscribe-time
+ * pull-through in feeds-v2.ts. Idempotent: a re-pushed item with the same content
+ * hash is a no-op, a changed one updates in place (seq unchanged → not
+ * re-delivered to clients that already saw it). That makes the proxy's
+ * at-least-once delivery safe.
+ *
+ * Throws on a D1 write failure so callers can surface a 5xx (which leaves the
+ * pusher's outbox state untouched, so it retries).
+ */
+export async function ingestBatch(
+ env: Env,
+ feeds: IngestFeed[],
+ items: IngestItem[]
+): Promise<{ inserted: number; updated: number }> {
+ const now = Math.floor(Date.now() / 1000);
+ const statements: D1PreparedStatement[] = [];
+
+ for (const feed of feeds) {
+ if (!feed?.feedUrl) continue;
+ statements.push(
+ env.DB.prepare(
+ `INSERT INTO feeds (feed_url, title, site_url, description, image_url, last_ingest_at)
+ VALUES (?, ?, ?, ?, ?, ?)
+ ON CONFLICT(feed_url) DO UPDATE SET
+ title = COALESCE(excluded.title, feeds.title),
+ site_url = COALESCE(excluded.site_url, feeds.site_url),
+ description = COALESCE(excluded.description, feeds.description),
+ image_url = COALESCE(excluded.image_url, feeds.image_url),
+ last_ingest_at = excluded.last_ingest_at`
+ ).bind(
+ feed.feedUrl,
+ feed.title ?? null,
+ feed.siteUrl ?? null,
+ feed.description ?? null,
+ feed.imageUrl ?? null,
+ now
+ )
+ );
+ }
+
+ // Max seq before this batch: everything above it in the RETURNING rows below is
+ // a fresh insert, everything at or below is an edit-in-place. Purely for the
+ // response's counters — no correctness depends on it.
+ const before = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{
+ max_seq: number | null;
+ }>();
+ const maxSeqBefore = before?.max_seq ?? 0;
+
+ const touchedFeeds = new Set();
+ for (const entry of items) {
+ if (!entry?.feedUrl || !entry.guid || !entry.item || !entry.contentHash) continue;
+ touchedFeeds.add(entry.feedUrl);
+ statements.push(
+ env.DB.prepare(
+ `INSERT INTO feed_items (feed_url, guid, item_json, published_at, first_seen_at, content_hash)
+ VALUES (?, ?, ?, ?, ?, ?)
+ ON CONFLICT(feed_url, guid) DO UPDATE SET
+ item_json = excluded.item_json,
+ content_hash = excluded.content_hash
+ WHERE feed_items.content_hash <> excluded.content_hash
+ RETURNING seq`
+ ).bind(
+ entry.feedUrl,
+ entry.guid,
+ JSON.stringify(capItemContent(entry.item)),
+ entry.publishedAt ?? null,
+ entry.firstSeenAt || Date.now(),
+ entry.contentHash
+ )
+ );
+ }
+
+ let inserted = 0;
+ let updated = 0;
+ for (let i = 0; i < statements.length; i += INGEST_BATCH_SIZE) {
+ const results = await env.DB.batch<{ seq: number }>(statements.slice(i, i + INGEST_BATCH_SIZE));
+ for (const result of results) {
+ for (const row of result.results ?? []) {
+ if (row.seq > maxSeqBefore) inserted++;
+ else updated++;
+ }
+ }
+ }
+
+ await trimFeedsToSanityCap(env, [...touchedFeeds]);
+
+ return { inserted, updated };
+}
+
+/**
+ * The ONLY pruning that happens at ingest. Under the cap the OFFSET subquery
+ * yields no row, so `seq <= NULL` matches nothing and the DELETE is a no-op
+ * costing a bounded index scan — which is the expected case for every healthy
+ * feed. `cap` is a parameter so tests can exercise the trim without writing
+ * 5,000 rows.
+ */
+export async function trimFeedsToSanityCap(
+ env: Env,
+ feedUrls: string[],
+ cap = SANITY_CAP
+): Promise {
+ if (feedUrls.length === 0) return;
+ const trims = feedUrls.map((feedUrl) =>
+ env.DB.prepare(
+ `DELETE FROM feed_items
+ WHERE feed_url = ?1
+ AND seq <= (SELECT seq FROM feed_items WHERE feed_url = ?1
+ ORDER BY seq DESC LIMIT 1 OFFSET ?2)`
+ ).bind(feedUrl, cap)
+ );
+ for (let i = 0; i < trims.length; i += INGEST_BATCH_SIZE) {
+ await env.DB.batch(trims.slice(i, i + INGEST_BATCH_SIZE));
+ }
+}
+
+/**
+ * Ingest a feed fetched straight from the proxy (the subscribe-time warm/ingest
+ * and the pull-through in feeds-v2.ts). Hashes match what the crawler will push
+ * later, so the first real push over these rows is a no-op rather than a phantom
+ * edit.
+ *
+ * Order matters: a proxy feed is newest-first, and `seq` is assigned in insert
+ * order, so we walk the array BACKWARDS — oldest first — exactly as the proxy's
+ * own `writeFeedItems` does. Ingesting forward would give the newest item the
+ * lowest seq, and since a re-push of an unchanged item is a no-op, that
+ * inversion would never heal: every later per-feed cold start (`ORDER BY seq
+ * DESC`) would serve the feed's OLDEST items, and the sanity-cap trim (deletes
+ * the lowest seqs) would delete its newest.
+ */
+export async function ingestProxyFeed(
+ env: Env,
+ feedUrl: string,
+ feed: {
+ title?: string;
+ description?: string;
+ siteUrl?: string;
+ imageUrl?: string;
+ items: FeedItem[];
+ }
+): Promise {
+ const nowMs = Date.now();
+ const oldestFirst = [...feed.items].reverse();
+ const items: IngestItem[] = await Promise.all(
+ oldestFirst.map(async (item) => {
+ const publishedMs = new Date(item.publishedAt).getTime();
+ return {
+ feedUrl,
+ guid: item.guid,
+ item,
+ publishedAt: Number.isNaN(publishedMs) ? null : publishedMs,
+ firstSeenAt: nowMs,
+ contentHash: await computeContentHash(item),
+ };
+ })
+ );
+
+ await ingestBatch(
+ env,
+ [
+ {
+ feedUrl,
+ title: feed.title ?? null,
+ siteUrl: feed.siteUrl ?? null,
+ description: feed.description ?? null,
+ imageUrl: feed.imageUrl ?? null,
+ },
+ ],
+ items
+ );
+}
+
+/**
+ * Record that this environment's crawler just talked to us. Both internal
+ * endpoints stamp it, so the signal survives a quiet period with no new items
+ * (the crawl-set pull runs every 5 minutes regardless of what the feeds do).
+ *
+ * This is what makes "is ingest live here?" server-authoritative instead of
+ * inferred from an empty archive: a Worker whose proxy has no INGEST_URL never
+ * gets a stamp, so `/api/v2/timeline` reports `ingestActive: false` and clients
+ * keep using the legacy batch path rather than committing a cursor against an
+ * archive nothing is filling.
+ */
+export async function stampCrawlerHeartbeat(env: Env): Promise {
+ try {
+ await env.DB.prepare(
+ `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch())
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`
+ )
+ .bind(CRAWLER_HEARTBEAT_KEY, String(Math.floor(Date.now() / 1000)))
+ .run();
+ } catch (error) {
+ // Observability only — never fail an ingest because the stamp didn't land.
+ console.error('[ingest] Failed to stamp crawler heartbeat:', error);
+ }
+}
+
+/**
+ * Per-feed crawl health, as the crawler reports it. Unix SECONDS on this wire
+ * (the proxy keeps milliseconds internally and converts on send); the timeline
+ * converts back to milliseconds for the client.
+ */
+export interface FeedHealthReport {
+ feedUrl: string;
+ errorCount: number;
+ lastError?: string | null;
+ lastErrorAt?: number | null;
+ nextRetryAt?: number | null;
+ lastFetchAt?: number | null;
+ // In the crawl set but going unfetched — starved, not failing. Independent of
+ // `errorCount`, which can be 0 while this is true.
+ crawlStale?: boolean;
+}
+
+// A report carries only the unhealthy feeds, so this is far above any plausible
+// real number — an abuse guard, not a tuning knob.
+const MAX_HEALTH_FEEDS = 2000;
+// Error strings are rendered in a popover and matched against substrings; the
+// crawler's messages are short, so anything longer is a runaway.
+const MAX_HEALTH_ERROR_CHARS = 500;
+// Feed URLs per recovery statement. Well under D1's per-statement bind limit,
+// and the recovered set is normally a handful anyway.
+const HEALTH_CLEAR_CHUNK = 50;
+
+/**
+ * A cheap fingerprint of the whole unhealthy set, used as the `feed_health_rev`
+ * token. Clients echo the revision they last saw and the timeline re-sends the
+ * health payload only when it differs, which keeps a steady-state poll at the
+ * one query the architecture promises.
+ *
+ * Aggregates rather than a real hash: the partial index makes this a scan of the
+ * handful of broken feeds, not of the archive. Two genuinely different sets can
+ * alias only if they share a count, an error-count sum, and both timestamps — and
+ * the cost of that is one poll showing yesterday's error, self-healing on the
+ * next change.
+ */
+async function readFeedHealthRev(env: Env): Promise {
+ const row = await env.DB.prepare(
+ `SELECT COUNT(*) AS n, COALESCE(SUM(error_count), 0) AS errors,
+ COALESCE(MAX(last_error_at), 0) AS newest, COALESCE(MAX(next_retry_at), 0) AS retry
+ FROM feeds WHERE error_count > 0`
+ ).first<{ n: number; errors: number; newest: number; retry: number }>();
+ return `${row?.n ?? 0}:${row?.errors ?? 0}:${row?.newest ?? 0}:${row?.retry ?? 0}`;
+}
+
+/**
+ * Clear one feed's error state because we just fetched it successfully.
+ *
+ * The pull-through in `/api/v2/feeds/fetch` is the user's explicit "retry this
+ * feed" action, and it is proof the feed works — but the crawler's next health
+ * report is up to five minutes away, so without this the reader would keep
+ * showing the error the user just cleared. Refreshes the revision so every other
+ * client picks the recovery up too.
+ */
+export async function clearFeedHealth(env: Env, feedUrl: string): Promise {
+ try {
+ const result = await env.DB.prepare(
+ `UPDATE feeds
+ SET error_count = 0, last_error = NULL, last_error_at = NULL, next_retry_at = NULL,
+ crawl_stale = 0, last_fetch_at = ?
+ WHERE feed_url = ? AND (error_count > 0 OR crawl_stale = 1)`
+ )
+ .bind(Math.floor(Date.now() / 1000), feedUrl)
+ .run();
+ if (!result.meta?.changes) return;
+
+ await env.DB.prepare(
+ `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch())
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`
+ )
+ .bind(FEED_HEALTH_REV_KEY, await readFeedHealthRev(env))
+ .run();
+ } catch (error) {
+ // Health is observability, not the read itself — never fail a fetch over it.
+ console.error('[ingest] Failed to clear feed health:', error);
+ }
+}
+
+/**
+ * POST /api/internal/feed-health
+ *
+ * The crawler's periodic report of every feed it currently considers broken.
+ *
+ * This is the timeline path's replacement for what the legacy batch response
+ * carried inline: on `/api/v2/feeds/batch` a failing feed came back with
+ * `status: 'error'` plus its error, count and retry time, which is what fed the
+ * reader's per-feed error badge and popover. Reads no longer touch the proxy, and
+ * a feed that fails to crawl pushes no items, so without this a broken feed is
+ * indistinguishable from a quiet one and the user gets no signal at all.
+ *
+ * The payload is the COMPLETE unhealthy set, not a delta: recovery is inferred
+ * from absence (see the sweep below), so a feed that starts working again clears
+ * without the crawler having to say anything about it.
+ */
+export async function handleFeedHealth(request: Request, env: Env): Promise {
+ if (request.method !== 'POST') return badRequest('Method not allowed', 405);
+ if (!isAuthorizedProxyRequest(request, env)) return unauthorized();
+
+ const declaredLength = Number(request.headers.get('Content-Length') ?? '0');
+ if (declaredLength > MAX_INGEST_BODY_BYTES) return badRequest('Payload too large', 413);
+
+ let body: { feeds?: FeedHealthReport[] };
+ try {
+ body = await request.json();
+ } catch {
+ return badRequest('Invalid JSON body');
+ }
+
+ // A report entry means "something is wrong with this feed" — it is erroring,
+ // or the crawler isn't reaching it, or both. An entry claiming neither is
+ // noise and would keep the feed flagged forever.
+ const reports = (Array.isArray(body.feeds) ? body.feeds : []).filter(
+ (f) => f?.feedUrl && ((Number.isFinite(f.errorCount) && f.errorCount > 0) || f.crawlStale)
+ );
+ if (reports.length > MAX_HEALTH_FEEDS) return badRequest('Too many feeds');
+
+ try {
+ const before = await readFeedHealthRev(env);
+
+ // Which feeds are flagged right now. Read BEFORE the upserts so the
+ // difference against this report is exactly "was in trouble, isn't any more".
+ // Small by construction (the partial index covers only flagged feeds), so the
+ // chunked IN list below can never approach the bound-parameter limit.
+ const flagged = await env.DB.prepare(
+ `SELECT feed_url FROM feeds WHERE error_count > 0 OR crawl_stale = 1`
+ ).all<{ feed_url: string }>();
+
+ const stillTroubled = new Set(reports.map((r) => r.feedUrl));
+ const recovered = flagged.results
+ .map((r) => r.feed_url)
+ .filter((url) => !stillTroubled.has(url));
+
+ // A feed can be broken from its very first crawl, in which case it has never
+ // been ingested and has no `feeds` row at all — so this inserts rather than
+ // assuming one exists. Such a row carries health only (NULL title/site_url),
+ // which the metadata backfill already skips.
+ const upserts = reports.map((report) =>
+ env.DB.prepare(
+ `INSERT INTO feeds (feed_url, error_count, last_error, last_error_at, next_retry_at,
+ last_fetch_at, crawl_stale)
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
+ ON CONFLICT(feed_url) DO UPDATE SET
+ error_count = excluded.error_count,
+ last_error = excluded.last_error,
+ last_error_at = excluded.last_error_at,
+ next_retry_at = excluded.next_retry_at,
+ last_fetch_at = COALESCE(excluded.last_fetch_at, feeds.last_fetch_at),
+ crawl_stale = excluded.crawl_stale`
+ ).bind(
+ report.feedUrl,
+ Number.isFinite(report.errorCount) ? Math.max(report.errorCount, 0) : 0,
+ report.lastError ? report.lastError.slice(0, MAX_HEALTH_ERROR_CHARS) : null,
+ report.lastErrorAt ?? null,
+ report.nextRetryAt ?? null,
+ report.lastFetchAt ?? null,
+ report.crawlStale ? 1 : 0
+ )
+ );
+ for (let i = 0; i < upserts.length; i += INGEST_BATCH_SIZE) {
+ await env.DB.batch(upserts.slice(i, i + INGEST_BATCH_SIZE));
+ }
+
+ // Recovery: a feed the crawler stopped listing is fine again. Absence is the
+ // entire signal, which is why the report has to be the complete set.
+ // Chunked well under D1's per-statement bind limit — the same limit a
+ // subscription-sized IN list walked into once already.
+ for (let i = 0; i < recovered.length; i += HEALTH_CLEAR_CHUNK) {
+ const chunk = recovered.slice(i, i + HEALTH_CLEAR_CHUNK);
+ await env.DB.prepare(
+ `UPDATE feeds
+ SET error_count = 0, last_error = NULL, last_error_at = NULL, next_retry_at = NULL,
+ crawl_stale = 0
+ WHERE feed_url IN (${chunk.map(() => '?').join(',')})`
+ )
+ .bind(...chunk)
+ .run();
+ }
+
+ const after = await readFeedHealthRev(env);
+ if (after !== before) {
+ await env.DB.prepare(
+ `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch())
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`
+ )
+ .bind(FEED_HEALTH_REV_KEY, after)
+ .run();
+ }
+
+ await stampCrawlerHeartbeat(env);
+ return new Response(JSON.stringify({ ok: true, unhealthy: reports.length, rev: after }), {
+ headers: { 'Content-Type': 'application/json' },
+ });
+ } catch (error) {
+ console.error('[ingest] Feed-health write error:', error);
+ return new Response(JSON.stringify({ error: 'Feed health update failed' }), {
+ status: 500,
+ headers: { 'Content-Type': 'application/json' },
+ });
+ }
+}
+
+/**
+ * POST /api/internal/ingest
+ *
+ * Upsert a batch of feed metadata + items pushed by the crawler. Any 5xx leaves
+ * the proxy's outbox state untouched — the idempotent upsert above makes
+ * at-least-once delivery safe.
+ */
+export async function handleIngest(request: Request, env: Env): Promise {
+ if (request.method !== 'POST') return badRequest('Method not allowed', 405);
+ if (!isAuthorizedProxyRequest(request, env)) return unauthorized();
+
+ const declaredLength = Number(request.headers.get('Content-Length') ?? '0');
+ if (declaredLength > MAX_INGEST_BODY_BYTES) return badRequest('Payload too large', 413);
+
+ let body: { feeds?: IngestFeed[]; items?: IngestItem[] };
+ try {
+ body = await request.json();
+ } catch {
+ return badRequest('Invalid JSON body');
+ }
+
+ const feeds = Array.isArray(body.feeds) ? body.feeds : [];
+ const items = Array.isArray(body.items) ? body.items : [];
+ if (items.length > MAX_INGEST_ITEMS) return badRequest('Too many items');
+
+ try {
+ const { inserted, updated } = await ingestBatch(env, feeds, items);
+ await stampCrawlerHeartbeat(env);
+ return new Response(JSON.stringify({ ok: true, inserted, updated }), {
+ headers: { 'Content-Type': 'application/json' },
+ });
+ } catch (error) {
+ console.error('[ingest] D1 WRITE ERROR:', error);
+ return new Response(JSON.stringify({ error: 'Ingest failed' }), {
+ status: 500,
+ headers: { 'Content-Type': 'application/json' },
+ });
+ }
+}
+
+/**
+ * GET /api/internal/crawl-set
+ *
+ * The feeds this environment wants crawled. Replaces the proxy's request-driven
+ * warmth: once clients stop reading through Fly, nothing stamps
+ * `last_requested_at`, so every feed would silently age out of the warm loop.
+ * The proxy polls this and stamps the rows itself.
+ */
+export async function handleCrawlSet(request: Request, env: Env): Promise {
+ if (request.method !== 'GET') return badRequest('Method not allowed', 405);
+ if (!isAuthorizedProxyRequest(request, env)) return unauthorized();
+
+ // The crawl-set pull is the crawler's liveness signal (it runs every 5 minutes
+ // whether or not any feed produced an item).
+ await stampCrawlerHeartbeat(env);
+
+ const rows = await env.DB.prepare(
+ `SELECT feed_url, COUNT(*) AS subscribers
+ FROM subscriptions_cache
+ WHERE active = 1
+ AND feed_url IS NOT NULL AND feed_url <> ''
+ AND ${rssSubscriptionPredicate()}
+ GROUP BY feed_url`
+ ).all<{ feed_url: string; subscribers: number }>();
+
+ const feeds = rows.results.map((row) => ({
+ feedUrl: row.feed_url,
+ subscribers: row.subscribers,
+ }));
+
+ return new Response(JSON.stringify({ feeds, count: feeds.length }), {
+ headers: { 'Content-Type': 'application/json' },
+ });
+}
diff --git a/backend/src/routes/subscriptions.ts b/backend/src/routes/subscriptions.ts
index 709d04a6..7d97aa07 100644
--- a/backend/src/routes/subscriptions.ts
+++ b/backend/src/routes/subscriptions.ts
@@ -1,6 +1,6 @@
import type { Env, Session } from '../types';
import { getSessionFromRequest } from '../services/oauth';
-import { warmProxyCache, warmProxyCacheBatch } from './feeds-v2';
+import { warmFeedIntoArchive, warmFeedsIntoArchive } from './feeds-v2';
import { getUserSettings } from './settings';
import { pushSubscriptionToPds, deleteSubscriptionFromPds } from '../services/subscription-sync';
import {
@@ -806,13 +806,14 @@ export async function handleCreateSubscription(
)
.run();
- // Warm up proxy cache for RSS subscriptions only
+ // Crawl the feed once and ingest it into the archive, so the client's first
+ // read of this brand-new subscription is a plain D1 query (RSS only).
if (!isAtProto && feedUrl) {
- const cacheResult = await warmProxyCache(env, feedUrl);
+ const cacheResult = await warmFeedIntoArchive(env, feedUrl);
if (cacheResult.success) {
- console.log(`Warmed proxy cache: ${feedUrl} (${cacheResult.itemCount} items)`);
+ console.log(`Ingested new feed: ${feedUrl} (${cacheResult.itemCount} items)`);
} else {
- console.error(`Failed to warm cache for ${feedUrl}: ${cacheResult.error}`);
+ console.error(`Failed to ingest ${feedUrl}: ${cacheResult.error}`);
}
}
@@ -1242,16 +1243,17 @@ export async function handleBulkCreateSubscriptions(
await env.DB.batch(batchStatements);
}
- // Warm up proxy cache for new subscriptions (batch is efficient)
+ // Crawl + ingest the first few new subscriptions (batch is efficient) so the
+ // client's backfill reads them straight from the archive.
const MAX_FEEDS_TO_WARM = 10;
const feedsToWarmNow = feedsToFetch.slice(0, MAX_FEEDS_TO_WARM);
if (feedsToWarmNow.length > 0) {
- const cacheResults = await warmProxyCacheBatch(env, feedsToWarmNow);
+ const cacheResults = await warmFeedsIntoArchive(env, feedsToWarmNow);
for (const [feedUrl, result] of Object.entries(cacheResults)) {
if (result.success) {
- console.log(`Warmed proxy cache: ${feedUrl} (${result.itemCount} items)`);
+ console.log(`Ingested new feed: ${feedUrl} (${result.itemCount} items)`);
} else {
- console.error(`Failed to warm cache for ${feedUrl}: ${result.error}`);
+ console.error(`Failed to ingest ${feedUrl}: ${result.error}`);
}
}
}
diff --git a/backend/src/routes/timeline.ts b/backend/src/routes/timeline.ts
new file mode 100644
index 00000000..cbeefa04
--- /dev/null
+++ b/backend/src/routes/timeline.ts
@@ -0,0 +1,506 @@
+import type { Env, FeedItem, Session } from '../types';
+import {
+ CRAWLER_HEARTBEAT_KEY,
+ CRAWLER_HEARTBEAT_FRESH_SECONDS,
+ FEED_HEALTH_REV_KEY,
+ TIMELINE_ENABLED_KEY,
+ rssSubscriptionPredicate,
+} from './ingest';
+
+/**
+ * GET /api/v2/timeline — the whole feed refresh, in one request.
+ *
+ * Replaces the `1 + ceil((N-8)/50)` batched `POST /api/v2/feeds/batch` calls
+ * (each a Worker → Fly hop plus `ceil(GUIDs/88)` chunked read-key queries). The
+ * crawler pushes items into D1; this serves them with subscriptions AND read
+ * state resolved in the same query, so `getReadKeys` never runs on the feed path.
+ *
+ * Named `/timeline` rather than `/sync` to avoid colliding with the existing
+ * PDS-subscription `/api/sync/*` routes.
+ */
+
+// Max items in one page. Also the cap on how much a single response can weigh —
+// items carry (capped) content bodies.
+const MAX_LIMIT = 200;
+const DEFAULT_LIMIT = 200;
+
+// Cold start delivers a per-feed newest slice, not a global one: a global
+// `ORDER BY seq DESC LIMIT n` would let one chatty feed starve every other.
+const COLD_START_PER_FEED = 30;
+// Statements per D1 batch on the cold-start path.
+const COLD_START_CHUNK = 25;
+// A cold start touches every subscribed feed; bound the work (and log if hit —
+// no silent truncation).
+const COLD_START_MAX_FEEDS = 500;
+// Rows one cold-start PAGE may accumulate. A cold start walks feeds in a stable
+// order and stops once it passes this budget, handing back `nextColdOffset` for
+// the next page — so a 150-feed reader (or everyone at once after a generation
+// bump) can't ask the Worker to buffer thousands of content-bearing items in a
+// single response. Checked per chunk, so a page can overshoot by at most
+// COLD_START_CHUNK × COLD_START_PER_FEED.
+const COLD_START_MAX_ITEMS = 750;
+
+export interface TimelineItem extends FeedItem {
+ seq: number;
+ feedUrl: string;
+ read: boolean;
+}
+
+interface ItemRow {
+ seq: number;
+ feed_url: string;
+ item_json: string;
+ read: number;
+}
+
+// The per-user read probe. An EXISTS (rather than a LEFT JOIN) keeps the result
+// one row per item even if a user somehow holds duplicate label rows, while
+// probing the same (user_did, item_key) index a join would.
+const READ_FLAG_SQL = `EXISTS (
+ SELECT 1 FROM item_labels_cache il
+ WHERE il.user_did = ?1 AND il.item_key = fi.guid
+ AND il.item_type = 'article' AND il.label = 'read' AND il.deleted_at IS NULL
+ ) AS read`;
+
+function toTimelineItems(rows: ItemRow[]): TimelineItem[] {
+ const items: TimelineItem[] = [];
+ for (const row of rows) {
+ try {
+ const item = JSON.parse(row.item_json) as FeedItem;
+ items.push({ ...item, seq: row.seq, feedUrl: row.feed_url, read: row.read === 1 });
+ } catch {
+ // A corrupt row must not poison the whole page; skip it. The cursor still
+ // advances past it, so it can't wedge the client's drain loop.
+ console.error(`[timeline] Unparseable item_json at seq ${row.seq}`);
+ }
+ }
+ return items;
+}
+
+/**
+ * Newest slice of one feed, read-annotated — the single-feed read path
+ * (`GET /api/v2/feeds/fetch`), which now serves D1 like everything else.
+ */
+export async function readFeedSlice(
+ env: Env,
+ userDid: string,
+ feedUrl: string,
+ limit: number
+): Promise {
+ const rows = await env.DB.prepare(
+ `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL}
+ FROM feed_items fi
+ WHERE fi.feed_url = ?2
+ ORDER BY fi.published_at DESC, fi.seq DESC
+ LIMIT ?3`
+ )
+ .bind(userDid, feedUrl, limit)
+ .all();
+ return toTimelineItems(rows.results);
+}
+
+export interface FeedMetadataRow {
+ title: string | null;
+ site_url: string | null;
+ description: string | null;
+ image_url: string | null;
+ last_ingest_at: number | null;
+ // Crawl health (unix seconds), so a single-feed read can report a broken feed
+ // even while it serves the archived items the feed still has.
+ error_count: number;
+ last_error: string | null;
+ last_error_at: number | null;
+ next_retry_at: number | null;
+ last_fetch_at: number | null;
+}
+
+export async function readFeedMetadata(env: Env, feedUrl: string): Promise {
+ return env.DB.prepare(
+ `SELECT title, site_url, description, image_url, last_ingest_at,
+ error_count, last_error, last_error_at, next_retry_at, last_fetch_at
+ FROM feeds WHERE feed_url = ?`
+ )
+ .bind(feedUrl)
+ .first();
+}
+
+export interface ArchiveState {
+ generation: string;
+ // True when this environment's crawler has checked in recently. False means
+ // nothing is filling this D1 (no INGEST_URL on the paired proxy, or the proxy
+ // is down), so the client must not treat an empty/partial archive as the truth.
+ crawlerFresh: boolean;
+ // The operator's rollout gate (`timeline_enabled`, migration 0071). A fresh
+ // heartbeat only proves a crawler is attached — it arrives seconds into a
+ // backfill that takes hours — so this is the switch that actually moves readers
+ // onto the timeline, and the switch that moves them back.
+ timelineEnabled: boolean;
+ // Revision of the unhealthy-feed set. A client that echoes this back unchanged
+ // already holds current health and is sent no health payload.
+ healthRev: string;
+}
+
+/**
+ * Generation token, crawler liveness, the rollout gate and the feed-health
+ * revision in one `sync_state` read (the timeline needs all four on every
+ * request, and they are four rows of the same small table).
+ */
+export async function readArchiveState(env: Env): Promise {
+ const rows = await env.DB.prepare(
+ `SELECT key, value FROM sync_state WHERE key IN ('items_generation', ?, ?, ?)`
+ )
+ .bind(CRAWLER_HEARTBEAT_KEY, FEED_HEALTH_REV_KEY, TIMELINE_ENABLED_KEY)
+ .all<{ key: string; value: string }>();
+
+ let generation = '';
+ let heartbeat = 0;
+ let healthRev = '';
+ // Absent means enabled: only an explicit '0' holds clients on the batch path,
+ // so an environment that never learned about this key behaves as it did before.
+ let timelineEnabled = true;
+ for (const row of rows.results) {
+ if (row.key === 'items_generation') generation = row.value;
+ else if (row.key === CRAWLER_HEARTBEAT_KEY) heartbeat = parseInt(row.value, 10) || 0;
+ else if (row.key === FEED_HEALTH_REV_KEY) healthRev = row.value;
+ else if (row.key === TIMELINE_ENABLED_KEY) timelineEnabled = row.value !== '0';
+ }
+
+ const age = Math.floor(Date.now() / 1000) - heartbeat;
+ return {
+ generation,
+ crawlerFresh: heartbeat > 0 && age <= CRAWLER_HEARTBEAT_FRESH_SECONDS,
+ timelineEnabled,
+ healthRev,
+ };
+}
+
+/**
+ * Per-feed crawl health for the caller's subscriptions — the timeline path's
+ * replacement for the per-feed `status: 'error'` the legacy batch response
+ * carried inline.
+ *
+ * Only broken feeds are returned: the client treats absence as healthy, which is
+ * what makes recovery work without sending a row per subscription. Driven from
+ * the (small, partially indexed) unhealthy set rather than from the caller's
+ * subscriptions, because on a healthy system that set is empty and this costs
+ * nothing.
+ *
+ * Timestamps go out in MILLISECONDS — D1 stores seconds like the rest of the
+ * backend, but the client's FeedStatus contract has always been ms.
+ */
+export async function subscribedFeedHealth(
+ env: Env,
+ userDid: string
+): Promise> {
+ const rows = await env.DB.prepare(
+ `SELECT f.feed_url, f.error_count, f.last_error, f.last_error_at, f.next_retry_at, f.last_fetch_at
+ FROM feeds f
+ WHERE f.error_count > 0
+ AND EXISTS (
+ SELECT 1 FROM subscriptions_cache sc
+ WHERE sc.user_did = ? AND sc.feed_url = f.feed_url AND sc.active = 1
+ AND ${rssSubscriptionPredicate('sc')}
+ )`
+ )
+ .bind(userDid)
+ .all<{
+ feed_url: string;
+ error_count: number;
+ last_error: string | null;
+ last_error_at: number | null;
+ next_retry_at: number | null;
+ last_fetch_at: number | null;
+ }>();
+
+ const health: Record = {};
+ for (const row of rows.results) {
+ health[row.feed_url] = {
+ errorCount: row.error_count,
+ error: row.last_error ?? undefined,
+ lastErrorAt: row.last_error_at ? row.last_error_at * 1000 : undefined,
+ nextRetryAt: row.next_retry_at ? row.next_retry_at * 1000 : undefined,
+ lastFetchedAt: row.last_fetch_at ? row.last_fetch_at * 1000 : undefined,
+ };
+ }
+ return health;
+}
+
+/** One broken feed as the timeline reports it. Timestamps are unix ms. */
+export interface FeedHealth {
+ errorCount: number;
+ error?: string;
+ lastErrorAt?: number;
+ nextRetryAt?: number;
+ lastFetchedAt?: number;
+}
+
+/** The archive's current head; 0 when nothing has ever been ingested. */
+async function archiveHead(env: Env): Promise {
+ const row = await env.DB.prepare('SELECT MAX(seq) AS max_seq FROM feed_items').first<{
+ max_seq: number | null;
+ }>();
+ return row?.max_seq ?? 0;
+}
+
+/**
+ * Subscribed RSS feed URLs in a STABLE order — the cold start pages through this
+ * list by index, so the ordering has to be the same from one page to the next.
+ */
+async function subscribedFeedUrls(env: Env, userDid: string): Promise {
+ const rows = await env.DB.prepare(
+ `SELECT DISTINCT feed_url FROM subscriptions_cache
+ WHERE user_did = ? AND active = 1
+ AND feed_url IS NOT NULL AND feed_url <> ''
+ AND ${rssSubscriptionPredicate()}
+ ORDER BY feed_url`
+ )
+ .bind(userDid)
+ .all<{ feed_url: string }>();
+ return rows.results.map((r) => r.feed_url);
+}
+
+/**
+ * Feed-level metadata for the feeds the caller subscribes to. Small (tens of
+ * rows) and only sent alongside a non-empty page, so a steady-state poll that
+ * returns nothing costs exactly one query.
+ */
+async function subscribedFeedMetadata(
+ env: Env,
+ userDid: string
+): Promise> {
+ const rows = await env.DB.prepare(
+ `SELECT f.feed_url, f.title, f.site_url, f.image_url
+ FROM feeds f
+ JOIN (SELECT DISTINCT feed_url FROM subscriptions_cache
+ WHERE user_did = ? AND active = 1
+ AND ${rssSubscriptionPredicate()}) sc
+ ON sc.feed_url = f.feed_url`
+ )
+ .bind(userDid)
+ .all<{
+ feed_url: string;
+ title: string | null;
+ site_url: string | null;
+ image_url: string | null;
+ }>();
+
+ const feeds: Record = {};
+ for (const row of rows.results) {
+ feeds[row.feed_url] = {
+ title: row.title ?? undefined,
+ siteUrl: row.site_url ?? undefined,
+ imageUrl: row.image_url ?? undefined,
+ };
+ }
+ return feeds;
+}
+
+export async function handleTimeline(
+ request: Request,
+ env: Env,
+ session: Session
+): Promise {
+ if (request.method !== 'GET') {
+ return new Response(JSON.stringify({ error: 'Method not allowed' }), {
+ status: 405,
+ headers: { 'Content-Type': 'application/json' },
+ });
+ }
+
+ const url = new URL(request.url);
+ const sinceSeqParam = url.searchParams.get('since_seq');
+ const generationParam = url.searchParams.get('generation');
+ const limitParam = url.searchParams.get('limit');
+ const coldOffsetParam = url.searchParams.get('cold_offset');
+ // The feed-health revision this client already holds. Absent (a fresh page
+ // load, whose status store is empty) means "send it".
+ const healthRevParam = url.searchParams.get('health_rev');
+
+ const parsedLimit = limitParam ? parseInt(limitParam, 10) : DEFAULT_LIMIT;
+ const limit = Number.isInteger(parsedLimit)
+ ? Math.min(Math.max(parsedLimit, 1), MAX_LIMIT)
+ : DEFAULT_LIMIT;
+
+ const parsedSince = sinceSeqParam !== null ? parseInt(sinceSeqParam, 10) : NaN;
+ const sinceSeq = Number.isInteger(parsedSince) && parsedSince >= 0 ? parsedSince : undefined;
+
+ // Continuation index into the caller's (stably ordered) subscribed-feed list.
+ const parsedColdOffset = coldOffsetParam !== null ? parseInt(coldOffsetParam, 10) : NaN;
+ const coldOffset =
+ Number.isInteger(parsedColdOffset) && parsedColdOffset > 0 ? parsedColdOffset : 0;
+
+ const { generation, crawlerFresh, timelineEnabled, healthRev } = await readArchiveState(env);
+ // Two independent conditions, one wire field. The client's contract is
+ // unchanged — `ingestActive: false` has always meant "stay on the batch path" —
+ // and it does not need to know WHY, only the operator does.
+ const ingestActive = timelineEnabled && crawlerFresh;
+
+ // Server time (unix seconds) at annotation. The client seeds its forward
+ // read-delta cursor from this, exactly as /batch does today, so the delta
+ // starts from bootstrap with no client/server clock skew.
+ const readCursor = Math.floor(Date.now() / 1000);
+
+ // A client told `ingestActive: false` discards the page and refetches through
+ // /batch, so building one is pure waste — and during a gated rollout that waste
+ // is every reader, every poll, for as long as the gate stays shut. Answer with
+ // the state and nothing else. `coldStart: true` keeps the older
+ // empty-cold-start heuristic pointing the same way as the flag, so a client
+ // reading either signal reaches the same conclusion.
+ if (!ingestActive) {
+ return json({
+ items: [],
+ cursor: 0,
+ generation,
+ ingestActive,
+ hasMore: false,
+ readCursor,
+ coldStart: true,
+ healthRev,
+ });
+ }
+
+ // Send health when the client's copy is stale. A cold start always gets it:
+ // it is the one page that delivers already-archived items for a feed that may
+ // have broken since, and its blanket "these feeds delivered, so they're fine"
+ // pass would otherwise clear a live error.
+ const healthStale = healthRevParam !== healthRev;
+
+ const incremental =
+ sinceSeq !== undefined &&
+ coldOffset === 0 &&
+ generationParam === generation &&
+ generation !== '';
+
+ try {
+ if (incremental) {
+ // Drain oldest-unseen first so a backlog larger than one page is paged
+ // across polls, never skipped. limit+1 probes hasMore without a second query.
+ //
+ // Scaling note: this walks the `feed_items` rowid range above the cursor and
+ // probes the subscription set per row, so the cost tracks GLOBAL ingest above
+ // the cursor rather than the caller's own new items. That is the accepted
+ // fan-out-on-read trade at ~1,300 feeds; if D1 row-reads ever become the
+ // constraint, the fix is to bound the scan with per-feed `(feed_url, seq)`
+ // seeks (idx_feed_items_feed_seq already supports them), not to materialize
+ // per-user timelines.
+ const rows = await env.DB.prepare(
+ `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL}
+ FROM feed_items fi
+ WHERE fi.seq > ?2
+ AND EXISTS (
+ SELECT 1 FROM subscriptions_cache sc
+ WHERE sc.user_did = ?1 AND sc.feed_url = fi.feed_url AND sc.active = 1
+ AND ${rssSubscriptionPredicate('sc')}
+ )
+ ORDER BY fi.seq ASC
+ LIMIT ?3`
+ )
+ .bind(session.did, sinceSeq, limit + 1)
+ .all();
+
+ const hasMore = rows.results.length > limit;
+ const page = hasMore ? rows.results.slice(0, limit) : rows.results;
+
+ // Rewound-archive guard, the D1 twin of the proxy's snapshot-restore check.
+ // A Time Travel restore (or a rebuild from export) rewinds `feed_items` seqs
+ // while `items_generation` comes back unchanged, leaving every client cursor
+ // above the head: `seq > ?` then returns nothing on every poll, forever. A
+ // cursor can never legitimately exceed the head, so treat that as a cold
+ // start instead of silently starving the client. Only costs a MAX(seq) on
+ // an otherwise empty page — never on a page that carried items.
+ if (page.length === 0 && sinceSeq > (await archiveHead(env))) {
+ console.warn(
+ `[timeline] Cursor ${sinceSeq} is above the archive head; cold-starting (archive rewound?).`
+ );
+ } else {
+ // Cursor comes from the returned rows, never a separate MAX(seq): the
+ // latter races ingest and would skip everything written in between.
+ const cursor = page.length > 0 ? page[page.length - 1].seq : sinceSeq;
+ const items = toTimelineItems(page);
+
+ return json({
+ items,
+ cursor,
+ generation,
+ ingestActive,
+ hasMore,
+ readCursor,
+ coldStart: false,
+ feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined,
+ healthRev,
+ feedHealth: healthStale ? await subscribedFeedHealth(env, session.did) : undefined,
+ });
+ }
+ }
+
+ // Cold start: no cursor, a generation mismatch (D1 recreated / restored), a
+ // rewound archive, or a continuation page of one already in progress.
+ const allFeedUrls = await subscribedFeedUrls(env, session.did);
+ const feedUrls = allFeedUrls.slice(0, COLD_START_MAX_FEEDS);
+ if (allFeedUrls.length > feedUrls.length) {
+ console.warn(
+ `[timeline] Cold start covering ${feedUrls.length} of ${allFeedUrls.length} feeds for ${session.did}; the rest fill in as new items arrive.`
+ );
+ }
+
+ // The cold-start cursor is the archive head read BEFORE the per-feed slices,
+ // and the client keeps the first page's value across a paged cold start.
+ // Reading it first is what makes it safe: an item ingested while we page gets
+ // a seq above this head, so it arrives on the first incremental poll instead
+ // of being skipped. (Re-delivering a handful of items is harmless — the merge
+ // dedupes by GUID.) The incremental path still derives its cursor from the
+ // rows it returned.
+ const cursor = await archiveHead(env);
+
+ // Walk feeds from the continuation point until the page budget is spent. The
+ // client re-requests with `cold_offset=nextColdOffset` until hasMore is false.
+ // A subscription added or removed mid-cold-start shifts the offsets by one, so
+ // a feed can be missed by this bootstrap; the client's per-feed backfill picks
+ // up any subscription that ends up with no articles.
+ const rows: ItemRow[] = [];
+ let nextIndex = Math.min(coldOffset, feedUrls.length);
+ while (nextIndex < feedUrls.length && rows.length < COLD_START_MAX_ITEMS) {
+ const chunk = feedUrls.slice(nextIndex, nextIndex + COLD_START_CHUNK);
+ const statements = chunk.map((feedUrl) =>
+ env.DB.prepare(
+ `SELECT fi.seq, fi.feed_url, fi.item_json, ${READ_FLAG_SQL}
+ FROM feed_items fi
+ WHERE fi.feed_url = ?2
+ ORDER BY fi.published_at DESC, fi.seq DESC
+ LIMIT ?3`
+ ).bind(session.did, feedUrl, COLD_START_PER_FEED)
+ );
+ const results = await env.DB.batch(statements);
+ for (const result of results) rows.push(...(result.results ?? []));
+ nextIndex += chunk.length;
+ }
+ const hasMore = nextIndex < feedUrls.length;
+
+ const items = toTimelineItems(rows);
+ return json({
+ items,
+ cursor,
+ generation,
+ ingestActive,
+ hasMore,
+ nextColdOffset: hasMore ? nextIndex : undefined,
+ readCursor,
+ coldStart: true,
+ feeds: items.length > 0 ? await subscribedFeedMetadata(env, session.did) : undefined,
+ healthRev,
+ feedHealth: await subscribedFeedHealth(env, session.did),
+ });
+ } catch (error) {
+ console.error('[timeline] Query error:', error);
+ return new Response(JSON.stringify({ error: 'Failed to load timeline' }), {
+ status: 500,
+ headers: { 'Content-Type': 'application/json' },
+ });
+ }
+}
+
+function json(body: unknown): Response {
+ return new Response(JSON.stringify(body), {
+ headers: { 'Content-Type': 'application/json' },
+ });
+}
diff --git a/backend/src/services/rate-limit.ts b/backend/src/services/rate-limit.ts
index e8119789..38a3241b 100644
--- a/backend/src/services/rate-limit.ts
+++ b/backend/src/services/rate-limit.ts
@@ -65,6 +65,14 @@ const RATE_LIMITS: Record = {
'/api/leaflet/subscriptions': STANDARD_LIMIT,
// Light operations (cached data, simple queries)
+ // The whole feed refresh is one D1 query per page now, and a returning reader
+ // drains several pages in a burst — a light limit, not a standard one.
+ '/api/v2/timeline': LIGHT_LIMIT,
+ // Single-feed reads are D1 queries now (the archive), and an OPML import fires
+ // one per imported feed to backfill it. The one expensive case — the
+ // pull-through crawl — only happens for a feed the caller subscribes to and
+ // only when the archive has nothing for it yet.
+ '/api/v2/feeds/fetch': LIGHT_LIMIT,
'/api/feeds/cached': LIGHT_LIMIT,
'/api/feeds/batch': LIGHT_LIMIT,
'/api/feeds/status': LIGHT_LIMIT,
diff --git a/backend/src/types.ts b/backend/src/types.ts
index 06521e1d..c08571e2 100644
--- a/backend/src/types.ts
+++ b/backend/src/types.ts
@@ -56,6 +56,9 @@ export interface FeedItem {
// Stamped by the authed batch fetch handler (feeds-v2.ts) from a per-user read
// join. Not a stored feed field — only present on annotated responses.
read?: boolean;
+ // Set at ingest when `content` exceeded the stored-content cap and was dropped
+ // (routes/ingest.ts). The reader falls back to /extract for the full text.
+ contentTruncated?: boolean;
}
export interface ParsedFeed {
diff --git a/backend/test/feed-timeline.spec.ts b/backend/test/feed-timeline.spec.ts
new file mode 100644
index 00000000..236fd4d7
--- /dev/null
+++ b/backend/test/feed-timeline.spec.ts
@@ -0,0 +1,894 @@
+import { env } from 'cloudflare:test';
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import {
+ handleIngest,
+ handleCrawlSet,
+ handleFeedHealth,
+ clearFeedHealth,
+ trimFeedsToSanityCap,
+ ingestProxyFeed,
+ CRAWLER_HEARTBEAT_KEY,
+ FEED_HEALTH_REV_KEY,
+ TIMELINE_ENABLED_KEY,
+} from '../src/routes/ingest';
+import { handleTimeline, readFeedSlice } from '../src/routes/timeline';
+import type { Env, FeedItem, Session } from '../src/types';
+
+const TEST_DID = 'did:plc:timeline123';
+const OTHER_DID = 'did:plc:timelineother';
+const SECRET = 'test-proxy-secret';
+
+const FEED_A = 'https://example.com/a.xml';
+const FEED_B = 'https://example.com/b.xml';
+
+const SESSION: Session = {
+ did: TEST_DID,
+ handle: 'timeline.bsky.social',
+ pdsUrl: 'https://test.pds.example',
+ accessToken: 'token',
+ refreshToken: 'refresh',
+ dpopPrivateKey: '{}',
+ expiresAt: Date.now() + 3600000,
+};
+
+function item(guid: string, overrides: Partial = {}): FeedItem {
+ return {
+ guid,
+ url: `https://example.com/${guid}`,
+ title: `Title ${guid}`,
+ publishedAt: '2026-01-01T00:00:00.000Z',
+ ...overrides,
+ };
+}
+
+function ingestRequest(
+ body: unknown,
+ secret: string | null = SECRET,
+ extraHeaders: Record = {}
+): Request {
+ const headers: Record = {
+ 'Content-Type': 'application/json',
+ ...extraHeaders,
+ };
+ if (secret !== null) headers['X-Proxy-Secret'] = secret;
+ return new Request('https://api.example/api/internal/ingest', {
+ method: 'POST',
+ headers,
+ body: JSON.stringify(body),
+ });
+}
+
+async function ingest(
+ feedUrl: string,
+ items: Array<{ item: FeedItem; contentHash: string }>
+): Promise<{ inserted: number; updated: number }> {
+ const res = await handleIngest(
+ ingestRequest({
+ feeds: [{ feedUrl, title: 'Feed A' }],
+ items: items.map((entry) => ({
+ feedUrl,
+ guid: entry.item.guid,
+ item: entry.item,
+ publishedAt: Date.parse(entry.item.publishedAt),
+ firstSeenAt: Date.now(),
+ contentHash: entry.contentHash,
+ })),
+ }),
+ env
+ );
+ expect(res.status).toBe(200);
+ return (await res.json()) as { inserted: number; updated: number };
+}
+
+async function addSubscription(
+ did: string,
+ feedUrl: string,
+ opts: { active?: number; sourceType?: string | null } = {}
+) {
+ await env.DB.prepare(
+ `INSERT INTO subscriptions_cache (user_did, record_uri, feed_url, title, source_type, active)
+ VALUES (?, ?, ?, ?, ?, ?)`
+ )
+ .bind(
+ did,
+ `at://${did}/app.skyreader.feed.subscription/${Math.random().toString(36).slice(2)}`,
+ feedUrl,
+ 'Feed',
+ opts.sourceType ?? null,
+ opts.active ?? 1
+ )
+ .run();
+}
+
+async function timeline(params: Record = {}) {
+ const search = new URLSearchParams(params).toString();
+ const res = await handleTimeline(
+ new Request(`https://api.example/api/v2/timeline${search ? `?${search}` : ''}`),
+ env,
+ SESSION
+ );
+ expect(res.status).toBe(200);
+ return (await res.json()) as {
+ items: Array<{ seq: number; feedUrl: string; guid: string; read: boolean; content?: string }>;
+ cursor: number;
+ generation: string;
+ hasMore: boolean;
+ nextColdOffset?: number;
+ ingestActive: boolean;
+ readCursor: number;
+ coldStart: boolean;
+ feeds?: Record;
+ healthRev?: string;
+ feedHealth?: Record<
+ string,
+ {
+ errorCount: number;
+ error?: string;
+ lastErrorAt?: number;
+ nextRetryAt?: number;
+ lastFetchedAt?: number;
+ }
+ >;
+ };
+}
+
+async function clearCrawlerHeartbeat() {
+ await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(CRAWLER_HEARTBEAT_KEY).run();
+}
+
+async function setTimelineGate(value: string) {
+ await env.DB.prepare(
+ `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch())
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`
+ )
+ .bind(TIMELINE_ENABLED_KEY, value)
+ .run();
+}
+
+async function clearTimelineGate() {
+ await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(TIMELINE_ENABLED_KEY).run();
+}
+
+async function reportHealth(feeds: unknown[], secret: string | null = SECRET) {
+ const headers: Record = { 'Content-Type': 'application/json' };
+ if (secret !== null) headers['X-Proxy-Secret'] = secret;
+ return handleFeedHealth(
+ new Request('https://api.example/api/internal/feed-health', {
+ method: 'POST',
+ headers,
+ body: JSON.stringify({ feeds }),
+ }),
+ env
+ );
+}
+
+function brokenFeed(feedUrl: string, overrides: Record = {}) {
+ const nowSeconds = Math.floor(Date.now() / 1000);
+ return {
+ feedUrl,
+ errorCount: 3,
+ lastError: 'Failed to fetch (HTTP 404)',
+ lastErrorAt: nowSeconds,
+ nextRetryAt: nowSeconds + 600,
+ lastFetchAt: nowSeconds - 7200,
+ ...overrides,
+ };
+}
+
+describe('feed timeline (D1 ingest + serve)', () => {
+ let savedSecret: string | undefined;
+
+ beforeEach(async () => {
+ savedSecret = (env as Env).FEED_PROXY_SECRET;
+ (env as Env).FEED_PROXY_SECRET = SECRET;
+ for (const did of [TEST_DID, OTHER_DID]) {
+ await env.DB.prepare(
+ `INSERT OR IGNORE INTO users (did, handle, pds_url, tier, created_at)
+ VALUES (?, ?, 'https://test.pds.example', 'free', unixepoch())`
+ )
+ .bind(did, `${did}.test`)
+ .run();
+ }
+ });
+
+ afterEach(async () => {
+ (env as Env).FEED_PROXY_SECRET = savedSecret as string;
+ await env.DB.prepare('DELETE FROM feed_items').run();
+ await env.DB.prepare('DELETE FROM feeds').run();
+ await env.DB.prepare('DELETE FROM subscriptions_cache').run();
+ await env.DB.prepare('DELETE FROM item_labels_cache').run();
+ await clearCrawlerHeartbeat();
+ await env.DB.prepare('DELETE FROM sync_state WHERE key = ?').bind(FEED_HEALTH_REV_KEY).run();
+ // Absent is the open position, so this restores the default the rest of the
+ // suite runs under.
+ await clearTimelineGate();
+ });
+
+ describe('ingest auth', () => {
+ it('rejects a request with no secret', async () => {
+ const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, null), env);
+ expect(res.status).toBe(401);
+ });
+
+ it('rejects a mismatched secret', async () => {
+ const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, 'nope'), env);
+ expect(res.status).toBe(401);
+ });
+
+ it('fails closed when the server secret is unset', async () => {
+ (env as Env).FEED_PROXY_SECRET = '' as string;
+ const res = await handleIngest(ingestRequest({ feeds: [], items: [] }, SECRET), env);
+ expect(res.status).toBe(401);
+ });
+
+ it('rejects an oversized body by Content-Length', async () => {
+ const res = await handleIngest(
+ ingestRequest({ feeds: [], items: [] }, SECRET, {
+ 'Content-Length': String(64 * 1024 * 1024),
+ }),
+ env
+ );
+ expect(res.status).toBe(413);
+ });
+ });
+
+ describe('ingest writes', () => {
+ it('inserts items and upserts feed metadata', async () => {
+ const result = await ingest(FEED_A, [
+ { item: item('g1'), contentHash: 'h1' },
+ { item: item('g2'), contentHash: 'h2' },
+ ]);
+ expect(result.inserted).toBe(2);
+ expect(result.updated).toBe(0);
+
+ const feed = await env.DB.prepare('SELECT * FROM feeds WHERE feed_url = ?')
+ .bind(FEED_A)
+ .first<{ title: string; last_ingest_at: number }>();
+ expect(feed?.title).toBe('Feed A');
+ expect(feed?.last_ingest_at).toBeGreaterThan(0);
+ });
+
+ it('is idempotent: re-pushing the same batch creates no duplicates', async () => {
+ await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]);
+ const before = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?')
+ .bind('g1')
+ .first<{ seq: number }>();
+
+ const second = await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]);
+ expect(second.inserted).toBe(0);
+ expect(second.updated).toBe(0);
+
+ const rows = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?')
+ .bind('g1')
+ .all<{ seq: number }>();
+ expect(rows.results.length).toBe(1);
+ expect(rows.results[0].seq).toBe(before?.seq);
+ });
+
+ it('edits in place: a changed hash rewrites the row without a new seq', async () => {
+ await ingest(FEED_A, [{ item: item('g1', { title: 'Old' }), contentHash: 'h1' }]);
+ const before = await env.DB.prepare('SELECT seq FROM feed_items WHERE guid = ?')
+ .bind('g1')
+ .first<{ seq: number }>();
+
+ const result = await ingest(FEED_A, [
+ { item: item('g1', { title: 'New' }), contentHash: 'h2' },
+ ]);
+ expect(result.updated).toBe(1);
+ expect(result.inserted).toBe(0);
+
+ const after = await env.DB.prepare('SELECT seq, item_json FROM feed_items WHERE guid = ?')
+ .bind('g1')
+ .first<{ seq: number; item_json: string }>();
+ expect(after?.seq).toBe(before?.seq);
+ expect(JSON.parse(after!.item_json).title).toBe('New');
+ });
+
+ it('never prunes: a feed accumulates past the proxy’s 200-item window', async () => {
+ // Five pushes of 50 distinct items each — well past the proxy's K = 200
+ // outbox window. D1 is the archive; ordinary ingest deletes nothing.
+ for (let batch = 0; batch < 5; batch++) {
+ await ingest(
+ FEED_A,
+ Array.from({ length: 50 }, (_, i) => ({
+ item: item(`b${batch}-i${i}`),
+ contentHash: `h${batch}-${i}`,
+ }))
+ );
+ }
+ const count = await env.DB.prepare('SELECT COUNT(*) AS c FROM feed_items WHERE feed_url = ?')
+ .bind(FEED_A)
+ .first<{ c: number }>();
+ expect(count?.c).toBe(250);
+ });
+
+ it('sanity cap trims oldest-first, and only above the cap', async () => {
+ await ingest(
+ FEED_A,
+ Array.from({ length: 5 }, (_, i) => ({ item: item(`s${i}`), contentHash: `h${i}` }))
+ );
+
+ // Under the cap: a no-op.
+ await trimFeedsToSanityCap(env, [FEED_A], 10);
+ let rows = await env.DB.prepare('SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq')
+ .bind(FEED_A)
+ .all<{ guid: string }>();
+ expect(rows.results.length).toBe(5);
+
+ // Above the cap: the oldest go, newest survive.
+ await trimFeedsToSanityCap(env, [FEED_A], 3);
+ rows = await env.DB.prepare('SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq')
+ .bind(FEED_A)
+ .all<{ guid: string }>();
+ expect(rows.results.map((r) => r.guid)).toEqual(['s2', 's3', 's4']);
+ });
+
+ it('caps stored content and marks it truncated, leaving small items alone', async () => {
+ const big = 'x'.repeat(9000);
+ await ingest(FEED_A, [
+ { item: item('big', { content: big, summary: 'kept' }), contentHash: 'hbig' },
+ { item: item('small', { content: 'tiny' }), contentHash: 'hsmall' },
+ ]);
+
+ const bigRow = await env.DB.prepare('SELECT item_json FROM feed_items WHERE guid = ?')
+ .bind('big')
+ .first<{ item_json: string }>();
+ const parsedBig = JSON.parse(bigRow!.item_json);
+ expect(parsedBig.content).toBeUndefined();
+ expect(parsedBig.contentTruncated).toBe(true);
+ expect(parsedBig.summary).toBe('kept');
+
+ const smallRow = await env.DB.prepare('SELECT item_json FROM feed_items WHERE guid = ?')
+ .bind('small')
+ .first<{ item_json: string }>();
+ const parsedSmall = JSON.parse(smallRow!.item_json);
+ expect(parsedSmall.content).toBe('tiny');
+ expect(parsedSmall.contentTruncated).toBeUndefined();
+ });
+ });
+
+ describe('pull-through ingest (proxy feed → archive)', () => {
+ it('assigns seq oldest→newest, so the newest entries are what a per-feed read serves', async () => {
+ // A proxy feed is newest-first. Ingesting it forward would give the newest
+ // item the lowest seq, and every later `ORDER BY seq DESC` read would then
+ // serve the feed's OLDEST items.
+ const newestFirst = Array.from({ length: 5 }, (_, i) => item(`p${5 - i}`));
+ await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: newestFirst });
+
+ const rows = await env.DB.prepare(
+ 'SELECT guid, seq FROM feed_items WHERE feed_url = ? ORDER BY seq ASC'
+ )
+ .bind(FEED_A)
+ .all<{ guid: string; seq: number }>();
+ expect(rows.results.map((r) => r.guid)).toEqual(['p1', 'p2', 'p3', 'p4', 'p5']);
+
+ const slice = await readFeedSlice(env, TEST_DID, FEED_A, 2);
+ expect(slice.map((i) => i.guid)).toEqual(['p5', 'p4']);
+ });
+
+ it('hashes identically to a later crawler push, so the push is a no-op', async () => {
+ await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: [item('same')] });
+ const before = await env.DB.prepare('SELECT seq, content_hash FROM feed_items WHERE guid = ?')
+ .bind('same')
+ .first<{ seq: number; content_hash: string }>();
+
+ await ingestProxyFeed(env, FEED_A, { title: 'Pulled', items: [item('same')] });
+ const rows = await env.DB.prepare('SELECT seq, content_hash FROM feed_items WHERE guid = ?')
+ .bind('same')
+ .all<{ seq: number; content_hash: string }>();
+ expect(rows.results.length).toBe(1);
+ expect(rows.results[0].seq).toBe(before?.seq);
+ expect(rows.results[0].content_hash).toBe(before?.content_hash);
+ });
+
+ it('serves newest publications when pull-through and crawler ingest interleave', async () => {
+ // Subscribe-time pull-through can write the newest proxy window before
+ // the crawler's initial backlog reaches this feed. The older backlog then
+ // has higher seq values, so per-feed reads must not use seq as recency.
+ await ingestProxyFeed(env, FEED_A, {
+ title: 'Pulled',
+ items: [
+ item('newest', { publishedAt: '2026-03-01T00:00:00.000Z' }),
+ item('newer', { publishedAt: '2026-02-01T00:00:00.000Z' }),
+ ],
+ });
+ await ingest(FEED_A, [
+ {
+ item: item('oldest', { publishedAt: '2025-12-01T00:00:00.000Z' }),
+ contentHash: 'oldest-hash',
+ },
+ {
+ item: item('older', { publishedAt: '2026-01-01T00:00:00.000Z' }),
+ contentHash: 'older-hash',
+ },
+ ]);
+
+ const slice = await readFeedSlice(env, TEST_DID, FEED_A, 2);
+ expect(slice.map((i) => i.guid)).toEqual(['newest', 'newer']);
+
+ await addSubscription(TEST_DID, FEED_A);
+ const page = await timeline();
+ expect(page.items.map((i) => i.guid)).toEqual(['newest', 'newer', 'older', 'oldest']);
+ });
+ });
+
+ describe('crawl set', () => {
+ it('requires the shared secret', async () => {
+ const res = await handleCrawlSet(
+ new Request('https://api.example/api/internal/crawl-set'),
+ env
+ );
+ expect(res.status).toBe(401);
+ });
+
+ it('returns active RSS feeds with subscriber counts', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await addSubscription(OTHER_DID, FEED_A);
+ await addSubscription(TEST_DID, FEED_B, { active: 0 });
+ await addSubscription(TEST_DID, 'at://did:plc:x/pub', { sourceType: 'atproto.documents' });
+
+ const res = await handleCrawlSet(
+ new Request('https://api.example/api/internal/crawl-set', {
+ headers: { 'X-Proxy-Secret': SECRET },
+ }),
+ env
+ );
+ expect(res.status).toBe(200);
+ const body = (await res.json()) as {
+ feeds: Array<{ feedUrl: string; subscribers: number }>;
+ };
+ expect(body.feeds).toEqual([{ feedUrl: FEED_A, subscribers: 2 }]);
+ });
+ });
+
+ describe('timeline serving', () => {
+ it('cold-starts with a per-feed newest slice and inline read flags', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await addSubscription(TEST_DID, FEED_B);
+ await ingest(FEED_A, [
+ { item: item('a1'), contentHash: 'h1' },
+ { item: item('a2'), contentHash: 'h2' },
+ ]);
+ await ingest(FEED_B, [{ item: item('b1'), contentHash: 'h3' }]);
+
+ await env.DB.prepare(
+ `INSERT INTO item_labels_cache (user_did, item_key, item_type, label, created_at, updated_at)
+ VALUES (?, 'a1', 'article', 'read', unixepoch(), unixepoch())`
+ )
+ .bind(TEST_DID)
+ .run();
+
+ const body = await timeline();
+ expect(body.coldStart).toBe(true);
+ expect(body.items.map((i) => i.guid).sort()).toEqual(['a1', 'a2', 'b1']);
+ const byGuid = Object.fromEntries(body.items.map((i) => [i.guid, i.read]));
+ expect(byGuid['a1']).toBe(true);
+ expect(byGuid['a2']).toBe(false);
+ expect(body.cursor).toBeGreaterThan(0);
+ expect(body.hasMore).toBe(false);
+ expect(typeof body.readCursor).toBe('number');
+ expect(body.feeds?.[FEED_A]?.title).toBe('Feed A');
+ });
+
+ it('excludes parked, unsubscribed and atproto sources', async () => {
+ await addSubscription(TEST_DID, FEED_A, { active: 0 });
+ await addSubscription(OTHER_DID, FEED_B);
+ await ingest(FEED_A, [{ item: item('a1'), contentHash: 'h1' }]);
+ await ingest(FEED_B, [{ item: item('b1'), contentHash: 'h2' }]);
+
+ const cold = await timeline();
+ expect(cold.items).toEqual([]);
+
+ // ...and the incremental path applies the same filter.
+ const incremental = await timeline({ since_seq: '0', generation: cold.generation });
+ expect(incremental.items).toEqual([]);
+ });
+
+ it('drains incrementally from the cursor, paging with hasMore', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [
+ { item: item('i1'), contentHash: 'h1' },
+ { item: item('i2'), contentHash: 'h2' },
+ { item: item('i3'), contentHash: 'h3' },
+ ]);
+
+ const generation = (await timeline()).generation;
+ const first = await timeline({ since_seq: '0', generation, limit: '2' });
+ expect(first.coldStart).toBe(false);
+ expect(first.items.map((i) => i.guid)).toEqual(['i1', 'i2']);
+ expect(first.hasMore).toBe(true);
+
+ const second = await timeline({
+ since_seq: String(first.cursor),
+ generation,
+ limit: '2',
+ });
+ expect(second.items.map((i) => i.guid)).toEqual(['i3']);
+ expect(second.hasMore).toBe(false);
+
+ // Steady state: nothing new, cursor held, one empty page.
+ const third = await timeline({ since_seq: String(second.cursor), generation, limit: '2' });
+ expect(third.items).toEqual([]);
+ expect(third.cursor).toBe(second.cursor);
+ expect(third.hasMore).toBe(false);
+ expect(third.feeds).toBeUndefined();
+ });
+
+ it('does not re-deliver an edited item (seq unchanged)', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('e1', { title: 'Old' }), contentHash: 'h1' }]);
+ const cold = await timeline();
+
+ await ingest(FEED_A, [{ item: item('e1', { title: 'New' }), contentHash: 'h2' }]);
+ const after = await timeline({
+ since_seq: String(cold.cursor),
+ generation: cold.generation,
+ });
+ expect(after.items).toEqual([]);
+ });
+
+ it('cold-starts again on a generation mismatch', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('g1'), contentHash: 'h1' }]);
+
+ const body = await timeline({ since_seq: '999999', generation: 'stale-generation' });
+ expect(body.coldStart).toBe(true);
+ expect(body.items.map((i) => i.guid)).toEqual(['g1']);
+ });
+
+ it('starts an empty account at the archive head, not at zero', async () => {
+ await addSubscription(TEST_DID, FEED_B);
+ await ingest(FEED_A, [{ item: item('other'), contentHash: 'h1' }]);
+
+ const body = await timeline();
+ expect(body.items).toEqual([]);
+ expect(body.cursor).toBeGreaterThan(0);
+ });
+
+ it('cold-starts a client whose cursor sits above the head (rewound archive)', async () => {
+ // The generation survives a Time Travel restore while the seqs rewind, so a
+ // cursor above the head is the only symptom — and `seq > cursor` would
+ // otherwise return nothing on every poll, forever.
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('r1'), contentHash: 'h1' }]);
+ const cold = await timeline();
+
+ const stalled = await timeline({
+ since_seq: String(cold.cursor + 5000),
+ generation: cold.generation,
+ });
+ expect(stalled.coldStart).toBe(true);
+ expect(stalled.items.map((i) => i.guid)).toEqual(['r1']);
+ expect(stalled.cursor).toBeLessThan(cold.cursor + 5000);
+ });
+
+ it('pages a large cold start and continues from nextColdOffset', async () => {
+ // 26 feeds × 30 items is past the per-page item budget, so the first page
+ // stops early and hands back a continuation index.
+ const feedUrls = Array.from({ length: 26 }, (_, i) => `https://example.com/paged${i}.xml`);
+ for (const feedUrl of feedUrls) {
+ await addSubscription(TEST_DID, feedUrl);
+ await ingest(
+ feedUrl,
+ Array.from({ length: 30 }, (_, i) => ({
+ item: item(`${feedUrl}#${i}`),
+ contentHash: `${feedUrl}-${i}`,
+ }))
+ );
+ }
+
+ const first = await timeline();
+ expect(first.coldStart).toBe(true);
+ expect(first.hasMore).toBe(true);
+ expect(first.nextColdOffset).toBeGreaterThan(0);
+ expect(first.items.length).toBeLessThan(26 * 30);
+
+ const second = await timeline({ cold_offset: String(first.nextColdOffset) });
+ expect(second.coldStart).toBe(true);
+ expect(second.hasMore).toBe(false);
+ expect(second.items.length).toBeGreaterThan(0);
+
+ // Every feed is covered across the two pages, each with its newest slice.
+ const seen = new Set([...first.items, ...second.items].map((i) => i.feedUrl));
+ expect(seen.size).toBe(26);
+ expect(first.items.length + second.items.length).toBe(26 * 30);
+ });
+ });
+
+ describe('crawler liveness (ingestActive)', () => {
+ it('is false until the crawler checks in', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await clearCrawlerHeartbeat();
+ const body = await timeline();
+ expect(body.ingestActive).toBe(false);
+ });
+
+ it('is true after an ingest push', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('h1'), contentHash: 'h1' }]);
+ expect((await timeline()).ingestActive).toBe(true);
+ });
+
+ it('is true after a crawl-set pull, even with nothing ingested', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await clearCrawlerHeartbeat();
+ const res = await handleCrawlSet(
+ new Request('https://api.example/api/internal/crawl-set', {
+ headers: { 'X-Proxy-Secret': SECRET },
+ }),
+ env
+ );
+ expect(res.status).toBe(200);
+
+ const body = await timeline();
+ expect(body.ingestActive).toBe(true);
+ expect(body.items).toEqual([]);
+ });
+
+ it('goes stale when the last heartbeat is old', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await env.DB.prepare(
+ `INSERT INTO sync_state (key, value, updated_at) VALUES (?, ?, unixepoch())
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value`
+ )
+ .bind(CRAWLER_HEARTBEAT_KEY, String(Math.floor(Date.now() / 1000) - 4 * 3600))
+ .run();
+
+ expect((await timeline()).ingestActive).toBe(false);
+ });
+ });
+
+ describe('rollout gate (timeline_enabled)', () => {
+ it('holds clients on the batch path while the gate is shut, however live the crawler', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('g1'), contentHash: 'g1' }]);
+ expect((await timeline()).ingestActive).toBe(true);
+
+ await setTimelineGate('0');
+ const gated = await timeline();
+ expect(gated.ingestActive).toBe(false);
+ // The whole point of the short-circuit: a page the client is about to throw
+ // away is never built, so the gated window costs one sync_state read.
+ expect(gated.items).toEqual([]);
+ // Both fallback signals agree, so a client reading either one stays put.
+ expect(gated.coldStart).toBe(true);
+ expect(gated.hasMore).toBe(false);
+ });
+
+ it('reopens without a deploy, serving the archive that filled while it was shut', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await setTimelineGate('0');
+ // Ingest keeps running while the gate is shut — that is the sequencing the
+ // gate exists to allow: fill the archive first, admit readers second.
+ await ingest(FEED_A, [{ item: item('g2'), contentHash: 'g2' }]);
+ expect((await timeline()).items).toEqual([]);
+
+ await setTimelineGate('1');
+ const open = await timeline();
+ expect(open.ingestActive).toBe(true);
+ expect(open.items.map((i) => i.guid)).toEqual(['g2']);
+ });
+
+ it('is open when the row is absent, so an environment that never set it is unaffected', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('g3'), contentHash: 'g3' }]);
+ await clearTimelineGate();
+ expect((await timeline()).ingestActive).toBe(true);
+ });
+
+ it('still requires a live crawler when open', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('g4'), contentHash: 'g4' }]);
+ await setTimelineGate('1');
+ await clearCrawlerHeartbeat();
+ expect((await timeline()).ingestActive).toBe(false);
+ });
+ });
+
+ describe('feed health', () => {
+ it('requires the shared secret', async () => {
+ expect((await reportHealth([brokenFeed(FEED_A)], null)).status).toBe(401);
+ expect((await reportHealth([brokenFeed(FEED_A)], 'nope')).status).toBe(401);
+ });
+
+ it('stamps the crawler heartbeat like the other internal endpoints', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await clearCrawlerHeartbeat();
+ expect((await reportHealth([])).status).toBe(200);
+ expect((await timeline()).ingestActive).toBe(true);
+ });
+
+ it('serves a broken feed to its subscribers, in milliseconds', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ const report = brokenFeed(FEED_A);
+ await reportHealth([report]);
+
+ const body = await timeline();
+ expect(body.feedHealth?.[FEED_A]).toEqual({
+ errorCount: 3,
+ error: 'Failed to fetch (HTTP 404)',
+ lastErrorAt: report.lastErrorAt * 1000,
+ nextRetryAt: report.nextRetryAt * 1000,
+ lastFetchedAt: report.lastFetchAt * 1000,
+ });
+ });
+
+ it('records a feed that has never ingested a single item', async () => {
+ // Broken from its first crawl: no items were ever pushed, so there is no
+ // `feeds` row for the health report to update.
+ await addSubscription(TEST_DID, FEED_B);
+ await reportHealth([brokenFeed(FEED_B)]);
+ expect((await timeline()).feedHealth?.[FEED_B]?.errorCount).toBe(3);
+ });
+
+ it('does not leak another user’s broken feeds', async () => {
+ await addSubscription(OTHER_DID, FEED_B);
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]);
+
+ const body = await timeline();
+ expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_A]);
+ });
+
+ it('clears a feed that recovers, by absence from the next report', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([brokenFeed(FEED_A)]);
+ expect((await timeline()).feedHealth?.[FEED_A]).toBeDefined();
+
+ // The crawler no longer lists it — that is the whole recovery signal.
+ await reportHealth([]);
+ expect((await timeline()).feedHealth?.[FEED_A]).toBeUndefined();
+ });
+
+ it('recovers one feed without disturbing another that is still broken', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await addSubscription(TEST_DID, FEED_B);
+ await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]);
+ await reportHealth([brokenFeed(FEED_B)]);
+
+ const body = await timeline();
+ expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_B]);
+ });
+
+ it('omits the payload when the client already holds the current revision', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([brokenFeed(FEED_A)]);
+
+ const first = await timeline();
+ expect(first.feedHealth).toBeDefined();
+ expect(first.healthRev).toBeTruthy();
+
+ // Steady state: the client echoes the revision back and pays nothing.
+ const second = await timeline({
+ since_seq: String(first.cursor),
+ generation: first.generation,
+ health_rev: first.healthRev!,
+ });
+ expect(second.feedHealth).toBeUndefined();
+ expect(second.healthRev).toBe(first.healthRev);
+ });
+
+ it('re-sends the payload once the unhealthy set changes', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await addSubscription(TEST_DID, FEED_B);
+ await reportHealth([brokenFeed(FEED_A)]);
+ const first = await timeline();
+
+ await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B, { errorCount: 1 })]);
+ const second = await timeline({
+ since_seq: String(first.cursor),
+ generation: first.generation,
+ health_rev: first.healthRev!,
+ });
+
+ expect(second.healthRev).not.toBe(first.healthRev);
+ expect(Object.keys(second.feedHealth ?? {}).sort()).toEqual([FEED_A, FEED_B].sort());
+ });
+
+ it('always sends health on a cold start, whatever revision the client claims', async () => {
+ // A cold start replays already-archived items, including from a feed that
+ // has broken since — so its blanket "these delivered, they're fine" pass
+ // must be corrected in the same response.
+ await addSubscription(TEST_DID, FEED_A);
+ await ingest(FEED_A, [{ item: item('c1'), contentHash: 'c1' }]);
+ await reportHealth([brokenFeed(FEED_A)]);
+
+ const rev = (await timeline()).healthRev!;
+ const cold = await timeline({ health_rev: rev });
+ expect(cold.coldStart).toBe(true);
+ expect(cold.items).toHaveLength(1);
+ expect(cold.feedHealth?.[FEED_A]).toBeDefined();
+ });
+
+ it('clearFeedHealth clears one feed and moves the revision', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await addSubscription(TEST_DID, FEED_B);
+ await reportHealth([brokenFeed(FEED_A), brokenFeed(FEED_B)]);
+ const before = (await timeline()).healthRev;
+
+ await clearFeedHealth(env as Env, FEED_A);
+
+ const body = await timeline();
+ expect(Object.keys(body.feedHealth ?? {})).toEqual([FEED_B]);
+ expect(body.healthRev).not.toBe(before);
+ });
+
+ it('flags a starved feed without telling readers anything', async () => {
+ // `crawl_stale` is an operator signal: the crawler isn't reaching the feed,
+ // but nothing is erroring, so the reader has no error to show.
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: true }]);
+
+ const body = await timeline();
+ expect(body.feedHealth?.[FEED_A]).toBeUndefined();
+
+ const row = await env.DB.prepare(
+ 'SELECT error_count, crawl_stale FROM feeds WHERE feed_url = ?'
+ )
+ .bind(FEED_A)
+ .first<{ error_count: number; crawl_stale: number }>();
+ expect(row).toMatchObject({ error_count: 0, crawl_stale: 1 });
+ });
+
+ it('clears a starved flag once the crawler catches up', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: true }]);
+ await reportHealth([]);
+
+ const row = await env.DB.prepare('SELECT crawl_stale FROM feeds WHERE feed_url = ?')
+ .bind(FEED_A)
+ .first<{ crawl_stale: number }>();
+ expect(row?.crawl_stale).toBe(0);
+ });
+
+ it('carries both flags for a feed that is erroring and starved', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([brokenFeed(FEED_A, { crawlStale: true })]);
+
+ const row = await env.DB.prepare(
+ 'SELECT error_count, crawl_stale FROM feeds WHERE feed_url = ?'
+ )
+ .bind(FEED_A)
+ .first<{ error_count: number; crawl_stale: number }>();
+ expect(row).toMatchObject({ error_count: 3, crawl_stale: 1 });
+ // Readers still see the error, which is the part they can act on.
+ expect((await timeline()).feedHealth?.[FEED_A]?.errorCount).toBe(3);
+ });
+
+ it('ignores an entry that claims no fault at all', async () => {
+ // Otherwise a healthy feed listed by mistake would stay flagged forever.
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([{ feedUrl: FEED_A, errorCount: 0, crawlStale: false }]);
+
+ const row = await env.DB.prepare('SELECT feed_url FROM feeds WHERE feed_url = ?')
+ .bind(FEED_A)
+ .first();
+ expect(row).toBeNull();
+ });
+
+ it('leaves the revision alone when only the starved flag moves', async () => {
+ // The reader payload holds erroring feeds only, so a crawl-capacity change
+ // must not make every client re-download it.
+ await addSubscription(TEST_DID, FEED_A);
+ await reportHealth([brokenFeed(FEED_A)]);
+ const before = (await timeline()).healthRev;
+
+ await addSubscription(TEST_DID, FEED_B);
+ await reportHealth([
+ brokenFeed(FEED_A),
+ { feedUrl: FEED_B, errorCount: 0, crawlStale: true },
+ ]);
+ expect((await timeline()).healthRev).toBe(before);
+ });
+
+ it('leaves the revision alone when a report changes nothing', async () => {
+ await addSubscription(TEST_DID, FEED_A);
+ const report = brokenFeed(FEED_A);
+ await reportHealth([report]);
+ const first = (await timeline()).healthRev;
+
+ await reportHealth([report]);
+ expect((await timeline()).healthRev).toBe(first);
+ });
+ });
+});
diff --git a/backend/test/feeds-v2-fetch.spec.ts b/backend/test/feeds-v2-fetch.spec.ts
new file mode 100644
index 00000000..1cb9f9c4
--- /dev/null
+++ b/backend/test/feeds-v2-fetch.spec.ts
@@ -0,0 +1,142 @@
+import { env } from 'cloudflare:test';
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+import { handleV2FeedFetch } from '../src/routes/feeds-v2';
+import type { Env, Session } from '../src/types';
+
+/**
+ * `GET /api/v2/feeds/fetch` reads the D1 archive, and pulls a feed through the
+ * crawler when the archive holds nothing for it. That pull-through is the one
+ * user-triggered write into a shared, never-pruned archive, so it is gated on the
+ * caller's own subscription.
+ */
+
+const TEST_DID = 'did:plc:fetchtester';
+const FEED_URL = 'https://example.com/fetch-feed.xml';
+
+const SESSION: Session = {
+ did: TEST_DID,
+ handle: 'fetch.bsky.social',
+ pdsUrl: 'https://test.pds.example',
+ accessToken: 'token',
+ refreshToken: 'refresh',
+ dpopPrivateKey: '{}',
+ expiresAt: Date.now() + 3600000,
+};
+
+function fetchRequest(params: Record): Request {
+ const search = new URLSearchParams(params).toString();
+ return new Request(`https://api.example/api/v2/feeds/fetch?${search}`);
+}
+
+function proxyFeed(guids: string[]): Response {
+ return new Response(
+ JSON.stringify({
+ feed: {
+ title: 'Pulled Feed',
+ siteUrl: 'https://example.com',
+ // The proxy serves newest-first.
+ items: guids.map((guid) => ({
+ guid,
+ url: `https://example.com/${guid}`,
+ title: `Title ${guid}`,
+ publishedAt: '2026-01-01T00:00:00.000Z',
+ })),
+ },
+ }),
+ { status: 200, headers: { 'Content-Type': 'application/json' } }
+ );
+}
+
+async function subscribe(feedUrl: string) {
+ await env.DB.prepare(
+ `INSERT OR IGNORE INTO users (did, handle, pds_url, tier, created_at)
+ VALUES (?, 'fetch.test', 'https://test.pds.example', 'free', unixepoch())`
+ )
+ .bind(TEST_DID)
+ .run();
+ await env.DB.prepare(
+ `INSERT INTO subscriptions_cache (user_did, record_uri, feed_url, title, active)
+ VALUES (?, ?, ?, 'Feed', 1)`
+ )
+ .bind(TEST_DID, `at://${TEST_DID}/app.skyreader.feed.subscription/${Date.now()}`, feedUrl)
+ .run();
+}
+
+describe('handleV2FeedFetch (archive + gated pull-through)', () => {
+ let originalFetch: typeof fetch;
+ let proxyCalls: string[];
+
+ beforeEach(() => {
+ originalFetch = globalThis.fetch;
+ proxyCalls = [];
+ (env as Env).FEED_PROXY_URL = 'https://proxy.example';
+ (env as Env).FEED_PROXY_SECRET = 'test-secret';
+ globalThis.fetch = vi.fn(async (input: RequestInfo | URL) => {
+ proxyCalls.push(typeof input === 'string' ? input : input.toString());
+ return proxyFeed(['newest', 'middle', 'oldest']);
+ }) as unknown as typeof fetch;
+ });
+
+ afterEach(async () => {
+ globalThis.fetch = originalFetch;
+ vi.restoreAllMocks();
+ await env.DB.prepare('DELETE FROM feed_items').run();
+ await env.DB.prepare('DELETE FROM feeds').run();
+ await env.DB.prepare('DELETE FROM subscriptions_cache').run();
+ });
+
+ it('pulls a subscribed feed through the crawler and serves it newest-first', async () => {
+ await subscribe(FEED_URL);
+
+ const res = await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION);
+ expect(res.status).toBe(200);
+ const body = (await res.json()) as { title: string; items: Array<{ guid: string }> };
+
+ expect(proxyCalls.length).toBe(1);
+ expect(body.title).toBe('Pulled Feed');
+ expect(body.items.map((i) => i.guid)).toEqual(['newest', 'middle', 'oldest']);
+
+ // Ingested oldest→newest, so the archive's seq order matches feed order.
+ const rows = await env.DB.prepare(
+ 'SELECT guid FROM feed_items WHERE feed_url = ? ORDER BY seq ASC'
+ )
+ .bind(FEED_URL)
+ .all<{ guid: string }>();
+ expect(rows.results.map((r) => r.guid)).toEqual(['oldest', 'middle', 'newest']);
+ });
+
+ it('does not write the shared archive for a feed the caller does not subscribe to', async () => {
+ const res = await handleV2FeedFetch(
+ fetchRequest({ url: FEED_URL, refresh: '1' }),
+ env,
+ SESSION
+ );
+ expect(res.status).toBe(200);
+ const body = (await res.json()) as { items: unknown[] };
+
+ expect(proxyCalls).toEqual([]);
+ expect(body.items).toEqual([]);
+ const count = await env.DB.prepare('SELECT COUNT(*) AS c FROM feed_items').first<{
+ c: number;
+ }>();
+ expect(count?.c).toBe(0);
+ });
+
+ it('serves the archive without a crawl once the feed is ingested', async () => {
+ await subscribe(FEED_URL);
+ await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION);
+ expect(proxyCalls.length).toBe(1);
+
+ const res = await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION);
+ const body = (await res.json()) as { items: Array<{ guid: string }> };
+ expect(proxyCalls.length).toBe(1);
+ expect(body.items.map((i) => i.guid)).toEqual(['newest', 'middle', 'oldest']);
+ });
+
+ it('re-crawls a subscribed feed on refresh=1', async () => {
+ await subscribe(FEED_URL);
+ await handleV2FeedFetch(fetchRequest({ url: FEED_URL }), env, SESSION);
+ await handleV2FeedFetch(fetchRequest({ url: FEED_URL, refresh: '1' }), env, SESSION);
+ expect(proxyCalls.length).toBe(2);
+ });
+});
diff --git a/backend/test/ops-metrics.spec.ts b/backend/test/ops-metrics.spec.ts
index 4ab44bda..84cc0363 100644
--- a/backend/test/ops-metrics.spec.ts
+++ b/backend/test/ops-metrics.spec.ts
@@ -49,6 +49,7 @@ const pollerStatusBody = (lagMs: number | null) => ({
beforeEach(async () => {
await env.DB.prepare('DELETE FROM system_status').run();
await env.DB.prepare('DELETE FROM metrics_snapshots').run();
+ await env.DB.prepare('DELETE FROM feeds').run();
});
afterEach(() => vi.restoreAllMocks());
@@ -226,17 +227,19 @@ describe('writeMetricsSnapshot', () => {
await writeSystemStatus(
env as Env,
'proxy_stats',
- { freshPct: 88.5 } as ProxyStatsValue,
+ { freshPct: 88.5, feedsInError: 7 } as ProxyStatsValue,
now - 60_000
);
await writeMetricsSnapshot(env as Env, now);
const row = await env.DB.prepare(
- 'SELECT firehose_lag_ms, proxy_fresh_pct FROM metrics_snapshots'
- ).first<{ firehose_lag_ms: number; proxy_fresh_pct: number }>();
+ 'SELECT firehose_lag_ms, proxy_fresh_pct, feeds_with_errors FROM metrics_snapshots'
+ ).first<{ firehose_lag_ms: number; proxy_fresh_pct: number; feeds_with_errors: number }>();
expect(row?.firehose_lag_ms).toBe(42_000);
expect(row?.proxy_fresh_pct).toBe(88.5);
+ // Per-feed errors come from the crawler now — `feeds` has no error column.
+ expect(row?.feeds_with_errors).toBe(7);
});
it('records null rather than a stale value nobody refreshed', async () => {
@@ -250,10 +253,31 @@ describe('writeMetricsSnapshot', () => {
await writeMetricsSnapshot(env as Env, now);
- const row = await env.DB.prepare('SELECT firehose_lag_ms FROM metrics_snapshots').first<{
- firehose_lag_ms: number | null;
- }>();
+ const row = await env.DB.prepare(
+ 'SELECT firehose_lag_ms, feeds_with_errors FROM metrics_snapshots'
+ ).first<{ firehose_lag_ms: number | null; feeds_with_errors: number | null }>();
expect(row?.firehose_lag_ms).toBeNull();
+ // No proxy row at all: unknown, not "zero feeds are erroring".
+ expect(row?.feeds_with_errors).toBeNull();
+ });
+
+ it('counts the feeds the crawler ingests into, not the dropped feed_metadata', async () => {
+ const now = 300 * HOUR_MS;
+ await env.DB.batch([
+ env.DB.prepare(
+ "INSERT INTO feeds (feed_url, title, last_ingest_at) VALUES ('https://a.example/f', 'A', 1)"
+ ),
+ env.DB.prepare(
+ "INSERT INTO feeds (feed_url, title, last_ingest_at) VALUES ('https://b.example/f', 'B', 2)"
+ ),
+ ]);
+
+ await writeMetricsSnapshot(env as Env, now);
+
+ const row = await env.DB.prepare('SELECT feeds FROM metrics_snapshots').first<{
+ feeds: number;
+ }>();
+ expect(row?.feeds).toBe(2);
});
it('prunes points past the retention window', async () => {
diff --git a/backend/wrangler.toml b/backend/wrangler.toml
index 659679a1..395cacaf 100644
--- a/backend/wrangler.toml
+++ b/backend/wrangler.toml
@@ -84,6 +84,17 @@ vars = { FRONTEND_URL = "http://127.0.0.1:5173" }
# Staging environment
[env.staging]
name = "skyreader-api-staging"
+# Staging is MEANT to talk to its own Fly proxy (skyreader-feed-proxy-staging),
+# which pushes only into the staging D1 — prod and staging as two fully isolated
+# pairs, no cross-links, distinct PROXY_SECRET/FEED_PROXY_SECRET values.
+#
+# The URL below still points at the prod proxy on purpose: staging deploys on
+# every push to main, so flipping it before the app exists would take /extract,
+# discovery, standard.site documents and social context down on staging. Flip it
+# to https://skyreader-feed-proxy-staging.fly.dev in the SAME deploy that rotates
+# FEED_PROXY_SECRET, once Phase 3 of docs/plans/D1_FEED_TIMELINE.md has actually
+# provisioned the app. Until then staging clients stay on the legacy batch path
+# by themselves (the timeline reports ingestActive: false).
vars = { FRONTEND_URL = "https://staging.skyreader.app", ALLOWED_ORIGINS = "https://staging.skyreader.app,https://staging-linkblogs.skyreader.app", FEED_PROXY_URL = "https://skyreader-feed-proxy.fly.dev", LINKBLOG_PUBLIC_URL = "https://staging-linkblogs.skyreader.app", SENTRY_ENVIRONMENT = "staging" }
routes = [{ pattern = "api-staging.skyreader.app", custom_domain = true }]
diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md
index cec9ff6b..bbf7863a 100644
--- a/docs/RUNBOOK.md
+++ b/docs/RUNBOOK.md
@@ -23,7 +23,7 @@ it rather than learning to ignore it.
| Backend Worker | Structured logs (Workers Logs) | One JSON object per event, keyed by `requestId`. |
| Backend cron (every minute) | Heartbeat ping (`HEARTBEAT_URL`) | Dead-man's switch; also guards the firehose. |
| Feed proxy | `GET /health` | Status, version, cached feed count. No auth. |
-| Feed proxy | `GET /stats` | Cache freshness + per-feed error counts. Secret. |
+| Feed proxy | `GET /stats` | Cache freshness, per-feed errors, ingest backlog. |
| Feed proxy | Sentry (`@sentry/bun`) | Route escapes, warmer failures. |
| Feed proxy warmer | Heartbeat ping (`WARM_HEARTBEAT_URL`) | Dead-man's switch for the warm loop. |
| Backend cron (every minute) | `system_status` rows (D1) | Cron liveness, poller lag, proxy cache stats. |
@@ -306,17 +306,17 @@ poll cycle — a `requestId`, plus `route` and `did` when known.
Query patterns that pay for themselves:
-| Question | Filter |
-| -------------------------------- | ------------------------------------------------- |
-| Everything about one failure | `requestId = ` |
-| Error rate on one endpoint | `event = request AND route = /api/v2/feeds/batch` |
-| Slow requests | `event = request AND durationMs > 3000` |
-| Is the cron doing its work? | `event = cron_run` |
-| Is the firehose keeping up? | `event = jetstream_poll` → `subscriptionsLagMs` |
-| What failed inside a cron run? | `event = cron_phase_failed` → `phase` |
-| Why is the ops panel stale? | `event = ops_metrics_failed` → `step` |
-| Did the hourly trend point land? | `event = metrics_snapshot` → `prunedRows` |
-| Are browsers throwing? | `event = client_error` → `kind`, `appVersion` |
+| Question | Filter |
+| -------------------------------- | ----------------------------------------------- |
+| Everything about one failure | `requestId = ` |
+| Error rate on one endpoint | `event = request AND route = /api/v2/timeline` |
+| Slow requests | `event = request AND durationMs > 3000` |
+| Is the cron doing its work? | `event = cron_run` |
+| Is the firehose keeping up? | `event = jetstream_poll` → `subscriptionsLagMs` |
+| What failed inside a cron run? | `event = cron_phase_failed` → `phase` |
+| Why is the ops panel stale? | `event = ops_metrics_failed` → `step` |
+| Did the hourly trend point land? | `event = metrics_snapshot` → `prunedRows` |
+| Are browsers throwing? | `event = client_error` → `kind`, `appVersion` |
The id is also returned to callers as the `X-Request-Id` response header (exposed
via CORS), so a user-reported failure can be traced if they can quote it. The
@@ -371,6 +371,9 @@ days, pruned by the job that writes it; the panel shows the most recent 30.
Cadence, if a number looks older than expected: poller status every minute, proxy
stats every 5th minute, snapshot once an hour on the hour.
+The **Feeds** tiles in the metrics section below the ops panel cover the ingest
+side; §4d says what they mean and how to check the pieces they can't see.
+
---
## 4c. Client signal
@@ -409,6 +412,85 @@ shape of most PWA weirdness, and a reload fixes it.
---
+## 4d. Ingest health
+
+Feed reads are served from D1, so what a reader sees is only as fresh as the
+crawler's **push**, not the crawler's cache. Proxy cache freshness (§4b) is the
+first hop of two; this section is the second. **None of it is wired to an alert
+yet** — the signals are on the admin and on the proxy, and this is the list to
+walk when the reader looks stale but every tile above is green.
+
+| Signal | Where | Healthy | How to check |
+| --------------------------- | ----------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
+| Crawler talking to us | `sync_state.crawler_heartbeat_at` (D1) | stamped within ~5 min | `npx wrangler d1 execute skyreader --remote --command "SELECT * FROM sync_state WHERE key = 'crawler_heartbeat_at'"` |
+| Clients reading the archive | `GET /api/v2/timeline` → `ingestActive` | `true` in an ingesting environment | Any authenticated timeline response |
+| Rollout gate | `sync_state.timeline_enabled` (D1) | `'1'` once rolled out | `npx wrangler d1 execute skyreader --remote --command "SELECT * FROM sync_state WHERE key = 'timeline_enabled'"` |
+| Push backlog (outbox) | proxy `GET /stats` → `ingest.pending` | near zero in steady state | `curl -H "X-Proxy-Secret: $SECRET" https://skyreader-feed-proxy.fly.dev/stats` |
+| Push failing | Sentry `source: ingest-push`, proxy logs | silent | `fly logs -a skyreader-feed-proxy` → `Ingest push failed` |
+| Feeds gone quiet | Admin → Feeds, "Subscribed Feeds Not Ingesting" | small and stable | The dashboard's Feeds metrics |
+
+Reading these correctly:
+
+- **`crawler_heartbeat_at` is per-environment and stamped by _both_ internal
+ endpoints** (`/api/internal/ingest` and the 5-minutely `/api/internal/crawl-set`
+ pull), so it keeps ticking through a quiet period with no new items. A missing
+ stamp means the proxy isn't talking to this Worker at all — usually `INGEST_URL`
+ unset or pointed at the wrong environment, which is also the deliberate state
+ before Phase 3. The consequence is not an outage: `ingestActive` goes false and
+ clients fall back to the legacy batch path.
+- **`ingest.pending` is the outbox depth**, and the push loop drains 100 items
+ every 15s (~400/min). A few hundred is churn. A number that holds in the
+ thousands across two readings means the push is erroring, not busy — check
+ Sentry before touching the machine. The exception is the **first** enablement of
+ `INGEST_URL` in an environment: the whole item log is dirty at once and the
+ backfill legitimately takes hours to drain.
+- **`feeds.last_ingest_at` is the last time that feed produced _new or changed_
+ items**, not the last time it was crawled — nothing stamps it on a fetch that
+ found nothing. A feed that publishes weekly therefore looks "not ingesting" for
+ a week, so read the admin's count as a trend (a jump means the crawl set or the
+ push stopped) rather than as a per-feed verdict.
+
+Fix, once you know which one it is: a wedged warm loop or push loop is
+`fly machine restart ` (never `fly scale count` — see §4's proxy-warmer entry);
+a heartbeat that never arrives is configuration, not a restart.
+
+### The timeline rollout gate
+
+`ingestActive` is the AND of two things: a fresh crawler heartbeat and
+`sync_state.timeline_enabled`. They are separate because the heartbeat lands
+_seconds_ into the first backfill of an environment and that backfill takes hours
+— without the gate, every reader switches to the timeline at the moment the
+archive is emptiest and then drags the whole backfill through the incremental
+drain (the expensive global scan, at its worst case, for the duration).
+
+Only an explicit `'0'` gates. Migration 0071 sets it for a database that already
+has users, so prod and staging start shut and a fresh environment (local dev,
+e2e, CI) starts open.
+
+```bash
+# Open it — after ingest.pending has trended to ~0.
+npx wrangler d1 execute skyreader --remote --command \
+ "UPDATE sync_state SET value='1', updated_at=unixepoch() WHERE key='timeline_enabled'"
+
+# Shut it — every client is back on the legacy batch path at its next poll.
+npx wrangler d1 execute skyreader --remote --command \
+ "UPDATE sync_state SET value='0', updated_at=unixepoch() WHERE key='timeline_enabled'"
+```
+
+Shutting it is the **fast rollback for the read path**, and the first thing to
+reach for if the timeline misbehaves after a rollout: no Worker deploy, no waiting
+out the 30-minute heartbeat freshness window, and the crawler keeps filling the
+archive the whole time. It is not a fix for a bad Worker deploy generally — only
+for "readers should not be on the timeline right now".
+
+One caveat when shutting it: clients hold a committed `timelineCursor` and stop
+advancing their per-subscription `feedCursors` while on the timeline, so the
+batch path re-drains from wherever those cursors were left. The proxy's K=200
+window bounds that, and the merge dedupes by GUID, so the cost is one heavier
+sync, not duplicates.
+
+---
+
## 5. Post-deploy smoke checks
Every deploy workflow except the admin's ends with `scripts/smoke-check.mjs`:
@@ -509,7 +591,7 @@ admire, and not promises to anyone outside the project. Their only job is to mak
| ------------------ | ------------------------------------ | ----------------------------------- | ------- |
| API availability | 99.5% | Uptime check on `/api/health` | 30 days |
| Firehose freshness | lag < 5 min, p95 of hourly snapshots | `metrics_snapshots.firehose_lag_ms` | 30 days |
-| Feed freshness | ≥95% fresh, p05 of hourly snapshots | `metrics_snapshots.proxy_fresh_pct` | 30 days |
+| Crawl freshness | ≥95% fresh, p05 of hourly snapshots | `metrics_snapshots.proxy_fresh_pct` | 30 days |
| Cron liveness | gap < 5 min | Heartbeat check history | 30 days |
99.5% is ~3.6 hours a month. That is a deliberately loose target for a
@@ -517,6 +599,15 @@ single-operator project on a singleton proxy: it says "a couple of short outages
month is survivable, a daily one is not." Tighten it only if you're also willing
to change the architecture it's measuring.
+**Crawl freshness is the first of two hops, not the whole of feed freshness.**
+`proxy_fresh_pct` says the crawler's own cache is current; since reads moved to D1,
+what a reader actually sees also depends on the push that carries those items into
+`feed_items` (§4d). The second hop has no SLO here because nothing records it
+hourly — the outbox depth lives on the proxy's `/stats` and is read by hand. Making
+it one means recording `ingest.pending` into `metrics_snapshots` beside the numbers
+above; until then, say "crawl freshness" and mean it, rather than claiming a
+reader-facing number this table can't compute.
+
Both freshness rows aggregate the **hourly** trend points, not the live tile: the
tile is a point-in-time reading and "95% fresh right now" says nothing about a
month. p95 for lag and p05 for freshness are the same statement pointed in
@@ -534,7 +625,7 @@ npx wrangler d1 execute skyreader --remote --command "
LIMIT 1 OFFSET (SELECT CAST(COUNT(*) * 95 / 100 AS INT) FROM metrics_snapshots
WHERE captured_at > (unixepoch() - 30*86400) * 1000 AND firehose_lag_ms IS NOT NULL)"
-# Proxy cache freshness, p05 (the 5th-worst-percent hour). Passes if >= 95.
+# Crawl freshness (proxy cache), p05 (the 5th-worst-percent hour). Passes if >= 95.
npx wrangler d1 execute skyreader --remote --command "
SELECT proxy_fresh_pct FROM metrics_snapshots
WHERE captured_at > (unixepoch() - 30*86400) * 1000 AND proxy_fresh_pct IS NOT NULL
diff --git a/docs/plans/D1_FEED_TIMELINE.md b/docs/plans/D1_FEED_TIMELINE.md
new file mode 100644
index 00000000..7d6217ec
--- /dev/null
+++ b/docs/plans/D1_FEED_TIMELINE.md
@@ -0,0 +1,267 @@
+# D1-Served Feed Timeline (ingest-push architecture)
+
+> Supersedes, in part, `RETENTION_SYNC_PLAN.md`. That plan made the Fly proxy a durable item
+> log with a monotonic cursor and it stays exactly as built — but its **serving** role moves
+> to D1. The proxy is now the **crawler and outbox**; D1 is the **archive and read path**.
+
+## What changed
+
+A refresh used to be `1 + ceil((N−8)/50)` sequential `POST /api/v2/feeds/batch` calls, each
+hopping Worker → Fly proxy and then paying `ceil(GUIDs/88)` sequential D1 queries to annotate
+read state (`getReadKeys`). Now:
+
+```
+ ┌────────────── crawl set (pull, every ~5 min) ──────────────┐
+ ▼ │
+ Fly proxy (crawler only) Cloudflare Worker + D1 (serving store)
+ ┌──────────────────────────┐ push deltas ┌──────────────────────────────────────┐
+ │ warm loop / etag / │ ──────────────▶ │ POST /api/internal/ingest │
+ │ backoff / UA fallback │ (at-least-once,│ upsert feeds + feed_items │
+ │ feed_items = outbox │ seq order, │ (archive: no pruning; sanity cap) │
+ │ (bounded, K = 200) │ ONE target) │ │
+ └──────────────────────────┘ │ GET /api/v2/timeline?since_seq=&… │
+ │ ONE query: feed_items ⋈ subs ⋈ read│
+ Client ◀───────────────── one paged request ─┴──────────────────────────────────────┘
+
+ (× 2: one such pair for prod, one for staging — no cross-links)
+```
+
+- **Reads never touch Fly.** If the Fly box is down, refresh still works from D1 (freshness
+ stalls, serving doesn't).
+- **D1 is the archive; the proxy log is a window.** The proxy keeps its K = 200 recent items per
+ feed as a delivery buffer; D1 keeps everything it has ever ingested. Prod D1 is therefore the
+ system of record for feed history — its Time Travel / export story matters.
+- The proxy keeps `/extract`, `/discover`, `/documents`, `/social-context`, `/mentions`, and the
+ single-feed `/feed` (now only for subscribe-time pull-through).
+
+## Shipped in this change
+
+**Backend** (`backend/`)
+
+- Migration `0068_feed_timeline.sql`: drops the dormant pre-Fly trio (`feed_items`, `feed_cache`,
+ `feed_metadata` — zero references in `src/`), creates `feeds` + `feed_items` (AUTOINCREMENT
+ `seq`, `UNIQUE(feed_url, guid)`, `content_hash NOT NULL`), the `(user_did, feed_url)`
+ subscription index, and mints `sync_state.items_generation`.
+- `routes/ingest.ts`: `POST /api/internal/ingest` and `GET /api/internal/crawl-set`, both
+ authenticated by a constant-time compare against `FEED_PROXY_SECRET` and **fail-closed** when
+ it is unset. Idempotent upsert (edit-in-place keeps the seq), per-item content cap
+ (`MAX_ITEM_CONTENT_BYTES = 8 KB`, drops `content` and sets `contentTruncated`), and the per-feed
+ `SANITY_CAP = 5000` trim — the only pruning that ever runs. Both endpoints stamp a **crawler
+ heartbeat** (`sync_state.crawler_heartbeat_at`); the crawl-set pull runs every 5 minutes, so the
+ stamp is fresh even when no feed produced an item.
+- `routes/timeline.ts`: `GET /api/v2/timeline?since_seq=&generation=&limit=&cold_offset=` —
+ incremental drain (cursor derived from returned rows, `hasMore` via `limit+1`) and a **paged**
+ per-feed newest-30 cold start (feeds walked in a stable order, `COLD_START_MAX_ITEMS` per page,
+ continuation via `nextColdOffset`). Read state is an `EXISTS` probe in the same query;
+ `getReadKeys` is never called on the feed path. Every response carries `ingestActive` — the AND of
+ a fresh crawler heartbeat (this deployment has a crawler filling D1) and the `timeline_enabled`
+ rollout gate (an operator has admitted readers to it; see Phase 4). False on either half means
+ clients stay on the legacy batch path, and the request short-circuits rather than building a page
+ they will discard. A cursor above the archive head cold-starts (rewound-archive guard).
+- `routes/ingest.ts` also serves `POST /api/internal/feed-health`: the crawler's periodic report of
+ every feed it currently considers broken, which the timeline hands to readers. On the batch path
+ a failing feed came back with `status: 'error'` inline; reads no longer touch the proxy, and a
+ broken feed pushes no items, so without this a dead feed and a quiet feed look identical to the
+ client. The payload is the **complete trouble set, not a delta** — recovery is inferred from a
+ feed's absence, so nothing has to announce that it started working. `feeds` carries the state
+ (`error_count`, `last_error`, `last_error_at`, `next_retry_at`, `last_fetch_at`, `crawl_stale`,
+ migration `0070`), and a report inserts rather than assumes a row, because a feed broken from its
+ first crawl has never ingested an item. Two distinct faults ride the same report: `error_count`
+ (the fetch fails — readers see this) and `crawl_stale` (the feed is in the crawl set but going
+ unfetched for hours — an operator signal only, invisible to readers and deliberately excluded from
+ `feed_health_rev` so crawl-capacity churn can't make every client re-download the payload).
+- `GET /api/v2/feeds/fetch` re-backed with D1 + **pull-through**: if a feed isn't in the archive
+ yet (first subscriber), fetch it from the proxy once, ingest it, then serve. The pull-through is
+ gated on the caller's own subscription, so the shared never-pruned archive can't be written with
+ arbitrary feeds. Subscribe time already crawls + ingests the feed (`warmFeedIntoArchive`, which
+ replaced the old warm-and-discard), so the pull-through is normally not needed at all. The
+ response carries the feed's `health` when it has any, and a successful pull-through clears it
+ (`clearFeedHealth`) — that path _is_ the user's "retry this feed" action, so it must show a result
+ now rather than after the crawler's next report.
+
+**Proxy** (`feed-proxy/`)
+
+- `ingest-push.ts`: the durable log _is_ the outbox. `push_state(seq, pushed_hash)` marks what has
+ been delivered; a row is dirty when it's missing there or the hashes differ (race-free against a
+ mid-flight edit). Seq-ordered batches of 100 every 15 s, capped exponential backoff on failure.
+- Crawl-set pull every 5 min: registers each feed's `cache` row and stamps `last_requested_at`, so
+ the existing warm loop / active window / eviction machinery keeps working now that read traffic
+ no longer stamps anything.
+- A feed-health report rides the same 5-minute timer, sent right **after** the crawl-set pull so a
+ feed registered this cycle is already in the set. `selectFeedHealth` reads the crawl-set `cache`
+ rows that are either erroring (`error_count > 0`) or unfetched for `CRAWL_STALE_MS` (2 h — the
+ warm loop works on a minutes-long cadence, so hours means the feed is losing its turn every tick),
+ converting its millisecond timestamps to seconds once, at this boundary.
+- `INGEST_URL` unset ⇒ both loops disabled. `push_state` cascades on the K = 200 cap trim, the
+ redelivery delete, and `cleanupCache` eviction. `/stats` reports `ingest.{items,pushed,pending}`.
+
+**Frontend** (`frontend/`)
+
+- `feedFetcher.fetchAllFeeds` prefers one `GET /api/v2/timeline` (plus drain pages) with a single
+ global cursor in Dexie `metadata` (`timelineCursor`). Pure helpers live in `timelineSync.ts`.
+- Falls back to the legacy `/api/v2/feeds/batch` path when the timeline 404s (old or rolled-back
+ backend) **or** whenever the server reports `ingestActive: false`. The cursor is never committed
+ in that case. (An `ingestActive`-less backend keeps the old empty-cold-start heuristic.)
+- A paged cold start commits only the FIRST page's cursor, and only after its last page merges, so
+ an interrupted bootstrap starts over instead of skipping feeds it never delivered.
+- Subscriptions that arrive from another device sit below the global cursor, so each is backfilled
+ once through the per-feed endpoint (`backfillMissingSubscriptions`, ≤ 10 per sync, attempts
+ recorded in Dexie `metadata.timelineBackfilledFeeds`).
+- OPML import backfills via the per-feed endpoint: a freshly imported feed's items sit below the
+ global cursor, so only the single-feed path can deliver them. The requests are paced (3 at a
+ time, 1 s apart) and no longer force a crawl, so a 250-feed import stays inside the rate limit.
+- An article whose body was dropped at ingest (`contentTruncated`) is extracted automatically when
+ its card opens, so the reader shows the whole article rather than an RSS summary.
+- Per-feed error badges come from the response's `feedHealth` (`reconcileFeedHealth` →
+ `feedStatusStore.applyHealthSnapshot`). Absence from that map is what CLEARS an error, so the
+ reconcile runs after the "these feeds delivered items, so they're fine" pass and overrides it — a
+ cold start replays archived items from feeds that may have broken since. To keep the steady-state
+ poll at one query, the payload is only sent when the client's echoed `health_rev` is stale, plus
+ unconditionally on a cold start (whose status store may be empty).
+
+**Admin** (`admin/`) — feed health is re-pointed at `feeds`/`feed_items`: crawled feeds, archived
+item count, estimated archive size with a 6 GB alert, and a churn detector for feeds nearing the
+sanity cap. Health itself is the **crawler's verdict**, not an inference:
+
+- **Subscribed Feeds Erroring** (`error_count > 0`) and **Subscribed Feeds Not Being Crawled**
+ (`crawl_stale = 1`) replace the single "Subscribed Feeds Not Ingesting" tile. That tile keyed off
+ `last_ingest_at`, which only moves when a fetch yields a NEW item — so it counted every feed that
+ simply hadn't published in an hour, warned permanently, and told an operator nothing.
+- The Feeds page filters on All / Erroring / Not Crawled / OK, sorts by `error_count`, and shows the
+ crawler's actual message, failure count, retry time and last good fetch on the row. `last_ingest_at`
+ stays, relabelled "Last Item", as what it really is: publishing cadence.
+- The existing "Proxy Feeds in Error" tile and its trend series are untouched. They count the
+ proxy's whole cache, orphaned feeds included, and stay sourced from `proxy_stats` so the series
+ keeps matching its tile; the new tiles are subscriber-scoped and read D1.
+
+## Operator steps (not code)
+
+### Phase 0 — measure (before enabling prod ingest)
+
+Snapshot prod so the growth math becomes a projection: proxy `feed_items` count and `item_json`
+size distribution (drives backfill time and confirms the 8 KB content cap), per-feed new-item
+velocity over a sample window (items/day, and it flags existing GUID-churn feeds), and current
+refresh request counts from logs (the baseline the win is measured against).
+
+### Phase 3 — provision the staging Fly proxy, then soak
+
+```bash
+# 1. Create the app (same org as skyreader-feed-proxy)
+fly apps create skyreader-feed-proxy-staging
+
+# 2. Create its volume (same region as primary_region; 1 GB is ample for staging)
+fly volumes create proxy_data -a skyreader-feed-proxy-staging -r sjc -s 1
+
+# 3. Mint a DISTINCT staging secret (don't reuse prod's — smaller blast radius)
+openssl rand -hex 32 # →
+fly secrets set PROXY_SECRET= -a skyreader-feed-proxy-staging
+
+# 4. Set the same value on the staging Worker (it is both the outbound proxy auth
+# and the inbound ingest auth)
+cd backend && npx wrangler secret put FEED_PROXY_SECRET --env staging
+
+# 5. First deploy (CI takes over afterwards, on every push to main)
+cd feed-proxy && fly deploy --remote-only --config fly.staging.toml
+```
+
+Sequencing matters: today's staging Worker authenticates to the **prod** proxy with prod's secret,
+so provision + deploy the staging proxy first, then flip `FEED_PROXY_URL` in
+`backend/wrangler.toml` (`[env.staging]`, still pointing at the prod proxy on purpose) and rotate
+`FEED_PROXY_SECRET` together in one staging Worker deploy. The CI staging Fly job skips itself with
+a notice until the app exists, so nothing breaks in the meantime.
+
+Verify: `fly status -a skyreader-feed-proxy-staging` shows exactly **one** machine (the singleton
+invariant applies to this app too — never `fly scale count`); `/stats` answers and its
+`ingest.pending` trends to ~0; staging D1 `feed_items` counts rise monotonically; the prod proxy's
+logs show no staging-origin traffic.
+
+**Then enable prod:** uncomment `INGEST_URL` in `feed-proxy/fly.toml` and cut a release. Prod's
+backfill drains through the normal pusher loop (≤ 200 × active feeds at 100 items / 15 s). Until
+that happens, prod clients keep using the legacy batch path automatically: with no crawler pushing,
+nothing stamps `crawler_heartbeat_at`, so `/api/v2/timeline` answers `ingestActive: false` and no
+client commits a cursor. That signal is server-side on purpose — subscribe-time ingest and the
+pull-through both write to the archive, so "the archive is empty for this user" would stop being
+true long before the crawler existed.
+
+### Phase 4 — open the gate
+
+The heartbeat means a crawler is attached; it does **not** mean the initial archive backfill is
+complete, and the first stamp lands _seconds_ after the release (the proxy pulls the crawl set
+immediately at boot). On its own it would therefore switch every reader onto the timeline at the
+moment the archive is emptiest, and each of them would then drag the entire backfill through the
+incremental drain — the fan-out-on-read scan at its worst case, for the hours the drain takes,
+surfacing back-catalogue items as unread along the way.
+
+`sync_state.timeline_enabled` (migration `0071`) separates the two. `ingestActive` is the AND of a
+fresh heartbeat and this flag, so:
+
+1. Enable `INGEST_URL`; the crawler fills the archive while every client stays on the batch path.
+2. Watch `/stats` until `ingest.pending` trends to ~0.
+3. Open the gate — one `UPDATE sync_state`, no deploy (commands in
+ [`RUNBOOK.md` §4d](../RUNBOOK.md)). Clients switch on their next poll.
+
+Only an explicit `'0'` gates; an absent row means enabled, so a hand-built schema or a future
+environment is never silently held back. The migration writes `'0'` for a database that already has
+users, so prod and staging start shut while local dev, e2e and CI start open.
+
+Setting it back to `'0'` is the **fast rollback for the read path** — every client returns to the
+batch path at its next poll, with no Worker deploy and no waiting out the 30-minute heartbeat
+freshness window, and the crawler keeps ingesting throughout. A gated timeline request
+short-circuits: it answers with the state and an empty page rather than building one the client is
+about to discard, so the gated window costs one `sync_state` read per poll.
+
+### Phase 5 — cleanup (a later release, once no legacy traffic remains)
+
+Remove `/api/v2/feeds/batch`'s proxy passthrough + its `getReadKeys` call (keep it for documents),
+the proxy's `POST /feeds` batch read endpoint, the `since_guids`/`since_seq` client plumbing, Dexie
+`feedCursors`, `liveDb.getRecentGuids`, and `fetchAllFeedsViaBatch`. Optionally add an hourly cron
+step deleting `feeds`/`feed_items` rows whose feed has had **zero active subscribers for > 90 days**
+— the one deliberate deletion path, and skippable if even orphans should stay.
+
+## Invariants worth keeping
+
+- **The incremental cursor comes from returned rows, never `MAX(seq)`** — the latter races ingest
+ and silently skips rows. A cold start is the one exception, and only because it reads the head
+ BEFORE its per-feed slices: anything ingested while it pages lands above that head and arrives on
+ the next poll.
+- **The heartbeat and the gate answer different questions.** "Is a crawler attached?" is not "is
+ the archive ready for readers?" — the first stamp arrives seconds into a backfill that takes
+ hours. Anything that collapses the two back into one signal reintroduces a rollout where every
+ client switches at the emptiest moment and then drains the entire backfill.
+- **Any D1 restore bumps `items_generation`** (one `UPDATE sync_state`): Time Travel rewinds seqs
+ while the token would otherwise stay the same. The timeline also self-heals a cursor that sits
+ above the head by cold-starting that client, so a forgotten bump degrades to one extra cold start
+ rather than a silent, permanent stall.
+- **Per-feed recency comes from `published_at`, then `seq`.** A proxy feed is newest-first, so each
+ ingest call still writes oldest→newest. But subscribe-time pull-through and the crawler backlog
+ can interleave, making archive `seq` different from publication order; per-feed slices must remain
+ correct in that case. Incremental delivery continues to use `seq` as its cursor.
+- **Writes to the archive need a subscription.** Ordinary ingest deletes nothing, so every
+ user-triggered write path (subscribe-time ingest, the pull-through) is gated on the caller's own
+ subscription list.
+- **The pusher sends `cache.url`**, the registered URL, never a post-redirect one: the timeline
+ joins on that exact string.
+- **Ordinary ingest deletes nothing.** A feed at the sanity cap is a bug signal (GUID churn), not
+ steady state — investigate the feed rather than letting it rotate.
+- **A health report is the whole trouble set, never a delta.** Recovery is inferred from absence,
+ so a partial report silently marks feeds healthy. The recovery sweep is a set difference against
+ the currently flagged feeds (small, partially indexed) rather than a `NOT IN` list of every
+ healthy feed, which at ~1,300 feeds is the bound-parameter wall `/batch` already hit once.
+- **`last_ingest_at` is publishing cadence, not health.** It only moves when a fetch produces a new
+ or edited item, so a healthy monthly newsletter looks identical to a feed that 404s. Anything
+ asking "is this feed alive?" has to read `error_count` / `crawl_stale`, which is exactly what the
+ admin's old stale-ingest alarm got wrong.
+- **`nextRetryAt` is milliseconds everywhere the client sees it.** The crawler computes it as
+ `Date.now() + backoff`; D1 stores seconds like the rest of the backend and converts back on the
+ way out. Rescaling it a second time is what put every retry ~50,000 years out and made
+ `canFetch` retire a feed permanently after one transient error.
+
+## Known scaling knob
+
+The incremental drain scans the `feed_items` rowid range above the caller's cursor and probes the
+subscription set per row, so its cost tracks **global** ingest above the cursor rather than the
+caller's own new items: a 5-feed reader returning after a week pays for everything the whole system
+ingested that week. That is the accepted fan-out-on-read trade at ~1,330 feeds. If D1 row-reads or
+CPU ever become the constraint, bound the scan with per-feed `(feed_url, seq)` seeks against the
+subscription set (`idx_feed_items_feed_seq` already supports them) rather than materializing
+per-user timelines.
diff --git a/docs/plans/RETENTION_SYNC_PLAN.md b/docs/plans/RETENTION_SYNC_PLAN.md
index a867484b..d961f86c 100644
--- a/docs/plans/RETENTION_SYNC_PLAN.md
+++ b/docs/plans/RETENTION_SYNC_PLAN.md
@@ -1,5 +1,11 @@
# Durable Item Retention: Catch Everything Since Last Visit
+> **Superseded in part by `D1_FEED_TIMELINE.md`.** Everything below shipped and still runs — but
+> the durable log's *serving* role has moved to D1. The proxy log is now the crawler's **outbox**
+> (a bounded K = 200 delivery window pushed into D1); the **archive** clients read from is
+> `feed_items` in D1, which never prunes, and the cursor clients hold is a single global one
+> against that table rather than one per feed.
+
> Supersedes `BATCH_CURSOR_PLAN.md`. That plan optimized the *request payload* (500 GUIDs →
> a cursor) but kept the proxy's replace-the-blob storage model, which structurally cannot
> retain items beyond the source feed's live window. This plan changes the storage model so
diff --git a/e2e/seed.ts b/e2e/seed.ts
index 8f2f77cc..13d72c5f 100644
--- a/e2e/seed.ts
+++ b/e2e/seed.ts
@@ -136,6 +136,65 @@ export async function seedSavedArticle(
return rkey;
}
+export interface SeedFeedItemOpts {
+ guid: string;
+ title: string;
+ url?: string;
+ publishedAt?: string;
+ summary?: string;
+}
+
+/**
+ * Seed the server-side archive the timeline serves from: a `feeds` row plus its
+ * `feed_items`. Feed-scoped rather than user-scoped (the archive is shared by
+ * every subscriber), so it's cleaned up with `cleanupFeedItems`.
+ */
+export async function seedFeedItems(
+ feedUrl: string,
+ items: SeedFeedItemOpts[],
+ opts: { title?: string; siteUrl?: string } = {}
+): Promise {
+ const nowMs = Date.now();
+ const nowSeconds = Math.floor(nowMs / 1000);
+
+ const statements = [
+ `INSERT OR REPLACE INTO feeds (feed_url, title, site_url, last_ingest_at, created_at) VALUES (${sqlString(feedUrl)}, ${sqlNullableString(opts.title ?? null)}, ${sqlNullableString(opts.siteUrl ?? null)}, ${nowSeconds}, ${nowSeconds})`,
+ // Stand in for the crawler's check-in. Without a fresh heartbeat the timeline
+ // reports `ingestActive: false` and the client (correctly) stays on the legacy
+ // batch path, which is not what these tests are exercising.
+ `INSERT INTO sync_state (key, value, updated_at) VALUES ('crawler_heartbeat_at', ${sqlString(String(nowSeconds))}, ${nowSeconds}) ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
+ // The other half of the same gate: a heartbeat says a crawler is attached,
+ // this says readers may use what it wrote. Migration 0071 leaves it open on a
+ // fresh database, but a test DB that predates the migration with users in it
+ // starts closed — so set it rather than depend on which case this one is.
+ `INSERT INTO sync_state (key, value, updated_at) VALUES ('timeline_enabled', '1', ${nowSeconds}) ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
+ ];
+
+ items.forEach((item, index) => {
+ const publishedAt = item.publishedAt ?? new Date(nowMs - index * 60_000).toISOString();
+ const itemJson = JSON.stringify({
+ guid: item.guid,
+ url: item.url ?? `https://example.com/${item.guid}`,
+ title: item.title,
+ summary: item.summary ?? `Summary for ${item.title}`,
+ publishedAt,
+ });
+ statements.push(
+ `INSERT OR REPLACE INTO feed_items (feed_url, guid, item_json, published_at, first_seen_at, content_hash) VALUES (${sqlString(feedUrl)}, ${sqlString(item.guid)}, ${sqlString(itemJson)}, ${Date.parse(publishedAt)}, ${nowMs}, ${sqlString(`hash-${item.guid}`)})`
+ );
+ });
+
+ await execD1(statements);
+}
+
+export async function cleanupFeedItems(feedUrl: string): Promise {
+ await execD1([
+ `DELETE FROM feed_items WHERE feed_url = ${sqlString(feedUrl)}`,
+ `DELETE FROM feeds WHERE feed_url = ${sqlString(feedUrl)}`,
+ `DELETE FROM sync_state WHERE key = 'crawler_heartbeat_at'`,
+ ]);
+}
+
export interface SeedItemLabelOpts {
itemKey: string;
itemType: string;
diff --git a/e2e/timeline.spec.ts b/e2e/timeline.spec.ts
new file mode 100644
index 00000000..124efbe1
--- /dev/null
+++ b/e2e/timeline.spec.ts
@@ -0,0 +1,105 @@
+import { test, expect } from './fixtures';
+import { seedSubscription, seedFeedItems, seedItemLabel, cleanupFeedItems } from './seed';
+
+/**
+ * The D1-served timeline: a refresh is ONE request that already carries read
+ * state, instead of a per-feed batch fan-out against the Fly proxy.
+ */
+test.describe('Timeline refresh', () => {
+ const FEED_URL = 'https://example.com/timeline-feed.xml';
+ const FEED_TITLE = 'Timeline Feed';
+
+ test.afterEach(async () => {
+ await cleanupFeedItems(FEED_URL);
+ });
+
+ test('renders archived items from a single timeline request, with read state', async ({
+ authedPage,
+ testUser,
+ }) => {
+ await seedSubscription(testUser, { feedUrl: FEED_URL, title: FEED_TITLE });
+ await seedFeedItems(
+ FEED_URL,
+ [
+ { guid: 'timeline-item-1', title: 'Archived Article One' },
+ { guid: 'timeline-item-2', title: 'Archived Article Two' },
+ ],
+ { title: FEED_TITLE, siteUrl: 'https://example.com' }
+ );
+ // Read on another device: the timeline join must stamp it inline.
+ await seedItemLabel(testUser, {
+ itemKey: 'timeline-item-1',
+ itemType: 'article',
+ label: 'read',
+ });
+
+ const timelineRequests: string[] = [];
+ const batchRequests: string[] = [];
+ authedPage.on('request', (request) => {
+ const url = request.url();
+ if (url.includes('/api/v2/timeline')) timelineRequests.push(url);
+ if (url.includes('/api/v2/feeds/batch')) batchRequests.push(url);
+ });
+
+ const timelineResponsePromise = authedPage.waitForResponse((response) =>
+ response.url().includes('/api/v2/timeline')
+ );
+ await authedPage.reload();
+ const timelineResponse = await timelineResponsePromise;
+ expect(timelineResponse.ok()).toBe(true);
+ const timelineBody = (await timelineResponse.json()) as {
+ items: Array<{ title?: string }>;
+ };
+ expect(timelineBody.items.map((item) => item.title)).toEqual(
+ expect.arrayContaining(['Archived Article One', 'Archived Article Two'])
+ );
+
+ // Both articles land in IndexedDB. Assert the durable client merge directly:
+ // a read article may be hidden by the reader's current filter, which made the
+ // old visibility assertion depend on incidental UI state.
+ await expect
+ .poll(async () => {
+ return authedPage.evaluate(async () => {
+ const request = indexedDB.open('skyreader');
+ const database: IDBDatabase = await new Promise((resolve, reject) => {
+ request.onsuccess = () => resolve(request.result);
+ request.onerror = () => reject(request.error);
+ });
+ const store = database.transaction('articles', 'readonly').objectStore('articles');
+ const all: Array<{ title: string }> = await new Promise((resolve, reject) => {
+ const req = store.getAll();
+ req.onsuccess = () => resolve(req.result);
+ req.onerror = () => reject(req.error);
+ });
+ return all.map((row) => row.title);
+ });
+ })
+ .toEqual(expect.arrayContaining(['Archived Article One', 'Archived Article Two']));
+
+ // The refresh is a single timeline call (at most one extra drain page), and
+ // the legacy per-feed fan-out never runs — the point of the architecture.
+ expect(timelineRequests.length).toBeGreaterThanOrEqual(1);
+ expect(timelineRequests.length).toBeLessThanOrEqual(2);
+ expect(batchRequests.length).toBe(0);
+
+ // Read state arrived with the articles (no separate read fetch).
+ const readGuids = await authedPage.evaluate(async () => {
+ const request = indexedDB.open('skyreader');
+ const database: IDBDatabase = await new Promise((resolve, reject) => {
+ request.onsuccess = () => resolve(request.result);
+ request.onerror = () => reject(request.error);
+ });
+ const store = database.transaction('itemLabels', 'readonly').objectStore('itemLabels');
+ const all: Array<{ itemKey: string; label: string }> = await new Promise(
+ (resolve, reject) => {
+ const req = store.getAll();
+ req.onsuccess = () => resolve(req.result);
+ req.onerror = () => reject(req.error);
+ }
+ );
+ return all.filter((row) => row.label === 'read').map((row) => row.itemKey);
+ });
+ expect(readGuids).toContain('timeline-item-1');
+ expect(readGuids).not.toContain('timeline-item-2');
+ });
+});
diff --git a/feed-proxy/README.md b/feed-proxy/README.md
index 583f32c2..bc6668a8 100644
--- a/feed-proxy/README.md
+++ b/feed-proxy/README.md
@@ -245,21 +245,6 @@ Cache statistics (requires authentication).
"fresh": 80,
"stale": 50,
"inFlight": 2,
- "extract": { "inUse": 1, "queued": 0 },
- "constellation": {
- "breakerOpen": false,
- "consecutiveFailures": 0,
- "inUse": 2,
- "queued": 0,
- "requests": 8421,
- "resets": 37,
- "retries": 37,
- "retriesRecovered": 36,
- "failures": 1,
- "shed": 0,
- "breakerOpens": 0,
- "shortCircuited": 0
- },
"cacheTtlSeconds": 900,
"staleTtlSeconds": 3600,
"errors": {
@@ -303,27 +288,48 @@ curl "/feed?url=...&since_guids=old-guid&limit=50"
## Configuration
-| Environment Variable | Default | Description |
-| -------------------- | -------- | ----------------------------------------- |
-| `PROXY_SECRET` | (none) | Shared secret for `X-Proxy-Secret` header |
-| `DATA_DIR` | `./data` | SQLite database location |
-| `CACHE_TTL_SECONDS` | `900` | Fresh cache duration (15 min) |
-| `STALE_TTL_SECONDS` | `3600` | Stale cache max age (1 hour) |
-| `PORT` | `3000` | HTTP server port |
-| `SENTRY_DSN` | (none) | Error reporting; unset ⇒ no-op |
-| `WARM_HEARTBEAT_URL` | (none) | Dead-man ping after each warm tick |
-| `GIT_COMMIT_SHA` | `dev` | Build stamp, reported by `/health` |
-
-Observability setup, alert thresholds, and incident procedures live in
-[`docs/RUNBOOK.md`](../docs/RUNBOOK.md).
-
-Constellation (mentions, social context, linkblog registry):
-
-| Environment Variable | Default | Description |
-| --------------------------- | ------- | ------------------------------------------------- |
-| `CONSTELLATION_CONCURRENCY` | `3` | Concurrent requests allowed against Constellation |
-| `CONSTELLATION_QUEUE_MAX` | `200` | Callers that may queue before requests are shed |
-| `WARM_MENTIONS` | `true` | Set `false` to stop pre-warming mentions entirely |
+| Environment Variable | Default | Description |
+| ---------------------------- | -------- | ------------------------------------------------------- |
+| `PROXY_SECRET` | (none) | Shared secret for `X-Proxy-Secret` (inbound + outbound) |
+| `DATA_DIR` | `./data` | SQLite database location |
+| `CACHE_TTL_SECONDS` | `900` | Fresh cache duration (15 min) |
+| `STALE_TTL_SECONDS` | `3600` | Stale cache max age (1 hour) |
+| `PORT` | `3000` | HTTP server port |
+| `INGEST_URL` | (none) | Paired Worker base URL. **Unset ⇒ ingest disabled** |
+| `INGEST_INTERVAL_SECONDS` | `15` | Push cycle |
+| `INGEST_BATCH_SIZE` | `100` | Items per push request |
+| `CRAWL_SET_INTERVAL_SECONDS` | `300` | How often to pull the crawl set |
+| `SENTRY_DSN` | (none) | Error reporting; unset ⇒ no-op |
+| `WARM_HEARTBEAT_URL` | (none) | Dead-man ping after each successful warm tick |
+| `GIT_COMMIT_SHA` | `dev` | Build stamp, reported by `/health` |
+
+Observability setup and incident procedures live in [`docs/RUNBOOK.md`](../docs/RUNBOOK.md).
+
+## Ingest push (crawler mode)
+
+With `INGEST_URL` set, this proxy stops being a read path for the reader and becomes the crawler
+for exactly ONE Worker + D1 pair (prod proxy → prod Worker, staging proxy → staging Worker):
+
+- **Push:** the durable item log _is_ the outbox. `push_state(seq, pushed_hash)` records what has
+ reached D1; a row is dirty when it's missing there or its `content_hash` has changed since. Dirty
+ rows drain in seq order to `POST {INGEST_URL}/api/internal/ingest`; delivery is at-least-once and
+ the Worker's upsert is idempotent. Failures back off and retry the same rows.
+- **Pull:** every `CRAWL_SET_INTERVAL_SECONDS` the proxy fetches
+ `GET {INGEST_URL}/api/internal/crawl-set` and stamps `last_requested_at` on each feed, which is
+ what keeps the warm loop working now that reads no longer touch this box.
+- **Health:** immediately after each crawl-set pull, the proxy posts every crawl-set feed with
+ something wrong with it to `POST {INGEST_URL}/api/internal/feed-health` — erroring
+ (`error_count > 0`), starved (not fetched in `CRAWL_STALE_MS`, 2 h), or both. The erroring ones
+ are how a reader learns its feed is dead rather than idle, since reads no longer come through here
+ and a failing feed is otherwise indistinguishable from one that hasn't published; the starved ones
+ are the admin's alarm that this box can't keep up with its crawl set. The payload is the
+ **complete** trouble set: the Worker infers recovery from a feed's absence, so a partial report
+ silently marks feeds healthy. Timestamps are converted from this box's milliseconds to seconds
+ here, once.
+- The per-feed cap (`FEED_ITEMS_CAP = 200`) bounds the **outbox**, not the archive: D1 retains
+ everything it has ingested. `push_state` cascades with every `feed_items` delete.
+
+See `docs/plans/D1_FEED_TIMELINE.md`.
## Cache Behavior
@@ -374,37 +380,6 @@ When a feed is in backoff:
3. After backoff expires, a single fetch is attempted
4. On success, error tracking resets to zero
-### Constellation Client
-
-Every Constellation caller (mentions, social context, mention lanes, the linkblog
-registry) shares one client with three defenses, since they all hit one
-small community-run host:
-
-1. **Circuit breaker** — 5 consecutive failing calls (timeout, network, 5xx/429)
- open it for 30 s; calls short-circuit to `null` instead of each eating the
- 10 s timeout. A clean 4xx is a healthy "no data" and does not count. The check
- runs again after a call gets its concurrency permit, so callers that were
- queued when the breaker opened short-circuit too.
-2. **Concurrency cap** — `CONSTELLATION_CONCURRENCY` requests in flight, the rest
- queued up to `CONSTELLATION_QUEUE_MAX` and then shed to `null`. Shedding is our
- own backpressure, so it does **not** count toward the breaker. Keep this low:
- it governs how many sockets we hold open, and resets scale with socket churn
- (measured reset rate over a fixed request count — 1 concurrent: 2%, 2: 0%,
- 4: 8%, 6: 8%, 12: 14%).
-3. **One retry on connection resets** — `ECONNRESET` / "socket connection was
- closed unexpectedly" is retried once after ~250 ms. Timeouts are not retried.
- A reset-then-reset call counts as exactly one breaker failure. What resets is
- connection _setup_, not idle keep-alive reuse: a warm pooled socket served 47
- of 48 sequential requests cleanly, while forcing a fresh connection per request
- (`keepalive: false` / `Connection: close`) reset 60-76% of the time. So fewer,
- longer-lived sockets is the fix; disabling keep-alive makes it far worse, and
- some residual failure remains because a retry also has to open a socket.
-
-Everything degrades to `null` (never throws); mentions and social adornments
-simply render empty. `GET /stats` reports `constellation`: breaker state, gate
-occupancy, and `resets` / `retriesRecovered` / `failures` / `shed` counters —
-`retriesRecovered` ≫ `failures` means resets are being absorbed.
-
### Error Response in Bulk Endpoint
The `/feeds` endpoint includes error information even when returning cached data:
@@ -448,6 +423,11 @@ The `/stats` endpoint includes error statistics:
### Fly.io
+Two apps, one per environment — `skyreader-feed-proxy` (prod, `fly.toml`) and
+`skyreader-feed-proxy-staging` (staging, `fly.staging.toml`). Each pushes into its own Worker's D1
+and holds its own `PROXY_SECRET`; keep the two config files in sync apart from `app` and
+`INGEST_URL`. Each app runs exactly ONE machine (see the singleton invariant in `fly.toml`).
+
```bash
# Create app
fly apps create skyreader-feed-proxy
@@ -455,7 +435,7 @@ fly apps create skyreader-feed-proxy
# Set secret
fly secrets set PROXY_SECRET=your-secret-here
-# Deploy
+# Deploy (staging: add --config fly.staging.toml)
fly deploy
```
diff --git a/feed-proxy/fly.staging.toml b/feed-proxy/fly.staging.toml
new file mode 100644
index 00000000..c234a55d
--- /dev/null
+++ b/feed-proxy/fly.staging.toml
@@ -0,0 +1,55 @@
+app = "skyreader-feed-proxy-staging"
+primary_region = "sjc"
+
+# KEEP IN SYNC WITH fly.toml. These two files must differ only in `app` and the
+# `INGEST_URL` in [env]; everything else (mounts, http_service, vm, tuning) is
+# deliberately identical so staging is a faithful soak of what prod will run.
+# Staging deploys on every push to main, prod on release — so drift shows up here
+# first, before a release ships it.
+#
+# SINGLETON INVARIANT — run exactly ONE machine. This app is not horizontally
+# scalable as written: the cache is a SQLite DB on the per-machine `proxy_data`
+# volume (volumes can't be shared between machines), and the Jetstream firehose,
+# the self-warming loop, the ingest pusher, and the in-memory request-coalescing
+# maps are all in-process singletons. A second machine would get its own empty
+# volume + its own firehose + its own warm loop, with no shared coalescing — i.e.
+# a split-brain cache, duplicate upstream fetches, duplicate pushes, and responses
+# that differ by which machine you hit. Scale VERTICALLY via the [vm] block below,
+# never by `fly scale count`. Keep only one machine provisioned; auto_start/stop
+# below toggle that one machine, they do not create new ones.
+
+[build]
+
+[mounts]
+ source = "proxy_data"
+ destination = "/data"
+
+[http_service]
+ internal_port = 3000
+ force_https = true
+ auto_stop_machines = true
+ auto_start_machines = true
+ # Keep one machine alive: the self-warming loop and the ingest pusher run
+ # in-process on timers and can only work while a machine is up.
+ min_machines_running = 1
+
+[vm]
+ cpu_kind = "shared"
+ cpus = 2
+ memory_mb = 1024
+
+[env]
+ DATA_DIR = "/data"
+ # This proxy pushes its item log into the STAGING Worker's D1 and pulls the
+ # staging crawl set from it. One proxy, one Worker — never both environments.
+ INGEST_URL = "https://api-staging.skyreader.app"
+ CACHE_TTL_SECONDS = "300"
+ STALE_TTL_SECONDS = "3600"
+ WARM_ENABLED = "true"
+ WARM_INTERVAL_SECONDS = "60"
+ WARM_ACTIVE_WINDOW_SECONDS = "1209600"
+ WARM_BATCH_CAP = "500"
+ WARM_CONCURRENCY = "16"
+ WARM_MENTIONS = "false"
+ EXTRACT_CONCURRENCY = "4"
+ EXTRACT_QUEUE_MAX = "20"
diff --git a/feed-proxy/fly.toml b/feed-proxy/fly.toml
index 8db17795..f3cc6663 100644
--- a/feed-proxy/fly.toml
+++ b/feed-proxy/fly.toml
@@ -1,6 +1,10 @@
app = "skyreader-feed-proxy"
primary_region = "sjc"
+# KEEP IN SYNC WITH fly.staging.toml. The two files must differ only in `app` and
+# the `INGEST_URL` in [env]; everything else is deliberately identical so staging
+# soaks what prod will run.
+#
# SINGLETON INVARIANT — run exactly ONE machine. This app is not horizontally
# scalable as written: the cache is a SQLite DB on the per-machine `proxy_data`
# volume (volumes can't be shared between machines), and the Jetstream firehose,
@@ -35,6 +39,11 @@ primary_region = "sjc"
[env]
DATA_DIR = "/data"
+ # Ingest push into the prod Worker's D1. Commented out until the staging pair
+ # has soaked (see docs/plans/D1_FEED_TIMELINE.md, Phase 3): with it unset this
+ # machine crawls exactly as before and pushes nothing. Enabling it is this one
+ # line plus a release — prod's backfill then drains through the normal pusher.
+ # INGEST_URL = "https://api.skyreader.app"
# Fresh window. The self-warming loop refreshes active feeds before they reach
# this age, so user requests stay HITs even at a tight 5-minute freshness.
CACHE_TTL_SECONDS = "300"
diff --git a/feed-proxy/src/app.ts b/feed-proxy/src/app.ts
index ce5a31fd..7ea87db0 100644
--- a/feed-proxy/src/app.ts
+++ b/feed-proxy/src/app.ts
@@ -64,6 +64,9 @@ export interface AppConfig {
// Defaults to "unhealthy / nothing subscribed" so behavior is unchanged when
// the firehose isn't wired in.
getFirehoseStatus?: () => FirehoseStatus;
+ // Whether this proxy pushes its item log into a paired Worker's D1 (INGEST_URL
+ // is set). Reported by /stats; the loops themselves live in index.ts.
+ ingestEnabled?: boolean;
}
export interface CacheRow {
@@ -607,12 +610,20 @@ export function writeFeedItems(
// delete) instead of an implicit commit per statement — a single fsync rather
// than ~100 every warm-refresh per feed. Single-writer SQLite makes it safe.
const deleteForRedelivery = db.query('DELETE FROM feed_items WHERE url_hash = ? AND guid = ?');
+ // Cascade for the two deletes below. push_state is keyed by seq, so a seq that
+ // disappears from feed_items would otherwise leave an orphan row forever.
+ const dropPushStateForItem = db.query(
+ 'DELETE FROM push_state WHERE seq IN (SELECT seq FROM feed_items WHERE url_hash = ? AND guid = ?)'
+ );
const writeBatch = db.transaction(() => {
if (redeliver) {
// Keep retained history that has aged out of the source feed, but remove
// every item in this parse so it receives a fresh seq and reaches clients
// that already consumed its pre-upgrade representation.
- for (const item of items) deleteForRedelivery.run(urlHash, item.guid);
+ for (const item of items) {
+ dropPushStateForItem.run(urlHash, item.guid);
+ deleteForRedelivery.run(urlHash, item.guid);
+ }
}
for (let i = items.length - 1; i >= 0; i--) {
@@ -631,6 +642,20 @@ export function writeFeedItems(
// Cheaper-than-anti-join cap: compute the (K+1)-th-newest seq once, range-delete
// below it. The OFFSET subquery yields nothing when the feed has <= K rows, so
// `seq <= NULL` matches nothing — a no-op.
+ //
+ // NOTE: this cap bounds the *outbox*, not the archive. Items trimmed here that
+ // already reached D1 stay there permanently (D1 does not prune); items trimmed
+ // before ever being pushed simply drop out of the dirty set — the same K-window
+ // bound the proxy has always had.
+ db.run(
+ `DELETE FROM push_state
+ WHERE seq IN (
+ SELECT seq FROM feed_items
+ WHERE url_hash = ?
+ AND seq <= (SELECT seq FROM feed_items WHERE url_hash = ? ORDER BY seq DESC LIMIT 1 OFFSET ?)
+ )`,
+ [urlHash, urlHash, FEED_ITEMS_CAP]
+ );
db.run(
`DELETE FROM feed_items
WHERE url_hash = ?
@@ -835,6 +860,18 @@ export function initDatabase(db: Database): void {
db.run(`CREATE INDEX IF NOT EXISTS idx_feed_items_feed_seq ON feed_items(url_hash, seq)`);
db.run(`CREATE INDEX IF NOT EXISTS idx_feed_items_first_seen ON feed_items(first_seen_at)`);
+ // Ingest delivery state (ingest-push.ts): which seqs, at which content hash,
+ // have reached the paired Worker's D1. A row is dirty when it's missing here or
+ // the hashes differ — race-free against a concurrent edit. Created
+ // unconditionally, even with pushing disabled, so the cascade deletes that keep
+ // this table bounded (writeFeedItems, cleanupCache) always have a table to hit.
+ db.run(`
+ CREATE TABLE IF NOT EXISTS push_state (
+ seq INTEGER PRIMARY KEY,
+ pushed_hash TEXT NOT NULL
+ )
+ `);
+
// Extracted article content (Defuddle output), keyed by source URL. Separate
// from the feed cache: article content is effectively immutable per URL, so it
// has its own long TTL and is not touched by the feed self-warming loop.
@@ -1855,11 +1892,37 @@ export function createApp(db: Database, config: AppConfig) {
>('SELECT COUNT(*) as count FROM cache WHERE next_retry_at > ? AND error_count >= 5')
.get(now + 6 * 24 * 60 * 60 * 1000); // More than 6 days means it's a permanent error
+ // Ingest push (see ingest-push.ts). `pending` is the outbox backlog — the
+ // number that should hover near zero once a push cycle keeps up.
+ const itemCount = db
+ .query<{ count: number }, []>('SELECT COUNT(*) as count FROM feed_items')
+ .get();
+ const pushedCount = db
+ .query<{ count: number }, []>('SELECT COUNT(*) as count FROM push_state')
+ .get();
+ // Inlined rather than imported from ingest-push.ts, which imports this
+ // module (hashUrl/itemContentHash) — keep the dependency one-directional.
+ const pendingCount = db
+ .query<{ count: number }, []>(
+ `SELECT COUNT(*) AS count
+ FROM feed_items fi
+ JOIN cache c ON c.url_hash = fi.url_hash
+ LEFT JOIN push_state ps ON ps.seq = fi.seq
+ WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '')`
+ )
+ .get();
+
return c.json({
total: total?.count || 0,
fresh: fresh?.count || 0,
stale: stale?.count || 0,
inFlight: inFlight.size,
+ ingest: {
+ enabled: config.ingestEnabled ?? false,
+ items: itemCount?.count || 0,
+ pushed: pushedCount?.count || 0,
+ pending: pendingCount?.count || 0,
+ },
extract: { inUse: extractSemaphore.inUse, queued: extractSemaphore.queued },
// Constellation health from *our* side: breaker state, gate occupancy, and
// the retry/shed counters. `retriesRecovered` ≫ `failures` means the
@@ -2772,6 +2835,13 @@ export function cleanupCache(db: Database): number {
// cache row goes (the delete keys off cache.fetched_at). Once the feed is cold
// the retained items are unreachable anyway, and this bounds feed_items growth
// to the active working set.
+ db.run(
+ `DELETE FROM push_state WHERE seq IN (
+ SELECT seq FROM feed_items
+ WHERE url_hash IN (SELECT url_hash FROM cache WHERE fetched_at < ?)
+ )`,
+ [threshold]
+ );
db.run(
'DELETE FROM feed_items WHERE url_hash IN (SELECT url_hash FROM cache WHERE fetched_at < ?)',
[threshold]
diff --git a/feed-proxy/src/index.ts b/feed-proxy/src/index.ts
index e233b570..1c8d398f 100644
--- a/feed-proxy/src/index.ts
+++ b/feed-proxy/src/index.ts
@@ -6,6 +6,7 @@ import { mkdirSync } from 'fs';
import { createApp, initDatabase, cleanupCache } from './app';
import { DocumentFirehose } from './jetstream';
import { pingHeartbeat } from './heartbeat';
+import { pushDirtyItems, pullCrawlSet, reportFeedHealth, type IngestConfig } from './ingest-push';
// Config
const PROXY_SECRET = process.env.PROXY_SECRET;
@@ -51,6 +52,20 @@ const WARM_MENTIONS_ENABLED = (process.env.WARM_MENTIONS ?? 'true') !== 'false';
// local dev, so the ping is a no-op there.
const WARM_HEARTBEAT_URL = process.env.WARM_HEARTBEAT_URL;
+// This proxy crawls for one Worker/D1 pair. Leaving INGEST_URL unset disables
+// both directions, which keeps local development and staged rollout safe.
+const INGEST_URL = process.env.INGEST_URL?.replace(/\/$/, '') || '';
+const INGEST_ENABLED = INGEST_URL.length > 0;
+const INGEST_INTERVAL_MS = parseInt(process.env.INGEST_INTERVAL_SECONDS || '15', 10) * 1000;
+const INGEST_BATCH_SIZE = parseInt(process.env.INGEST_BATCH_SIZE || '100', 10);
+const CRAWL_SET_INTERVAL_MS = parseInt(process.env.CRAWL_SET_INTERVAL_SECONDS || '300', 10) * 1000;
+const PUSH_BACKOFF_BASE_MS = 30 * 1000;
+const PUSH_BACKOFF_MAX_MS = 10 * 60 * 1000;
+
+function pushBackoff(failures: number): number {
+ return Math.min(PUSH_BACKOFF_BASE_MS * 2 ** (failures - 1), PUSH_BACKOFF_MAX_MS);
+}
+
// /extract is the heaviest request (fetch + Defuddle DOM build). Cap concurrent
// extractions so a burst of distinct heavy articles can't OOM the 512MB machine;
// excess callers queue, then are shed with a 503 once the queue fills.
@@ -96,6 +111,7 @@ const { app, warmStaleFeeds, warmStaleDocuments } = createApp(db, {
extractConcurrency: EXTRACT_CONCURRENCY,
extractQueueMax: EXTRACT_QUEUE_MAX,
getFirehoseStatus: () => firehose?.status() ?? { healthy: false, isSubscribed: () => false },
+ ingestEnabled: INGEST_ENABLED,
});
// Document firehose: push-based freshness for standard.site documents.
@@ -158,6 +174,70 @@ if (WARM_ENABLED) {
console.log('[Proxy] Warmer: disabled');
}
+// Push the durable item log into D1 and pull the registered crawl set back.
+if (INGEST_ENABLED) {
+ const ingestConfig: IngestConfig = {
+ ingestUrl: INGEST_URL,
+ secret: PROXY_SECRET,
+ batchSize: INGEST_BATCH_SIZE,
+ };
+ let pushRunning = false;
+ let pushFailures = 0;
+ let pushBlockedUntil = 0;
+ setInterval(() => {
+ if (pushRunning || Date.now() < pushBlockedUntil) return;
+ pushRunning = true;
+ pushDirtyItems(db, ingestConfig)
+ .then((result) => {
+ if (result.error) {
+ pushFailures++;
+ pushBlockedUntil = Date.now() + pushBackoff(pushFailures);
+ console.error(`[Proxy] Ingest push failed (${pushFailures}): ${result.error}`);
+ } else {
+ pushFailures = 0;
+ if (result.pushed > 0) console.log(`[Proxy] Ingest pushed ${result.pushed} item(s)`);
+ }
+ })
+ .catch((error) => {
+ console.error('[Proxy] Ingest push error:', error);
+ reportError(error, { tags: { source: 'ingest-push' } });
+ })
+ .finally(() => {
+ pushRunning = false;
+ });
+ }, INGEST_INTERVAL_MS);
+
+ let crawlSetRunning = false;
+ const refreshCrawlSet = () => {
+ if (crawlSetRunning) return;
+ crawlSetRunning = true;
+ pullCrawlSet(db, ingestConfig)
+ .then((result) => {
+ if (result.error) console.error(`[Proxy] Crawl-set pull failed: ${result.error}`);
+ else console.log(`[Proxy] Crawl set: ${result.registered} feed(s) registered`);
+ // Report health AFTER the pull, so a feed registered for the first time
+ // this cycle is already in the crawl set and its errors are reportable.
+ // Reads no longer pass through here, so this is the only way a broken
+ // feed reaches the reader's error badge.
+ return reportFeedHealth(db, ingestConfig).then((health) => {
+ if (health.error) console.error(`[Proxy] Feed-health report failed: ${health.error}`);
+ else console.log(`[Proxy] Feed health: ${health.reported} feed(s) in error`);
+ });
+ })
+ .catch((error) => {
+ console.error('[Proxy] Crawl-set pull error:', error);
+ reportError(error, { tags: { source: 'crawl-set' } });
+ })
+ .finally(() => {
+ crawlSetRunning = false;
+ });
+ };
+ refreshCrawlSet();
+ setInterval(refreshCrawlSet, CRAWL_SET_INTERVAL_MS);
+} else {
+ console.log('[Proxy] Ingest push: disabled (INGEST_URL unset)');
+}
+
// Flush the firehose cursor + close its socket cleanly on shutdown so we resume
// where we left off instead of replaying.
for (const signal of ['SIGTERM', 'SIGINT'] as const) {
diff --git a/feed-proxy/src/ingest-push.test.ts b/feed-proxy/src/ingest-push.test.ts
new file mode 100644
index 00000000..c3ca58e3
--- /dev/null
+++ b/feed-proxy/src/ingest-push.test.ts
@@ -0,0 +1,461 @@
+import { describe, expect, it, beforeEach, afterEach, spyOn } from 'bun:test';
+import { Database } from 'bun:sqlite';
+import {
+ initDatabase,
+ hashUrl,
+ writeFeedItems,
+ cleanupCache,
+ itemContentHash,
+ FEED_ITEMS_CAP,
+} from './app';
+import {
+ pushDirtyItems,
+ pullCrawlSet,
+ registerCrawlFeeds,
+ reportFeedHealth,
+ CRAWL_STALE_MS,
+ selectDirtyRows,
+ selectFeedHealth,
+ countDirtyRows,
+ type IngestConfig,
+} from './ingest-push';
+import type { FeedItem } from './types';
+
+const FEED_URL = 'https://example.com/feed.xml';
+const URL_HASH = hashUrl(FEED_URL);
+
+const CONFIG: IngestConfig = {
+ ingestUrl: 'https://api.example',
+ secret: 'test-secret',
+ batchSize: 2,
+};
+
+function item(guid: string, overrides: Partial = {}): FeedItem {
+ return {
+ guid,
+ url: `https://example.com/${guid}`,
+ title: `Title ${guid}`,
+ publishedAt: '2026-01-01T00:00:00.000Z',
+ ...overrides,
+ };
+}
+
+function seedCache(db: Database, url = FEED_URL, feedTitle = 'Test Blog'): void {
+ db.run(
+ `INSERT INTO cache (url_hash, url, parsed_json, parser_version, cached_at, fetched_at, last_requested_at)
+ VALUES (?, ?, ?, 1, ?, ?, ?)`,
+ [
+ hashUrl(url),
+ url,
+ JSON.stringify({
+ title: feedTitle,
+ siteUrl: 'https://example.com',
+ description: 'A test blog',
+ items: [],
+ }),
+ Date.now(),
+ Date.now(),
+ Date.now(),
+ ]
+ );
+}
+
+interface CapturedRequest {
+ url: string;
+ headers: Record;
+ body: {
+ feeds: Array<{ feedUrl: string; title?: string | null; siteUrl?: string | null }>;
+ items: Array<{ feedUrl: string; guid: string; contentHash: string; item: FeedItem }>;
+ };
+}
+
+// Capture what the pusher sends and control the Worker's reply.
+function mockIngestEndpoint(status = 200): { calls: CapturedRequest[]; restore: () => void } {
+ const calls: CapturedRequest[] = [];
+ const spy = spyOn(globalThis, 'fetch').mockImplementation((async (
+ input: string,
+ init?: RequestInit
+ ) => {
+ calls.push({
+ url: String(input),
+ headers: (init?.headers ?? {}) as Record,
+ body: init?.body ? JSON.parse(String(init.body)) : { feeds: [], items: [] },
+ });
+ return new Response(JSON.stringify({ ok: status === 200 }), { status });
+ }) as unknown as typeof fetch);
+ return { calls, restore: () => spy.mockRestore() };
+}
+
+describe('ingest push', () => {
+ let db: Database;
+
+ beforeEach(() => {
+ db = new Database(':memory:');
+ initDatabase(db);
+ seedCache(db);
+ });
+
+ afterEach(() => {
+ db.close();
+ });
+
+ it('pushes dirty rows in seq order and acks them', async () => {
+ writeFeedItems(db, URL_HASH, [item('g2'), item('g1')], Date.now());
+ expect(countDirtyRows(db)).toBe(2);
+
+ const endpoint = mockIngestEndpoint();
+ const result = await pushDirtyItems(db, CONFIG);
+ endpoint.restore();
+
+ expect(result.pushed).toBe(2);
+ expect(endpoint.calls.length).toBe(1);
+ expect(endpoint.calls[0].url).toBe('https://api.example/api/internal/ingest');
+ expect(endpoint.calls[0].headers['X-Proxy-Secret']).toBe('test-secret');
+ // Items are newest-first in the feed, written oldest-first, so seq order is g1, g2.
+ expect(endpoint.calls[0].body.items.map((i) => i.guid)).toEqual(['g1', 'g2']);
+ // Always the registered URL, never a post-redirect one.
+ expect(endpoint.calls[0].body.items.every((i) => i.feedUrl === FEED_URL)).toBe(true);
+ expect(endpoint.calls[0].body.feeds[0]).toMatchObject({
+ feedUrl: FEED_URL,
+ title: 'Test Blog',
+ siteUrl: 'https://example.com',
+ });
+
+ expect(countDirtyRows(db)).toBe(0);
+ });
+
+ it('pages a backlog larger than one batch', async () => {
+ writeFeedItems(db, URL_HASH, [item('g3'), item('g2'), item('g1')], Date.now());
+
+ const endpoint = mockIngestEndpoint();
+ const first = await pushDirtyItems(db, CONFIG);
+ expect(first.pushed).toBe(2);
+ expect(first.hasMore).toBe(true);
+
+ const second = await pushDirtyItems(db, CONFIG);
+ endpoint.restore();
+ expect(second.pushed).toBe(1);
+ expect(second.hasMore).toBe(false);
+ expect(countDirtyRows(db)).toBe(0);
+ });
+
+ it('leaves state untouched on a 5xx so the same rows retry', async () => {
+ writeFeedItems(db, URL_HASH, [item('g1')], Date.now());
+
+ const failing = mockIngestEndpoint(500);
+ const result = await pushDirtyItems(db, CONFIG);
+ failing.restore();
+ expect(result.pushed).toBe(0);
+ expect(result.error).toContain('500');
+ expect(countDirtyRows(db)).toBe(1);
+
+ const ok = mockIngestEndpoint();
+ const retry = await pushDirtyItems(db, CONFIG);
+ ok.restore();
+ expect(retry.pushed).toBe(1);
+ expect(countDirtyRows(db)).toBe(0);
+ });
+
+ it('re-qualifies an item edited after it was pushed', async () => {
+ writeFeedItems(db, URL_HASH, [item('g1', { title: 'Old' })], Date.now());
+ const first = mockIngestEndpoint();
+ await pushDirtyItems(db, CONFIG);
+ first.restore();
+ expect(countDirtyRows(db)).toBe(0);
+
+ // An edit rewrites content_hash in place (seq unchanged) → dirty again.
+ writeFeedItems(db, URL_HASH, [item('g1', { title: 'New' })], Date.now());
+ expect(countDirtyRows(db)).toBe(1);
+
+ const second = mockIngestEndpoint();
+ const result = await pushDirtyItems(db, CONFIG);
+ second.restore();
+ expect(result.pushed).toBe(1);
+ expect(second.calls[0].body.items[0].item.title).toBe('New');
+ });
+
+ it('acks the hash it pushed, so an edit mid-flight is not lost', async () => {
+ writeFeedItems(db, URL_HASH, [item('g1', { title: 'Old' })], Date.now());
+ const rows = selectDirtyRows(db, 10);
+ expect(rows.length).toBe(1);
+
+ const endpoint = spyOn(globalThis, 'fetch').mockImplementation((async () => {
+ // The warm loop rewrites the item while the request is in flight.
+ writeFeedItems(db, URL_HASH, [item('g1', { title: 'Newer' })], Date.now());
+ return new Response('{}', { status: 200 });
+ }) as unknown as typeof fetch);
+ await pushDirtyItems(db, CONFIG);
+ endpoint.mockRestore();
+
+ // The acked hash is the one we sent, which no longer matches — still dirty.
+ expect(countDirtyRows(db)).toBe(1);
+ });
+
+ it('supplies a hash for legacy rows that have none', async () => {
+ const parsed = item('legacy');
+ db.run(
+ `INSERT INTO feed_items (url_hash, guid, item_json, published_at, first_seen_at, content_hash)
+ VALUES (?, ?, ?, ?, ?, NULL)`,
+ [URL_HASH, 'legacy', JSON.stringify(parsed), Date.now(), Date.now()]
+ );
+
+ const endpoint = mockIngestEndpoint();
+ await pushDirtyItems(db, CONFIG);
+ endpoint.restore();
+
+ expect(endpoint.calls[0].body.items[0].contentHash).toBe(itemContentHash(parsed));
+ // Acked as '' (matching COALESCE), so it doesn't re-push every cycle.
+ expect(countDirtyRows(db)).toBe(0);
+ });
+
+ it('cascades push_state when the per-feed cap trims items', async () => {
+ // Fill past the cap so the oldest rows are trimmed on the next write.
+ const initial = Array.from({ length: FEED_ITEMS_CAP }, (_, i) => item(`old-${i}`));
+ writeFeedItems(db, URL_HASH, initial, Date.now());
+ const endpoint = mockIngestEndpoint();
+ let guard = 0;
+ while (countDirtyRows(db) > 0 && guard++ < 200) {
+ await pushDirtyItems(db, { ...CONFIG, batchSize: 100 });
+ }
+ endpoint.restore();
+ expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe(
+ FEED_ITEMS_CAP
+ );
+
+ writeFeedItems(db, URL_HASH, [item('fresh-1'), item('fresh-2')], Date.now());
+
+ const itemCount = db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM feed_items').get()?.c;
+ const stateCount = db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c;
+ expect(itemCount).toBe(FEED_ITEMS_CAP);
+ // Two trimmed rows dropped their delivery state with them (2 fresh rows are
+ // not yet pushed, so push_state holds cap - 2).
+ expect(stateCount).toBe(FEED_ITEMS_CAP - 2);
+ });
+
+ it('cascades push_state when an idle feed is evicted', async () => {
+ writeFeedItems(db, URL_HASH, [item('g1')], Date.now());
+ const endpoint = mockIngestEndpoint();
+ await pushDirtyItems(db, CONFIG);
+ endpoint.restore();
+ expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe(1);
+
+ // Age the feed past the 7-day eviction window.
+ db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [
+ Date.now() - 8 * 24 * 60 * 60 * 1000,
+ URL_HASH,
+ ]);
+ cleanupCache(db);
+
+ expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM feed_items').get()?.c).toBe(0);
+ expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM push_state').get()?.c).toBe(0);
+ });
+});
+
+describe('crawl-set registration', () => {
+ let db: Database;
+
+ beforeEach(() => {
+ db = new Database(':memory:');
+ initDatabase(db);
+ });
+
+ afterEach(() => {
+ db.close();
+ });
+
+ it('creates a cache row for an unknown feed, due for the warm loop', () => {
+ const now = Date.now();
+ expect(registerCrawlFeeds(db, [FEED_URL], now)).toBe(1);
+
+ const row = db
+ .query<
+ { url: string; fetched_at: number; parser_version: number; last_requested_at: number },
+ [string]
+ >('SELECT url, fetched_at, parser_version, last_requested_at FROM cache WHERE url_hash = ?')
+ .get(URL_HASH);
+ expect(row?.url).toBe(FEED_URL);
+ expect(row?.last_requested_at).toBe(now);
+ // fetched_at 0 + a stale parser version make it immediately warm-eligible.
+ expect(row?.fetched_at).toBe(0);
+ expect(row?.parser_version).toBe(0);
+ });
+
+ it('keeps a known feed warm without any read traffic', () => {
+ seedCache(db);
+ db.run('UPDATE cache SET last_requested_at = ? WHERE url_hash = ?', [1000, URL_HASH]);
+
+ const now = Date.now();
+ registerCrawlFeeds(db, [FEED_URL], now);
+
+ const row = db
+ .query<
+ { last_requested_at: number; parsed_json: string; parser_version: number },
+ [string]
+ >('SELECT last_requested_at, parsed_json, parser_version FROM cache WHERE url_hash = ?')
+ .get(URL_HASH);
+ expect(row?.last_requested_at).toBe(now);
+ // The existing cached parse is untouched.
+ expect(row?.parser_version).toBe(1);
+ expect(JSON.parse(row!.parsed_json).title).toBe('Test Blog');
+ });
+
+ it('pulls the crawl set from the paired Worker', async () => {
+ const spy = spyOn(globalThis, 'fetch').mockImplementation((async (
+ input: string,
+ init?: RequestInit
+ ) => {
+ expect(String(input)).toBe('https://api.example/api/internal/crawl-set');
+ expect((init?.headers as Record)['X-Proxy-Secret']).toBe('test-secret');
+ return new Response(
+ JSON.stringify({
+ feeds: [
+ { feedUrl: FEED_URL, subscribers: 3 },
+ { feedUrl: 'https://other.example/f.xml' },
+ ],
+ })
+ );
+ }) as unknown as typeof fetch);
+
+ const result = await pullCrawlSet(db, CONFIG);
+ spy.mockRestore();
+
+ expect(result.registered).toBe(2);
+ expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM cache').get()?.c).toBe(2);
+ });
+
+ it('reports a failed pull without touching the cache', async () => {
+ const spy = spyOn(globalThis, 'fetch').mockImplementation((async () => {
+ return new Response('nope', { status: 503 });
+ }) as unknown as typeof fetch);
+
+ const result = await pullCrawlSet(db, CONFIG);
+ spy.mockRestore();
+
+ expect(result.registered).toBe(0);
+ expect(result.error).toContain('503');
+ expect(db.query<{ c: number }, []>('SELECT COUNT(*) AS c FROM cache').get()?.c).toBe(0);
+ });
+});
+
+describe('feed health reporting', () => {
+ let db: Database;
+
+ beforeEach(() => {
+ db = new Database(':memory:');
+ initDatabase(db);
+ seedCache(db);
+ });
+
+ afterEach(() => {
+ db.close();
+ });
+
+ function breakFeed(
+ url: string,
+ opts: { errorCount?: number; error?: string; at?: number; retryAt?: number } = {}
+ ): void {
+ const at = opts.at ?? Date.now();
+ db.run(
+ 'UPDATE cache SET error_count = ?, last_error = ?, last_error_at = ?, next_retry_at = ? WHERE url_hash = ?',
+ [
+ opts.errorCount ?? 3,
+ opts.error ?? 'Failed to fetch (HTTP 404)',
+ at,
+ opts.retryAt ?? at + 600_000,
+ hashUrl(url),
+ ]
+ );
+ }
+
+ it('reports only broken feeds, converting milliseconds to seconds', () => {
+ const at = 1_770_000_000_000;
+ breakFeed(FEED_URL, { errorCount: 4, at, retryAt: at + 600_000 });
+
+ const health = selectFeedHealth(db);
+ expect(health).toHaveLength(1);
+ expect(health[0]).toMatchObject({
+ feedUrl: FEED_URL,
+ errorCount: 4,
+ lastError: 'Failed to fetch (HTTP 404)',
+ lastErrorAt: Math.floor(at / 1000),
+ nextRetryAt: Math.floor((at + 600_000) / 1000),
+ });
+ });
+
+ it('leaves healthy feeds out entirely — absence is the recovery signal', () => {
+ seedCache(db, 'https://other.example/feed.xml', 'Other');
+ breakFeed(FEED_URL);
+
+ expect(selectFeedHealth(db).map((f) => f.feedUrl)).toEqual([FEED_URL]);
+ });
+
+ it('flags a crawl-set feed the warm loop has not fetched in hours', () => {
+ // Nothing is erroring; the feed is simply losing its turn every tick, which
+ // is what a capped warm batch does and what the admin needs to see.
+ const now = Date.now();
+ db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [
+ now - CRAWL_STALE_MS - 60_000,
+ URL_HASH,
+ ]);
+
+ const health = selectFeedHealth(db, now);
+ expect(health).toHaveLength(1);
+ expect(health[0]).toMatchObject({ feedUrl: FEED_URL, errorCount: 0, crawlStale: true });
+ });
+
+ it('does not flag a feed fetched within the window', () => {
+ const now = Date.now();
+ db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [now - 60_000, URL_HASH]);
+ expect(selectFeedHealth(db, now)).toHaveLength(0);
+ });
+
+ it('reports a feed that is both erroring and starved', () => {
+ const now = Date.now();
+ breakFeed(FEED_URL);
+ db.run('UPDATE cache SET fetched_at = ? WHERE url_hash = ?', [
+ now - CRAWL_STALE_MS - 60_000,
+ URL_HASH,
+ ]);
+
+ const health = selectFeedHealth(db, now);
+ expect(health[0]).toMatchObject({ errorCount: 3, crawlStale: true });
+ });
+
+ it('skips feeds that have dropped out of the crawl set', () => {
+ // last_requested_at NULL = evicted / never registered, so nobody is
+ // subscribed and its errors are not the reader's problem.
+ breakFeed(FEED_URL);
+ db.run('UPDATE cache SET last_requested_at = NULL WHERE url_hash = ?', [URL_HASH]);
+
+ expect(selectFeedHealth(db)).toHaveLength(0);
+ });
+
+ it('posts the set to the Worker, and posts an empty set too', async () => {
+ const endpoint = mockIngestEndpoint();
+ await reportFeedHealth(db, CONFIG);
+
+ expect(endpoint.calls[0].url).toBe('https://api.example/api/internal/feed-health');
+ expect(endpoint.calls[0].headers['X-Proxy-Secret']).toBe('test-secret');
+ // Everything is healthy: the empty list is what clears the Worker's flags.
+ expect((endpoint.calls[0].body as unknown as { feeds: unknown[] }).feeds).toEqual([]);
+
+ breakFeed(FEED_URL);
+ const result = await reportFeedHealth(db, CONFIG);
+ endpoint.restore();
+
+ expect(result.reported).toBe(1);
+ expect(
+ (endpoint.calls[1].body as unknown as { feeds: Array<{ feedUrl: string }> }).feeds[0].feedUrl
+ ).toBe(FEED_URL);
+ });
+
+ it('surfaces a rejected report instead of pretending it landed', async () => {
+ breakFeed(FEED_URL);
+ const endpoint = mockIngestEndpoint(503);
+ const result = await reportFeedHealth(db, CONFIG);
+ endpoint.restore();
+
+ expect(result.reported).toBe(0);
+ expect(result.error).toContain('503');
+ });
+});
diff --git a/feed-proxy/src/ingest-push.ts b/feed-proxy/src/ingest-push.ts
new file mode 100644
index 00000000..64260a1e
--- /dev/null
+++ b/feed-proxy/src/ingest-push.ts
@@ -0,0 +1,348 @@
+import type { Database } from 'bun:sqlite';
+import type { FeedItem } from './types';
+import { hashUrl, itemContentHash } from './app';
+
+/**
+ * Ingest push: the proxy stops being a read path and becomes a crawler that
+ * pushes deltas into its paired Worker's D1.
+ *
+ * The durable item log IS the outbox — there is no separate queue. A row is
+ * *dirty* when `push_state` has no row for its seq, or the hash we last pushed
+ * differs from the row's current `content_hash`. That comparison is race-free
+ * against concurrent edits: if `writeFeedItems` rewrites an item between select
+ * and ack, the hashes no longer match and the row simply re-qualifies.
+ *
+ * One proxy pushes to exactly ONE Worker (prod proxy → prod Worker, staging
+ * proxy → staging Worker), so delivery state is keyed by seq alone. With
+ * `INGEST_URL` unset both loops are disabled — the safe default for local dev
+ * and the gate for a staged rollout.
+ */
+
+export interface IngestConfig {
+ // Base URL of the paired Worker, e.g. https://api.skyreader.app
+ ingestUrl: string;
+ // Shared secret, sent as X-Proxy-Secret (the same PROXY_SECRET the Worker uses
+ // to authenticate to us — one secret per environment, no new names).
+ secret?: string;
+ // Items per push request. The Worker's own cap is well above this.
+ batchSize?: number;
+ timeoutMs?: number;
+}
+
+const DEFAULT_BATCH_SIZE = 100;
+const DEFAULT_TIMEOUT_MS = 30_000;
+
+interface DirtyRow {
+ seq: number;
+ url_hash: string;
+ guid: string;
+ item_json: string;
+ published_at: number | null;
+ first_seen_at: number;
+ content_hash: string;
+ feed_url: string;
+}
+
+interface FeedMetaRow {
+ url_hash: string;
+ url: string;
+ title: string | null;
+ site_url: string | null;
+ description: string | null;
+ image_url: string | null;
+}
+
+export interface PushResult {
+ pushed: number;
+ // True when the log still holds dirty rows beyond this batch.
+ hasMore: boolean;
+ error?: string;
+}
+
+export function selectDirtyRows(db: Database, limit: number): DirtyRow[] {
+ return db
+ .query(
+ `SELECT fi.seq, fi.url_hash, fi.guid, fi.item_json, fi.published_at, fi.first_seen_at,
+ COALESCE(fi.content_hash, '') AS content_hash, c.url AS feed_url
+ FROM feed_items fi
+ JOIN cache c ON c.url_hash = fi.url_hash
+ LEFT JOIN push_state ps ON ps.seq = fi.seq
+ WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '')
+ ORDER BY fi.seq ASC
+ LIMIT ?`
+ )
+ .all(limit);
+}
+
+export function countDirtyRows(db: Database): number {
+ return (
+ db
+ .query<{ count: number }, []>(
+ `SELECT COUNT(*) AS count
+ FROM feed_items fi
+ JOIN cache c ON c.url_hash = fi.url_hash
+ LEFT JOIN push_state ps ON ps.seq = fi.seq
+ WHERE ps.seq IS NULL OR ps.pushed_hash <> COALESCE(fi.content_hash, '')`
+ )
+ .get()?.count ?? 0
+ );
+}
+
+function feedMetadata(db: Database, urlHashes: string[]): FeedMetaRow[] {
+ if (urlHashes.length === 0) return [];
+ const placeholders = urlHashes.map(() => '?').join(',');
+ return db
+ .query(
+ `SELECT url_hash, url,
+ json_extract(parsed_json, '$.title') AS title,
+ json_extract(parsed_json, '$.siteUrl') AS site_url,
+ json_extract(parsed_json, '$.description') AS description,
+ json_extract(parsed_json, '$.imageUrl') AS image_url
+ FROM cache
+ WHERE url_hash IN (${placeholders})`
+ )
+ .all(...urlHashes);
+}
+
+function ackPushed(db: Database, rows: DirtyRow[]): void {
+ const upsert = db.query(
+ `INSERT INTO push_state (seq, pushed_hash) VALUES (?, ?)
+ ON CONFLICT(seq) DO UPDATE SET pushed_hash = excluded.pushed_hash`
+ );
+ // Ack the hash we actually PUSHED, not the row's hash right now: if the item
+ // was edited mid-flight the two differ and the row stays dirty for the next
+ // cycle, which is exactly what we want.
+ db.transaction(() => {
+ for (const row of rows) upsert.run(row.seq, row.content_hash);
+ })();
+}
+
+/**
+ * Drain one batch of dirty rows to the paired Worker. Rows go in seq order, so
+ * within-feed order in D1 matches proxy first-seen order — all the cursor
+ * semantics need. On any failure nothing is acked and the same rows retry.
+ */
+export async function pushDirtyItems(db: Database, config: IngestConfig): Promise {
+ const batchSize = config.batchSize ?? DEFAULT_BATCH_SIZE;
+ const rows = selectDirtyRows(db, batchSize);
+ if (rows.length === 0) return { pushed: 0, hasMore: false };
+
+ const urlHashes = [...new Set(rows.map((r) => r.url_hash))];
+ const feeds = feedMetadata(db, urlHashes).map((meta) => ({
+ feedUrl: meta.url,
+ title: meta.title,
+ siteUrl: meta.site_url,
+ description: meta.description,
+ imageUrl: meta.image_url,
+ }));
+
+ const items = rows.map((row) => {
+ const item = JSON.parse(row.item_json) as FeedItem;
+ return {
+ // ALWAYS the registered/requested URL (cache.url), never a post-redirect
+ // one: D1 joins subscriptions on this exact string.
+ feedUrl: row.feed_url,
+ guid: row.guid,
+ item,
+ publishedAt: row.published_at,
+ firstSeenAt: row.first_seen_at,
+ // A pre-hash legacy row still needs a hash for D1's NOT NULL column.
+ contentHash: row.content_hash || itemContentHash(item),
+ };
+ });
+
+ const headers: Record = { 'Content-Type': 'application/json' };
+ if (config.secret) headers['X-Proxy-Secret'] = config.secret;
+
+ try {
+ const response = await fetch(`${config.ingestUrl}/api/internal/ingest`, {
+ method: 'POST',
+ headers,
+ body: JSON.stringify({ feeds, items }),
+ signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS),
+ });
+
+ if (!response.ok) {
+ const body = await response.text().catch(() => '');
+ return {
+ pushed: 0,
+ hasMore: true,
+ error: `HTTP ${response.status}${body ? `: ${body.slice(0, 200)}` : ''}`,
+ };
+ }
+
+ ackPushed(db, rows);
+ return { pushed: rows.length, hasMore: rows.length >= batchSize };
+ } catch (error) {
+ return {
+ pushed: 0,
+ hasMore: true,
+ error: error instanceof Error ? error.message : String(error),
+ };
+ }
+}
+
+/**
+ * Register a feed in the crawl set: create its cache row if missing and stamp
+ * `last_requested_at`. That single stamp is the whole trick — the existing warm
+ * loop, active window and eviction machinery then work unchanged, with the
+ * Worker's subscription table (rather than read traffic) driving warmth.
+ *
+ * A newly created row starts with parser_version 0 and fetched_at 0 so the warm
+ * loop treats it as due immediately.
+ */
+export function registerCrawlFeeds(db: Database, feedUrls: string[], now: number): number {
+ if (feedUrls.length === 0) return 0;
+ const upsert = db.query(
+ `INSERT INTO cache (url_hash, url, parsed_json, parser_version, parser_upgrade_attempted_version,
+ cached_at, fetched_at, error_count, last_requested_at)
+ VALUES (?, ?, '{"title":"","items":[],"fetchedAt":0}', 0, 0, 0, 0, 0, ?)
+ ON CONFLICT(url_hash) DO UPDATE SET last_requested_at = excluded.last_requested_at`
+ );
+ let registered = 0;
+ db.transaction(() => {
+ for (const url of feedUrls) {
+ if (!url) continue;
+ upsert.run(hashUrl(url), url, now);
+ registered++;
+ }
+ })();
+ return registered;
+}
+
+export interface CrawlSetResult {
+ registered: number;
+ error?: string;
+}
+
+/**
+ * One broken feed, as reported to the Worker. Timestamps are unix SECONDS on the
+ * wire — the cache stores milliseconds (everything here compares against
+ * `Date.now()`), and the Worker's `feeds` table is in seconds like the rest of
+ * the backend, so the conversion happens once, here.
+ */
+export interface FeedHealthReport {
+ feedUrl: string;
+ errorCount: number;
+ lastError: string | null;
+ lastErrorAt: number | null;
+ nextRetryAt: number | null;
+ lastFetchAt: number | null;
+ // In the crawl set but not fetched in CRAWL_STALE_MS — starved by a saturated
+ // warm loop rather than failing. `errorCount` can be 0 while this is true.
+ crawlStale: boolean;
+}
+
+// How long a crawl-set feed may go unfetched before it counts as starved. The
+// warm loop works on a minutes-long cadence (a 300s cache TTL refreshed at
+// ~180s), so hours without a fetch means this feed is losing its turn every
+// tick — the failure mode a capped warm batch produces.
+export const CRAWL_STALE_MS = 2 * 60 * 60 * 1000;
+
+function toSeconds(ms: number | null | undefined): number | null {
+ return ms ? Math.floor(ms / 1000) : null;
+}
+
+/**
+ * Every crawl-set feed with something wrong with it: failing to fetch, starved
+ * of fetches, or both.
+ *
+ * Deliberately the whole trouble set rather than a delta: the Worker infers
+ * recovery from a feed's ABSENCE here, so a feed that starts working again needs
+ * no message of its own. Only rows still in the crawl set count — a feed evicted
+ * by `cleanupCache` is nobody's problem any more.
+ */
+export function selectFeedHealth(db: Database, now = Date.now()): FeedHealthReport[] {
+ const staleBefore = now - CRAWL_STALE_MS;
+ return db
+ .query<
+ {
+ url: string;
+ error_count: number;
+ last_error: string | null;
+ last_error_at: number | null;
+ next_retry_at: number | null;
+ fetched_at: number | null;
+ },
+ [number]
+ >(
+ `SELECT url, error_count, last_error, last_error_at, next_retry_at, fetched_at
+ FROM cache
+ WHERE last_requested_at IS NOT NULL
+ AND (error_count > 0 OR COALESCE(fetched_at, 0) < ?)`
+ )
+ .all(staleBefore)
+ .map((row) => ({
+ feedUrl: row.url,
+ errorCount: row.error_count,
+ lastError: row.last_error,
+ lastErrorAt: toSeconds(row.last_error_at),
+ nextRetryAt: toSeconds(row.next_retry_at),
+ // 0 means never fetched — an error placeholder from a first-crawl failure,
+ // or a row the crawl set just registered. Not a timestamp.
+ lastFetchAt: toSeconds(row.fetched_at),
+ crawlStale: (row.fetched_at ?? 0) < staleBefore,
+ }));
+}
+
+export interface FeedHealthResult {
+ reported: number;
+ error?: string;
+}
+
+/**
+ * Push the trouble set to the paired Worker: the erroring feeds it serves to
+ * readers, and the starved ones the admin alarms on. An empty list is a
+ * meaningful report — it is how "everything recovered" is communicated — so this
+ * always posts.
+ */
+export async function reportFeedHealth(
+ db: Database,
+ config: IngestConfig
+): Promise {
+ const feeds = selectFeedHealth(db);
+ const headers: Record = { 'Content-Type': 'application/json' };
+ if (config.secret) headers['X-Proxy-Secret'] = config.secret;
+
+ try {
+ const response = await fetch(`${config.ingestUrl}/api/internal/feed-health`, {
+ method: 'POST',
+ headers,
+ body: JSON.stringify({ feeds }),
+ signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS),
+ });
+ if (!response.ok) return { reported: 0, error: `HTTP ${response.status}` };
+ return { reported: feeds.length };
+ } catch (error) {
+ return {
+ reported: 0,
+ error: error instanceof Error ? error.message : String(error),
+ };
+ }
+}
+
+/**
+ * Pull the crawl set from the paired Worker and stamp every feed in it.
+ */
+export async function pullCrawlSet(db: Database, config: IngestConfig): Promise {
+ const headers: Record = {};
+ if (config.secret) headers['X-Proxy-Secret'] = config.secret;
+
+ try {
+ const response = await fetch(`${config.ingestUrl}/api/internal/crawl-set`, {
+ headers,
+ signal: AbortSignal.timeout(config.timeoutMs ?? DEFAULT_TIMEOUT_MS),
+ });
+ if (!response.ok) {
+ return { registered: 0, error: `HTTP ${response.status}` };
+ }
+ const body = (await response.json()) as { feeds?: Array<{ feedUrl: string }> };
+ const urls = (body.feeds ?? []).map((f) => f.feedUrl).filter(Boolean);
+ return { registered: registerCrawlFeeds(db, urls, Date.now()) };
+ } catch (error) {
+ return {
+ registered: 0,
+ error: error instanceof Error ? error.message : String(error),
+ };
+ }
+}
diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md
index 888e67d3..7fabdb0e 100644
--- a/frontend/CLAUDE.md
+++ b/frontend/CLAUDE.md
@@ -39,25 +39,42 @@ All stores use Svelte 5 runes (`.svelte.ts` files):
### Services
-| Service | Purpose |
-| --------------- | -------------------------------------------------- |
-| `api.ts` | HTTP client for backend API |
-| `db.ts` | Dexie (IndexedDB) schema for offline storage |
-| `sync-queue.ts` | Queue operations when offline, process when online |
-| `realtime.ts` | WebSocket connection management |
-| `telemetry.ts` | Sampled client error reports → our backend |
-
-### Error reporting
-
-`hooks.client.ts` wires three channels — SvelteKit's `handleError`, `window.onerror`
-and `unhandledrejection` — into `reportClientError()`, which posts to the backend's
-`/api/telemetry/error`. No Sentry SDK ships in the bundle and no third party is
-contacted; the backend decides what reaches the error tracker.
-
-It is sampled at 10%, capped per page load, silent in dev and offline, and sends
-no query strings. The exception is `preload_recovery_failed` — the stale-chunk
-guard tripping twice, i.e. a deploy that bricked the PWA — which always sends.
-See [`docs/RUNBOOK.md`](../docs/RUNBOOK.md) §4c.
+| Service | Purpose |
+| ----------------- | -------------------------------------------------- |
+| `api.ts` | HTTP client for backend API |
+| `db.ts` | Dexie (IndexedDB) schema for offline storage |
+| `feedFetcher.ts` | Feed refresh (timeline sync + legacy batch path) |
+| `timelineSync.ts` | Pure helpers for the timeline sync (unit-tested) |
+| `sync-queue.ts` | Queue operations when offline, process when online |
+| `realtime.ts` | WebSocket connection management |
+| `telemetry.ts` | Sampled client error reports to the backend |
+
+### Feed refresh
+
+A refresh is **one** `GET /api/v2/timeline` request (plus drain pages while `hasMore`), served from
+the backend's D1 archive with read state already stamped on each item. The client holds a single
+global cursor in Dexie `metadata` (`timelineCursor` = `{cursor, generation}`), committed only after
+a successful merge; a `generation` change cold-starts. A cold start is paged (`cold_offset`), and
+commits the first page's cursor only once the last page has merged. New subscriptions are
+backfilled through `fetchSingleFeed` → `GET /api/v2/feeds/fetch`, since their items sit below the
+global cursor — including ones that arrive from another device (`backfillMissingSubscriptions`,
+once per feed, ≤ 10 per sync).
+
+Per-feed error state (the sidebar badge and the Manage Sources popover) comes from the response's
+`feedHealth`: the set of feeds the crawler currently considers broken. Reads are served from the
+archive, so "this feed returned nothing" means nothing — a dead feed and a quiet feed look
+identical, and only the server's health verdict distinguishes them. **Absence from that map is what
+clears an error**, so `applyHealthSnapshot` runs after the per-feed `markReady` pass and overrides
+it. The payload is sent on every cold start and whenever the echoed `health_rev` is stale, so a
+steady-state poll pays nothing for it.
+
+The legacy per-feed `/api/v2/feeds/batch` path (`fetchAllFeedsViaBatch`, with per-subscription
+`feedCursors`) is kept for one release as a fallback: it runs when the timeline 404s and whenever
+the server reports `ingestActive: false` (this environment's crawler isn't pushing into D1). See
+`docs/plans/D1_FEED_TIMELINE.md`.
+
+Client errors flow through `/api/telemetry/error`; no Sentry SDK ships in the frontend bundle.
+See [`docs/RUNBOOK.md`](../docs/RUNBOOK.md) for sampling and recovery details.
### Key Routes
diff --git a/frontend/src/lib/components/ArticleCard.svelte b/frontend/src/lib/components/ArticleCard.svelte
index 8a32766e..d8331afd 100644
--- a/frontend/src/lib/components/ArticleCard.svelte
+++ b/frontend/src/lib/components/ArticleCard.svelte
@@ -1,4 +1,5 @@